tokens/s/GPU, including the nominal bubble
Roofline: Qwen3.8-Flash-Next
on 16 GB300s
Six billion activated parameters per token, 51 billion that cost no FLOPs at all, and a router that sends ten copies of every token across NVLink. This study turns the shapes from study 01 and the lifetimes from study 02 into an analytic ceiling for one recipe.
Anchor recipe, as in study 02: 16 GB300s in one NVLink domain, PP2 with interleaved 1F1B and two virtual stages, DP-shard8 / EP8, 16 microbatches of 16,384 packed tokens per DP rank in 4K documents, full layer recompute, Dist-MoE with slot factor 0.5, BF16 compute. The result is an optimistic analytic ceiling: FLOPs over practical compute rates, operand bytes over HBM, payload bytes over NVLink, plus the nominal pipeline bubble. Communication is priced byte by byte but assumed to overlap perfectly with compute, so it sets the step only if it exceeds compute. No measured throughput is used as an input or shown as a comparison.
Three roofs, and only one network
A GEMM is limited by its FLOPs or by the HBM bytes it must move. Distributed work is limited by the link that carries it. All 16 GPUs of the anchor sit in one NVLink domain, so expert dispatch, the PLE lookup, FSDP and pipeline packets all share the NVLink roof; no traffic reaches the scale-out network.
| Resource | GB300 input | How to read it |
|---|---|---|
| BF16 Tensor Cores | 1,800 TFLOP/s / GPU | Practical sustained target shared with the Kimi K3 roofline study, not the advertised peak. Every GEMM here is BF16. |
| HBM | 8 TB/s / GPU | The operand-bandwidth roof. Capacity is a separate question, answered by study 02. |
| NVLink | 900 GB/s / GPU, one direction | Collectives and contention can lower effective service; opposite-direction traffic is not added to the denominator. |
| Nominal dense BF16 peak | 2,500 TFLOP/s / GPU | Only the denominator of the conventional MFU metric. It never enters a time. |
F = multiply-add FLOPs; B = operand bytes; intensity I = F / B.
Below the ridge, fewer bytes help. Above it, the compute rate is the limit.
For every GEMM phase: forward, input gradient (DGRAD), weight gradient (WGRAD), and recompute replay t_op = max(F_op / BF16_rate, operand_bytes / HBM_bandwidth) T_stage = microbatches * sum(t_op over local chunks) + attention_core_FLOPs / BF16_rate T_nvl = NVLink_bytes / NVLink_bandwidth T_resource = max over PP ranks of max(T_stage, T_nvl) (assumes perfect overlap) T_step = T_resource + (PP - 1) / (M * VPP) * max(T_stage) T_exposed = max over PP ranks of (T_stage + T_nvl) + bubble (no overlap at all) rho = global_tokens_per_update / (GPUs * T_step) model TFLOP/s/GPU = rho * conventional_FLOPs_per_token / 1e12
Configure a recipe and calculate its roofline
Start from the anchor and change the layout, batch, recompute policy or hardware rates. Every result updates from those inputs. Memory feasibility is not checked here; use the memory lab for the same layout.
| PP rank | Layers | GEMM time | Attention cores | Compute / HBM | NVLink service | Limiter |
|---|
2(MK + KN + MN) with one
operand pass and no cross-GEMM reuse. Routed experts use the balanced rows per local expert, EP × T × 10 / 512.
| GEMM (one layer, forward) | [M, K] x [K, N] | Groups | Intensity | Bound | Time per microbatch, all phases |
|---|
Stored parameters are not activated FLOPs
For Y = XW with X [M, K] and W [K, N], forward costs 2MNK FLOPs and backward adds
dX = dY WT and dW = XT dY of the same size: 6MNK, or
6 Nactive FLOPs per token. Of 176.94B trainable parameters, only 6.67B are activated per token.
| Parameter group | Stored | Activated / token | Why |
|---|---|---|---|
| Routed experts | 120,795,955,200 | 2,359,296,000 | 10 of 512 experts per layer: 48 × 10 × 3 × 2560 × 640 |
| PLE table | 51,200,245,760 | 0 | A lookup: 16 rows are read, none multiplied. Its key and value projections are counted below. |
| Input embedding | 635,699,200 | 0 | A lookup, as in TorchTitan's convention. |
| Everything else: GDN, QSA, indexers, shared experts, routers, residual mixers, PLE projections, LM head | 4,311,999,360 | 4,311,999,360 | Dense: every token uses every weight. |
| Total | 176,943,899,520 | 6,671,295,360 | 6 × 6.67B = 40.03 GFLOP per token before attention. |
Attention adds work without adding a weight matrix
| Core, per token per layer | Conventional (TorchTitan) | Issued-work sketch | What changes |
|---|
| Per token | GFLOP |
|---|
The router moves more bytes than everything else together
Flash-Next dispatches the full 2,560-wide token, ten times, to experts on up to seven other GPUs. With uniform routing, a fraction
1 − 1/EP of the copies leaves the GPU. Each MoE layer sends them out (dispatch), back (combine), and reverses both in
backward. Under full recompute, Dist-MoE re-dispatches the rows its slot did not keep.
Expert payload / layer / microbatch = exchanges * T * 10 * (1 - 1/EP) * 2560 * 2 bytes
exchanges = 4 (dispatch, combine, and their reverses) + (1 - slot factor) under full recompute
= 4.5 in the anchor: 16384 * 10 * 7/8 * 5120 * 4.5 = 3.08 GiB per layer per microbatch
| Traffic per GPU per update | PP rank 0 | PP rank 1 | How it is counted |
|---|
IDs out, rows back
Each token asks for 16 rows. Without deduplication, 7/8 of them live on another owner: an 8-byte ID out, a 320-byte BF16 row back, and a 320-byte row gradient in backward. That is under 1% of the expert payload, and the 95-GiB table itself never crosses a link.
One gather, one reduction
PP keeps every local chunk gathered and defers gradient reduction to the end of the schedule, so dense weights cost one BF16 all-gather
and one FP32 reduce-scatter per update: 6P(1 − 1/8) bytes. Experts have expert FSDP 1 and no collective at all.
Without PP, each microbatch gathers twice and reduces once.
The whole four-stream state
A stage boundary carries [16384, 4 × 2560] BF16, 320 MiB per microbatch forward and the same gradient backward. VPP2
puts three boundaries in every microbatch's path; each rank touches all three.
The LM head makes the last stage the slow one
Nominal bubble overhead = (PP - 1) / (M * VPP) Anchor: (2 - 1) / (16 * 2) = 3.13% of the slowest rank's compute VPP1: (2 - 1) / 16 = 6.25%
The bubble formula assumes balanced chunks. With the LM head and the final mixer on chunk 3, rank 1 has more work than rank 0 even though both own 24 decoder layers, and the faster rank waits. Moving the head's work, or giving rank 1 fewer decoder layers, would balance compute but shift memory: rank 0 already carries the table (study 02). Balancing one resource unbalances the other.
Muon is a matrix workload of its own
DistMuon orthogonalizes each logical matrix with 8 Polar Express iterations on a BF16 working copy. For an expert matrix oriented as
X [r, c] with r = 640 and c = 2560, an iteration forms A = XXT, a polynomial in A, and the update BX: 4r2c + 2r3 FLOPs.
The routed experts dominate: each GPU owns 24 layers × 64 experts × 3 matrices. The dense Muon matrices add little, and the narrow 640 side keeps every Gram matrix small. Muon is a small fraction of the ideal step, and more tokens per update amortize it further. The redistribution between storage and compute layouts is communication not priced here.
Sources and accounting notes
- Qwen3.8-Flash-Next configuration and the shape audit in study 01; the lifetimes and ownership in study 02.
-
TorchTitan's Flash-Next model, recipe and FLOP convention (
torchtitan/models/common/flops.pyand the QSA attention config): the conventional column and its MFU denominator. - Practical GB300 rates as used in the Kimi K3 roofline study.
- Williams, Waterman and Patterson, Roofline: the compute and bandwidth ceilings.
The ceiling omits non-matmul work (norms, gates, convolutions, top-k, softmax), launch and CPU overhead, grouped-GEMM inefficiency, routing imbalance, collective protocol overhead, and the optimizer step and its communication. It assumes communication overlaps compute perfectly; the no-overlap bound is reported alongside. GDN cores use TorchTitan's state-contraction formula; intra-chunk terms are not priced. Routed experts assume balanced routing without token dropping. Every number is computed from the stated inputs; none is measured.