Qwen3.8-Flash-Next studies / 03

Roofline: Qwen3.8-Flash-Next
on 16 GB300s

Six billion activated parameters per token, 51 billion that cost no FLOPs at all, and a router that sends ten copies of every token across NVLink. This study turns the shapes from study 01 and the lifetimes from study 02 into an analytic ceiling for one recipe.

6.67Bactivated parameters per token
42.39model GFLOP per token (TorchTitan convention)
30,457recipe roofline, tokens/s/GPU
1,291model TFLOP/s/GPU at that ceiling

Anchor recipe, as in study 02: 16 GB300s in one NVLink domain, PP2 with interleaved 1F1B and two virtual stages, DP-shard8 / EP8, 16 microbatches of 16,384 packed tokens per DP rank in 4K documents, full layer recompute, Dist-MoE with slot factor 0.5, BF16 compute. The result is an optimistic analytic ceiling: FLOPs over practical compute rates, operand bytes over HBM, payload bytes over NVLink, plus the nominal pipeline bubble. Communication is priced byte by byte but assumed to overlap perfectly with compute, so it sets the step only if it exceeds compute. No measured throughput is used as an input or shown as a comparison.

Practical rates, not advertised peaks

Three roofs, and only one network

A GEMM is limited by its FLOPs or by the HBM bytes it must move. Distributed work is limited by the link that carries it. All 16 GPUs of the anchor sit in one NVLink domain, so expert dispatch, the PLE lookup, FSDP and pipeline packets all share the NVLink roof; no traffic reaches the scale-out network.

Resource GB300 input How to read it
BF16 Tensor Cores 1,800 TFLOP/s / GPU Practical sustained target shared with the Kimi K3 roofline study, not the advertised peak. Every GEMM here is BF16.
HBM 8 TB/s / GPU The operand-bandwidth roof. Capacity is a separate question, answered by study 02.
NVLink 900 GB/s / GPU, one direction Collectives and contention can lower effective service; opposite-direction traffic is not added to the denominator.
Nominal dense BF16 peak 2,500 TFLOP/s / GPU Only the denominator of the conventional MFU metric. It never enters a time.
One GEMMt ≥ max(F / P, B / βHBM)

F = multiply-add FLOPs; B = operand bytes; intensity I = F / B.

The BF16 ridgeI* = P / βHBM = 225 FLOP/B

Below the ridge, fewer bytes help. Above it, the compute rate is the limit.

For every GEMM phase: forward, input gradient (DGRAD), weight gradient (WGRAD), and recompute replay
  t_op       = max(F_op / BF16_rate, operand_bytes / HBM_bandwidth)
T_stage      = microbatches * sum(t_op over local chunks) + attention_core_FLOPs / BF16_rate
T_nvl        = NVLink_bytes / NVLink_bandwidth
T_resource   = max over PP ranks of max(T_stage, T_nvl)        (assumes perfect overlap)
T_step       = T_resource + (PP - 1) / (M * VPP) * max(T_stage)
T_exposed    = max over PP ranks of (T_stage + T_nvl) + bubble  (no overlap at all)
rho          = global_tokens_per_update / (GPUs * T_step)
model TFLOP/s/GPU = rho * conventional_FLOPs_per_token / 1e12
Change a shape, watch a roof move

Configure a recipe and calculate its roofline

Start from the anchor and change the layout, batch, recompute policy or hardware rates. Every result updates from those inputs. Memory feasibility is not checked here; use the memory lab for the same layout.

Recipe roofline

tokens/s/GPU, including the nominal bubble

Model throughput at the ceiling

Step time

PP rank Layers GEMM time Attention cores Compute / HBM NVLink service Limiter
Forward GEMMs on the BF16 roofline
Points are modeled roof positions for one microbatch, not measured kernels. Forward bytes are 2(MK + KN + MN) with one operand pass and no cross-GEMM reuse. Routed experts use the balanced rows per local expert, EP × T × 10 / 512.
GEMM (one layer, forward) [M, K] x [K, N] Groups Intensity Bound Time per microbatch, all phases
Take 6ND apart

Stored parameters are not activated FLOPs

For Y = XW with X [M, K] and W [K, N], forward costs 2MNK FLOPs and backward adds dX = dY WT and dW = XT dY of the same size: 6MNK, or 6 Nactive FLOPs per token. Of 176.94B trainable parameters, only 6.67B are activated per token.

Parameter group Stored Activated / token Why
Routed experts 120,795,955,200 2,359,296,000 10 of 512 experts per layer: 48 × 10 × 3 × 2560 × 640
PLE table 51,200,245,760 0 A lookup: 16 rows are read, none multiplied. Its key and value projections are counted below.
Input embedding 635,699,200 0 A lookup, as in TorchTitan's convention.
Everything else: GDN, QSA, indexers, shared experts, routers, residual mixers, PLE projections, LM head 4,311,999,360 4,311,999,360 Dense: every token uses every weight.
Total 176,943,899,520 6,671,295,360 6 × 6.67B = 40.03 GFLOP per token before attention.

Attention adds work without adding a weight matrix

Core, per token per layer Conventional (TorchTitan) Issued-work sketch What changes

Per token GFLOP
Follow each byte to its destination

The router moves more bytes than everything else together

Flash-Next dispatches the full 2,560-wide token, ten times, to experts on up to seven other GPUs. With uniform routing, a fraction 1 − 1/EP of the copies leaves the GPU. Each MoE layer sends them out (dispatch), back (combine), and reverses both in backward. Under full recompute, Dist-MoE re-dispatches the rows its slot did not keep.

Expert payload / layer / microbatch = exchanges * T * 10 * (1 - 1/EP) * 2560 * 2 bytes
exchanges = 4 (dispatch, combine, and their reverses) + (1 - slot factor) under full recompute
          = 4.5 in the anchor: 16384 * 10 * 7/8 * 5120 * 4.5 = 3.08 GiB per layer per microbatch
Traffic per GPU per update PP rank 0 PP rank 1 How it is counted
PLE

IDs out, rows back

Each token asks for 16 rows. Without deduplication, 7/8 of them live on another owner: an 8-byte ID out, a 320-byte BF16 row back, and a 320-byte row gradient in backward. That is under 1% of the expert payload, and the 95-GiB table itself never crosses a link.

FSDP under PP

One gather, one reduction

PP keeps every local chunk gathered and defers gradient reduction to the end of the schedule, so dense weights cost one BF16 all-gather and one FP32 reduce-scatter per update: 6P(1 − 1/8) bytes. Experts have expert FSDP 1 and no collective at all. Without PP, each microbatch gathers twice and reduces once.

PP packets

The whole four-stream state

A stage boundary carries [16384, 4 × 2560] BF16, 320 MiB per microbatch forward and the same gradient backward. VPP2 puts three boundaries in every microbatch's path; each rank touches all three.

The useful FLOPs still need a schedule

The LM head makes the last stage the slow one

Compute per PP rank, one update
Nominal bubble overhead = (PP - 1) / (M * VPP)
Anchor: (2 - 1) / (16 * 2) = 3.13% of the slowest rank's compute
VPP1:   (2 - 1) / 16       = 6.25%

The bubble formula assumes balanced chunks. With the LM head and the final mixer on chunk 3, rank 1 has more work than rank 0 even though both own 24 decoder layers, and the faster rank waits. Moving the head's work, or giving rank 1 fewer decoder layers, would balance compute but shift memory: rank 0 already carries the table (study 02). Balancing one resource unbalances the other.

Once per update, not once per microbatch

Muon is a matrix workload of its own

DistMuon orthogonalizes each logical matrix with 8 Polar Express iterations on a BF16 working copy. For an expert matrix oriented as X [r, c] with r = 640 and c = 2560, an iteration forms A = XXT, a polynomial in A, and the update BX: 4r2c + 2r3 FLOPs.


        

The routed experts dominate: each GPU owns 24 layers × 64 experts × 3 matrices. The dense Muon matrices add little, and the narrow 640 side keeps every Gram matrix small. Muon is a small fraction of the ideal step, and more tokens per update amortize it further. The redistribution between storage and compute layouts is communication not priced here.

Sources and accounting notes

The ceiling omits non-matmul work (norms, gates, convolutions, top-k, softmax), launch and CPU overhead, grouped-GEMM inefficiency, routing imbalance, collective protocol overhead, and the optimizer step and its communication. It assumes communication overlaps compute perfectly; the no-overlap bound is reported alongside. GDN cores use TorchTitan's state-contraction formula; intra-chunk terms are not priced. Routed experts assume balanced routing without token dropping. Every number is computed from the stated inputs; none is measured.