Qwen3.8-Flash-Next studies / 02

Memory: Qwen3.8-Flash-Next
on 16 GB300s

A 125B-A6B decoder with a 51-billion-parameter lookup table. The table is free to compute and expensive to keep. The question is which GPU keeps it, and what else is live there at the same time.

176.94Btrainable text-decoder parameters
16GB300 GPUs, one NVLink domain
288 GiBHBM3e per GPU (eight 36-GiB stacks)
2.13 TiBlogical FP32 update state

Anchor recipe: PP2 with interleaved 1F1B and two virtual stages per rank, DP-shard8 / EP8 / TP1 / CP1, 16 microbatches of 16,384 packed tokens per DP rank (documents split at 4K), full layer recompute, Dist-MoE with an activation-slot capacity factor of 0.5, DistMuon plus AdamW plus Adam for the PLE table, FP32 master weights and gradient reduction, BF16 compute. No vision encoder, no MTP head. Every number on this page is derived from shapes, dtypes and schedules; none is measured.

First, name the copies

FP32 history, BF16 compute, and one table without a BF16 view

The FSDP storage shard is the master weight: the optimizer updates it in FP32, and FSDP casts a BF16 compute view for forward and backward. There is no second, full-model BF16 master. The PLE n-gram table is the exception that shapes this whole study: FSDP ignores it, so it never gets a BF16 view. Each data-parallel rank owns a contiguous range of rows in FP32 and only the rows a microbatch looks up are cast to BF16.

Resident masterFP32FSDP shard, EP-local experts, owned table rows
Compute viewBF16Gathered dense and expert weights; table rows only per lookup
Gradient historyFP32Reduce-scatter in FP32; table gradients stay on the row owner
Optimizer stateFP32One Muon momentum, or two Adam/AdamW moments
Object Dtype Lifetime / ownership
Dense resident parameter FP32 Sharded over the 8 DP ranks of its pipeline stage
Routed expert parameter FP32 64 of 512 experts per EP rank; expert FSDP degree 8 / 8 = 1
PLE table rows FP32 40,000,192 rows per owner on the stage that runs layer 2; ignored by FSDP
Forward/backward compute view BF16 Unsharded over FSDP; under PP it stays live from the first forward to the end of the schedule
Gradient accumulation and reduction FP32 Under PP, dense gradients stay unsharded until the last backward of the schedule
Muon momentum FP32 One tensor with the storage shard's layout
Adam / AdamW first and second moments FP32 Two tensors: the PLE table (Adam, no weight decay) and the AdamW remainder
Model GEMMs and main activations BF16 Four-stream residual [T, 4 × 2560], projections, expert rows
The unavoidable bill

A 2.13-TiB bill, and a third of it is one table

The architecture study audited 179,999,981,424 checkpoint elements. Remove the 448,931,056 vision-encoder and 2,607,150,848 MTP parameters and 176,943,899,520 trainable parameters remain. The recipe's first-match optimizer patterns split them three ways:

Optimizer Parameters Share Bytes / parameter FP32 state Share of state
DistMuon: projections, shared and routed experts 123,003,207,680 69.5% 12 1.342 TiB 63.1%
Adam: the PLE table 51,200,245,760 28.9% 16 0.745 TiB 35.0%
AdamW: embeddings, head, routers, gates, mixers, norms, convolutions 2,740,446,080 1.5% 16 0.040 TiB 1.9%
DistMuon parameter4 + 4 + 4 = 12 bytes

FP32 master + FP32 gradient + FP32 momentum

Adam or AdamW parameter4 + 4 + 4 + 4 = 16 bytes

FP32 master + gradient + two FP32 moments

Update-state tally = 12 P_muon + 16 (P_adam + P_adamw)
                   = 2,339,089,561,600 bytes
                   = 2.13 TiB across the logical model
Ideal 16-way average = 136.15 GiB / GPU

The table is free to compute: a token reads 16 rows of 160 values, 5 KiB in BF16. But every one of its 51.2 billion parameters needs a master, a gradient and two moments, because Adam is applied to the whole table. Sparse access reduces arithmetic, not optimizer state. The same is true of the routed experts: choosing 10 of 512 per token does not shrink their 120.8 billion stored weights.

Who holds what

PP, EP, FSDP and row owners divide different things

PP2 × VPP2

Four chunks of 12 layers

Chunk 0 holds the embedding and layers 1–12, including the PLE layer. Chunk 3 holds layers 37–48, the final mixer and the LM head. PP rank 0 runs chunks 0 and 2; rank 1 runs chunks 1 and 3.

EP8

64 experts per layer per GPU

512 / 8 = 64. With 8 DP ranks per stage, expert FSDP is 8 / 8 = 1: EP alone divides the expert bank. Each GPU stores 24 layers × 64 experts of its stage.

FSDP8

Dense weights over 8 ranks

Attention, GDN, shared experts, routers, mixers, embedding and head are sharded over the 8 DP ranks of their stage. They are 2.8% of the parameters, so FSDP barely matters for the state bill here.

Row owners

The table lives on PP rank 0

The 320,001,536 rows are split over the 8 DP ranks that run layer 2: 40,000,192 rows, or 6.4 billion parameters, per GPU. At 16 bytes per parameter that is 95.37 GiB per rank-0 GPU, and nothing on rank 1.

On PP rank r with dense parameters D_r, routed parameters X_r and table rows P_r:
Resident parameters / GPU = D_r / 8 + X_r / EP + P_r * 160 / 8
FP32 state / GPU          = 16 * (Adam + AdamW part) + 12 * (Muon part)
BF16 views / GPU          = 2 * (D_r + X_r / EP)        # the table has no view
Per GPU, after reduction PP rank 0 (chunks 0, 2) PP rank 1 (chunks 1, 3)
FP32 master 53.13 GiB 29.27 GiB
FP32 Muon momentum 28.65 GiB 28.63 GiB
FP32 Adam / AdamW moments 48.96 GiB 1.28 GiB
FP32 gradient shards 53.13 GiB 29.27 GiB
Update-state tally 183.85 GiB 88.45 GiB

Rank 0 carries 95 GiB more persistent state than rank 1, and the table accounts for all of it. The experts are symmetric: each rank holds 24 layers of 64 experts, 28.1 GiB of master and the same of momentum. The pipeline balances layers, not bytes.

Explore the ownership

A memory lab, with its assumptions exposed

Pick a pipeline rank and a lifecycle phase. The ledger follows the real 48-layer structure: 36 GDN and 12 QSA layers, the PLE layer, the embedding, the final mixer and the LM head. Structural state, schedule-derived activations, the Dist-MoE planner's own buffer formulas and a chosen workspace allowance are kept as separate rows.

After-reduction update-state tally183.85 GiB

Planning subtotal + chosen allowance

Schedule steady-state planning budget

The dashed line is the 288-GiB physical HBM. The CUDA context and driver keep part of it, so the usable budget is smaller. Deferred gradients replace reduced shards in the steady-state phase; they are not added on top of them.

Allocation category Modeled size What the number means
Follow the bytes through one layer

What a layer saves, and what it only borrows

With full layer recompute, a layer keeps only its four-stream input until backward. Everything else is regenerated one layer at a time, with two deliberate exceptions: the Dist-MoE call keeps what its slot allows, and the QSA indexer keeps its no-grad KL target instead of redoing the teacher. Sizes follow the lab's tokens per microbatch.

Tensor Logical shape Dtype Size Lifetime
Parameter group Shape Parameters Optimizer FP32 master / GPU BF16 view / GPU

Per-GPU sizes use the lab's EP and FSDP degrees. Fused storage such as [512, 2, 640, 2560] holds two logical Muon matrices per expert; DistMuon orthogonalizes them separately and no extra copy is counted for the view.

The allocations a state formula misses

Four temporary-memory stories

These worked sizes follow the lab's token shape. They are separate examples, not four extra bars: concurrency, buffer reuse and fusion decide how many are live at once.

GDN: training is not inference

Token chunks carry matrix state

Decode carries one [48, 128, 128] FP32 state per layer. Chunked training materializes one state per 64-token chunk: ceil(T / 64) × 48 × 128 × 128 × 4 bytes. Splitting documents at 4K resets the recurrence but not the number of chunks. Full recompute makes this one layer's transient, not a saved tensor.

QSA: selection is data too

Indices and a teacher chunk

Each query reads 512 selected blocks of four keys plus up to three tail keys: [T, 2051] int64 indices. The KL teacher gathers those keys for 512 queries at a time and keeps only a [T, 512] FP32 target, which is cheaper to save than to recompute.

Dist-MoE: a planner, not autograd

Slots scale with the capacity factor

A layer must keep its input [T, D]. Saving everything for balanced routing adds x_gathered, h1 and the staged output: R × (D + 2F + D) × 2 bytes with R = 10T. The factor interpolates between the two; overflow is recomputed. Scratch is sized for the worst routed row count the run must tolerate.

Loss: a 248K vocabulary

Never hold the full logits

A full [T, 248320] logit tensor would rival a whole chunk's saved inputs. The chunked loss computes logits and their gradient one token chunk at a time, so only a slice is ever live. Its saved input is the final [T, 2560] hidden state.

Memory follows lifetime, not global batch

16 microbatches do not mean 16 saved forwards

A forward adds one saved set for its chunk; the matching backward releases it. With interleaving, a rank alternates between its two chunks, and its warmup decides how many forwards are unfinished at once. In PyTorch's interleaved 1F1B with PP2, VPP2 and 16 microbatches, rank 0 warms up with 4 forwards and peaks at 5 unfinished chunk forwards; rank 1 peaks at 3.

Interleaved 1F1B warmup on rank r = (VPP - 1) * (M / rounds) + 2 * (PP - 1 - r),   rounds = floor(M / PP)
Peak unfinished chunk forwards    = warmup + 1   (when M is large enough)
Saved per chunk forward           = 12 * T * 10240 * 2  +  3 * T * 512 * 4   bytes
Dist-MoE slots                    = peak unfinished chunk forwards (one 12-layer slot each)
Unfinished chunk forwards by logical action

The horizontal axis is the order of forward and backward actions on each rank, not elapsed time. No overlap or speed is inferred.

Schedule Layers per chunk Peak chunk forwards (rank 0 / last) Saved inputs, rank 0 Dist-MoE slots, rank 0 Nominal bubble

Interleaving halves the nominal bubble, from (PP − 1) / M to (PP − 1) / (M × VPP), but it does not halve activation memory: rank 0 holds five 12-layer chunk forwards instead of two 24-layer stage forwards, 60 layers of saved inputs instead of 48. And the stage boundary packet is the whole four-stream state: [16384, 10240] BF16, 320 MiB per microbatch per hop forward, with a matching gradient backward. VPP2 has three hops per microbatch instead of one.

An interactive fitting walkthrough

From a 2.13-TiB bill to one 288-GiB GPU

Each step changes one ownership or activation assumption for the most loaded GPU. The FP32-master / FP32-state / BF16-compute policy is kept throughout. Bars use a logarithmic axis; add the numbers, not the bar lengths.

1 MiB1 GiB64 GiB64 TiB

What would make rank 0 fit?

The anchor's rank-0 planning budget exceeds 288 GiB while rank 1 sits far below it. Each lever below changes one input of the anchor and recomputes the rank-0 budget. They are accounting alternatives, not validated recipes.

Change from the anchor Rank-0 budget Last-rank budget What it trades

Sources and accounting notes

GiB = 230 bytes; TiB = 240 bytes. Shards are modeled by ideal division without per-tensor padding. Decoder layers are split equally across PP × VPP chunks in loop order (chunk i on rank i mod PP), with the embedding on the first chunk and the final mixer and LM head on the last. The table's row owners are the DP ranks of the stage running layer 2; the "every GPU" placement is a hypothetical alternative. The PLE lookup output, router statistics, the indexer's own activations and FSDP communication buffers are omitted from saved activations and covered only by the workspace allowance. No measured run, profile or cluster information is used.