Four chunks of 12 layers
Chunk 0 holds the embedding and layers 1–12, including the PLE layer. Chunk 3 holds layers 37–48, the final mixer and the LM head. PP rank 0 runs chunks 0 and 2; rank 1 runs chunks 1 and 3.
A 125B-A6B decoder with a 51-billion-parameter lookup table. The table is free to compute and expensive to keep. The question is which GPU keeps it, and what else is live there at the same time.
Anchor recipe: PP2 with interleaved 1F1B and two virtual stages per rank, DP-shard8 / EP8 / TP1 / CP1, 16 microbatches of 16,384 packed tokens per DP rank (documents split at 4K), full layer recompute, Dist-MoE with an activation-slot capacity factor of 0.5, DistMuon plus AdamW plus Adam for the PLE table, FP32 master weights and gradient reduction, BF16 compute. No vision encoder, no MTP head. Every number on this page is derived from shapes, dtypes and schedules; none is measured.
The FSDP storage shard is the master weight: the optimizer updates it in FP32, and FSDP casts a BF16 compute view for forward and backward. There is no second, full-model BF16 master. The PLE n-gram table is the exception that shapes this whole study: FSDP ignores it, so it never gets a BF16 view. Each data-parallel rank owns a contiguous range of rows in FP32 and only the rows a microbatch looks up are cast to BF16.
| Object | Dtype | Lifetime / ownership |
|---|---|---|
| Dense resident parameter | FP32 | Sharded over the 8 DP ranks of its pipeline stage |
| Routed expert parameter | FP32 | 64 of 512 experts per EP rank; expert FSDP degree 8 / 8 = 1 |
| PLE table rows | FP32 | 40,000,192 rows per owner on the stage that runs layer 2; ignored by FSDP |
| Forward/backward compute view | BF16 | Unsharded over FSDP; under PP it stays live from the first forward to the end of the schedule |
| Gradient accumulation and reduction | FP32 | Under PP, dense gradients stay unsharded until the last backward of the schedule |
| Muon momentum | FP32 | One tensor with the storage shard's layout |
| Adam / AdamW first and second moments | FP32 | Two tensors: the PLE table (Adam, no weight decay) and the AdamW remainder |
| Model GEMMs and main activations | BF16 | Four-stream residual [T, 4 × 2560], projections, expert rows |
The architecture study audited 179,999,981,424 checkpoint elements. Remove the 448,931,056 vision-encoder and 2,607,150,848 MTP parameters and 176,943,899,520 trainable parameters remain. The recipe's first-match optimizer patterns split them three ways:
| Optimizer | Parameters | Share | Bytes / parameter | FP32 state | Share of state |
|---|---|---|---|---|---|
| DistMuon: projections, shared and routed experts | 123,003,207,680 | 69.5% | 12 | 1.342 TiB | 63.1% |
| Adam: the PLE table | 51,200,245,760 | 28.9% | 16 | 0.745 TiB | 35.0% |
| AdamW: embeddings, head, routers, gates, mixers, norms, convolutions | 2,740,446,080 | 1.5% | 16 | 0.040 TiB | 1.9% |
FP32 master + FP32 gradient + FP32 momentum
FP32 master + gradient + two FP32 moments
Update-state tally = 12 P_muon + 16 (P_adam + P_adamw)
= 2,339,089,561,600 bytes
= 2.13 TiB across the logical model
Ideal 16-way average = 136.15 GiB / GPU
The table is free to compute: a token reads 16 rows of 160 values, 5 KiB in BF16. But every one of its 51.2 billion parameters needs a master, a gradient and two moments, because Adam is applied to the whole table. Sparse access reduces arithmetic, not optimizer state. The same is true of the routed experts: choosing 10 of 512 per token does not shrink their 120.8 billion stored weights.
Chunk 0 holds the embedding and layers 1–12, including the PLE layer. Chunk 3 holds layers 37–48, the final mixer and the LM head. PP rank 0 runs chunks 0 and 2; rank 1 runs chunks 1 and 3.
512 / 8 = 64. With 8 DP ranks per stage, expert FSDP is 8 / 8 = 1: EP alone divides the expert bank. Each GPU stores 24 layers × 64 experts of its stage.
Attention, GDN, shared experts, routers, mixers, embedding and head are sharded over the 8 DP ranks of their stage. They are 2.8% of the parameters, so FSDP barely matters for the state bill here.
The 320,001,536 rows are split over the 8 DP ranks that run layer 2: 40,000,192 rows, or 6.4 billion parameters, per GPU. At 16 bytes per parameter that is 95.37 GiB per rank-0 GPU, and nothing on rank 1.
On PP rank r with dense parameters D_r, routed parameters X_r and table rows P_r: Resident parameters / GPU = D_r / 8 + X_r / EP + P_r * 160 / 8 FP32 state / GPU = 16 * (Adam + AdamW part) + 12 * (Muon part) BF16 views / GPU = 2 * (D_r + X_r / EP) # the table has no view
| Per GPU, after reduction | PP rank 0 (chunks 0, 2) | PP rank 1 (chunks 1, 3) |
|---|---|---|
| FP32 master | 53.13 GiB | 29.27 GiB |
| FP32 Muon momentum | 28.65 GiB | 28.63 GiB |
| FP32 Adam / AdamW moments | 48.96 GiB | 1.28 GiB |
| FP32 gradient shards | 53.13 GiB | 29.27 GiB |
| Update-state tally | 183.85 GiB | 88.45 GiB |
Rank 0 carries 95 GiB more persistent state than rank 1, and the table accounts for all of it. The experts are symmetric: each rank holds 24 layers of 64 experts, 28.1 GiB of master and the same of momentum. The pipeline balances layers, not bytes.
Pick a pipeline rank and a lifecycle phase. The ledger follows the real 48-layer structure: 36 GDN and 12 QSA layers, the PLE layer, the embedding, the final mixer and the LM head. Structural state, schedule-derived activations, the Dist-MoE planner's own buffer formulas and a chosen workspace allowance are kept as separate rows.
The dashed line is the 288-GiB physical HBM. The CUDA context and driver keep part of it, so the usable budget is smaller. Deferred gradients replace reduced shards in the steady-state phase; they are not added on top of them.
| Allocation category | Modeled size | What the number means |
|---|
With full layer recompute, a layer keeps only its four-stream input until backward. Everything else is regenerated one layer at a time, with two deliberate exceptions: the Dist-MoE call keeps what its slot allows, and the QSA indexer keeps its no-grad KL target instead of redoing the teacher. Sizes follow the lab's tokens per microbatch.
| Tensor | Logical shape | Dtype | Size | Lifetime |
|---|
| Parameter group | Shape | Parameters | Optimizer | FP32 master / GPU | BF16 view / GPU |
|---|
Per-GPU sizes use the lab's EP and FSDP degrees. Fused storage such as [512, 2, 640, 2560] holds two logical Muon matrices per
expert; DistMuon orthogonalizes them separately and no extra copy is counted for the view.
These worked sizes follow the lab's token shape. They are separate examples, not four extra bars: concurrency, buffer reuse and fusion decide how many are live at once.
Decode carries one [48, 128, 128] FP32 state per layer. Chunked training materializes one state per 64-token chunk:
ceil(T / 64) × 48 × 128 × 128 × 4 bytes. Splitting documents at 4K resets the recurrence but not
the number of chunks. Full recompute makes this one layer's transient, not a saved tensor.
Each query reads 512 selected blocks of four keys plus up to three tail keys: [T, 2051] int64 indices. The KL teacher
gathers those keys for 512 queries at a time and keeps only a [T, 512] FP32 target, which is cheaper to save than to
recompute.
A layer must keep its input [T, D]. Saving everything for balanced routing adds x_gathered,
h1 and the staged output: R × (D + 2F + D) × 2 bytes with R = 10T. The factor
interpolates between the two; overflow is recomputed. Scratch is sized for the worst routed row count the run must tolerate.
A full [T, 248320] logit tensor would rival a whole chunk's saved inputs. The chunked loss computes logits and their
gradient one token chunk at a time, so only a slice is ever live. Its saved input is the final [T, 2560] hidden state.
A forward adds one saved set for its chunk; the matching backward releases it. With interleaving, a rank alternates between its two chunks, and its warmup decides how many forwards are unfinished at once. In PyTorch's interleaved 1F1B with PP2, VPP2 and 16 microbatches, rank 0 warms up with 4 forwards and peaks at 5 unfinished chunk forwards; rank 1 peaks at 3.
Interleaved 1F1B warmup on rank r = (VPP - 1) * (M / rounds) + 2 * (PP - 1 - r), rounds = floor(M / PP) Peak unfinished chunk forwards = warmup + 1 (when M is large enough) Saved per chunk forward = 12 * T * 10240 * 2 + 3 * T * 512 * 4 bytes Dist-MoE slots = peak unfinished chunk forwards (one 12-layer slot each)
The horizontal axis is the order of forward and backward actions on each rank, not elapsed time. No overlap or speed is inferred.
| Schedule | Layers per chunk | Peak chunk forwards (rank 0 / last) | Saved inputs, rank 0 | Dist-MoE slots, rank 0 | Nominal bubble |
|---|
Interleaving halves the nominal bubble, from (PP − 1) / M to (PP − 1) / (M × VPP), but it does
not halve activation memory: rank 0 holds five 12-layer chunk forwards instead of two 24-layer stage forwards, 60 layers of saved inputs
instead of 48. And the stage boundary packet is the whole four-stream state: [16384, 10240] BF16, 320 MiB per microbatch per
hop forward, with a matching gradient backward. VPP2 has three hops per microbatch instead of one.
Each step changes one ownership or activation assumption for the most loaded GPU. The FP32-master / FP32-state / BF16-compute policy is kept throughout. Bars use a logarithmic axis; add the numbers, not the bar lengths.
The anchor's rank-0 planning budget exceeds 288 GiB while rank 1 sits far below it. Each lever below changes one input of the anchor and recomputes the rank-0 budget. They are accounting alternatives, not validated recipes.
| Change from the anchor | Rank-0 budget | Last-rank budget | What it trades |
|---|
torchtitan/models/qwen3_8_flash_next,
torchtitan_recipes/models/qwen3_8_flash_next.py): parameter shapes, the optimizer patterns, the row-owned table and the fixed
PLE request capacity.
torch.distributed.pipelining: the interleaved 1F1B action order, and PipelineStage disabling FSDP
gradient sync and resharding until the last backward of a schedule. TorchTitan keeps reshard_after_forward off under PP by
default.
GiB = 230 bytes; TiB = 240 bytes. Shards are modeled by ideal division without per-tensor padding. Decoder layers are
split equally across PP × VPP chunks in loop order (chunk i on rank i mod PP), with the
embedding on the first chunk and the final mixer and LM head on the last. The table's row owners are the DP ranks of the stage running layer
2; the "every GPU" placement is a hypothetical alternative. The PLE lookup output, router statistics, the indexer's own activations and FSDP
communication buffers are omitted from saved activations and covered only by the workspace allowance. No measured run, profile or cluster
information is used.