Compression and sparsity are different
CSA stores one entry per four tokens, then chooses only a subset to read. All compressed entries remain in the cache, even those not selected by this query.
01 / The big picture
The model at a glance
Every decoder block has an MoE feed-forward network. The first three blocks use token-ID-based hash routing, not dense FFNs. Attention alternates between a lightly compressed sparse view and a heavily compressed global view, always retaining a local 128-token window.
Text inputToken embedding
129,280 vocabulary entries
T × 7,168
Replicate into 4 residual streams
Sequential decoder61 blocks at width 7,168
CSA / 4:1 + top-K
HCA / 128:1 + dense
384 routed experts, choose 6 + 1 shared
mHC around attention and MoE · 4 streams, same inner width D
Each layer owns its own compressed memory; V4 does not use V4.1-style cross-layer KV sharing.
PredictionMix back to D
Final mHC reduction + RMSNorm
Untied vocabulary head
Separate MTP block; shares embedding / output weights
CSA / 4:1 HCA / 128:1 SWA only
Gold top border = hash-routed MoE
Click a layer to update the computation diagram. The release config has one extra compression-ratio entry of 0 for the MTP block; it is not an extra decoder layer.
HCA is not fixed-state attention
Its memory still grows with context, at one entry per 128 tokens. It reads all causally available compressed entries instead of running an indexer.
Four streams do not make GEMMs four times wider
mHC contracts the residual state to one D-wide input before each attention or MoE call, then broadcasts the result back with learned mixing.
Most parameters are inactive for any one token
One token touches six routed experts and one shared expert per layer. It does not execute all 384 expert networks in Pro.
02 / From tensors to kernels
Follow one decoder block
Choose a layer and click an operation to inspect its weight shape, logical output, and backward dependencies. Heavy borders mark matrix multiplies and attention; dashed borders mark distributed expert movement.
GEMM / attentionNorm / layout / elementwiseExpert dispatch / combine
T denotes packed tokens in a logical invocation. R = 6T is the number of routed token-expert assignments across the expert group, not the workload on every GPU. The diagram assumes one continuous sequence only when showing compressed-entry counts; real packed sequences require independent causal boundaries.
03 / Two resolutions of history
Compress first. Select only when needed.
Both paths use shared-key-value multi-query attention: one 512-dimensional memory vector serves as both K and V for all query heads. The last 64 dimensions carry RoPE; attention output receives inverse RoPE before the grouped projection.
CSA: four-token compression, sparse reading
Two KV projections and two gate projections create previous/current candidates. A per-channel softmax pools eight candidates from two adjacent four-token groups into one compressed entry. Groups overlap, so the storage reduction is 4:1, not 8:1.
t0t1t2t3
+
t4t5t6t7
→
1 entry
A separate lightning indexer reuses the low-rank query latent, makes 64 query heads of dimension 128, and scores causally available compressed index keys. Top-K selects positions for the main attention.
score(t, s) = Σh wt,h ReLU(qIt,h · kIs)
- Compression4:1, overlapping
- Indexer64 × 128, FP4
- Selected entries1,024 in Pro
- At 1M context262,144 stored / layer
Report §2.3.1. The indexer still examines the compressed candidate pool; top-K bounds the expensive main attention, not every operation in the system.
HCA: 128-token compression, global reading
One KV projection and one gate projection pool each non-overlapping group of 128 tokens into one entry. No lightning indexer is needed: each query reads every completed, causally available compressed block.
128 consecutive tokens
→
1 entry
The longer pooling interval makes its global memory much smaller. The tradeoff is coarser historical detail; CSA supplies a finer-grained view in the alternating layers.
cblock = Σt in block softmaxt(zt + bt) ⊙ vt
- Compression128:1, no overlap
- Indexer / top-KNeither
- At 1M context8,192 stored / layer
- Local window128 uncompressed tokens
Report §2.3.2. The global path is dense over compressed entries, not dense over the original million-token sequence.
The local window preserves fine detail and causality. Incomplete future blocks are not available to a query. The
128-token uncompressed window covers recent tokens that compression would otherwise hide. Local and compressed candidates feed the same
attention softmax, with a learned per-head attention sink.
| Configuration | Pro | Flash |
|---|---|---|
| Decoder layers / hidden width | 61 / 7,168 | 43 / 4,096 |
| CSA / HCA / SWA-only layers | 30 / 31 / 0 | 21 / 20 / 2 |
| Query heads / head dimension | 128 / 512 | 64 / 512 |
| Query latent rank | 1,536 | 1,024 |
| Output groups / rank per group | 16 / 1,024 | 8 / 1,024 |
| Routed experts / active / shared | 384 / 6 / 1 | 256 / 6 / 1 |
| Expert intermediate width | 3,072 | 2,048 |
| CSA top-K | 1,024 | 512 |
04 / Depth and width
Four residual streams. Seven active experts.
mHC: constrain the residual highway
The residual state is T × 4 × D. Before each sublayer, a normalized flattened 4D vector produces 24 mixing values: 4 input coefficients, 4 output coefficients, and a 4 × 4 residual matrix.
stream 1
stream 2
stream 3
stream 4
Xnext = B X + C F(A X)
Sigmoid bounds A and 2 × sigmoid bounds C. Twenty Sinkhorn iterations make B approximately doubly stochastic: positive entries, with each row and column summing to one. The ideal constrained residual mapping is non-expansive and remains stable under composition.
The expensive sublayer F still receives T × D. The four-way expansion affects residual storage and the small mixing operation, not the attention and expert input widths.
Report §2.2. This is standard V4 mHC, not V4.1 Single-Pass mHC.
DeepSeekMoE: capacity behind sparse routing
Every block owns a bank of fine-grained routed experts and one always-active shared expert. Scores use sqrt(softplus), with a balancing bias affecting selection but not the gathered expert weights.
expert(x) = W2 [SiLU(W1x) ⊙ W3x]
The first three layers select experts using a token-ID hash table; later layers use score-based top-6 selection. Both execute real MoE networks. The reference clips SwiGLU activations to limit outliers.
A packed input of T tokens generates 6T routed assignments. Expert parallelism redistributes those assignments to expert owners, combines their weighted outputs, and adds the shared expert result.
- Routed assignmentsR = 6T
- Weight matrices / expert3 × D × F
- Always active1 shared expert
05 / Where the weights live
Parameter capacity is mostly expert capacity
This ledger reconstructs logical trainable parameter elements from the released dimensions and inference modules. It counts all routed experts, not just the six used by a token. Quantization changes their stored bytes, not their logical parameter count.
reconstructed decoder + embedding / head
share in routed expert banks
additional MTP parameters
| Component | Logical formula | Parameters | Share |
|---|
D = hidden width; F = expert intermediate width; L = decoder layers; E = routed experts; H/K = query heads/head dimension; q = query rank; G/r = output groups/rank; V = vocabulary. Biases, RMSNorm weights and attention sinks are included. Frozen token-ID lookup tables, quantization scales and storage padding are excluded. MTP shares the embedding and vocabulary head and is listed separately, avoiding double-counting. Headline 1.6T/284B and 49B/13B are the publisher's reported figures, not exact outputs of this calculator.
Do not turn FP4 checkpoint size into a training-memory estimate. In the instruction-tuned release, routed expert weights
use FP4 and most other weights use FP8; the base releases use mixed FP8. Gradients, optimizer state, master weights, activations,
communication buffers, and quantization metadata need separate accounting.
06 / What grows with context
A smaller cache, not a constant-size cache
Only completed blocks occupy compressed history. CSA keeps N/4 main entries and N/4 index entries per layer; HCA keeps N/128 main entries. The local ring buffer saturates at 128 entries per layer.
Mixed payload = 448 FP8 + 64 BF16 main dimensions (576 bytes), and 128 FP4 index dimensions (64 bytes). One main entry serves as both K and V, shared across all query heads.
CSA main KV
HCA main KV
CSA index keys
Local windows
V4 total
GQA8 baseline
Bars share the GQA8 baseline scale: the same number of layers, separate K and V, eight KV heads, head dimension 128, BF16. Estimates are unsharded decoder cache payload only; exclude MTP, quantization scales, page tables, unfinished compressor state, workspaces, and allocator overhead. The public Python reference uses BF16 buffers while simulating quantization, so these mixed-format payload numbers are not its measured GPU allocations.
Report §2.3.4 reports roughly 2% of this GQA8 baseline at 1M context. Separately, the model card reports Pro's single-token inference FLOPs and KV cache at 27% and 10% of DeepSeek-V3.2; that is a different baseline, not a prediction from this calculator.
07 / The optimizer is part of the recipe
Eight fast steps. Two stabilizing steps.
The V4 report explicitly uses hybrid Newton–Schulz orthogonalization inside every Muon update: eight iterations of the fast quintic, followed by two of a convergent polynomial. This is V4's published choice, not a universal Muon default.
Per-iteration polynomial coefficients
X ← aX + b(XXT)X + c(XXT)2X
- Iterations 1–8(3.4445, -4.7750, 2.0315)
- Iterations 9–10(2.0, -1.5, 0.5)
- Muon momentum0.95 + Nesterov
- Weight decay0.1
A singular value s follows p(s) = as + bs3 + cs5. The fast polynomial expands small values quickly but does not fix s = 1. The final polynomial has p(1) = 1 and p′(1) = 0.
Report §2.4, Hybrid Newton-Schulz Iterations. The schedule repeats inside each optimizer update; it is not a schedule over training steps.
Follow one normalized singular value
Illustration in JavaScript scalar arithmetic, not a BF16 matrix-kernel simulation or a training-loss comparison. Very small singular values may not reach 1 in ten iterations.
Muon needs the whole logical matrix. Sharding storage is compatible with Muon only if the distributed optimizer
reconstructs the required matrix view for orthogonalization. The report optimizes experts independently; a physical shard or a stack of
unrelated experts is not automatically the right matrix.
08 / Making the model runnable
Balance matrices, tokens, and saved tensors
The report's system combines expert dispatch with a hybrid ZeRO strategy for Muon. Dense matrices are assigned to bounded-size groups; MoE matrices are split by expert identity so that each expert can be orthogonalized independently.
1 / RouteToken scores or hash IDs choose six experts
2 / DispatchMove token assignments to expert owners
3 / ComputeExpert GEMMs and local shared expert
4 / CombineReduce weighted expert outputs
5 / UpdateWhole-matrix Muon, then restore shards
Dense and expert parameters have different ownership
Knapsack-style assignment balances dense matrix sizes. Experts are assigned independently, with data-parallel owners redundant only where the report's hybrid ZeRO policy calls for it.
Batch equal-shaped NS work
The report merges consecutive same-shaped matrices into batched Newton–Schulz computations. This improves utilization without making those matrices a single larger orthogonalization problem.
Low-precision communication still needs accurate reduction
MoE gradients are stochastically rounded to BF16 for communication. The report uses all-to-all exchange and a local FP32 sum to avoid low-precision collective accumulation errors.
Checkpoint tensor dependencies, not just modules
Tensor-level checkpointing, fused MoE kernels, and recomputed mHC intermediates reduce saved activation memory. The diagram's checkpoint option illustrates dependencies, not a measured peak-memory budget.
Report §3.4: efficient Muon, mHC, long-context communication, and extended automatic differentiation.
Sources and scope
- DeepSeek V4 Pro model card: reported model sizes, context, releases, precision and reasoning modes.
- DeepSeek-V4 technical report, HTML version: mHC, CSA/HCA, hybrid Muon and training infrastructure.
- Released Pro configuration and Flash configuration: exact layer maps and dimensions, checked October 26, 2026.
- Reference inference implementation: projection shapes, shared K/V, hash routing, compressor state, mHC and MTP weight sharing.
The formulas are educational reconstructions, not measured throughput, training convergence, or memory requirements for a specific deployment. Parallelism, padding, quantization metadata, fusion and checkpoint policies change physical ownership and allocations.