40 local caches, 4 global cache pairs
Every layer makes its own local KV. Only L3, L9, L15, and L21 produce persistent global KV and indexer K.
The model at a glance
Both halves are causal: a token can read only its own position and earlier positions. All 40 blocks have MoE, local sliding-window attention, and four residual streams. CSA2 gives the global branch a much smaller set of cache producers.
Native vision input
DeepSeek-ViT
32 layers · width 1024
14×14 patches · 2D RoPE
14×14 patches · 2D RoPE
3×3 pixel unshuffle
MLP projector → 5120
MLP projector → 5120
Language backbone · width 5120
One sequential stack, two sources of memory
Encoder / L1–20
2 SWA-only layers
3 groups of Full + 5 Reuse
Global compression 2:1
Decoder / L21–40
Full + 3 Reuse, then
4 groups of Reindex + 3 Reuse
Global compression 1:1
Encoder output H20 → global projection at L21 → shared by L21–40
Every layer: 384 routed experts, choose 6 + 1 shared · expert width 2304
Single-Pass mHC · 4 residual streams · Engram at L2 and L15
Output
Next-token head
Carried mHC input mix
Final RMSNorm
Final RMSNorm
129,280-way LM head
DSpark drafts separately
DSpark drafts separately
Encoder
Decoder
F = Full · I = Reindex · R = Reuse · S = SWA only
E badge = Engram
Layer 30 / ReuseDecoder · release index 29
Global KV: layer 21Derived from encoder output H20
Top-K indices: layer 29Own main query and local KV
Click a layer to highlight its KV and index producers and update the computation diagram below.
Layer 30 still receives layer 29
The hidden-state path remains sequential. Layer 30 also reads global memory derived from layer 20.
KV sharing and selection sharing are separate
A Reindex layer keeps the same global memory but chooses new positions. A Reuse layer keeps the latest selection too.
Prefill savings concern old prompt positions
Optimized serving skips most decoder work for the prompt. Each generated token and each teacher-forced training token runs all 40 layers.
Follow one block through its tensors
Choose a layer, then select an operation to inspect its weights, precision, and backward dependencies. Solid heavy borders mark matrix multiplies or attention; dashed boxes show shared state or communication.
T counts current query/local positions; G counts new global source positions; S counts retained global entries. R = 6T counts routed token-expert assignments. During optimized decoder prefill, the global projection can process the full encoder prompt while the query/local path processes 128 replay positions. Encoder pooling emits only completed pairs, including any carried partial pair. Batch/document boundaries are suppressed here; causal masks preserve them. Deployment can fuse several logical operations into one kernel.
Traced from inference/model.py. Its
small default ModelArgs describe a debug model; the dimensions here come from the released config.json.
Encode the prompt, then read its memory
CED changes where decoder global KV comes from. It is projected from the last encoder output H20, so the global memory of an old prompt position can exist without computing that position through L21–40.
The current hidden state always follows the layer chain
L1 → L20 → L21 → … → L40
Encoder / 20 layers
All 65,536 prompt tokens
Decoder / 20 layers
Last 128 prompt tokens
Global KV from encoder outputs across the full prompt → decoder attention
Prefill computes H20 for the full prompt, projects decoder global KV, then runs the decoder on the last 128 positions.
h30[t] = Block30(h29[t], shared_global_KV(H20), local_KV30)
Layer 30 depends on both the sequential path through L21–29 and the encoder-derived global memory.
Illustrative token-layer work
50.1%of a full 40-layer pass over the same positions
Encoder
Decoder
Full-stack reference
20N + 20 min(N, 128)
Counts token-layer executions, treating layers equally. Projection costs, sparse indexing, attention work, and hardware scheduling determine actual FLOPs and latency.
The 128-token decoder replay is approximate. Every decoder layer generates its own local KV from its own incoming hidden state.
Local dependencies accumulate across depth, so exact reconstruction can require about 20 × 128 = 2,560 recent positions. Bounded replay processes only 128, truncating local attention at the replay boundary while preserving access to global memory. The report finds negligible quality degradation and simulates this replay during post-training.
Report §2.2 and §3.2.2. The released minimal inference example runs the full stack; bounded replay describes the optimized serving system.
CSA2: separate producing, selecting, and reading
Each attention layer reads its local window plus at most 512 selected global entries. Main queries are always layer-specific. The three modes decide who makes the global memory and who chooses its positions.
| Mode | Examples | Global KV + indexer K | Indexer query + Top-K | Main query + local KV |
|---|---|---|---|---|
| Full | L3, L9, L15, L21 | Generate new memory | Compute fresh selection | Compute independently |
| Reindex | L25, L29, L33, L37 | Reuse L21 memory | Own query; fresh selection | Compute independently |
| Reuse | 30 layers, including L30 | Reuse the group's memory | Reuse latest Top-K; no indexer query | Compute independently |
Encoder groups compress pairs
L3–8, L9–14, and L15–20 each have a separate global store at ratio 2:1. A learned channel-wise softmax pools each pair; groups do not overlap.
One global store spans the decoder
L21 projects H20 at ratio 1:1. All 20 decoder layers share this store. Four deeper indexers update which entries their groups read.
Hierarchical indexing bounds the deeper searches
L21 / initial searchFull visible context
Score causally visible positions. Select Top-512 for L21 and score each 8-position block by its maximum.
Shared candidate pool≤16,384
Keep up to 2,048 blocks × 8 positions. The pool is larger than the final attention selection.
L25 / L29 / L33 / L37≤512
Each Reindex layer rescores only the candidate pool with its own indexer query, then selects its own Top-512.
The first search still grows with context length. Later decoder searches are bounded by the fixed candidate pool. Reuse layers perform no search. The candidate restriction is introduced during post-training and applied during inference.
Relative to V4 CSA, the compressor removes overlap and learned absolute position embeddings. Indexer K is projected from the main KV latent before RoPE. The core attention still gathers local and selected global latents, with a per-head attention sink.
Report §2.3 and the released compressor, indexer, and attention implementations.
Engram: a huge phrase embedding with a gate
An ordinary embedding maps a token ID to a learned vector. Engram maps recent 2-, 3-, and 4-token groups to learned vectors. The current hidden state decides how much of that memory to add to each residual stream.
One Engram module / addressing demonstration
24 fetched vectors × 256 channels = 6,144 channels
Concatenate 8 hash-head vectors for each of 3 orders. A learned projection produces four keys and one shared value.
[T, 6144] → Wkv → keys [T, 4, 5120] + value [T, 5120]
Updated stream = H + 0.60 × V
The actual model computes a gate separately for each stream from normalized hidden/key compatibility. Words here are illustrative tokens; the release hashes compressed tokenizer IDs using multiple heads and distinct prime table sizes. The slider demonstrates gating and does not run the model.
48 rowsaccessed per text token across both modules
12 KiBraw FP8 lookup payload, before scales and communication
196.6Btable elements derived from the released config
Capacity grows faster than arithmetic
Only a few rows are accessed per token. The huge table adds learned pattern capacity with small projections instead of evaluating all table parameters.
Addresses are available early
Lookup addresses depend only on token IDs. Embeddings can be prefetched before a stage runs, overlapping transfers with GPU computation.
Context controls the contribution
The same short phrase can appear in different contexts. Gating allows its retrieved vectors to contribute differently to the current hidden state.
Stored model weights, trained end to end
Engram entries persist across requests as model parameters. Training updates selected rows; inference reads them. KV caches instead hold request-specific activations.
Modules sit at L2 and L15 (release indices 1 and 14), with 8 hash heads per order and about 16M entries per head. The V4.1 variant omits the original Engram short causal convolution and uses momentum plus Sinkhorn balancing for table updates. Table storage and lookup bandwidth remain substantial; the report does not give an isolated Engram accuracy ablation.
Two kinds of model capacity
The backbone stores most of its weights in routed expert matrices. Engram adds a second form of sparsity: token-addressed table rows. Both activate a small subset of their stored capacity at each position.
Reported headline counts / approximately 748B combined
Routed expert bank: 543.58B
40 layers × 384 experts × 3 SwiGLU matrices. Each expert has gate/up/down matrices between model width 5120 and expert width 2304.
40 × 384 × 3 × 5120 × 2304
= 543,581,798,400 matrix parameters
= 543,581,798,400 matrix parameters
A token selects 6/384 of the routed bank, plus one shared expert in every layer. Main attention, norms, projections, embeddings, and output head add the remaining backbone weights.
Engram tables: 196.61B
The released config specifies two row totals and 256 channels per row. This tally counts table elements, separately from projection and gating parameters.
(384,006,168 + 384,016,682) × 256
= 196,613,849,600 table elements
= 196,613,849,600 table elements
At one FP8 byte per element, raw tables occupy about 183.11 GiB, before scales. Only 48 rows are selected per text token across the two modules.
The report's 552B and 196B are headline counts rather than a complete checkpoint tensor audit. Its 8B prefill / 16B decode figures describe the optimized execution path. Parameter counts, active arithmetic, stored weight bytes, training optimizer state, and KV memory are different quantities.
Reconstruct the 890-byte global cache
The three encoder producers each retain one entry per pair of tokens; the decoder producer retains one per token. That is about 2.5 global entries per context token across the model.
Single request / logical serving KV storage
892.50 MiBshared global KV + local FP8 payload
Main global KV
Global indexer K
40 local windows
2.5 × (256 + 32 + 64 + 4) = 890 bytes per context token
Global values include their quantization scales. Local values count payload only.
Before parallel replication or sharding. Excludes local quantization scales, allocator/page padding, sparse-index and candidate buffers, compressor state, and activations used for training.
Four global memory producers
3 × floor(N/2) + N entries
Main KV: 512 channels / FP4
256 packed bytes + 32 E4M3 scales
Indexer K: 128 channels / MXFP4
64 packed bytes + 4 UE8M0 scales
256 packed bytes + 32 E4M3 scales
Indexer K: 128 channels / MXFP4
64 packed bytes + 4 UE8M0 scales
The main format has one scale per 16 channels. The indexer uses one scale per 32 channels. Both keys and values of main attention use the stored latent.
The independent-cache comparison uses the same 38 CSA2 layers, widths, compression ratios, and precision, with a separate global store at every layer.
Reported runtime KV: roughly 1/4 of V4 Flash
CSA2 sharing and FP4 main KV jointly reduce the actual long-context serving footprint. At 1M tokens, global KV is 890 MiB in the logical model.
Reported persistent KV: roughly 1/8
Global KV stays in persistent storage. Encoder SWA state uses a short-lived memory pool; decoder SWA is rebuilt from the prompt tail. Bounded replay recovers missing local state.
Model card and report Figure 1; §2.4.4 and §3.2. The minimal reference simulates cache quantization in dequantized tensors; its allocated buffers do not directly demonstrate packed-cache memory usage.
The other pieces of the design
CED and CSA2 target long-context compute and cache storage. These additions address residual traffic, numerical formats, vision, and generation throughput.
Single-Pass mHC
Each sublayer predicts input, output, and residual mixing coefficients. The input mix used now comes from the previous sublayer, removing a reduction dependency from the critical path.
X[l+1] = B[l] X[l] + C[l] F(A[l-1] X[l])
Mega-mHC fuses residual update, input mixing, coefficient prediction, pre-norm, and FP8 conversion. The report halves mHC activation traffic relative to its original implementation; this is a residual-module saving.
The final stream merge consumes the carried mixing coefficients too.
FP4 main KV
Quantize the full 512-channel latent after RoPE into E2M1, with one E4M3 scale per 16 channels and no second-level global scale.
512 × 4 bits + 32 × 8-bit scales
= 288 bytes per main KV entry
= 288 bytes per main KV entry
Attention dequantizes cached values before using them. Main KV QAT is introduced during post-training. Local SWA KV stays FP8 because it is more sensitive to quantization.
DeepSeek-ViT
A 32-layer, width-1024 ViT uses 16 heads, 2D RoPE, RMSNorm, and SwiGLU. Linear patch embedding keeps the projection compatible with Muon.
1344×1344 image → 96×96 patches
3×3 unshuffle → 32×32 visual tokens
MLP projector → 5120 channels
3×3 unshuffle → 32×32 visual tokens
MLP projector → 5120 channels
At that resolution, 9,216 patch tokens become 1,024 language-model visual tokens. Image and text expert routing have separate load-balancing correction biases.
DSpark
Three drafter blocks produce base logits for five draft positions in parallel. A lightweight Markov head models token dependencies; a confidence head estimates acceptance probabilities.
3 draft blocks → 5 positions
Confidence + engine load → verification length
Confidence + engine load → verification length
The scheduler adapts verification length to expected system throughput. DSpark is trained after backbone pre-training, then kept aligned during post-training without sending its objective's gradients into the backbone.
Report §2.4 and §2.1.1.
Training needs explicit ownership and lifetimes
The report pre-trains on 45T multimodal tokens, using sparse attention from the start at 64K context. Context expands to 1M at 34T tokens. Teacher-forced execution keeps the full 40-layer graph; shared activations introduce distributed responsibilities.
Carry values and their gradients
Shared KV, indexer keys, and selections can cross pipeline stages. Per-microbatch state tracks their consumers through forward, recomputation, and backward, retaining them until the last consumer finishes.
Shard rows, prefetch lookups
Dedicated process groups own table row partitions. Prefetch begins before stage microbatches. Lookup gradients are buffered and returned to their owning ranks after backbone backward; optimizer states are further sharded.
Match updates to the matrices
Backbone linear matrices use Muon, with head-wise treatment for Q/K. Engram tables, token embeddings, and prediction heads use momentum plus Sinkhorn balancing. RMSNorm and other non-matrix parameters use AdamW.
What this means for existing V4 code
| Piece | Reusable foundation | V4.1 integration work |
|---|---|---|
| Attention kernel | Gather local + selected global latents and normalize with the sink | Support packed FP4 reads and the chosen layout; expose shared KV and supplied selections |
| Compressor / indexer | Projection, normalization, pooling, and Top-K primitives | Ratio 2 encoder / ratio 1 global decoder; new indexer K source, three modes, hierarchical candidates |
| MoE / residuals | Routed and shared experts; Sinkhorn and residual combination math | Modality-specific routing biases; carried Single-Pass mHC input coefficients |
| Model / distributed runtime | Common decoder, linear, normalization, and sharding infrastructure | CED source relationships, shared-state ownership and gradients, new checkpoint mappings |
| Engram / vision / drafter | Embedding, ViT, and transformer building blocks | New table lookup integration, DeepSeek vision configuration, and DSpark algorithm |
The existing TorchTitan V4 attention wrapper selects CSA for ratio 4, HCA for other compressed ratios, and local-only attention for ratio 1. V4.1 requires indexed global attention at both ratios 2 and 1, so the model needs new control flow and configuration wiring.
Report §3.1 and TorchTitan V4 implementation. The reuse table is an implementation assessment.
Sources and accounting notes
- DeepSeek-V4.1-Flash Technical Report: CED (§2.2), CSA2 (§2.3), architectural extensions (§2.4), optimizer (§2.5), distributed training and serving (§3). An HTML version on arXiv provides section links.
- Hugging Face model card and weights and release files: headline counts, 890 bytes per global context token, model evaluation, quantized release, and inference entry points.
- Released config.json: layer and source indices, head dimensions, expert sizes, Engram row counts, vision configuration, and DSpark settings. Its compression list also contains three drafter blocks; this map covers only the 40-layer backbone.
- Reference model, Engram addressing, and kernels: tensor paths, gating, quantization blocks, and shared attention interfaces.
- The Kimi K3 architecture study supplies this page's presentation style. The logical kernel-diagram format follows the approach of Edward Yang's DeepSeek diagram.
Source review: October 26, 2026. Calculations use one context and one logical model copy. Global KV includes specified quantization scales; local KV counts FP8 payload. The CED work plot is an equal-layer execution illustration, not a performance benchmark. Engram addressing and gating controls are schematic examples, not model inference.