Most layers carry a recurrent state
Thirty-six GDN layers summarize history in fixed-size matrices; twelve QSA layers retain token-growing K/V caches.
The model at a glance
Forty-eight decoder layers repeat a four-layer rhythm twelve times: GDN, GDN, GDN, QSA. Every layer has 512 routed experts and one shared expert. Lexical embeddings enter at layer 2; separate gated residual modules surround attention and MoE in every layer. [2]
Text + visual inputNative multimodal input
248,320-token table
token vectors → 2560
token vectors → 2560
27-layer vision encoder
width 1152 → 2560
width 1152 → 2560
Decoder · sublayer width 256012 repeated groups · 48 layers
GDN
GDN
GDN
QSA
Every layer: 512 choose 10 routed experts + 1 gated shared expert · intermediate width 640
Four 2560-wide streams · gated read → attention / MoE → gated write
Layer 2: bigram/trigram lookup → lexical injection
OutputFinal gated read
4 streams → 2560
248,320-way LM head
Layer 2Lexical injection, then GDN and MoE, each with gated residual reads and writes.
Select a layer to update the computation diagram. Gold marks the only lexical embedding layer.
Sparsity selects what to read
A lightweight indexer scores four-token blocks. The main attention reads up to 512 complete blocks, plus the current partial block.
Residual capacity grows to four streams
Each sublayer reads a learned channel-wise mixture of the streams, computes at width 2560, and writes through one scalar gate per stream.
Lexical capacity uses sparse lookups
Sixteen bigram/trigram hash heads retrieve 160 features each. A 51.2B table supplies 2560 lexical features per token.
Follow the kernels, one layer at a time
Choose a layer and click an operation to inspect its tensor shape, weight count, and backward dependencies. Thick borders mark matrix multiplies or attention kernels.
GEMM / attentionNorm / elementwise / layoutExpert dispatch / combineBackward uses this output
T is the packed token count; R = 10T is the logical number of token-expert assignments. The diagram shows grouped operations, not a measured GPU schedule. Shapes can describe logical tensors processed in chunks.
One decoder, six views
The three main additions are QSA, gated residuals, and lexical embeddings. GDN, routed experts, and the vision pathway remain the foundation of the hybrid decoder. [1]
Gated DeltaNet: history in a fixed-size state
GDN is a gated delta-rule recurrence. Short causal convolutions prepare Q/K/V; a decay gate forgets old state, and a beta gate controls a corrective write. Chunked training computes this recurrence in parallel; token-by-token decoding carries the state forward.
Sixteen Q/K heads feed forty-eight value heads, so each Q/K head serves three value heads. The output uses per-head RMS normalization and a sigmoid gate in this release. [4]
x [T, 2560]Q/K/V + causal convQ/K L2 normGated delta ruleRMSNorm × sigmoid gateOutput → 2560
S̃t = αt St-1
St = S̃t + βt kt (vt - S̃tT kt)T
ot = StT qt
St = S̃t + βt kt (vt - S̃tT kt)T
ot = StT qt
Persistent state
The thirty-six recurrent matrices occupy 108 MiB at FP32 per sequence, independent of context length. Short-convolution history adds a small fixed term.
Compute and normalization
The main projections produce QKV width 10,240 and gate width 6,144. The beta and decay projections each produce 48 scalars per token. State accumulation and normalization use higher precision than BF16 activations.
Qwen Sparse Attention: select blocks, then read tokens
A small multi-query indexer predicts relevance for complete four-token blocks. It averages raw keys inside each block, then normalizes and applies positional encoding using the first token's position. Four query heads score each block; their rectified scores are summed.
Top-k chooses up to 512 blocks. Main GQA uses the original, unpooled K/V of the selected 2048 tokens, plus up to three tokens in the current incomplete block. Causality and document boundaries apply to both selection and attention. [3]
Indexer Q / raw KAverage 4 raw keysNorm + RoPETop 512 blocksExpand to token IDsSelected-token GQA
k̄b = RoPE(RMSNorm(mean(raw_k[4b : 4b+4])), position[4b])
score(q, b) = Σh=1..4 ReLU(qh · k̄b) / √128
selected = tokens(top_512_complete_blocks) ∪ incomplete_tail
score(q, b) = Σh=1..4 ReLU(qh · k̄b) / √128
selected = tokens(top_512_complete_blocks) ∪ incomplete_tail
Two training stages
The report introduces QSA during continued pretraining: first distill a dense-attention teacher into the indexer, then jointly train the backbone and indexer with sparse attention and a selected-block KL objective.
Two different costs
The sparse attention core reads a bounded token budget. The indexer still scores the growing set of compressed blocks; whole-sequence prefill indexing remains quadratic in sequence length, reduced by the factor-four compression.
Gated residual: four streams, one sublayer input
The persistent residual is [T, 4, 2560], or [T, 10240] when flattened. Each stream is normalized independently. A 10,240 → 320 → 10,240 bottleneck predicts an element-wise sigmoid read gate. Averaging the gated streams produces a width-2560 input for the sublayer.
Four scalar write gates send the sublayer's output back into the four streams. Attention and MoE have independent read/write modules. A final read-only mixer reduces the streams before the LM head; the reference has no separate final RMSNorm. [4]
4 residual streamsGrouped RMSNormLow-rank read gatesMean gated read → DSublayer F(x)4 scalar writes
R̂i = GroupRMSNorm(Ri)
G = σ(Wup SiLU(Wdown vec(R̂) / 4))
x = meani(Gi ⊙ R̂i), s = 2σ(Wwrite vec(R̂) / 4)
R′i = Ri + si F(x)
G = σ(Wup SiLU(Wdown vec(R̂) / 4))
x = meani(Gi ⊙ R̂i), s = 2σ(Wwrite vec(R̂) / 4)
R′i = Ri + si F(x)
Width in the residual path
Attention and expert projections operate at D = 2560. The wider residual path increases activation traffic and gives the model different learned reads of accumulated information.
Compared with Kimi's Attention Residuals
Kimi K3 retrieves earlier depth summaries using attention. GR keeps four streams and learns channel-wise read gates plus scalar writes. Its design omits an additional stream-to-stream residual mixing matrix.
Lexical embeddings: a large table with a small read
Eight bigram and eight trigram hash heads each address a distinct prime-sized vocabulary near twenty million slots. Their tables are concatenated into 320,001,536 padded rows, each with 160 features. Sixteen lookups concatenate to [T, 2560].
The lexical features project to one shared value and four stream-specific keys. Normalized residual streams gate the value. A width-four causal convolution with dilation three adds local lexical context. The result is added before layer 2's attention read. [2]
Original token IDsDocument-local n-gram hashes16 sparse lookupsKey / value projectionsStream gates + causal convAdd [T, 10240]
et = concat(lookuph(hashh(token history))) ∈ R2560
gatei = σ(sign(ai) √max(|ai|, 10-6)), ai = norm(keyi) · norm(Ri) / √2560
ui = gatei value; PLEi = ui + SiLU(causal_conv(norm(ui)))
gatei = σ(sign(ai) √max(|ai|, 10-6)), ai = norm(keyi) · norm(Ri) / √2560
ui = gatei value; PLEi = ui + SiLU(causal_conv(norm(ui)))
Weight capacity and bandwidth
The BF16 table alone is 95.37 GiB. Each token reads 5 KiB of lexical features before projections. The report places PLE at layer 2 so predictable lookups can be prefetched from host memory while layer 1 runs.
Token metadata must survive
Hashing requires original IDs and resets history at EOS. Packed training also needs document boundaries, including when PLE lives on a later pipeline stage. Multimodal rotary coordinates do not replace that lexical history.
MoE: 512 experts behind a width-640 intermediate
Every layer routes each token to ten of 512 SwiGLU experts and also evaluates one shared expert. Each expert has gate, up, and down matrices at widths 2560 and 640. Selected router probabilities are normalized; the shared expert has its own sigmoid output gate.
Routing uses FP32 softmax on activation-rounded logits, then casts expert mixing weights to the activation dtype. Small BF16 rounding differences near a tie can change the discrete expert IDs. [4]
Router [T, 512]Top 10 + normalizeDispatch R tokensGrouped expert GEMMsWeighted combine+ gated shared expert
expert(x) = Wdown(SiLU(Wgatex) ⊙ Wupx)
MoE(x) = Σe ∈ top10 pe experte(x) + σ(wsharedx) shared(x)
routed bank = 48 × 512 × 3 × 2560 × 640 = 120.796B weights
MoE(x) = Σe ∈ top10 pe experte(x) + σ(wsharedx) shared(x)
routed bank = 48 × 512 × 3 × 2560 × 640 = 120.796B weights
Storage versus selected matrices
The routed bank holds 120.796B weights. Ten experts per token select 2.359B of those weights across all layers. Routers, attention, residual mixers, and the LM head add to the token's active path.
Expert parallelism
Dispatch and combine exchange token activations across expert owners. R is a logical group-wide assignment count; expert imbalance and placement determine how many assignments each GPU receives.
Vision: patch tokens enter the same decoder
A twenty-seven-layer vision transformer uses width 1152, sixteen heads, and intermediate width 4304. A spatiotemporal patch embedding consumes [3, 2, 16, 16] patches for images and video. A 2 × 2 spatial merge projects visual features to width 2560.
The decoder uses interleaved multimodal rotary coordinates: 64 rotary channels in a 256-channel main attention head, with temporal/height/width sections [11, 11, 10]. The released configuration has no deepstack visual injection layers. [2]
Images / video framesSpatiotemporal patches27 vision blocks2 × 2 merge + MLPReplace visual token embeddingsShared decoder
Variable visual token counts
Image resolution, video sampling, and patch merging determine the decoder's visual token count. Visual tokens consume context and follow the same decoder attention and MoE paths.
Native and extended context
The checkpoint sets 262,144 positions and RoPE theta 10,000,000. The model card describes YaRN extension to 1,000,000 tokens. The hosted Qwen3.8-Flash product advertises a default 1M context and additional built-in tools.
See complete-block selection happen
Move the query forward. Complete blocks compete for a fixed budget; the partial tail is always included. This 64-token illustration uses synthetic scores and a reduced budget to make selection visible.
Selected complete blockIncluded partial tailVisible, unselectedHatched: future token
The released model uses the same block size of four, with a 2048-token budget. The scores here illustrate selection mechanics and are not predictions from the checkpoint.
Why the checkpoint is larger than 125B
The model card reports a 125B backbone, 6B activated, plus 51B n-gram embeddings and 4B MTP. The checkpoint contains 179,999,981,424 BF16 weight elements. Choose a counting scope to see where those weights live. [5]
180.000B
| Scope | Exact count | What it includes |
|---|---|---|
| Text backbone | 125,743,653,760 | Decoder, input/output tables, gated residuals, and PLE projections; excludes lexical lookup table, vision, and MTP. |
| Backbone + lexical + vision | 177,392,830,576 | The released backbone and vision weights, before MTP. |
| MTP unique weights | 2,607,150,848 | 31 checkpoint tensors under mtp.*. |
| Full BF16 checkpoint | 179,999,981,424 | All unique BF16 weight elements across 131 shards. |
The MTP configuration shares embeddings. Its checkpoint contains no dedicated copies of the input table or LM head. Counting the shared 1.271B input/output tables as part of the draft's footprint gives 3.879B; full-checkpoint storage counts each table once. The card's 4B MTP label and unique stored MTP weights therefore use different scopes.
The checkpoint also stores 35 int64 hash-buffer elements. We exclude these from trainable parameter counts. Its total tensor payload is 359,999,963,128 bytes: about 360 GB or 335.28 GiB, before runtime and allocator overhead.
Sparse reads still use a growing history cache
The 2048-token budget limits which K/V entries a query reads. A future query can select any earlier complete block, so the main K/V history remains token-growing. GDN's recurrent matrices stay constant. The pinned Transformers reference also caches raw indexer keys. [4]
12 QSA K/V caches
12 indexer key caches
36 GDN recurrent matrices
Flash-Next structural total
48-layer dense GQA baseline
Main K/V = N × 12 × 2 (K + V) × 2 KV heads × 256 × element bytes
Raw indexer keys = N × 12 × 128 × element bytes
GDN matrices = 36 × 48 × 128 × 128 × state bytes
Raw indexer keys = N × 12 × 128 × element bytes
GDN matrices = 36 × 48 × 128 × 128 × state bytes
These are logical per-sequence cache sizes before sharding. The comparison baseline uses 48 dense GQA layers with the same two KV heads and head width. Estimates exclude convolution history, PLE history, speculative draft caches, model weights, prefill activations, quantization metadata, and allocator overhead.
The training recipe follows the architecture
The report treats optimizer choice, residual stability, and memory movement as part of the design. These are findings from the released model's training study. [3]
Muon for genuine linear maps
Attention/GDN projections, expert matrices, and PLE key/value projections use Muon. Fused matrices are split into their semantic operators before orthogonalization; the report uses per-head splitting for attention and GDN.
AdamW where the structure differs
Input/output tables, routers, and GR's elongated low-rank projections use AdamW. The n-gram table uses Adam with weight decay disabled. The optimizer assignment follows the role and shape of each weight.
Prefetch lexical rows early
Hash addresses depend on token IDs, so lookups can be prepared before the decoder reaches PLE. Placing PLE at layer 2 creates overlap with layer 1's computation and makes host-memory storage practical in the report's system.
Distill before sparse training
A dense-attention teacher first trains the indexer. Sparse continued pretraining then adapts both the backbone and indexer. The report's QSA objective learns block relevance from detached main-attention probabilities.
Reported kernel efficiency: at a 1M context, the report measures 7.6× prefill and 4.9× decode speedups for
the attention module, including indexing and sparse attention. Its settings use 16K chunked prefill at batch size 1, and decode at batch
size 4 with speculative steps. These measurements apply to the report's fused kernels and benchmark settings.
The report also refits batch-size and learning-rate scaling laws and starts at the target batch size without traditional batch-size warmup. GR's gated normalization is evaluated for training stability under larger learning rates.
Sources and counting notes
- Official Qwen3.8-Flash-Next model card and release collection: features, published size labels, model overview, serving guidance, native context, and RoPE extension.
- Pinned checkpoint configuration: layer pattern, head counts, PLE placement, residual width, vision dimensions, MTP sharing, and context settings.
- On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability: GDN/QSA, GR equations, lexical ablations, sparse distillation, optimizer assignments, and kernel measurements.
- Pinned Transformers Qwen4Exp reference: exact projection shapes, gate dtypes, indexer caching, lexical injection, and final residual mixer.
- Checkpoint shape audit and configuration snapshot against the pinned safetensors index: all 1658 tensor shapes from all 131 file headers; 31 MTP tensors; BF16 weights and int64 hash buffers counted separately.
The audit reads only bounded header byte ranges and materializes no checkpoint weight tensors. Every tensor name is checked against the index, and shape-derived payload bytes match the index's total_size. The downloadable snapshot records source hashes, grouping, individual shapes, and the exact revision.
Full checkpoint: 179,999,981,424 BF16 weights. Backbone + vision before MTP: 177,392,830,576. Text backbone after removing the 51,200,245,760-weight lexical table and 448,931,056 vision weights: 125,743,653,760. Parameter count is distinct from active computation, cache memory, and physical GPU placement.
Presentation follows the Kimi K3 architecture study, which was inspired by Edward Yang's DeepSeek diagram. This page's dimensions, graph, and accounting are specific to Qwen3.8-Flash-Next.