NVIDIA
SEEDShallow Encoding Emits Deep Memories
Hybrid language models keep a memory of every earlier token in each layer. SEED serves them with a shallow prefill and a shallow recurrent state, on frozen checkpoints and nearly without loss. This page builds the idea step by step, and every figure on it can be played with.
NVIDIA
* Equal contribution: Mingyuan Ma and Yangming Li co-led the project, and their order was decided at random. † Corresponding author.
01 Background
What a hybrid model remembers.
A layer's memory is what it keeps of earlier tokens. An attention layer caches a key and a value for every token, so its cache grows with the context. A linear-attention or state-space layer folds every token into a fixed-size recurrent state, a dk × dv matrix per head. Hybrid models typically keep a few attention layers and make the rest recurrent.
Fixed-size does not mean small. Drag the context length and switch models to see where a request's bytes go.
Serving spends compute and memory on these memories in both phases. The prefill runs every layer for every prompt token, although only the last token's output is used. Every decode step reads and rewrites every recurrent state in full. SEED makes the first shallow in depth and the second shallow in representation.
02 Shallow prefill
Writes are easier than reads.
A deep layer writes its memory of a token from the hidden state that enters the layer, whereas predicting the next token needs the attention and feed-forward blocks of every deep layer. So SEED lets every prompt token but the last stop at an anchor depth k. Trained copies of the deeper layers' write paths, the emitters, write those layers' attention caches and recurrent states from the anchor layer's output h(k). The last prompt token and every generated token still run all layers.
Skip a layer for the prompt, or for the answer?
Remove one layer's output only for the prompt tokens, while the layer still writes its memory from the hidden state that enters it, or only for the generated tokens. Hover over the points.
Write the deep memories, or read the output?
Take the hidden state at the anchor depth and, without any training, either feed it to the deep layers' own write paths (an untrained shallow prefill) or apply the output head to it (an untrained early exit).
The risk is retrieval.
An average loss hardly registers a miscopied identifier, which is copied only if every one of its tokens is. A copy probe hides one record with a unique identifier in 4K–16K tokens of text. Too shallow an anchor depth fails to copy it, which is why the anchor depth also has to keep three quarters of the copies.
03 The state sink
A recurrent state is a sink plus a working memory.
A gated-delta head updates its state by St = gt(I − βtκtκt⊤)St−1 + βtκtvt⊤. The update is linear in the values, so splitting every value along a fixed direction v̄, the head's mean value, splits the state the same way: S = a v̄⊤ + C. We call a v̄⊤, the state's projection on v̄, the state sink, and C the content. When the values share a mean, the sink collects it, while the content holds mainly the recent writes, because older ones decay.
In Qwen3.8-Flash-Next
The same structure appears in the 1,728 gated-delta heads of Qwen3.8-Flash-Next, on held-out web prompts. In the median head one direction holds 85% of the state's energy; it is nearly the same direction for unrelated prompts and even for random tokens; and the content left over needs a median of 6 directions for 90% of its energy and 17 for 99%. Removing the sink costs 0.073 nats per token, while replacing it by its mean over other prompts costs only 0.002.
One direction carries most paper data
Energy share of each singular direction; one grey line per layer, the median in green.
The same across prompts paper data
prompt A vs B vs random tokens unrelated vectors
Low-rank content paper data
Directions for 90% and 99% of the content's energy: box 25–75%, whiskers 5–95% of the heads.
Qwen3.8-Flash-Next, 36 gated-delta layers of 48 heads, held-out web prompts (Figure 3 and Section 4 of the paper).
04 Shallow state
Keep the sink and a few factors, and update them exactly.
SEED keeps each state as a v̄⊤ + UV⊤: the sink as one term and the content as rank-R factors. The model's own update extends the factors exactly, since each step only decays U and appends one column:
U ← [g U, κ], V ← [V, β(ε − g VU⊤κ)], a ← g a + βκ(ω − g κ⊤a)
where ω and ε are the value's parts along v̄ and orthogonal to it. A decode step reads and writes the factors instead of the dense state, and every W steps the factors are compressed back to rank R; the sink is never truncated.
Exact between compressions, bounded across them
Run the dense state and the shallow state side by side on the same inputs. Between compressions the factored update adds no error, so the difference only decays. Each compression adds a truncation error, and decay forgets old errors: for the same inputs and decays below some g < 1, the accumulated error stays below the largest single compression error times 1/(1 − gW), however many compressions occur.
Prefill · compute
Emitters
For every layer at or above the anchor depth that keeps a memory, a trained copy of the layer's write path writes its attention cache or recurrent state from h(k).
Prefill · storage
Prompt code
Except on GLM-5.3-Flash, the emitters read h(k) through a compact code, r NVFP4 coordinates plus m exact values, trained in the loop; it is also a stored form of a prompt, up to 22% smaller than its bf16 caches and shallow states.
Decode · memory
Shallow state
Sink plus rank-R factors with the exact update above, compressed every W steps; the state is 2.0–3.9× smaller right after a compression and 1.3–2.6× smaller on average.
Unchanged: the base weights stay frozen, no attention cache is compressed, and every generated token runs all layers. One reference implementation serves Gated DeltaNet, Mamba-2 and Kimi Delta Attention layers.
05 Results
Nearly lossless on six checkpoints of five production hybrid models, from 30B to over 300B parameters.
Each component set is compared with its original model on eleven official exams, question by question and with the same sampling seeds. No exam's mean score falls more than 1.5 percentage points below the original model's, and 3 of the 66 checkpoint–exam cells lie more than two standard errors below it.
Accuracy
| Checkpoint | GPQA-D | AIME | MATH | MMLU | IFEval | LCB | HE+ | LB-v2 | R-32K | R-64K | R-128K |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-Flash-Nextbf16 | −0.3± 0.491.5orig. 91.9 | −0.6± 0.598.8orig. 99.4 | −0.5± 0.198.8orig. 99.3 | +0.1± 0.192.6orig. 92.5 | −0.6± 0.391.1orig. 91.7 | +0.4± 0.488.9orig. 88.5 | +0.9± 0.494.8orig. 93.9 | +0.9± 0.965.3orig. 64.3 | +3.2± 0.696.4orig. 93.2 | +1.2± 0.595.5orig. 94.2 | −0.5± 0.495.0orig. 95.5 |
| Qwen3.8-Flash-NextNVFP4 | −1.1± 0.591.2orig. 92.4 | +0.0± 0.499.6orig. 99.6 | −0.3± 0.298.9orig. 99.2 | +0.2± 0.292.5orig. 92.3 | −0.1± 0.391.5orig. 91.6 | −0.1± 0.488.8orig. 88.9 | +0.2± 0.494.6orig. 94.4 | −0.7± 0.964.6orig. 65.3 | +1.4± 0.596.1orig. 94.7 | +0.5± 0.595.4orig. 94.9 | −0.7± 0.494.6orig. 95.3 |
| Nemotron-3.5-Lightningbf16 | −1.3± 1.075.0orig. 76.3 | −0.8± 1.188.8orig. 89.6 | +0.1± 0.298.3orig. 98.2 | −0.2± 0.288.0orig. 88.1 | +0.4± 0.492.0orig. 91.5 | −0.6± 0.677.6orig. 78.2 | +0.1± 0.692.0orig. 91.9 | −1.4± 1.145.7orig. 47.1 | +0.0± 0.296.4orig. 96.5 | −0.3± 0.296.0orig. 96.4 | −0.3± 0.395.6orig. 95.9 |
| Kimi-Linearbf16 | +0.4± 1.468.0orig. 67.6 | −0.6± 2.566.9orig. 67.5 | −0.8± 0.592.4orig. 93.2 | +0.6± 0.478.3orig. 77.7 | −1.5± 0.671.7orig. 73.2 | −0.9± 0.843.4orig. 44.3 | −0.4± 0.878.3orig. 78.7 | −0.1± 1.134.6orig. 34.7 | +0.0± 0.195.0orig. 95.0 | −0.1± 0.293.9orig. 94.0 | −0.4± 0.294.1orig. 94.5 |
| GLM-5.3-FlashFP8 | −0.2± 0.591.1orig. 91.3 | +2.1± 1.398.1orig. 96.0 | −0.1± 0.298.6orig. 98.7 | −0.1± 0.192.8orig. 93.0 | +0.8± 0.490.7orig. 89.9 | +0.4± 0.488.2orig. 87.8 | −0.2± 0.394.7orig. 94.8 | +0.8± 0.864.1orig. 63.2 | −0.2± 0.196.5orig. 96.7 | +0.2± 0.296.5orig. 96.3 | −0.3± 0.296.1orig. 96.4 |
| Nemotron-3.5-Superbf16 | +1.0± 0.690.1orig. 89.1 | +0.0± 0.798.5orig. 98.5 | −0.1± 0.199.4orig. 99.4 | +0.0± 0.192.1orig. 92.1 | +0.5± 0.492.1orig. 91.7 | +0.0± 0.486.3orig. 86.3 | +0.5± 0.495.5orig. 95.0 | −1.2± 0.858.1orig. 59.3 | +0.0± 0.296.7orig. 96.8 | −0.1± 0.294.8orig. 94.9 | −0.1± 0.295.9orig. 96.0 |
Paired difference SEED − original in percentage points, mean with its standard error over questions below; or the SEED score with the original model's below. Official graders; eight repetitions of the sampled exams, four of LongBench v2, one of MMLU and RULER. GPQA-D: GPQA-Diamond; LCB: LiveCodeBench v6; HE+: HumanEval+; LB-v2: LongBench v2; R: RULER. On Nemotron-3.5-Super, its stage-1 set. On RULER the original Qwen3.8-Flash-Next checkpoints return an empty reply on 8 to 46 of the 1,300 documents per length, which raises the official differences.
Serving
Served with SGLang on NVIDIA GB300 GPUs, Qwen3.8-Flash-Next with SEED reaches 1.16–1.42× the original model's peak throughput and 1.2–1.5× its throughput at an equal per-user rate of 40 tokens/s. Smaller states leave room for more cached prefixes and more running requests. Pick a deployment and a view; hover over the points.
06 Release
We will release the code, the trained components and their training data.
Links will appear here as each item becomes public.
Paper
The full paper with proofs, measurements and the training recipe.
Code
The reference implementation, training and evaluation, and the SGLang integration.
Checkpoints
The six component sets on Hugging Face, one per original checkpoint.
Training data
The training data of every component set.
07 Citation
Cite this work
@misc{ma2026seed,
title = {{SEED}: Shallow Encoding Emits Deep Memories},
author = {Ma, Mingyuan and Li, Yangming and Nover, Harris and Fu, Zheyu and
Huang, Po-Han and Settle, Alex and Nanjappa, Ashwin},
year = {2026}
}