NVIDIA

NVIDIA

SEEDShallow Encoding Emits Deep Memories

Hybrid language models keep a memory of every earlier token in each layer. SEED serves them with a shallow prefill and a shallow recurrent state, on frozen checkpoints and nearly without loss. This page builds the idea step by step, and every figure on it can be played with.

Mingyuan Ma* Yangming Li* Harris Nover Zheyu Fu Po-Han Huang Alex Settle Ashwin Nanjappa†

NVIDIA

* Equal contribution: Mingyuan Ma and Yangming Li co-led the project, and their order was decided at random.  † Corresponding author.

PaperSoon CodeSoon CheckpointsSoon Training dataSoon

01 Background

What a hybrid model remembers.

A layer's memory is what it keeps of earlier tokens. An attention layer caches a key and a value for every token, so its cache grows with the context. A linear-attention or state-space layer folds every token into a fixed-size recurrent state, a dk × dv matrix per head. Hybrid models typically keep a few attention layers and make the rest recurrent.

Fixed-size does not mean small. Drag the context length and switch models to see where a request's bytes go.

Bytes of one requestpaper data
tokens in context
recurrent statesattention caches
Original
SEED

Table 2 of the paper: fp32 recurrent states and bf16 attention caches of one request; SEED's state is shown at its mean over a compression window, as in the paper's account of resident bytes. Short-convolution windows (0.8–5.0 MB) are not included, and the bytes are those of the representation, not of an engine's allocation.

Serving spends compute and memory on these memories in both phases. The prefill runs every layer for every prompt token, although only the last token's output is used. Every decode step reads and rewrites every recurrent state in full. SEED makes the first shallow in depth and the second shallow in representation.

02 Shallow prefill

Writes are easier than reads.

A deep layer writes its memory of a token from the hidden state that enters the layer, whereas predicting the next token needs the attention and feed-forward blocks of every deep layer. So SEED lets every prompt token but the last stop at an anchor depth k. Trained copies of the deeper layers' write paths, the emitters, write those layers' attention caches and recurrent states from the anchor layer's output h(k). The last prompt token and every generated token still run all layers.

The prefillschematic
layer L−1layer klayer k−1layer 0anchor layeremitters write layers k…L−1 from h(k)state Sshallow statestate Sshallow stateK,V cachestate Sshallow statestate Sshallow statestate Sshallow stateK,V cachestate Sshallow statestate Sshallow statestate Sshallow state⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯✕✕✕✕y₁x₁x₂x₃xN−1xNprompt tokensmemorieslast token1Prompt tokensrun all L layersrun layers 0…k−1 only2Deep memorieswritten by layers k…L−1written by emitters from h(k)3Last prompt tokenruns all L layersOutputs of x₁ … xN−1 are discarded.Every deep layer still holdsits caches and states.

The rows follow Qwen3.8-Flash-Next's mix of three recurrent layers to one attention layer. The layer and FLOP counts under the buttons are those of each model's released set (Table 2 of the paper).

2.1

Skip a layer for the prompt, or for the answer?

Remove one layer's output only for the prompt tokens, while the layer still writes its memory from the hidden state that enters it, or only for the generated tokens. Hover over the points.

One layer skipped in one phasepaper data
skipped for the prompt tokens, memory still writtenskipped for the generated tokens

KL from the original model over 48 held-out 4K windows (continuation steps 1–255), mean ± standard error. Figure 2 (a, b) of the paper.

2.2

Write the deep memories, or read the output?

Take the hidden state at the anchor depth and, without any training, either feed it to the deep layers' own write paths (an untrained shallow prefill) or apply the output head to it (an untrained early exit).

The same hidden state, two usespaper data
deep write paths fed h(k)early exit: output head on h(k)

KL from the original model in nats per token (log scale): 48 held-out 4K windows for the write paths, 64 for the early exit. Figure 2 (c) of the paper.

2.3

The risk is retrieval.

An average loss hardly registers a miscopied identifier, which is copied only if every one of its tokens is. A copy probe hides one record with a unique identifier in 4K–16K tokens of text. Too shallow an anchor depth fails to copy it, which is why the anchor depth also has to keep three quarters of the copies.

Identifiers copied by the untrained shallow prefillpaper data

Copied identifiers over those the original model copies, at each probed anchor depth; dashed: three quarters. Figure 2 (d) of the paper.

03 The state sink

A recurrent state is a sink plus a working memory.

A gated-delta head updates its state by St = gt(I − βtκtκt⊤)St−1 + βtκtvt⊤. The update is linear in the values, so splitting every value along a fixed direction v̄, the head's mean value, splits the state the same way: S = a v̄⊤ + C. We call a v̄⊤, the state's projection on v̄, the state sink, and C the content. When the values share a mean, the sink collects it, while the content holds mainly the recent writes, because older ones decay.

Build a state, one token at a timetoy head
decay g
write strength β
values' energy in their mean
state S
sink a v̄⊤
content C
energy share of the state's 12 strongest singular directions
–state energy in the sink
–in the strongest direction
–|cos| of that direction with v̄
–directions for 90% / 99% of the content
0tokens written

A toy, not model data: one head with dk = dv = 24, random unit keys, values made of a fixed mean plus random noise, constant decay and write strength. Raise the mean's share and the sink takes the state; shorten the decay horizon and the content needs fewer directions.

3.1

In Qwen3.8-Flash-Next

The same structure appears in the 1,728 gated-delta heads of Qwen3.8-Flash-Next, on held-out web prompts. In the median head one direction holds 85% of the state's energy; it is nearly the same direction for unrelated prompts and even for random tokens; and the content left over needs a median of 6 directions for 90% of its energy and 17 for 99%. Removing the sink costs 0.073 nats per token, while replacing it by its mean over other prompts costs only 0.002.

One direction carries most paper data

Energy share of each singular direction; one grey line per layer, the median in green.

Over the heads paper data

Share of the strongest direction, per head.

The same across prompts paper data

prompt A vs B   vs random tokens   unrelated vectors

Low-rank content paper data

Directions for 90% and 99% of the content's energy: box 25–75%, whiskers 5–95% of the heads.

Qwen3.8-Flash-Next, 36 gated-delta layers of 48 heads, held-out web prompts (Figure 3 and Section 4 of the paper).

04 Shallow state

Keep the sink and a few factors, and update them exactly.

SEED keeps each state as a v̄⊤ + UV⊤: the sink as one term and the content as rank-R factors. The model's own update extends the factors exactly, since each step only decays U and appends one column:

U ← [g U, κ],   V ← [V, β(ε − g VU⊤κ)],   a ← g a + βκ(ω − g κ⊤a)

where ω and ε are the value's parts along v̄ and orthogonal to it. A decode step reads and writes the factors instead of the dense state, and every W steps the factors are compressed back to rank R; the sink is never truncated.

The decodeschematic
Original modeldense state S per headread and rewritten in fullevery step: 16,384 numbersSEED: shallow statesink a v̄⊤never truncated+factors U V⊤, rank R + wafter a compression: (R+1)dk + Rdv numbers,4,224 at R = 161Each stepdecay U, appendone column: exact2Every W stepscompress the factorsback to rank Rw = 0 of W

Every generated token runs all layers, and the attention caches stay uncompressed. Schematic with R = W = 4; the released sets use R = W = 16 or 32.

4.1

Exact between compressions, bounded across them

Run the dense state and the shallow state side by side on the same inputs. Between compressions the factored update adds no error, so the difference only decays. Each compression adds a truncation error, and decay forgets old errors: for the same inputs and decays below some g < 1, the accumulated error stays below the largest single compression error times 1/(1 − gW), however many compressions occur.

Dense against shallow, step by steptoy head
rank R
compression period W
decay g
–largest error before the first compression (step ): rounding only
–largest error over the run
–the bound
–error at the last step, relative to the state
–numbers a 128 × 128 head would keep at the peak with these R and W: (R + W + 1) dk + (R + W) dv, against dk dv dense

A toy, not model data: one head with dk = dv = 16, random unit keys, β = 0.6, values with 80% of their energy in their mean, 160 steps, in double precision. Amber ticks mark the compressions. The error depends on how low-rank the toy's content is, which is not the model's.

Prefill · compute

Emitters

For every layer at or above the anchor depth that keeps a memory, a trained copy of the layer's write path writes its attention cache or recurrent state from h(k).

Prefill · storage

Prompt code

Except on GLM-5.3-Flash, the emitters read h(k) through a compact code, r NVFP4 coordinates plus m exact values, trained in the loop; it is also a stored form of a prompt, up to 22% smaller than its bf16 caches and shallow states.

Decode · memory

Shallow state

Sink plus rank-R factors with the exact update above, compressed every W steps; the state is 2.0–3.9× smaller right after a compression and 1.3–2.6× smaller on average.

Unchanged: the base weights stay frozen, no attention cache is compressed, and every generated token runs all layers. One reference implementation serves Gated DeltaNet, Mamba-2 and Kimi Delta Attention layers.

05 Results

Nearly lossless on six checkpoints of five production hybrid models, from 30B to over 300B parameters.

Each component set is compared with its original model on eleven official exams, question by question and with the same sampling seeds. No exam's mean score falls more than 1.5 percentage points below the original model's, and 3 of the 66 checkpoint–exam cells lie more than two standard errors below it.

Accuracy

CheckpointGPQA-DAIMEMATHMMLUIFEvalLCBHE+LB-v2R-32KR-64KR-128K
Qwen3.8-Flash-Nextbf16−0.3± 0.491.5orig. 91.9−0.6± 0.598.8orig. 99.4−0.5± 0.198.8orig. 99.3+0.1± 0.192.6orig. 92.5−0.6± 0.391.1orig. 91.7+0.4± 0.488.9orig. 88.5+0.9± 0.494.8orig. 93.9+0.9± 0.965.3orig. 64.3+3.2± 0.696.4orig. 93.2+1.2± 0.595.5orig. 94.2−0.5± 0.495.0orig. 95.5
Qwen3.8-Flash-NextNVFP4−1.1± 0.591.2orig. 92.4+0.0± 0.499.6orig. 99.6−0.3± 0.298.9orig. 99.2+0.2± 0.292.5orig. 92.3−0.1± 0.391.5orig. 91.6−0.1± 0.488.8orig. 88.9+0.2± 0.494.6orig. 94.4−0.7± 0.964.6orig. 65.3+1.4± 0.596.1orig. 94.7+0.5± 0.595.4orig. 94.9−0.7± 0.494.6orig. 95.3
Nemotron-3.5-Lightningbf16−1.3± 1.075.0orig. 76.3−0.8± 1.188.8orig. 89.6+0.1± 0.298.3orig. 98.2−0.2± 0.288.0orig. 88.1+0.4± 0.492.0orig. 91.5−0.6± 0.677.6orig. 78.2+0.1± 0.692.0orig. 91.9−1.4± 1.145.7orig. 47.1+0.0± 0.296.4orig. 96.5−0.3± 0.296.0orig. 96.4−0.3± 0.395.6orig. 95.9
Kimi-Linearbf16+0.4± 1.468.0orig. 67.6−0.6± 2.566.9orig. 67.5−0.8± 0.592.4orig. 93.2+0.6± 0.478.3orig. 77.7−1.5± 0.671.7orig. 73.2−0.9± 0.843.4orig. 44.3−0.4± 0.878.3orig. 78.7−0.1± 1.134.6orig. 34.7+0.0± 0.195.0orig. 95.0−0.1± 0.293.9orig. 94.0−0.4± 0.294.1orig. 94.5
GLM-5.3-FlashFP8−0.2± 0.591.1orig. 91.3+2.1± 1.398.1orig. 96.0−0.1± 0.298.6orig. 98.7−0.1± 0.192.8orig. 93.0+0.8± 0.490.7orig. 89.9+0.4± 0.488.2orig. 87.8−0.2± 0.394.7orig. 94.8+0.8± 0.864.1orig. 63.2−0.2± 0.196.5orig. 96.7+0.2± 0.296.5orig. 96.3−0.3± 0.296.1orig. 96.4
Nemotron-3.5-Superbf16+1.0± 0.690.1orig. 89.1+0.0± 0.798.5orig. 98.5−0.1± 0.199.4orig. 99.4+0.0± 0.192.1orig. 92.1+0.5± 0.492.1orig. 91.7+0.0± 0.486.3orig. 86.3+0.5± 0.495.5orig. 95.0−1.2± 0.858.1orig. 59.3+0.0± 0.296.7orig. 96.8−0.1± 0.294.8orig. 94.9−0.1± 0.295.9orig. 96.0

Paired difference SEED − original in percentage points, mean with its standard error over questions below; or the SEED score with the original model's below. Official graders; eight repetitions of the sampled exams, four of LongBench v2, one of MMLU and RULER. GPQA-D: GPQA-Diamond; LCB: LiveCodeBench v6; HE+: HumanEval+; LB-v2: LongBench v2; R: RULER. On Nemotron-3.5-Super, its stage-1 set. On RULER the original Qwen3.8-Flash-Next checkpoints return an empty reply on 8 to 46 of the 1,300 documents per length, which raises the official differences.

Qwen3.8-Flash-Nextbf16 and NVFP4 · Gated DeltaNet
Nemotron-3.5-Lightning30B-A3B · Mamba-2
Nemotron-3.5-Super120B-A12B · Mamba-2
Kimi-Linear48B-A3B · Kimi Delta Attention
GLM-5.3-FlashFP8 · Kimi Delta Attention, sparse attention
Frozen weightsonly the emitters and the prompt code are trained
5.1

Serving

Served with SGLang on NVIDIA GB300 GPUs, Qwen3.8-Flash-Next with SEED reaches 1.16–1.42× the original model's peak throughput and 1.2–1.5× its throughput at an equal per-user rate of 40 tokens/s. Smaller states leave room for more cached prefixes and more running requests. Pick a deployment and a view; hover over the points.

original modelshallow statesshallow prefillours: the best measured configuration

Qwen3.8-Flash-Next against the original model, all with fp32 recurrent states: aggregated NVFP4 on one GPU, aggregated bf16 on two GPUs, and disaggregated bf16 with one prefill and one decode instance (1P:1D). "Ours" is, in each view and at each concurrency, the best measured configuration for that view's metric; the text quotes the combination of both components. Per-user rates below 10 tokens/s and TTFTs above 20 s are not drawn. Figure 4 of the paper.

06 Release

We will release the code, the trained components and their training data.

Links will appear here as each item becomes public.

Soon

Paper

The full paper with proofs, measurements and the training recipe.

Soon

Code

The reference implementation, training and evaluation, and the SGLang integration.

Soon

Checkpoints

The six component sets on Hugging Face, one per original checkpoint.

Soon

Training data

The training data of every component set.

07 Citation

Cite this work

@misc{ma2026seed,
  title  = {{SEED}: Shallow Encoding Emits Deep Memories},
  author = {Ma, Mingyuan and Li, Yangming and Nover, Harris and Fu, Zheyu and
            Huang, Po-Han and Settle, Alex and Nanjappa, Ashwin},
  year   = {2026}
}