Architecture notes · Nemotron 3

Nemotron 3: How Mamba-2, Experts and a Little Attention Serve Agents

NVIDIA built the Nemotron 3 family for agents, and almost every unusual choice in its architecture lines up with a specific cost that agents run into. Here is how the pieces fit, from the 31.6B Nano to the 550B Ultra.

October 2026 · 9 min read

Picture a coding agent working through a bug report. It reads a few thousand lines of the repository, runs the test suite, gets back forty thousand tokens of logs, and then writes a long reasoning trace before it touches a single file. A planner might run eight of these agents at once, each with its own history, and each history keeps growing until the task is done. Every one of those numbers turns into a cost on the GPU, and most of the cost lands in the same place.

Nemotron 3 is NVIDIA's open model family for this kind of work. Nano, with 31.6B parameters and about 3.2B active per token, came out in December 2025. Super, at 120B total and 12B active, followed in March 2026, and Ultra, at 550B total and 55B active, arrived in June. All three share one backbone, a hybrid of Mamba-2 layers, mixture-of-experts layers and a small number of attention layers, and all three support context windows of up to a million tokens. The design makes the most sense when you read it as a set of answers to the costs in that agent loop.

The cost that grows with every token

When a transformer generates text, each attention layer saves a key and a value for every token it has seen, so that the next token can look back at all of them. That store is the KV cache, and it grows by the same amount for every token in the context. Qwen3-30B-A3B, a popular open model close to Nano in size, has 48 attention layers with 4 key-value heads of 128 dimensions each. In 16-bit precision that works out to 96 KiB of cache for every token. A single session at 128K tokens holds 12 GiB of cache, about a fifth of what the model's own weights take up, and the whole cache has to be read again every time the model produces one more token.

Nemotron 3 Nano has 52 layers, and only 6 of them are attention layers. Those six use just 2 key-value heads each, so the cache costs 6 KiB per token, sixteen times less than Qwen3-30B-A3B. The same 128K-token session holds about three quarters of a gigabyte instead of twelve.

Context length 131,072 tokens

Figure 1. Cache held by one session as its context grows to a million tokens, in 16-bit precision. Each row is one layer, with layer 1 at the bottom. The left stack has Qwen3-30B-A3B's shape, with all 48 layers growing. In Nemotron 3 Nano on the right, only the 6 orange attention rows grow, the 23 teal Mamba-2 rows hold a fixed state of about 1 MiB each, and the 23 blue mixture-of-experts rows keep nothing.

What Mamba-2 keeps instead

Twenty-three of Nano's layers are Mamba-2 layers, and they do the job attention would otherwise do, which is carrying information from earlier tokens to later ones. A Mamba-2 layer keeps a fixed-size state, a set of matrices that it updates with every new token. Each token fades the state a little and then writes its own contribution into it, and the layer reads its output back out of the updated state. In Nano that state is 64 heads of 64 by 128 numbers, about 1 MiB per layer in 16-bit precision, and it stays exactly that size whether the context is ten tokens or a million. The work per token stays flat as well, because the layer reads the same small state every time instead of a cache that keeps getting longer.

The state does cost something up front. All 23 states together take about as much memory as 260 tokens of Qwen3-30B-A3B's cache, so a pure transformer is cheaper for very short prompts. Agents almost never work in that range. A system prompt, a few tool definitions and the first round of tool output usually push past it before the model has written anything.

Why keep any attention at all

A fixed-size state is a summary, and summaries lose detail. If an agent needs the exact ticket number from a log it read 200K tokens ago, a state that has been faded and rewritten two hundred thousand times since then is a poor place to look for it. Attention layers can retrieve that token exactly, because they kept a copy of it. Nemotron 3 keeps a handful of them for that reason and spreads them through the stack, so that every stage of the network has exact recall available somewhere above it.

Tokens read 0

Figure 2. The Mamba-2 state blends every token into the same fixed set of numbers, so the ticket number written early on fades as more tokens arrive. The attention cache keeps one entry per token and goes straight back to the ticket number when it is asked for. In a real model the decay is chosen per token, which lets important tokens fade more slowly, but the state never grows.

The attention layers also run without positional encodings. Nemotron 3 drops RoPE and lets the Mamba layers, which process tokens strictly in order, supply the sense of position, which removes one of the usual obstacles to stretching a context window. On the RULER long-context benchmark, the Nano base model scores 54.2 at a million tokens, while its predecessor, Nemotron Nano 12B v2, falls to 23.4 at the same length. The trade shows up at shorter lengths, where the older and denser model scores higher, 85.1 to 74.5 at 128K. The new model is built to degrade gradually as the context grows, which matters more to an agent that has no way of knowing in advance how long its history will get.

What fits on one GPU

The memory savings turn into speed through batching. A GPU generating text spends most of its time reading weights and cache from memory, so the cheapest way to get more tokens per second is to serve many sessions at once and share each pass over the weights across all of them. The number of sessions you can serve at once is set by how much memory is left for cache after the weights are loaded.

On a single 141 GB H200, Nano and Qwen3-30B-A3B each take a little over 60 GB for their weights in 16-bit precision, which leaves a little over 70 GiB for cache. At 128K tokens per session, that is room for about six Qwen3 sessions or more than ninety Nano sessions. Real serving stacks reserve memory for activations and can store the cache in 8 bits, which Nano's own recipe does, so the exact counts move around, but the ratio holds.

Context per session

Figure 3. Sessions that fit in one 141 GB H200 after loading 16-bit weights, with the cache also in 16 bits. Pick a context length to refill the bars. These are back-of-envelope counts that ignore activations and framework overhead. Qwen3-30B-A3B does not support a million tokens natively, so its 1M row only shows what the cache would need.

NVIDIA's own benchmark points the same way. With 8K tokens of input and 16K tokens of output on one H200, Nano produces 3.3× the throughput of Qwen3-30B-A3B and 2.2× that of GPT-OSS-20B. The 16K output length is the telling part of that setup. Long outputs are what reasoning models and agents produce, and every one of those output tokens pays the cache-reading cost again.

Experts, and why Super and Ultra shrink them

The other half of each Nemotron 3 block is a mixture-of-experts layer. Nano has 128 experts per MoE layer and a small learned router that sends each token to 6 of them, so only 3.2B of its 31.6B parameters do any work on a given token. You get the knowledge of a large model at the compute cost of a small one, and in an agent loop, where the same model may be called thousands of times per task, that difference shows up directly in the bill.

Super and Ultra change how the experts are built. At their scale the experts are spread across many GPUs, and the cost of an MoE layer is dominated by moving data, both reading each expert's weights from memory and shipping tokens between devices to reach the experts they were routed to. LatentMoE goes after that cost by projecting each token into a smaller latent space before routing it, typically about four times smaller, and running the experts in that space. Every expert gets cheaper to load and every token gets cheaper to send, and NVIDIA spends the savings on more experts and on more of them per token. In the paper's ablation, a standard layer with 128 experts and 6 active per token was replaced by a latent one with 512 experts and 22 active, at nearly the same active parameter count, and MMLU-Pro rose from 48.3 to 52.9 after a trillion tokens of training.

6 of 128 experts · 4,096 numbers per token

Figure 4. A standard MoE layer sends the full 4,096-number token to 6 of 128 experts. LatentMoE compresses the token to 1,024 numbers first, routes it to 22 of 512 experts, and projects the result back up. Both configurations come from NVIDIA's ablation, where they had nearly the same number of active parameters. The view switches between them on its own until you pick one.

Guessing ahead

Generation has one cost that batching cannot hide. Each token depends on the one before it, so a single long answer takes as many sequential passes through the model as it has tokens. Super and Ultra add multi-token prediction layers, small heads that predict several tokens past the next one. At inference time those predictions serve as drafts. The model checks a whole draft in a single pass, keeps the tokens it agrees with, and moves on, so a pass that would have produced one token can produce several. In NVIDIA's ablation the drafts were accepted about 97 percent of the time for the first two predicted tokens, and training with the extra heads also raised benchmark scores by about 2.4 percent on average.

Same 24-token answer, two ways

Figure 5. The top row produces one token per pass through the model. The bottom row drafts three extra tokens on each pass and checks them on the next, keeping the ones that match and dropping the rest, shown in gray. The acceptance pattern here is illustrative.

This matters most where batching helps least, in single long sessions such as a coding agent writing a large patch or a planner working through a long reasoning trace with nobody else sharing the GPU. The paper names batch size one and long-form generation as the cases the feature was built for.

Three sizes, one shape

The family splits the work by size. Nano is small enough for a single H100 or a DGX Spark, and it is built to be the model you run many copies of, such as the retrievers, tool callers and verifiers that run constantly inside a pipeline. Super is tuned for collaborative agents and high-volume work, with IT ticket automation as NVIDIA's own example. Ultra is the frontier-scale reasoner, and NVIDIA lists four B200s or eight H100s as the minimum hardware to serve it.

Model	Total / active	Released	Adds
Nano	31.6B / 3.2B	Dec 2025	Hybrid Mamba-2, MoE and attention backbone, 1M context
Super	120B / 12B	Mar 2026	LatentMoE, multi-token prediction, NVFP4 pretraining
Ultra	550B / 55B	Jun 2026	The same additions at frontier scale

Super and Ultra were also trained differently. Most of their pretraining arithmetic ran in NVFP4, a 4-bit floating-point format, over as many as 25 trillion tokens. On a GB300, 4-bit math runs at three times the peak rate of 8-bit math, and the paper reports a training loss gap against 16-bit of under one percent. The parts of the network that proved sensitive, including the attention projections, the latent projections and the MTP layers, stayed in higher precision.

Most of the decisions in Nemotron 3 come from asking what an agent actually does all day. It reads far more than it writes, it keeps its history open for a long time, it runs alongside copies of itself, and every so often it needs one exact detail from a long way back. A transformer pays for that whole history in every layer on every token. Nemotron 3 pays for a summary in most of its layers and keeps the exact record in a few, and the difference is the room where the other ninety sessions fit.