Looped Transformers: Depth Without New Parameters

Paper: Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model

Model card: Nanbeige/Nanbeige4.2-3B

This is a paper note, but the architecture is the thing I actually want to sit with. Nanbeige4.2-3B is the latest compact model that ships a looped transformer in the open. The useful question is not "is 3B the new 12B." It is: what does looping the stack change, and what does that buy a local agent.

The Claim

The authors claim that a 3B non-embedding-parameter model can be a general agent — code, office work, and multi-step tool use — without giving up competitive math, coding, and science scores.

The architectural claim underneath that is older and more interesting. A looped transformer reuses the same layer stack for extra passes. Hidden states go bottom to top, then back through the same weights. You get more serial computation without a second copy of the parameters.

Nanbeige4.2-3B is pretrained from scratch on 28T tokens with two fixed passes through a 22-layer stack. That is 44 block applications from 22 distinct layers. The released config matches that: num_hidden_layers: 22, num_loops: 2, hidden size 3072, 48 query heads / 8 KV heads, 256K context.

Those are source claims. Whether the agent numbers travel outside the authors' harnesses is a separate question, and I treat it that way below.

How a Looped Transformer Works

A normal decoder stack applies each block once:

effective applications = L × T

For a standard transformer, T = 1. For a looped transformer, T > 1, and the extra passes share the same weights. The hidden state after pass 1 is the input to pass 2. Nothing new is allocated in the parameter table. What grows is compute, activation memory, and usually the KV cache.

That is depth-wise weight tying. It is not mixture-of-experts, and it is not a bigger model with more unique layers. It is the same transformation applied again.

The idea goes back to Universal Transformers (2018): a recurrent transition function plus, in that paper, adaptive halting per token. Later work keeps the reuse and changes the control policy:

• Recurrent depth unrolls a shared block to add test-time compute.
• Mixture-of-Recursions routes different tokens to different recursion depths.
• Ouro / LoopLMs add a learned exit gate over several passes.
• Nanbeige 4.2 is the simple case: two fixed passes, no router, no early exit.

The simple case is the one I want people to have in their head first. If you understand two fixed loops, the fancier control policies are extra knobs on the same machine.

Two practical consequences follow immediately.

1. Parameters and FLOPs are no longer the same story. A 3B looped model can spend closer to 6B-class compute per token while remaining 3B on disk. That is the point. It is also the cost: decode gets slower, and peak attention memory grows with the extra pass.

2. Sharing weights does not automatically share the KV cache. Nanbeige investigated sharing KV across loops and dropped it. The paper says the sharing variant cut the cache in half but underperformed the full loop, so 4.2 keeps a separate cache per pass. A later note on Apple Silicon makes the memory penalty concrete: the second pass roughly doubles peak attention memory unless you change the prefill strategy. Looping is not a free lunch. It moves the load from parameters to compute and cache.

The Method

Nanbeige is useful as a worked example because the authors actually trained the loop from scratch instead of upcycling a finished one-pass model. They report that from-scratch looping beat conversion after the fact. That matches the intuition: the representation has to learn to be reused, not just tolerate reuse later.

They also report a loop-depth sweep. Two passes kept about 75% of the token efficiency of a standard transformer and gave the best trade-off. Three loops helped a little, spiked more often, and were not worth the extra FLOPs. In other words, the loop is an efficiency choice, not "more recurrence is always better."

The rest of the paper is post-training, and it is doing a lot of the agent work:

• Hybrid environments and trajectory synthesis for repo-level coding, tool use, and office artifacts.
• Turn-level loss masking so a bad intermediate tool call is not trained as if it were correct.
• Two-stage RLHF on Think and Non-Think responses, aimed at loops, malformed output, and failure to terminate.
• Length-controlled reasoning RL.
• Agentic RL with outcome rewards plus turn-level process rewards.

I would not collapse that into "the loop did it." The architecture is the capacity trick. The agent numbers also depend on environments, filters, and RL. Those should stay separate when we read the table.

The Hugging Face modeling code already contains later ideas — LoopSplit, mHC with depth attention, concatenated n-gram embeddings — that the authors say land in Nanbeige 4.5. 4.2 itself is the two-pass stack.

Here is the 4.2 block, drawn from the released config. Green is the second pass through the same 22 layers.

Figure from Sebastian Raschka's LLM Architecture Gallery. The numbers match config.json on the model card: embedding 3072, 48/8 GQA, head dim 128, FFN 10752, vocab 166144, context 262144.

The Results

These are the authors' numbers, from the technical report and the Hugging Face model card. Office and co-work tasks use their in-house scaffold. SWE-Bench Verified uses OpenHands, SWE-Bench Pro uses SWE-agent, Terminal-Bench 2.0 uses Terminus 2. All Nanbeige evals are in thinking mode with preserve_thinking=true. That is not an independent bake-off. It is the lab leaderboard. I still want the table on the page, because this is what they are claiming a 3B looped model can do.

Base model, before post-training

If those hold up, the loop-plus-data-recipe change from 4.1 to 4.2 is doing real work on the base checkpoint, not only on chat RL.

Instruct / agent model

Gemma4-12B still wins IF-Bench and Recruit-Bench on this sheet. SciCode also stays with Gemma. Everywhere else in the published grid, the 3B looped model is on top of the named peers.

Under a shared OpenClaw scaffold they also report:

Limitations I would keep next to the table

• Author-reported, author-scaffolded. Several office numbers are in-house.
• Looping doubles attention work and, in the released design, KV memory. That matters on 16–32GB boxes more than the parameter count does.
• The Hugging Face transformers path has needed patches; a follow-up note on Apple Silicon lists bugs that prevent the checkpoint from running out of the box.
• 4.2 is two fixed loops. It is not adaptive compute. Hard tokens and easy tokens pay the same second pass.
• Alignment is competitive, not dominant. Instruction-following is not where they claim the win.

My Read

I keep coming back to the same question I had with HRM-Text: how much of a useful agent can live in a small, reusable core if you spend compute as depth instead of as parameters.

Looped transformers are one clean answer. They are also closer to how I already think about agents. An agent is already a loop — observe, act, read the environment, try again. Putting a loop inside the network is the same instinct at a different timescale. Pass 1 does not have to be "coarse reasoning" and pass 2 "refinement." The Nanbeige authors say they do not constrain the passes that way. The first pass looks more like forming a representation than like a sketch of the answer. Still, the model is being given a second look with the same weights. That is a structural prior for iterative work.

For Oxygen AI, the local-assistant angle is the one that matters. A 3B model that is actually good at tools is in the size class you can run beside a memory system instead of renting a 70B. The architecture thesis is: keep the reasoning core compact, make memory and tools explicit, and spend extra FLOPs only where reuse helps. Looping is one way to spend those FLOPs without growing the checkpoint.

I would not ship this as "the 3B that beats 12B" without running the agent suites myself. I would ship it as: here is a readable looped stack, trained from scratch, aimed at agents, with a config you can hold in one screen. That is already enough to learn from.

Open Questions

1. How much of the agent lift is the loop versus the 28T mixture, the trajectory factory, and the RL stack?
2. Does two-pass looping still win if you match FLOPs against a deeper one-pass 3B, not against a larger unique-parameter model?
3. Can KV reuse be made good enough that the memory penalty disappears for local 256K prefill?
4. When LoopSplit lands in 4.5 — some layers looped, some not — does that recover the three-loop gains without the instability?
5. What would a trustworthy interface even show for a looped model? Intermediate states after pass 1 are sitting there. We should be able to inspect them.

Primary sources: arXiv:2607.22083, Nanbeige/Nanbeige4.2-3B, config.json. Architecture figure after Raschka. All benchmark rows above are the authors' reported numbers unless noted.