Özer Cem KELAHMETKELAHMET
Articles · Trends

SenseNova Loop-9B explained: how a looped transformer runs 32 layers as 56

SenseTime has released SenseNova Loop-9B, an open-weight multimodal model that runs its middle 24 layers twice. It is a practical test of an old idea: buy depth with compute instead of with more parameters. Here is how looping works and what it costs you.

At a glance
Reading time
16 min
Sections
10
Code examples
1
Tables
3
Sources
10
Published
20 Nov 2026
In 30 seconds
What changed
SenseTime ships SenseNova Loop-9B Preview
Why it matters
Loop-9B gives teams with one workstation GPU a new option.
What to do
Separate the two parameter counts

SenseTime has released SenseNova Loop-9B, an open-weight model that takes text and images and produces text. On paper it is small: about 9 billion parameters and 32 decoder layers. But it does not use those layers the normal way. A block of 24 middle layers runs twice in every forward pass, so each token goes through 56 layer computations instead of 32.

This matters for two reasons. First, it is one of the clearest public releases of a "looped" or "recurrent-depth" transformer that you can download and run. Second, it offers a different trade-off. Most open models get stronger by adding parameters, which costs memory. A looped model gets deeper by spending more compute on the same weights.

What happened

SenseTime ships SenseNova Loop-9B Preview (30 October 2026)

The first public version was a Preview. Its Hugging Face card dates the release to 30 October 2026 and describes it as "a looped multimodal model for long-context and agentic workflows." It already had the looped layout of 32 distinct layers executed as 56 layer computations, was trained on sequences up to 262,144 tokens, and was released under the Apache License 2.0.

The official release: Loop-9B with 512K context (17 November 2026)

SenseTime published the official release on Tuesday 17 November 2026. TechNode reported it the next day, describing a model that "increases computational depth by running some of its layers more than once." According to the model card, the context length was extended to 512K tokens, and the release "outperforms the Preview on all seven benchmarks shown." The license remains Apache 2.0, but the Hugging Face repository is gated: you must agree to share your contact details before downloading.

Quantized AWQ and GGUF builds follow (18 November 2026)

SenseTime also published a 4-bit AWQ version and GGUF files. The release note says the AWQ version was available on launch day; the card's news log records the AWQ and GGUF links as added on 18 November. The GGUF files come in BF16, Q8_0, Q5_K_M and Q4_K_M. The GGUF card says to build a SenseTime fork of llama.cpp at a pinned commit, which suggests the looped layout is not yet a standard architecture in mainline llama.cpp.

How a looped transformer works

A standard transformer is a stack of layers, each with its own weights. A token's representation enters layer 1, gets refined, moves to layer 2, and so on. To make the model deeper, you add layers, and each new layer adds parameters.

A looped transformer breaks the link between depth and parameters by using some layers more than once. Think of a cook who reads a recipe once but stirs the pot several times. The instructions (the weights) stay the same. The food (the hidden state) changes with each pass.

The Loop-9B layout

The model card describes the layout precisely:

  • Layers 0 to 3 run once, turning tokens into a working representation.
  • Layers 4 to 27 form the looped block. These 24 layers run twice in sequence with the same weights. The output of the first pass is the input of the second.
  • Layers 28 to 31 run once, followed by the output head that predicts the next token.

So the count is 4 + (24 x 2) + 4 = 56 layer computations from 32 physical layers. Both the Preview and the release use two loops by default. The card also lists a hidden size of 4,096, a feed-forward size of 12,288, a vocabulary of 248,320 tokens, and 16 full-attention heads with 4 KV heads. The backbone is stored in BF16.

Hybrid attention inside the loop

Loop-9B does not use the same attention in every layer. The card says the decoder repeats a four-layer pattern: three linear-attention layers followed by one full-attention layer. Loop boundaries line up with complete groups, so each pass through the looped block contains six groups.

Full attention lets every token look at every earlier token. It is accurate but grows expensive with long inputs and needs a growing KV cache. Linear attention keeps a fixed-size running summary instead, so its cost grows more gently. Using full attention in only one layer of every four is a common way to make very long contexts such as 512K affordable.

Two residual streams and mHC

The card also lists "two-stream mHC." In a normal transformer, each layer adds its output to one running vector, the residual stream. This path lets information and gradients flow through a deep network.

mHC stands for Manifold-Constrained Hyper-Connections, described in a December 2025 paper by Zhenda Xie and colleagues. Hyper-connections widen the residual stream into several parallel streams with learned mixing. The paper says plain hyper-connections can cause "severe training instability and restricted scalability," and that mHC constrains the mixing so the residual path keeps its identity-mapping property.

SenseTime's card says Loop-9B has two residual streams, and the map that mixes them is constrained with Sinkhorn normalization using 20 iterations, described as "repeated row and column normalization." The card notes that these 20 iterations are separate from the two loop passes. The mHC routing parameters are kept in FP32.

Compute versus parameters: what "14.6B inference-equivalent" means

The card lists the model as "approximately 9B total parameters; approximately 14.6B inference-equivalent parameters." The two numbers measure different things.

Stored parameters are what you download and hold in memory. For Loop-9B, about 9 billion. They set file size and the GPU memory for weights.

Inference-equivalent parameters describe how much computation each token goes through. Because 24 layers run twice, each token does roughly the work of a larger non-looped model, which SenseTime puts at about 14.6 billion. Research papers often call this "effective parameters."

A gym analogy helps. Lifting the same set of weights twice gives you more exercise, but it does not take up more floor space. Floor space is memory. Exercise is compute.

Table 1. Compute versus parameters: what "14.6B inference-equivalent" means
Property Normal 9B model Loop-9B (2 loops) Normal ~15B model
Weights stored in memory ~9B ~9B (card) ~15B
Compute per token baseline higher, roughly like ~14.6B (card) higher
Layer computations one per layer 56 from 32 layers (card) one per layer
Extra loops at inference not possible SenseTime tested 1 to 4 loops (card) not possible

Why reusing layers can help reasoning

Looped transformers have a clear research history, and Loop-9B is best read as a large-scale product version of it.

Universal Transformers (2018)

In 2018, Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit and Łukasz Kaiser published "Universal Transformers," described as a "parallel-in-time self-attentive recurrent sequence model." The same block is applied again and again over depth, the way a recurrent neural network applies the same cell over time. The paper added a dynamic per-position halting mechanism, so each position could decide when to stop refining. The authors also showed that, under certain assumptions, the Universal Transformer is Turing-complete, unlike the standard transformer.

Depth matters more than parameters for some tasks (2025)

In February 2025, Nikunj Saunshi and colleagues published "Reasoning with Latent Thoughts: On the Power of Looped Transformers." Their central claim is that "many reasoning problems require a large depth but not necessarily many parameters." On synthetic tasks such as addition and multi-hop lookups, a k-layer transformer looped L times nearly matched a kL-layer model and clearly beat a plain k-layer model. They also proved that looped models can simulate T steps of chain-of-thought with T loops.

Many reasoning tasks are step-by-step procedures: carry a digit, follow a link, apply a rule, repeat. A procedure that repeats the same kind of step suits a network that repeats the same layers. Extra parameters help store more facts. Extra depth helps run more steps.

Recurrent depth as test-time compute (2025)

Also in February 2025, Jonas Geiping and colleagues published "Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach." Their proof-of-concept model had 3.5 billion parameters, was trained on 800 billion tokens, and repeats a recurrent block that can be unrolled to arbitrary depth at inference. The authors report that reasoning scores rise with more iterations, reaching a compute load equivalent to a 50-billion-parameter model.

This links looping to test-time compute. Today's reasoning models mostly think by writing out more tokens. A recurrent-depth model can think by running more internal passes. The two can be combined: Loop-9B also has a thinking mode, on by default.

If you read our piece on Titans and MIRAS, note the difference. Those designs change how a model remembers across a sequence. Looping changes how much computation each token gets across depth.

What SenseTime reports about loop count

SenseTime tested Loop-9B with different loop counts against a single-pass baseline. These are company figures:

  • At three loops, AIDABench-QA improved 6.42% and Workbuddy-office improved 16.93% over one loop.
  • At four loops, LiveCodeBench V6 improved 8.10% and AIDABench data visualization improved 8.55%.
  • Going from two loops to four added 1.04 points on LiveCodeBench V6 and 1.35 points on data visualization.

The card says "the best results at three or four loops," yet both versions ship with two loops for efficiency.

The costs of looping

The memory saving on weights is paid for elsewhere.

Latency and FLOPs grow with loops. Every pass through the looped block is a full set of layer computations: 56 per token with two loops, and a third loop would add another 24. Time per token rises roughly with the number of layer computations, so a looped 9B model will generally decode more slowly than a plain 9B model of the same width.

The KV cache may grow too. Full-attention layers store keys and values for earlier tokens. In a looped model, the second pass through a layer sees a different input than the first, so in general its keys and values differ, and a straightforward implementation would cache them per pass. The card does not say how SenseTime's code handles this, so treat it as a general property of looping, not a confirmed Loop-9B detail. The hybrid design limits the effect, since only one layer in four uses full attention.

Less runtime support. The card asks you to build vLLM from pull request #64832 and SGLang from #47281, and the GGUF build needs a SenseTime fork of llama.cpp. SenseTime's release note adds that the vLLM pull request currently supports unquantized text inference only, not the AWQ 4-bit configuration. The card's own examples are less clear, so check before you deploy.

Memory, context and quantization

SenseTime's main pitch is that Loop-9B runs on modest hardware. The card gives these estimated minimum GPU memory figures for multimodal inference with 4-bit weights:

Table 2. Memory, context and quantization
Context length 8-bit KV cache 4-bit KV cache
64K ~10 GiB ~9 GiB
128K ~12 GiB ~10 GiB
256K ~15 GiB ~12 GiB

The card says that "with 4-bit weight quantization, the model weights occupy approximately 4.9 GB." The release note describes deployment memory "as low as under 10GB," which matches the low end of the table. The card gives no memory figure for the full 512K context.

The table shows a basic lesson about long context: once weights are compressed, the KV cache becomes the main growing cost. Going from 64K to 256K adds about 5 GiB with an 8-bit cache. Quantization of the cache, not just the weights, keeps long contexts within reach.

Table 3. Memory, context and quantization
Format Size (card) Main runtime
BF16 safetensors full precision Transformers 5.18.0, vLLM and SGLang branches, LightLLM
AWQ 4-bit ~4.9 GB weights LightLLM example on one GPU; includes an MTP draft
GGUF Q4_K_M 5.82 GB SenseTime llama.cpp fork
GGUF Q5_K_M 6.67 GB SenseTime llama.cpp fork
GGUF Q8_0 9.8 GB SenseTime llama.cpp fork
GGUF BF16 18.4 GB SenseTime llama.cpp fork

The AWQ repository includes an optional multi-token prediction (MTP) draft for speculative decoding, where a quick draft proposes several tokens that the main model checks in one go. For a looped model, where each token costs more, that speed-up is especially useful. The card says further draft weights and launch settings "will be updated." For background on GGUF quant names like Q4_K_M, see our article on llama.cpp 1.0.

Benchmarks as SenseTime reports them

All scores below are SenseTime's own. The full tables on the card are images, so we list only figures stated in text. Against Qwen3.8-27B:

  • NL2Repo: 41.36 for Loop-9B versus 44.60 for Qwen3.8-27B.
  • AIDABench: 65.31 versus 65.01.
  • AutomationBench: 36.67 for both.

SenseTime describes the release as "on par with Qwen3.8-27B" on agent, coding and data analysis tasks. TechNode reports the narrower phrase "competitive with Qwen3.8-27B on selected agent and data-analysis benchmarks," which is the fairer reading of three numbers, one of which Loop-9B loses.

Against its own Preview, the release improved on AutomationBench (14.33 to 36.67), Terminal-Bench 2.1 (40.45 to 59.55), GAIA (37.86 to 69.90) and BIRD (71.38 to 72.34).

The card also lists limitations: the model can struggle with many tool definitions or constraints, its coding and general agent performance is inconsistent, long-horizon tasks still need work, and complex 3D or animated web pages may need two to three rounds of feedback.

What it means in practice

Loop-9B gives teams with one workstation GPU a new option. Instead of choosing between a small, fast model and a large model that does not fit, you can run a model whose weights fit like a 9B model but whose per-token work is closer to a mid-sized one. The cost is speed, not memory.

For document and data-analysis work, the 512K context and low-memory KV cache options are the practical draw. If SenseTime's estimates hold, a 256K context in about 12 to 15 GiB puts long reports, codebases or spreadsheets within reach of a single mid-range card.

For latency-sensitive uses, such as chat that must feel instant or high-volume serving, the looped design fits less well. You pay for 56 layer computations per token while getting the memory profile of 32. Whether that is a good deal depends on whether your bottleneck is memory or time.

What to do now

  1. Separate the two parameter counts. Use about 9B for memory planning and about 14.6B as a rough guide to compute per token.
  2. Measure latency on your hardware. Compare tokens per second against a normal 9B model and any larger model you already run.
  3. Start with the AWQ build for single-GPU tests. The card puts 4-bit weights at about 4.9 GB, and its LightLLM example runs on one GPU.
  4. Quantize the KV cache for long contexts. At 256K, the card's estimate falls from about 15 GiB to about 12 GiB with a 4-bit cache. Check that quality holds on your tasks.
  5. Pin runtime versions. vLLM, SGLang and llama.cpp support depends on specific branches and a fork. Do not assume mainline releases load the model correctly.
  6. Run your own evaluation. The benchmark figures are SenseTime's. Test on your real agent, coding or data tasks, keeping the card's warnings about long-horizon work in mind.
  7. Try speculative decoding. The MTP draft can offset some of the extra per-token cost of looping.
  8. Review the gated access terms. The license is Apache 2.0, but downloads require sharing your details on Hugging Face. Check that this fits your policy.

Risks and open questions

Company figures only. Every benchmark number comes from SenseTime, and full tables appear only as images. Treat claims of parity with 27B models as unconfirmed until independent runs appear.

KV cache behaviour across loops is undocumented. The card does not say whether the second pass through each full-attention layer keeps its own cache, and it gives no memory estimate at the full 512K context.

Tooling is young. Branches, forks and pinned commits can break or lag behind mainline releases. Vision support in the GGUF build is not described.

Adjustable loops are a research result, not a feature. SenseTime showed gains at three and four loops, but the card does not document a supported way to change the loop count in production runtimes.

Watch next: whether looped-architecture support is merged into mainline vLLM, SGLang and llama.cpp; independent benchmarks against Qwen3.8-27B; a technical report on training and the KV cache design; and whether other labs ship recurrent-depth models with a user-facing loop setting.

Frequently asked questions

What is a looped transformer?

A looped transformer runs some of its layers more than once per forward pass, with the same weights each time. This adds computational depth without adding stored parameters. SenseNova Loop-9B runs its middle 24 layers twice, for 56 layer computations from 32 physical layers.

Is Loop-9B really a 9B model or a 14.6B model?

Both numbers are correct but measure different things. It stores about 9 billion parameters, which sets its memory footprint. SenseTime says its per-token computation is equivalent to about 14.6 billion parameters.

Does looping make the model slower?

Generally yes. Each extra pass through the looped block adds layer computations for every token. SenseTime found better scores at three or four loops but ships two by default for efficiency.

Can I run SenseNova Loop-9B on a single GPU?

According to the card, 4-bit weights take about 4.9 GB, and estimated minimum GPU memory is about 9 to 10 GiB at 64K context and 12 to 15 GiB at 256K, depending on KV cache precision. You need a runtime that supports the looped layout, such as the listed LightLLM, vLLM or SGLang options, or the llama.cpp fork for GGUF.

How is this different from a Universal Transformer?

The 2018 Universal Transformer applied one shared block repeatedly and let each position decide when to stop. Loop-9B keeps unshared entry and exit layers, loops a 24-layer middle block a fixed number of times, and adds hybrid attention and two-stream mHC. The core idea of reusing weights across depth is the same.

Summary

SenseNova Loop-9B is SenseTime's open-weight, Apache 2.0 multimodal model that turns 32 physical layers into 56 layer computations by running a 24-layer middle block twice. It stores about 9 billion parameters but, by SenseTime's count, computes like a model of about 14.6 billion. The design follows a line of research from Universal Transformers to recent work showing that many reasoning tasks need depth more than parameters, and it pairs looping with hybrid attention, two-stream mHC, a 512K context and 4-bit weights of about 4.9 GB. You save memory and pay in compute and latency. SenseTime's benchmark claims, including near-parity with Qwen3.8-27B on some tasks, are its own, and runtime support still lives on special branches.

Sources

Cite this article

Kelahmet, Ö. C. (2026, November 20). SenseNova Loop-9B explained: how a looped transformer runs 32 layers as 56. Özer Cem KELAHMET. https://www.ozercemkelahmet.com/articles/sensenova-loop-9b-looped-transformer

BibTeX
@online{kelahmet2026sensenova,
  author = {Kelahmet, {\"O}zer Cem},
  title  = {{SenseNova Loop-9B explained: how a looped transformer runs 32 layers as 56}},
  year   = {2026},
  month  = {nov},
  url    = {https://www.ozercemkelahmet.com/articles/sensenova-loop-9b-looped-transformer}
}
Version historyfirst version
  1. Published

Factual corrections are listed here with what changed. Wording and formatting edits are not listed.

Last updated Nov 20, 2026
Type to search.
    ↑↓ navigate↵ open