# Tencent Hy4 flash open weights explained: Engram memory, LatentMoE and hybrid attention

Tencent has published the weights of Hy4 flash, a 220B-parameter multimodal model that activates about 12B per token. Its design moves facts into a lookup table, keeps most attention layers short-sighted, and drafts tokens from a shared cache.

Tencent has released the weights of Hy4 flash, the cheaper and faster sibling of its Hy4 model. The model launched in Tencent's apps and cloud on 17 November 2026, and TechNode reported on 19 November that the weights were downloadable for local deployment. Three versions are out: the instruction-tuned model, an FP8 version and a base model for further training.

The interesting part is the parts list. Hy4 flash combines five ideas that have each appeared separately in research papers or other open models: a large hashed n-gram lookup table (Tencent calls it Engram), a mixture of experts that works in a compressed space (LatentMoE), attention that mixes sparse and sliding-window layers, multi-stream residual connections, and a small drafting model that borrows the main model's memory. Together they show where model design is heading: fewer computations per token, more cheap memory.

## What happened

### Tencent launches Hy4 flash in its apps and cloud (17 November 2026)

In a press release dated 17 November 2026, Tencent introduced Hy4 flash as "the most cost-effective Hy model to date". It made the model available in its WorkBuddy, Yuanbao, WeCom, QQ, ima and Marvis apps and through Tencent Cloud TokenHub. Tencent recommends it for everyday work such as deep research and document processing, and points long-horizon software engineering to the larger Hy4.

The press release gives approximate prices of about $0.070 per million input tokens, $0.281 per million output tokens and $0.003 per million cached input tokens. Tencent also claims decoding speeds above 150 tokens per second, which it attributes to a megakernel, 4-bit quantization and its DFly drafting technique.

### Weights published on Hugging Face (19 November 2026)

TechNode reported on 19 November that Tencent released the weights two days after the product launch, under the Tencent Hy License. The Hugging Face model card lists Hy4 flash, Hy4 flash-FP8 and Hy4 flash-base, with the release also announced on ModelScope and GitCode. A GGUF build is provided by the AngelSlim project.

Parameter counts differ slightly between sources. The model card says 220B total parameters including 64B Engram parameters. Tencent's press release and TechNode say about 224B, and the Hugging Face listing for the base model shows about 219B. All agree on roughly 12B active parameters per token.

### Plotly runs an outside test (November 2026)

Plotly ran Hy4 flash on its private PlotlyDataAnalyticsBench, a 43-question data analysis exam. Hy4 flash scored 39 of 43 for $0.13 per full run, with a median of 29 seconds per question. Plotly uses one run per model and treats gaps of one or two points as ties. It is a single, narrow test.

## The parts list: what is inside Hy4 flash

| Component | Hy4 flash setting | What it does |
|---|---|---|
| Total parameters | 220B (model card), of which 63.5B Engram | Everything stored on disk and in memory |
| Active per token | About 12B (11.3B text + 0.43B vision) | What is actually computed for each token |
| Layers | 42 (1 dense, 41 mixture-of-experts) | Depth of the language backbone |
| Experts per MoE layer | 512 routed + 1 shared, 16 routed active | Specialist sub-networks chosen per token |
| Expert width | 1,536-dimensional latent space | Experts run on a compressed version of the hidden state |
| Attention | 12 sparse layers + 30 sliding-window layers | Long-range vs local context |
| Residual connections | Identity Hyper-Connections, 4 streams | How information flows between layers |
| Drafter | DFly2, 5 layers, 0.85B parameters | Proposes tokens for speculative decoding |
| Vision encoder | 0.43B, adapted from SigLIP2-So400m | Turns images into tokens |
| Context window | 1,048,576 tokens | Maximum input length |

### Mixture of experts: total versus active parameters

A [mixture of experts](/ai-simulations/mixture-of-experts) model splits its feed-forward layers into many small sub-networks called experts. A router looks at each token and sends it to a few of them. Think of a large hospital: it employs hundreds of specialists, but each patient only sees two or three. The hospital's payroll is the total parameter count. The doctors you actually meet are the active parameters.

Hy4 flash has 512 routed experts plus one shared expert in each MoE layer, and 16 routed experts fire per token. The shared expert sees every token, so routed experts can specialise. You can watch routing happen step by step in the [mixture-of-experts simulation](/ai-simulations/mixture-of-experts).

In a standard MoE, each expert reads the full hidden state, which here is 3,072 numbers wide. In LatentMoE, the model first compresses the hidden state into a 1,536-dimensional latent space, and the routed experts work there. Smaller inputs mean smaller expert weight matrices, so Tencent can afford more experts for the same compute.

### Engram: memory lookup instead of computation

The most unusual part of Hy4 flash is its Engram module, which holds about 63.5B parameters, almost 30% of the model. The name and idea match DeepSeek's January 2026 paper "Conditional Memory via Scalable Lookup", which argued that transformers lack a native lookup primitive and must simulate retrieval through computation.

Many token patterns are fixed, such as "New York" or "according to the". A normal transformer rebuilds their meaning in early layers every time. Engram instead keeps a huge table of learned vectors keyed by short token sequences. When the model sees the last two or three tokens, it hashes them into an index, fetches the stored vectors and adds them into the hidden state. That costs a memory read, not a large matrix multiplication.

Think of a phrasebook: instead of building a sentence from grammar rules, you look up the whole phrase.

The model card describes the Hy4 flash version precisely:

- It stores bigram and trigram representations, retrieved through multi-head hash lookups.
- Each n-gram order uses 8 hash heads, each with a hash vocabulary of 10.3M entries.
- Each order produces 768 dimensions (8 heads × 96).
- The lookups are injected at backbone layers 1, 11, 21 and 31.

Why several hash heads? There are far more possible trigrams than table slots, so phrases sometimes collide on one slot. With eight independent hash functions, two phrases are unlikely to collide on all eight.

DeepSeek's paper reported that, at equal parameter count and equal compute, adding Engram beat a plain MoE baseline, with gains on reasoning, code and maths as well as knowledge. The authors also noted that because the lookup address depends only on the input tokens, the system can prefetch table rows from host memory with little overhead. The Hy4 flash model card does not say whether vLLM or SGLang offload the Engram table this way, so do not assume it.

The key point for deployment: Engram parameters add to total memory but barely add to compute. They are cheap to use and expensive to store.

### Hybrid attention: a few long views, many short ones

Attention lets each token look back at earlier tokens. Full attention over a 1M-token context is costly in two ways: compute grows with context length, and the [KV cache](/ai-simulations/kv-cache), the stored keys and values for every past token, grows with it too.

Hy4 flash uses two kinds of attention layers:

| Layer type | Count | What each token can see | Heads | KV cache growth |
|---|---|---|---|---|
| Sparse attention | 12 | Up to 2,048 selected tokens from the whole context, chosen by group-level selection | 32 | Grows with context length |
| Sliding-window attention | 30 | Only the last 512 tokens (511 before plus itself) | 64 | Fixed at 512 tokens per layer |

The layout is two sparse layers first, then ten repeats of three sliding-window layers followed by one sparse layer. Think of reading a contract: mostly line by line, with an occasional flip back to the definitions. The sliding-window layers read line by line; the sparse layers flip back.

The sparse layers do not read every past token. A selection step groups the context and picks the most relevant groups, up to 2,048 tokens. So the per-token compute in those layers stays roughly flat even as the context grows, although the full history must still be kept so it can be searched.

Both layer types use Multi-head Latent Attention (MLA), which stores a compressed latent vector per token instead of full keys and values. In Hy4 flash the KV latent rank is 512, plus a 64-dimensional rotary position part. Images and text share a position scheme called interleaved MRoPE.

Tencent claims the resulting KV cache is 1/24 the size of Hy3's and 1/6 of Hy4's. The [KV cache simulation](/ai-simulations/kv-cache) shows why this matters: on a long conversation, the cache, not the weights, often decides how many users fit on a GPU.

To make that concrete, here is a rough estimate. It is my own arithmetic, not a Tencent figure, and it assumes each layer caches one 576-value MLA latent per token (512 + 64) in 16-bit precision and ignores any extra index data. At 1M tokens, the 12 sparse layers would hold about 12 × 576 × 2 bytes × 1,048,576 ≈ 14.5 GB. The 30 sliding-window layers would hold only about 18 MB in total, because each keeps just 512 tokens. If all 42 layers cached the full history, the figure would be about 3.5 times larger.

### Identity Hyper-Connections: four lanes instead of one

Every transformer layer adds its output back onto a running "residual stream", a single lane of information that flows from bottom to top. Hyper-connections, introduced in a 2024 paper by Zhu and colleagues, widen this into several parallel streams and let the network learn how layers read from and write to each lane. The original paper reported clear gains over standard residual connections in dense and sparse language models.

Hy4 flash uses "identity Hyper-Connections" with four residual streams. The model card does not explain the "identity" variant in detail, and I could not find a Tencent paper describing it, so treat any deeper description of it as speculation.

## Speculative decoding with a shared-cache drafter

Generating text is slow because a model produces one token at a time, and each step must read the active weights from memory. [Speculative decoding](/ai-simulations/speculative-decoding) speeds this up. A small, fast "drafter" model guesses the next few tokens, and the large model checks all the guesses in one pass. Correct guesses are kept; the first wrong one is replaced. The output is the same as the large model would have produced on its own, only faster when the drafter guesses well. See the [speculative decoding simulation](/ai-simulations/speculative-decoding).

Normally the drafter needs its own context: on a 1M-token prompt, its own long KV cache and its own prefill.

DFly2, Tencent's drafter for Hy4 flash, avoids this. According to the model card, it is a 5-layer model with 0.85B parameters, trained with AngelSpec, and it:

- reuses the target model's KV cache ("KV Share"), so it keeps no cache of its own;
- reuses the sparse attention indices the target already computed ("Index Share");
- attends to at most about 2K keys per draft token, so its cost stays constant regardless of context length;
- does not take part in prefill.

It is like an apprentice who drafts using the senior writer's notes instead of keeping a separate notebook.

The vLLM example configures DFly speculation with 5 speculative tokens; the SGLang example uses a block size of 6.

## What it means in practice

**Memory, not compute, is the gating factor.** Tencent does not publish a GPU memory requirement. A simple estimate of weight memory is parameters times bytes per parameter. The table below is my arithmetic using the model card's 220B total; it excludes KV cache, activations and runtime overhead, so real needs are higher.

| Precision | Bytes per parameter | Approximate weight memory (220B total) | Per-token weight reads (about 12B active) |
|---|---|---|---|
| BF16 | 2 | About 440 GB | About 24 GB |
| FP8 | 1 | About 220 GB | About 12 GB |
| 4-bit | 0.5 | About 110 GB | About 6 GB |

Plotly states that the model needs about 132 GB of memory at 4-bit, which fits the 4-bit row once overhead is added. The gap between the third and fourth columns is the whole MoE trade: you must store everything, but you only touch a small slice per token. See the [quantization simulation](/ai-simulations/quantization) for how fewer bits per weight shrink memory and what accuracy risk comes with it.

**The official serving recipe targets an 8-GPU server.** The model card's examples serve the FP8 checkpoint with tensor parallelism of 8 on both [vLLM](/tools/vllm) (with a FLASHMLA_SPARSE attention backend) and [SGLang](/tools/sglang), using dedicated Docker images. The card does not name a GPU model. Hugging Face Transformers support was still pending a merge. For single-machine experiments, the community [GGUF](/tools/gguf) build is the likely path, with the usual caveat that low-bit quantization can cost quality.

**The licence is permissive for most users, with two large-company triggers.** The Tencent Hy License, Version 1.0, says it is derived from Apache 2.0 but is not the Apache License. Section 2 grants a worldwide, royalty-free copyright licence to reproduce, modify and distribute, and Section 3 grants a patent licence. Derivative works explicitly include fine-tuned, quantized, distilled and merged weights. The key extra is Section 10: if you operate a "MaaS" business, meaning you give third parties remote access to inference or fine-tuning, for example through an API, you need a separate written licence from Tencent if either your worldwide gross revenue exceeds US$50 million in any 12-month period or the model is used in a product with more than 100 million monthly active users. Purely internal use is exempt, as long as the model, derivatives and outputs are not made available to third parties. I found no geographic exclusions in the text. Read the full LICENSE file before deploying; this summary is not legal advice.

**Benchmarks are company-run.** On the base-model table, Tencent's own few-shot evaluations show Hy4 flash Base ahead of DeepSeek-V4-flash-Base on many rows, for example MMLU-Pro 71.1 versus 68.6, SimpleQA 37.5 versus 31.3 and U-MATH 69.8 versus 57.4, but behind it on MMLU (88.7 versus 89.7) and MBPP-plus (73.7 versus 76.8). Tencent warns that heavy synthetic instruction data in mid-training sometimes breaks few-shot answer formats, which may distort scores.

For the instruction model, Tencent reports a WorkIndex score, an internal average across 8 evaluation sets, of 42.1, versus 39.7 for Hy4 preview and 51.4 for Hy4. It also reports a blind evaluation with 85 internal WorkBuddy users on 259 tasks in 27 categories: satisfaction of 2.79 out of 4 against 2.81 for Claude Haiku 5.5, a median completion time of 6.7 minutes and a cost of RMB 0.26 per task. Tencent claims a per-task cost of 1/13 of Hy4 preview, 1/20 of Hy4 and 1/8 of DeepSeek-V4.1-Flash-1017 at Max tier, peak-hour pricing. All of these are Tencent's figures. For context on how Hy4 flash compares with other big open MoE releases, see [sizing giant open MoE models](/articles/open-weight-giant-moe-models).

## What to do now

1. **Decide whether you need the weights or the API.** If you just want cheap everyday tasks, the hosted service may be simpler; self-hosting makes sense for data control, as covered in [self-hosted open-weight coding agents](/articles/self-hosted-open-weight-coding-agents).
2. **Size memory from total parameters, not active ones.** Start from about 220 GB of weights at FP8 or about 110 GB at 4-bit, then add KV cache and overhead.
3. **Read Section 10 of the licence with your legal team.** Check whether you count as a MaaS business and whether either the revenue or the user threshold applies.
4. **Start with the FP8 checkpoint on vLLM or SGLang.** These are the paths Tencent documents, with tensor parallelism of 8.
5. **Test speculative decoding on your own prompts.** Measure acceptance rate and tokens per second with DFly turned on and off, because speed-ups depend on how predictable your text is.
6. **Benchmark long context separately.** Run your real long documents, since sliding-window layers see only 512 tokens and the sparse layers select at most 2,048.
7. **Try both reasoning modes.** Compare `reasoning_effort` set to `high` (the default) and `no_think` for latency, cost and quality on your tasks.
8. **Validate any GGUF or 4-bit build against FP8.** Compare outputs on a fixed evaluation set before swapping quantized weights into production.

## Risks and open questions

**No broad independent benchmark exists yet.** Plotly's single 43-question run is the only outside score found, and every other number comes from Tencent's own evaluations.

**The memory requirement is not officially stated.** The weight figures above are arithmetic, and real deployments need extra room for KV cache, activations and batching.

**Parameter counts disagree.** The model card says 220B, Tencent's press release says 224B and the base model listing shows about 219B; it is unclear what each count includes.

**Engram offloading is unproven here.** DeepSeek's paper suggests the lookup table can live in host memory, but the Hy4 flash card does not document this for the serving stacks.

**Sparse selection can miss things.** If the selection step ranks the wrong groups, a detail deep in a long document may be missed; no published needle tests for this model were found.

**Internal evaluations measure internal tasks.** WorkIndex and the blind test use Tencent's own users and tasks.

**Watch next:** independent long-context and agentic benchmarks, merged Hugging Face Transformers support, community memory measurements for the FP8 and GGUF builds, any Tencent technical report explaining identity Hyper-Connections and LatentMoE, and whether vLLM or SGLang add Engram table offloading.

## Frequently asked questions

### Is Hy4 flash really a 12B model?

No. It computes with about 12B parameters per token, but it stores about 220B. You need memory for all of them, far more than for a dense 12B model.

### What is Engram in simple terms?

Engram is a very large table of learned vectors keyed by the last two or three tokens. The model hashes those tokens into an index and fetches the stored vectors instead of recomputing common phrase meanings. In Hy4 flash it holds about 63.5B parameters.

### Why use both sparse and sliding-window attention?

Sliding-window layers see only the last 512 tokens, so their cost and cache stay small. The 12 sparse layers can reach anywhere in the 1M-token context but only read up to 2,048 selected tokens. The mix aims to keep long-range recall while cutting cost.

### Does speculative decoding change the answers?

When it is implemented correctly, no. The large model verifies every drafted token and keeps only those it would have produced itself. The drafter changes speed, not output.

### Can I use Hy4 flash commercially?

The Tencent Hy License grants broad, royalty-free rights, including for derivatives. If you offer the model to third parties as a service and your company's revenue exceeds US$50 million or your product has over 100 million monthly active users, you need a separate licence from Tencent. Read the full licence before relying on this summary.

## Summary

Tencent's Hy4 flash, launched on 17 November 2026 with weights released two days later, stores about 220B parameters but computes with about 12B per token. It moves fixed phrase knowledge into a hashed Engram table, runs 512 experts in a compressed latent space, keeps 30 of 42 attention layers to a 512-token window, and lets a 0.85B drafter reuse the main model's cache. The licence is permissive except for large service providers. Apart from one Plotly test, the benchmarks and cost claims are Tencent's own, and memory must be estimated: roughly 220 GB of weights at FP8 or 110 GB at 4-bit before overhead.

## Sources

- [Hugging Face: tencent/Hy4-flash model card](https://huggingface.co/tencent/Hy4-flash)
- [Hugging Face: tencent/Hy4-flash-base model card](https://huggingface.co/tencent/Hy4-flash-base)
- [Hugging Face: Tencent Hy License, Hy4-flash LICENSE file](https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE)
- [TechNode: Tencent releases Hy4 flash model weights for local deployment](https://technode.com/2026/11/19/tencent-releases-hy4-flash-model-weights-for-local-deployment/)
- [TNGlobal: Tencent Launches Hy4 flash to Power High-Speed, Cost-Efficient AI Deployment for Everyday Workflows](https://technode.global/?p=997942)
- [Plotly: Hy4 flash on the Plotly Data Analytics Bench](https://plotly.com/blog/hy4-flash-plotly-data-analytics-bench/)
- [arXiv: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://arxiv.org/abs/2601.07372)
- [arXiv: Hyper-Connections](https://arxiv.org/abs/2409.19606)
