How cache-aware routing and batching cut LLM serving costs

For three years the main question in serving large language models was "how fast is the GPU?". The first week of November 2026 suggested a different one: "what is already in the GPU's memory, and are we using it?". Buoyant open-sourced Kinference, a control plane that sends each vLLM request to the replica whose cache already holds its context. Fireworks AI announced a 1,500 tokens-per-second preview, a half-price Background Mode and multi-day prompt caching. AWS researchers published HOPE, a way to remove idle experts from Mixture-of-Experts models.

These are one story. A model spends money on compute to read a prompt and on memory to hold what it has read and the weights it reads with. Cache-aware routing avoids re-reading prompts. Continuous batching keeps the GPU busy while it waits on memory. Expert pruning shrinks weights that sit in memory but rarely work.

This article covers the news, then the mechanics, and ends with a checklist for teams running their own models. For wider context, see Generative AI and LLMs.

What happened

Fireworks AI announces Ultra-fast, Background Mode and multi-day caching at Forge (3 November 2026)

On 3 November Fireworks AI held Forge, its first conference, in San Francisco, and published its announcements the same day.

Ultra-fast inference is in public preview on NVIDIA Blackwell GPUs. Fireworks says it reaches "speeds up to 1,500 tokens per second", shown on DeepSeek V4.1 Flash, and that a 30-turn agent task fell from one minute to eight seconds. These are the company's own measurements. Its documentation says the tier targets "6-8x generation speed (up to 10x for agentic coding tasks)" at a higher price and may change quickly.

Background Mode runs work that can wait, such as generating evaluation data or overnight research, "at half price". No completion window or full model list was given.

Multi-day prompt caching targets agents that pause and resume with the same long context. No retention period or price was published. Existing pricing shows how big cache discounts can be: DeepSeek V4.1 Flash on Fireworks serverless lists $0.30 per million input tokens and $0.006 per million cached input tokens, one fiftieth of the price.

Buoyant open-sources Kinference for vLLM on Kubernetes (4 November 2026)

On 4 November the vLLM blog published "Introducing Kinference". The post says: "We created Kinference at Buoyant after watching our customers struggle to run open weight models in their environments." Buoyant created the Linkerd service mesh. Kinference is Apache 2.0 licensed.

Kinference sits in front of vLLM workers on Kubernetes as a traffic gateway and an inference handler, installed with Helm. Clients send OpenAI- or Anthropic-compatible requests to one endpoint. Load balancing considers "GPU queue sizes, historical performance, and the contents of the KV cache on the GPU", so a request reaches "the vLLM worker that will do the least work for that specific request". Policies are Kubernetes resources (currently v1alpha1) that match on model, tags such as PII classification, and backend state, and can send overflow to providers such as Bedrock. It also adds token rate limits and per-tenant cost tracking, including cached versus uncached tokens.

Buoyant's benchmarks compare it with standard Kubernetes service load balancing. On one unloaded workload, median time to first token fell 95% and median end-to-end latency 45%. Measured as goodput within a latency target, it reached 7.4 times the capacity on a shared-prefix workload and about 3.5 times on a multi-turn one; the default setup missed the target at every concurrency tested. The post does not name the model, hardware or target.

Analysis splits vLLM's gain into scheduler and engine (reported 10 November 2026)

A Towards Data Science analysis, "Continuous Batching Beats Static and Dynamic by Up to 2.91× in vLLM", tried to separate where vLLM's speed comes from. As summarised by the Rio Rundown newsletter on 10 November, it attributes a 2.91 times throughput gain to the continuous-batching scheduler over static and dynamic batching, and 1.33 times to the engine. Together, throughput rose 3.86 times against naive Hugging Face transformers. It reportedly used five traffic traces and a control that came out at 1.00 times. We could not retrieve the original, so we rely on that summary.

AWS researchers publish HOPE for pruning MoE experts (reported 10 November 2026)

Researchers at AWS Agentic AI published "Higher-Order Pruning of Experts in Mixture-of-Experts Language Models" on arXiv, and Amazon described it on the Hugging Face blog. HOPE chooses which experts to remove by looking at how experts work together, not only how useful each is alone.

The motivation is memory. In Qwen3.5-122B-A10B, about 116 billion of 122 billion parameters, roughly 240 GB, are expert weights, around 95% of the model, yet each token uses only 8 of 256 experts per layer. HOPE needs no retraining, only one pass over a calibration set. At 50% pruning it ranked 1.58 on average among five methods against 2.42 for REAP, the strongest first-order method, with gains of "up to +6.1%" on agentic coding.

Prefill, decode and the KV cache

Two phases, two bottlenecks

Prefill reads the prompt. All its tokens are processed in parallel, which is heavy maths, so prefill is limited by compute. It sets the time to first token.

Decode writes the answer one token at a time. Each token requires reading the model's weights from memory but does little maths, so decode is limited by memory bandwidth. It sets the inter-token latency.

Think of a student in an exam. Reading the paper is fast and intensive. Writing the answer is slow, one word at a time, with constant glances back at the question.

What the KV cache stores

Each attention layer computes a "key" and a "value" for every token, and later tokens need them. Rather than recompute them, the server keeps them in GPU memory: the KV cache. Its size per token is roughly two times layers times key-value heads times head size times bytes per number. A long agent conversation can need gigabytes. The cache competes with the weights for GPU memory and limits how many requests can run at once.

Why reusing it saves money

If two requests share a start, such as the same system prompt, tools and documents, their keys and values for that part are identical. A server that kept them can skip that prefill. This is prefix caching. vLLM does it automatically by dividing the cache into blocks identified by a hash of their tokens and everything before them; SGLang keeps a tree of shared prefixes. Hosted APIs pass the saving on as cheaper cached input tokens; see prompt caching.

Two rules follow. Only an exact prefix counts: a timestamp near the top of a system prompt makes everything after it miss. And caches are evicted when memory fills, which is why "multi-day" caching is a product, not a given. Holding a cache costs memory, and someone pays for it.

Why round-robin routing wastes the cache

A cache helps only if the next request reaches the machine holding it.

The problem

A standard Kubernetes Service spreads requests with no knowledge of their content. With three replicas, an agent's next step has about a two-in-three chance of landing where its context has never been seen, and pays full prefill again. Adding replicas makes it worse: the cache is split into more pieces.

It is like a clinic that sends a returning patient to whichever doctor is free. The new doctor has no file, so the patient repeats their history, and more doctors means more strangers.

Nebius measured this in February 2026 with 300 concurrent coding-agent runs on three vLLM replicas. About 62% of tokens came from cache with default routing and about 95% with cache-aware routing; average step time fell from 9.0 to 4.8 seconds. With one replica, routing made little difference.

How cache-aware routing works

The router keeps an approximate map of which replica holds which prefixes, either by asking workers or by remembering where it sent recent ones. For each request it estimates how much prefill each replica could skip and how busy each is, then picks the lowest predicted cost. Kinference expresses this as minimising a metric such as predicted time to first token.

The tension is affinity against balance. Pure affinity sends every request with a popular prompt to one replica, which overloads. Pure balance destroys the cache. Good routers prefer the cached replica until its queue passes a threshold, then fall back, paying one prefill and warming a second copy. The vLLM production-stack documentation adds a refinement: routing by prefix alone can keep targeting a replica that has already evicted that prefix, so routing by actual cache state is more accurate. This is a special case of LLM routing, choosing between copies of one model rather than between models.

How continuous batching keeps the GPU busy

Decode reads all the weights to make one token for one request. Run 32 requests together and the same read makes 32 tokens. Batching turns a memory-bound job into a useful one.

Static batching runs a fixed group until all finish. A 20-token answer waits for a 2,000-token one, and its slot sits empty.

Dynamic batching waits briefly to collect arriving requests, which fills batches better, but each batch still runs until its longest member ends.

Continuous batching decides after every decoding step. A finished request leaves at once and a waiting one takes its place.

The difference is a tour coach against a city bus. The coach does not return until the last tourist finishes; the bus lets people off and on at every stop, so its seats stay full. vLLM pairs this with PagedAttention, which stores the KV cache in small pages so a new request can use any free memory.

The reported split matters. Most of the gain came from the scheduler (2.91 times), not faster kernels (1.33 times). A tuned engine running static batches can lose to a plain one running continuous batches.

There is a trade-off. A new long prompt entering the batch can slow everyone already decoding. Engines use chunked prefill to split long prompts, and some deployments put prefill and decode on separate machines. Cache hits help too, because a cached request brings little prefill.

MoE models: lots of memory, little compute

Why MoE is memory-heavy

A Mixture-of-Experts model replaces each feed-forward layer with many experts and a router that picks a few per token. Qwen3.5-122B-A10B does the compute of about a 10B model per token, but all 122 billion parameters must stay in memory because the next token may need any expert.

Picture a hospital with 256 specialists where each patient sees 8. Appointments are quick, but every specialist must be on site all day. The guide to sizing giant open MoE models shows how this drives hardware. Every gigabyte of idle expert is a gigabyte unavailable for KV cache, which means smaller batches and more prefill.

How expert pruning works

Pruning removes the parts that contribute least. For MoE, the unit is a whole expert: run a few thousand typical prompts, record which experts fire and how much they contribute, then remove the weakest and let the router choose among the rest.

First-order methods such as REAP score each expert alone. HOPE also records how pairs of experts that fire together reinforce or duplicate each other, then chooses the prune set by solving a small optimisation per layer. REAP is the special case with the pair terms set to zero. HOPE may keep an expert that looks weak alone but matters to a strong partner, as a team keeps a quiet colleague who makes the star player effective.

Halving the experts in Qwen3.5-122B-A10B would free on the order of 120 GB. The paper says expert pruning composes with quantisation, citing earlier work, but does not test that combination itself.

The risks

Calibration bias. An expert idle on your calibration set may be the one that handles a rare language or tool format.
Uneven results. HOPE's worst single condition was 4.5% below the best method.
Cost of calibration. HOPE needed about 10,000 prompts against 8,000 for REAP, and runs 6 to 7% slower.
Layer by layer. Interactions between layers are not modelled.

Technique	What it saves	Main risk	Where it lives
Prefix or prompt caching	Prefill on repeated context	Misses on any prefix change	Engine, API provider
Cache-aware routing	Repeated prefill across replicas	Hot spots on popular prefixes	Gateway or control plane
Continuous batching	Idle GPU slots in decode	Prefill disturbs decodes	Engine scheduler
Background or batch tiers	Price	Uncertain completion time	Provider pricing
Expert pruning	Weight memory	Quality loss on uncovered tasks	Offline preparation

What it means in practice

Memory is the real budget. Weights, KV cache and batch size share one pool. Plan capacity in gigabytes of cache per user, not only in GPUs.

The load balancer is part of the model stack. A router that ignores the cache can double latency on the same hardware.

Prompt layout is a cost lever. Stable content first, variable content last.

Agents gain most. They resend long, growing contexts many times, which is why Fireworks framed its features around agents.

Price follows urgency. If a job can wait, it should not pay for speed.

Self-hosting is easier but not free. An open control plane lowers the barrier, but you inherit the tuning a provider would do.

What to do now

Measure prefill and decode separately. Track time to first token, inter-token latency and cache hit share, exported through OpenTelemetry or your monitoring.
Restructure prompts for caching. System prompt, tools and stable documents first; timestamps and user data last.
Verify provider cache behaviour. Check minimum cacheable length, retention and price, and confirm cached tokens appear on bills.
Replace round-robin for multi-replica self-hosting. Test a cache-aware router against your current setup with your own traffic.
Confirm continuous batching and chunked prefill. Standard in vLLM and SGLang; often missing in custom servers built on plain transformers.
Split urgent and patient work. Send evaluations and overnight jobs to batch or background tiers, with cost and latency budgets per workload.
Size MoE by total parameters. Budget memory for all experts plus cache at target concurrency.
Treat pruning as an experiment. Calibrate on real traffic, prune modestly first, and evaluate rare cases before shipping.
Load-test hot prefixes. Check the router spreads a popular system prompt instead of overloading one replica.

Risks and open questions

Headline numbers are vendor figures. Kinference's 7.4 times and Fireworks' 1,500 tokens per second come from the companies' own tests, and Kinference's post names no model or hardware.

Preview means change. Ultra-fast is a preview and Kinference's policy API is v1alpha1.

Multi-day caching is under-specified. No retention period or price is published, and longer-lived caches hold customer context longer, which raises retention and tenant-isolation questions.

Routing can concentrate failures. If one replica holds the only warm copy of a popular prefix and fails, every request pays full prefill at once.

Pruned models may fail where benchmarks do not look. HOPE was tested on math, instruction following, coding and agentic coding only.

Watch next: independent benchmarks of Kinference, published terms for Fireworks' caching and Background Mode, HOPE-pruned checkpoints on Hugging Face, and cache-aware routing in standard Kubernetes gateways.

Frequently asked questions

What is the difference between prefix caching and prompt caching?

They are one mechanism seen from two sides. Prefix caching is what the engine does: it keeps keys and values for a shared prompt start. Prompt caching is how hosted APIs expose and bill that saving.

Why can adding replicas make latency worse?

With content-blind routing, each extra replica splits the cache further, so fewer requests find their context loaded and more pay full prefill. Cache-aware routing sends related requests to the same replica where possible.

Do I need to enable continuous batching in vLLM or SGLang?

No, both use it by default. It matters for custom serving loops on plain Hugging Face transformers, where static batching is common and, by the reported analysis, leaves much throughput unused.

Does expert pruning make an MoE model faster?

Mainly it makes it smaller. The same number of experts still run per token, so compute barely changes. The freed memory can hold more KV cache, which allows larger batches and indirectly more throughput.

Is Kinference only for vLLM?

Its cache-aware load balancing applies to the vLLM workers it manages on Kubernetes. Its policies can also forward requests to cloud providers such as Bedrock or Databricks.

Summary

Early November 2026 brought three moves aimed at the memory side of LLM serving. Buoyant open-sourced Kinference, an Apache 2.0 control plane that routes vLLM requests by KV-cache contents and queue state, with vendor benchmarks showing large gains over content-blind load balancing. Fireworks AI used its first Forge conference to announce a 1,500 tokens-per-second preview, a half-price Background Mode and multi-day prompt caching. An independent analysis attributed most of vLLM's speed-up to continuous batching, and AWS researchers showed that pruning MoE experts by their pairwise interactions beats scoring them alone. For teams the lesson is practical: keep prompts cache-friendly, route by cache, batch continuously, separate urgent from patient work, and treat pruning as a carefully tested trade.

The new LLM serving stack: cache-aware routing, continuous batching and expert pruning