# llama.cpp reaches 1.0: how GGUF, quantization, ggml backends and llama-server fit together

On 13 November 2026 ggml-org tagged llama.cpp and ggml v1.0.0. The release prunes the deprecated API, adds an OpenAI-style speech endpoint and hardens llama-server. Here is how local inference works underneath, and what a 1.0 promise means for your app.

llama.cpp is the C/C++ engine that runs a large share of local language models, from laptops to phones to small servers. On 13 November 2026 its maintainers at ggml-org tagged version 1.0.0 of both libllama (the model runtime) and libggml (the tensor library underneath it). After roughly three and a half years of rapid, unversioned change, the project now says it will signal breaking API changes with major version numbers.

The release itself is a clean-up and a hardening pass. It removes dozens of deprecated functions from the public C header, adds an OpenAI-compatible `/v1/audio/speech` endpoint to the built-in server, and closes several network security gaps in `llama-server`. For anyone building on [llama.cpp](/tools/llama-cpp), the version number matters more than any single feature: it is the first time the project has made a written promise about when your code will break.

## What happened

### ggml-org tags llama.cpp and ggml v1.0.0 (13 November 2026)

The GitHub release for llama.cpp v1.0.0 lists a set of breaking clean-ups. It removes the deprecated public API from `llama.h`, including `llama_set_warmup`, 35 deprecated functions, and the `defrag_thold` parameter with its `--defrag-thold` flag. `llama_set_embd_state_src` is renamed to `llama_set_ctx_state_src`, and the separate `clip.h` header is folded into `mtmd`, the multimodal library. In ggml, the legacy `DIAG_MASK` operations are removed, and the ggml release notes say that "ggml 1.0.0 prunes the deprecated public API."

New features include FP8 KV caches (`f8_e4m3`) with optional static per-tensor scales, a host-memory cache for mixture-of-experts weights with new `moe_cache_size` and `moe_cache_cpu_share` context parameters, new models such as Kimi-K3.1-Pro, and a redesigned web UI.

### The server gains /v1/audio/speech and a security hardening pass (13 November 2026)

`llama-server` adds an OpenAI-compatible `/v1/audio/speech` endpoint and a `temperature` option for its existing `/tts` endpoint, and the notes list drama-3 text-to-speech model support.

On security, the release adds two opt-in flags, `--allow-unauthenticated-tools` and `--media-allow-private-urls`, so that risky behaviour is off unless you ask for it. The notes list fixes for `--cors-origins`, DNS rebinding, server-side request forgery through the CORS proxy, and direct browser access to child instances. The server now sends `HF_TOKEN` only to huggingface.co hosts, and cancels MCP tool calls when a client disconnects or times out.

### Hugging Face publishes the versioning policy (13 November 2026)

A post on the Hugging Face blog, published under the ggml-org account and crediting Georgi Gerganov and Daniel Bevenius, explains the reasoning. "Today we release libraries `v1.0.0`, highlighting our responsibility to those developers," it says, referring to downstream projects. The post defines the rules: a MAJOR bump for a breaking change to the public C API in `./include`, a MINOR bump for backward-compatible features, backend updates, model support or API additions, and a PATCH bump for bug fixes with no API change. According to the post, the project supports more than 150 architecture variants.

## How a model becomes a file: GGUF and quantization

### GGUF: one self-contained file

Every model llama.cpp runs is a GGUF file. GGUF is a binary format designed for fast loading. Think of it as a shipping container with the packing list taped inside the door: the weights and everything needed to interpret them travel together.

The [GGUF](/tools/gguf) specification lays the file out in a fixed order:

1. **Header.** The magic bytes `GGUF`, a version number (the current spec is version 3), the number of tensors and the number of metadata entries.
2. **Metadata.** Typed key-value pairs such as `general.architecture` (for example `llama` or `mamba`), context length, layer counts, and the full tokenizer: vocabulary, merges, special token IDs and often the chat template.
3. **Tensor info.** For each tensor: its name, shape, data type (for example `Q4_K` or `F16`) and an offset into the data section.
4. **Tensor data.** The raw weights, each aligned to a fixed boundary (32 bytes by default).

Because the tokenizer and chat template live in the file, a GGUF needs no companion files. Alignment makes the file friendly to memory mapping: the operating system maps it and loads pages only when touched, so startup is fast.

By contrast, safetensors, the common format for training checkpoints, stores tensors only, with tokenizer and config in separate files.

### Quantization: storing weights in fewer bits

A model weight is normally a 16-bit number. Quantization stores it with fewer bits, much as a JPEG stores a photo with less data than the raw camera file. You lose some detail, but often less than you would expect.

llama.cpp quantizes in blocks. A block of weights shares one scale factor (and sometimes a minimum). Each weight is stored as a small integer `q`, and the runtime rebuilds an approximate value as `w = q × scale` or `w = q × scale + min`. The scale is what lets 4 bits cover the real range of values in that block.

The Hugging Face GGUF documentation describes three families:

| Family | Examples | How it works | Typical use |
|---|---|---|---|
| Legacy | Q4_0, Q5_1, Q8_0 | 32-weight blocks, one scale (plus a minimum for the _1 variants) | Q8_0 is still a near-lossless baseline; others are described as rarely used |
| K-quants | Q2_K to Q6_K | Super-blocks of 256 weights split into sub-blocks, with the sub-block scales themselves quantized | General-purpose default, especially Q4_K_M and Q5_K_M |
| IQ types | IQ1_S to IQ4_XS | 256-weight super-blocks guided by an importance matrix | Very low bit widths where every bit counts |

The `_S`, `_M` and `_L` suffixes on K-quants are mixes: a `Q4_K_M` file keeps some tensors at higher precision, so its effective bits per weight sits above the nominal 4.5 of Q4_K.

The importance matrix, or imatrix, holds statistics gathered by running sample text through the model, telling the quantizer where precision matters most. The quantize tool's documentation says accuracy loss "can be minimized by using a suitable imatrix file."

The quantize README gives concrete sizes for Llama-3.1-8B:

| Type | Bits per weight | Size (GiB) |
|---|---|---|
| F16 | 16.00 | 14.96 |
| Q8_0 | 8.50 | 7.95 |
| Q6_K | 6.56 | 6.14 |
| Q5_K_M | 5.70 | 5.33 |
| Q4_K_M | 4.89 | 4.58 |
| IQ3_M | 3.76 | 3.52 |
| IQ2_M | 2.93 | 2.74 |

The same page says a Llama 3.1 70B model shrinks from 280.9 GB to 43.1 GB at Q4_K_M. That is the difference between needing a multi-GPU server and fitting on one large workstation. For GPU-oriented alternatives, see [GPTQ](/tools/gptq) and [AWQ](/tools/awq), often used with [vLLM](/tools/vllm).

## How a model runs: ggml, backends and the KV cache

### ggml builds a graph, a backend executes it

llama.cpp does not hand-write a forward pass for each model. It describes each architecture as a graph of ggml operations: matrix multiplications, normalisation, rotary position embeddings, attention, and so on. ggml then schedules that graph onto one or more backends.

A backend is a set of kernels for one kind of hardware. The project README lists, among others:

| Backend | Target hardware |
|---|---|
| CPU | x86 (AVX, AVX2, AVX512, AMX), ARM NEON, RISC-V |
| Metal | Apple silicon |
| CUDA | Nvidia GPUs |
| HIP | AMD GPUs |
| Vulkan | Most GPUs with a Vulkan driver |
| SYCL | Intel GPUs |
| WebGPU | Browsers and other WebGPU hosts |
| RPC | Remote machines over the network |

According to the build documentation, Metal is enabled by default on macOS, while CUDA, Vulkan and HIP are switched on with CMake flags such as `-DGGML_CUDA=ON`. Several backends can be compiled into one build, and the `GGML_BACKEND_DL` option builds them as libraries loaded at runtime, so one binary can run on machines with different GPUs.

The key practical feature is hybrid CPU+GPU inference. The `-ngl` (GPU layers) setting decides how many transformer layers live in GPU memory; the rest run on the CPU. If a model is larger than your VRAM, you can still run it, just more slowly. The new host-memory MoE expert cache in 1.0 extends this idea to [mixture-of-experts](/tools/mixture-of-experts) models, where only some experts are active per token and the rest can wait in system RAM.

Each quantization type needs its own kernels on each backend, which read the packed blocks directly instead of expanding weights back to 16 bits in memory.

### Two phases: prefill and decode

Generation has two phases. In prefill, the model reads your whole prompt at once. This is compute-heavy and parallel, which is why the release mentions a Vulkan FlashAttention path for prefill. In decode, the model produces one token at a time. Each step must read essentially every active weight from memory, so decode speed is usually limited by memory bandwidth rather than arithmetic. This is the main reason quantization speeds things up: fewer bytes per weight means fewer bytes to move per token.

### The KV cache

During decode, the model must attend to every earlier token. Recomputing all of them each step would be wasteful, so the runtime stores each layer's key and value vectors for past tokens. This store is the [KV cache](/tools/kv-cache). It works like the notes you take while reading a long report: you consult your notes instead of rereading every page.

The cost is memory that grows with context length. A rough formula is:

`KV bytes = 2 × layers × KV heads × head size × context tokens × bytes per value`

For an illustrative model with 32 layers, 8 KV heads and a head size of 128, each token needs 2 × 32 × 8 × 128 = 65,536 values. At 16 bits that is 128 KiB per token, or 4 GiB for a 32,768-token context, on top of the weights. With several parallel users, multiply again.

llama.cpp gives you three levers. The `--cache-type-k` and `--cache-type-v` flags change the cache data type from the default f16 to smaller types. Version 1.0 adds an 8-bit floating point option, `f8_e4m3`, which halves the cache size compared with f16. Flash Attention (`-fa`) computes attention in tiles without materialising the full attention matrix, as described in our [FlashAttention](/tools/flashattention) explainer. And the server's prompt cache reuses KV entries from a previous request, so only the part of a prompt that changed has to be processed.

## The OpenAI-compatible server

### What llama-server exposes

`llama-server` wraps all of the above in an HTTP service. Its documentation lists OpenAI-compatible chat completions, completions, responses and embeddings routes, an Anthropic Messages-compatible route, and a `/health` endpoint that returns 503 while the model loads and 200 when it is ready. With v1.0 the list gains `/v1/audio/speech`. Most OpenAI SDKs work by changing the base URL.

Under the hood, the server runs a fixed number of slots, set with `-np`. Each slot is one in-flight conversation with its own KV cache region. Continuous batching is on by default: new requests join the running batch between decode steps instead of waiting for others to finish. Other documented features include JSON-schema constrained output, [speculative decoding](/tools/speculative-decoding) with a draft model, and a Prometheus `/metrics` endpoint behind `--metrics`.

### Prompt caching and its catch

The `cache_prompt` option is on by default. If a new request shares a prefix with the last one in that slot, the server skips the shared part. For chat and agent loops that resend a long system prompt each turn, this saves a lot of prefill work. The documentation warns that it can make results non-deterministic, because logits are not always bit-identical across batch sizes. If you need reproducible outputs for testing, turn it off for those runs.

### Why the hardening matters

By default the server binds to 127.0.0.1. Many users expose it on a network anyway, and the server has grown features that fetch things: media URLs, tool calls, MCP servers. Each is a path an attacker could abuse. Server-side request forgery means tricking the server into fetching an internal address for you. DNS rebinding means a web page in your browser resolving a hostname to your local server and talking to it. The 1.0 fixes and the new opt-in flags close these paths by default. For larger shared deployments, our piece on the [LLM serving stack](/articles/llm-inference-serving-stack-cache-routing) covers routing and batching beyond a single process.

## What a 1.0 stability promise means

Before 1.0, functions were renamed and removed often, and downstream bindings had to track the main branch closely. The 1.0 release clears the accumulated deprecated surface, then promises that future removals will come with a major version bump.

Read the scope carefully. The blog post defines MAJOR as a breaking change to "the public C API in `./include`." That covers `llama.h` and the ggml headers you link against. It also commits to signalling ABI incompatibilities, so a binary built against 1.x should keep working with a later 1.x shared library. It does not explicitly mention the HTTP API of `llama-server`, command-line flags, the web UI, or the numerical output of a given model.

The MINOR category is also broad. New model support and backend updates land as minor releases. A backend kernel change can alter speed or, in edge cases, output, without being a "breaking change" in the semver sense. Semver protects your build, not your benchmark numbers.

Finally, GGUF has its own version field, separate from the library version. The current GGUF spec is version 3. Your existing model files are not invalidated by the 1.0 tag, though individual model architectures can still gain new metadata keys over time.

## What it means in practice

For application developers, pinning becomes meaningful. You can depend on `libllama` 1.x and expect your C or binding code to compile across minor upgrades.

For people running local models, very little changes on day one. The same GGUF files load, the same quantization choices apply, and the server speaks the same OpenAI dialect. What improves is the default security posture of `llama-server`, plus FP8 KV caches and the MoE expert cache for fitting larger models or longer contexts into the same memory.

For teams weighing local inference against hosted APIs, a 1.0 C API lowers the risk of building on a fast-moving engine, but the trade-offs remain. llama.cpp shines on single machines, edge devices and mixed hardware. High-throughput multi-user GPU serving is still the home ground of engines like vLLM and [SGLang](/tools/sglang). Our article on [self-hosted open-weight coding agents](/articles/self-hosted-open-weight-coding-agents) covers where companies draw that line.

## What to do now

1. **Pin a version.** Move from tracking the main branch or build numbers to a 1.x release, and record it in your build or container file.
2. **Search your code for removed symbols.** Check for `llama_set_warmup`, `defrag_thold`, `llama_set_embd_state_src` and includes of `clip.h` before you upgrade. The release notes list the full set.
3. **Update your bindings.** If you use a language wrapper, check that it supports 1.0 before upgrading the shared library underneath it.
4. **Review server exposure.** If `llama-server` listens on anything other than localhost, set `--api-key`, set `--cors-origins` explicitly, and leave the new opt-in flags off unless you need them.
5. **Re-measure memory with FP8 KV cache.** Try the new `f8_e4m3` cache type through `--cache-type-k` and `--cache-type-v`, and check output quality on your own prompts before adopting it.
6. **Choose quantization by test, not by habit.** Start with Q4_K_M or Q5_K_M, compare against Q8_0 on a small evaluation set of your real tasks, and consider imatrix quantizations for 3-bit and below.
7. **Use prompt caching deliberately.** Keep stable system prompts at the start of requests so prefix reuse works, and disable `cache_prompt` in tests that need deterministic output.
8. **Try the speech endpoint behind a flag.** If you already call OpenAI's speech API, point a test client at `/v1/audio/speech` with a supported TTS model and compare latency and quality.

## Risks and open questions

**The promise covers the C API, not everything.** Server routes, flags and output quality are not explicitly part of the written semver rule, so apps that depend on `llama-server` behaviour should still test each upgrade.

**Minor releases can change numbers.** Backend kernel updates are classed as minor. Speed and, rarely, outputs can shift between 1.x versions, so keep a regression set if your product depends on stable answers.

**Low-bit quantization has hidden costs.** Size tables show bits per weight, not task accuracy. Two-bit and three-bit models can look fine in chat and fail on reasoning or code, so measure on your workload.

**Network exposure remains your job.** The hardening closes known paths, but a local server with tools, media fetching and MCP enabled is a real attack surface. Treat it like any internal web service.

**FP8 KV cache is new.** Long-context quality across model families will only become clear with wider use.

**Watch next:** how fast downstream bindings and desktop apps adopt 1.x, whether the project extends a stability statement to the server's HTTP API, the first 2.0 discussion and what triggers it, and independent quality measurements for FP8 KV caches.

## Frequently asked questions

### Do I need to re-download or re-convert my GGUF models for llama.cpp 1.0?

The release notes do not announce a GGUF format change, and GGUF carries its own version number, currently 3. Existing files for supported architectures should keep loading.

### What is the difference between llama.cpp and ggml?

ggml is the tensor library: it defines operations, data types and quantization formats, and runs them on backends like CUDA or Metal. llama.cpp (libllama) sits on top and knows about model architectures, tokenizers, sampling and the KV cache. Both reached v1.0.0 on the same day.

### Which quantization type should I start with?

Q4_K_M is a common starting point because it cuts an 8B model from about 15 GiB to about 4.6 GiB while keeping more precision on sensitive tensors. If you have memory to spare, Q5_K_M or Q8_0 lose less quality. Below 4 bits, prefer IQ types made with an importance matrix, and test on your own tasks.

### Is llama-server safe to expose to the internet?

It binds to localhost by default for a reason. The 1.0 release fixes several network attacks and makes risky features opt-in, but you should still add an API key, restrict CORS origins and put it behind a reverse proxy with TLS. Leave tool and private-URL features off unless you need them.

### Does semantic versioning mean my outputs will not change between versions?

No. The policy defines breaking changes in terms of the public C API in the include directory. Kernel optimisations and backend updates can ship in minor releases and may change speed or produce slightly different numbers. Keep your own evaluation set to catch changes that matter to you.

## Summary

llama.cpp and ggml reached v1.0.0 on 13 November 2026. The release removes deprecated C API functions, adds an OpenAI-compatible `/v1/audio/speech` endpoint, introduces FP8 KV caches and an MoE expert cache, and closes several security gaps in `llama-server`. Underneath, a GGUF file holds block-quantized weights and the tokenizer, ggml runs the model graph on CPU, Metal, CUDA, Vulkan or other backends, the KV cache trades memory for speed, and the server exposes it through OpenAI-style routes. The new part is the promise: breaking changes to the public C API will now come with a major version bump. That makes llama.cpp easier to build products on, as long as you remember the promise covers the headers, not server behaviour or model outputs.

## Sources

- [GitHub: llama.cpp release v1.0.0](https://github.com/ggml-org/llama.cpp/releases/tag/v1.0.0)
- [Hugging Face: llama.cpp v1.0.0, a stronger foundation for the local AI ecosystem](https://huggingface.co/blog/ggml-org/llama-cpp-v1)
- [GitHub: ggml releases](https://github.com/ggml-org/ggml/releases)
- [GitHub: llama.cpp README](https://github.com/ggml-org/llama.cpp)
- [GitHub: GGUF specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)
- [Hugging Face: GGUF documentation](https://huggingface.co/docs/hub/gguf)
- [GitHub: llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md)
- [GitHub: llama.cpp build documentation](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)
- [GitHub: llama-server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
