Özer Cem KELAHMETKELAHMET
Articles · Trends

llama.cpp reaches 1.0: how GGUF, quantization, ggml backends and llama-server fit together

On 13 November 2026 ggml-org tagged llama.cpp and ggml v1.0.0. The release prunes the deprecated API, adds an OpenAI-style speech endpoint and hardens llama-server. Here is how local inference works underneath, and what a 1.0 promise means for your app.

At a glance
Reading time
17 min
Sections
8
Code examples
1
Tables
3
Sources
9
Published
16 Nov 2026

llama.cpp is the C/C++ engine that runs a large share of local language models, from laptops to phones to small servers. On 13 November 2026 its maintainers at ggml-org tagged version 1.0.0 of both libllama (the model runtime) and libggml (the tensor library underneath it). After roughly three and a half years of rapid, unversioned change, the project now says it will signal breaking API changes with major version numbers.

The release itself is a clean-up and a hardening pass. It removes dozens of deprecated functions from the public C header, adds an OpenAI-compatible /v1/audio/speech endpoint to the built-in server, and closes several network security gaps in llama-server. For anyone building on llama.cpp, the version number matters more than any single feature: it is the first time the project has made a written promise about when your code will break.

What happened

ggml-org tags llama.cpp and ggml v1.0.0 (13 November 2026)

The GitHub release for llama.cpp v1.0.0 lists a set of breaking clean-ups. It removes the deprecated public API from llama.h, including llama_set_warmup, 35 deprecated functions, and the defrag_thold parameter with its --defrag-thold flag. llama_set_embd_state_src is renamed to llama_set_ctx_state_src, and the separate clip.h header is folded into mtmd, the multimodal library. In ggml, the legacy DIAG_MASK operations are removed, and the ggml release notes say that "ggml 1.0.0 prunes the deprecated public API."

New features include FP8 KV caches (f8_e4m3) with optional static per-tensor scales, a host-memory cache for mixture-of-experts weights with new moe_cache_size and moe_cache_cpu_share context parameters, new models such as Kimi-K3.1-Pro, and a redesigned web UI.

The server gains /v1/audio/speech and a security hardening pass (13 November 2026)

llama-server adds an OpenAI-compatible /v1/audio/speech endpoint and a temperature option for its existing /tts endpoint, and the notes list drama-3 text-to-speech model support.

On security, the release adds two opt-in flags, --allow-unauthenticated-tools and --media-allow-private-urls, so that risky behaviour is off unless you ask for it. The notes list fixes for --cors-origins, DNS rebinding, server-side request forgery through the CORS proxy, and direct browser access to child instances. The server now sends HF_TOKEN only to huggingface.co hosts, and cancels MCP tool calls when a client disconnects or times out.

Hugging Face publishes the versioning policy (13 November 2026)

A post on the Hugging Face blog, published under the ggml-org account and crediting Georgi Gerganov and Daniel Bevenius, explains the reasoning. "Today we release libraries v1.0.0, highlighting our responsibility to those developers," it says, referring to downstream projects. The post defines the rules: a MAJOR bump for a breaking change to the public C API in ./include, a MINOR bump for backward-compatible features, backend updates, model support or API additions, and a PATCH bump for bug fixes with no API change. According to the post, the project supports more than 150 architecture variants.

How a model becomes a file: GGUF and quantization

GGUF: one self-contained file

Every model llama.cpp runs is a GGUF file. GGUF is a binary format designed for fast loading. Think of it as a shipping container with the packing list taped inside the door: the weights and everything needed to interpret them travel together.

The GGUF specification lays the file out in a fixed order:

  1. Header. The magic bytes GGUF, a version number (the current spec is version 3), the number of tensors and the number of metadata entries.
  2. Metadata. Typed key-value pairs such as general.architecture (for example llama or mamba), context length, layer counts, and the full tokenizer: vocabulary, merges, special token IDs and often the chat template.
  3. Tensor info. For each tensor: its name, shape, data type (for example Q4_K or F16) and an offset into the data section.
  4. Tensor data. The raw weights, each aligned to a fixed boundary (32 bytes by default).

Because the tokenizer and chat template live in the file, a GGUF needs no companion files. Alignment makes the file friendly to memory mapping: the operating system maps it and loads pages only when touched, so startup is fast.

By contrast, safetensors, the common format for training checkpoints, stores tensors only, with tokenizer and config in separate files.

Quantization: storing weights in fewer bits

A model weight is normally a 16-bit number. Quantization stores it with fewer bits, much as a JPEG stores a photo with less data than the raw camera file. You lose some detail, but often less than you would expect.

llama.cpp quantizes in blocks. A block of weights shares one scale factor (and sometimes a minimum). Each weight is stored as a small integer q, and the runtime rebuilds an approximate value as w = q × scale or w = q × scale + min. The scale is what lets 4 bits cover the real range of values in that block.

The Hugging Face GGUF documentation describes three families:

Table 1. Quantization: storing weights in fewer bits
Family Examples How it works Typical use
Legacy Q4_0, Q5_1, Q8_0 32-weight blocks, one scale (plus a minimum for the _1 variants) Q8_0 is still a near-lossless baseline; others are described as rarely used
K-quants Q2_K to Q6_K Super-blocks of 256 weights split into sub-blocks, with the sub-block scales themselves quantized General-purpose default, especially Q4_K_M and Q5_K_M
IQ types IQ1_S to IQ4_XS 256-weight super-blocks guided by an importance matrix Very low bit widths where every bit counts

The _S, _M and _L suffixes on K-quants are mixes: a Q4_K_M file keeps some tensors at higher precision, so its effective bits per weight sits above the nominal 4.5 of Q4_K.

The importance matrix, or imatrix, holds statistics gathered by running sample text through the model, telling the quantizer where precision matters most. The quantize tool's documentation says accuracy loss "can be minimized by using a suitable imatrix file."

The quantize README gives concrete sizes for Llama-3.1-8B:

Table 2. Quantization: storing weights in fewer bits
Type Bits per weight Size (GiB)
F16 16.00 14.96
Q8_0 8.50 7.95
Q6_K 6.56 6.14
Q5_K_M 5.70 5.33
Q4_K_M 4.89 4.58
IQ3_M 3.76 3.52
IQ2_M 2.93 2.74

The same page says a Llama 3.1 70B model shrinks from 280.9 GB to 43.1 GB at Q4_K_M. That is the difference between needing a multi-GPU server and fitting on one large workstation. For GPU-oriented alternatives, see GPTQ and AWQ, often used with vLLM.

How a model runs: ggml, backends and the KV cache

ggml builds a graph, a backend executes it

llama.cpp does not hand-write a forward pass for each model. It describes each architecture as a graph of ggml operations: matrix multiplications, normalisation, rotary position embeddings, attention, and so on. ggml then schedules that graph onto one or more backends.

A backend is a set of kernels for one kind of hardware. The project README lists, among others:

Table 3. ggml builds a graph, a backend executes it
Backend Target hardware
CPU x86 (AVX, AVX2, AVX512, AMX), ARM NEON, RISC-V
Metal Apple silicon
CUDA Nvidia GPUs
HIP AMD GPUs
Vulkan Most GPUs with a Vulkan driver
SYCL Intel GPUs
WebGPU Browsers and other WebGPU hosts
RPC Remote machines over the network

According to the build documentation, Metal is enabled by default on macOS, while CUDA, Vulkan and HIP are switched on with CMake flags such as -DGGML_CUDA=ON. Several backends can be compiled into one build, and the GGML_BACKEND_DL option builds them as libraries loaded at runtime, so one binary can run on machines with different GPUs.

The key practical feature is hybrid CPU+GPU inference. The -ngl (GPU layers) setting decides how many transformer layers live in GPU memory; the rest run on the CPU. If a model is larger than your VRAM, you can still run it, just more slowly. The new host-memory MoE expert cache in 1.0 extends this idea to mixture-of-experts models, where only some experts are active per token and the rest can wait in system RAM.

Each quantization type needs its own kernels on each backend, which read the packed blocks directly instead of expanding weights back to 16 bits in memory.

Two phases: prefill and decode

Generation has two phases. In prefill, the model reads your whole prompt at once. This is compute-heavy and parallel, which is why the release mentions a Vulkan FlashAttention path for prefill. In decode, the model produces one token at a time. Each step must read essentially every active weight from memory, so decode speed is usually limited by memory bandwidth rather than arithmetic. This is the main reason quantization speeds things up: fewer bytes per weight means fewer bytes to move per token.

The KV cache

During decode, the model must attend to every earlier token. Recomputing all of them each step would be wasteful, so the runtime stores each layer's key and value vectors for past tokens. This store is the KV cache. It works like the notes you take while reading a long report: you consult your notes instead of rereading every page.

The cost is memory that grows with context length. A rough formula is:

KV bytes = 2 × layers × KV heads × head size × context tokens × bytes per value

For an illustrative model with 32 layers, 8 KV heads and a head size of 128, each token needs 2 × 32 × 8 × 128 = 65,536 values. At 16 bits that is 128 KiB per token, or 4 GiB for a 32,768-token context, on top of the weights. With several parallel users, multiply again.

llama.cpp gives you three levers. The --cache-type-k and --cache-type-v flags change the cache data type from the default f16 to smaller types. Version 1.0 adds an 8-bit floating point option, f8_e4m3, which halves the cache size compared with f16. Flash Attention (-fa) computes attention in tiles without materialising the full attention matrix, as described in our FlashAttention explainer. And the server's prompt cache reuses KV entries from a previous request, so only the part of a prompt that changed has to be processed.

The OpenAI-compatible server

What llama-server exposes

llama-server wraps all of the above in an HTTP service. Its documentation lists OpenAI-compatible chat completions, completions, responses and embeddings routes, an Anthropic Messages-compatible route, and a /health endpoint that returns 503 while the model loads and 200 when it is ready. With v1.0 the list gains /v1/audio/speech. Most OpenAI SDKs work by changing the base URL.

Under the hood, the server runs a fixed number of slots, set with -np. Each slot is one in-flight conversation with its own KV cache region. Continuous batching is on by default: new requests join the running batch between decode steps instead of waiting for others to finish. Other documented features include JSON-schema constrained output, speculative decoding with a draft model, and a Prometheus /metrics endpoint behind --metrics.

Prompt caching and its catch

The cache_prompt option is on by default. If a new request shares a prefix with the last one in that slot, the server skips the shared part. For chat and agent loops that resend a long system prompt each turn, this saves a lot of prefill work. The documentation warns that it can make results non-deterministic, because logits are not always bit-identical across batch sizes. If you need reproducible outputs for testing, turn it off for those runs.

Why the hardening matters

By default the server binds to 127.0.0.1. Many users expose it on a network anyway, and the server has grown features that fetch things: media URLs, tool calls, MCP servers. Each is a path an attacker could abuse. Server-side request forgery means tricking the server into fetching an internal address for you. DNS rebinding means a web page in your browser resolving a hostname to your local server and talking to it. The 1.0 fixes and the new opt-in flags close these paths by default. For larger shared deployments, our piece on the LLM serving stack covers routing and batching beyond a single process.

What a 1.0 stability promise means

Before 1.0, functions were renamed and removed often, and downstream bindings had to track the main branch closely. The 1.0 release clears the accumulated deprecated surface, then promises that future removals will come with a major version bump.

Read the scope carefully. The blog post defines MAJOR as a breaking change to "the public C API in ./include." That covers llama.h and the ggml headers you link against. It also commits to signalling ABI incompatibilities, so a binary built against 1.x should keep working with a later 1.x shared library. It does not explicitly mention the HTTP API of llama-server, command-line flags, the web UI, or the numerical output of a given model.

The MINOR category is also broad. New model support and backend updates land as minor releases. A backend kernel change can alter speed or, in edge cases, output, without being a "breaking change" in the semver sense. Semver protects your build, not your benchmark numbers.

Finally, GGUF has its own version field, separate from the library version. The current GGUF spec is version 3. Your existing model files are not invalidated by the 1.0 tag, though individual model architectures can still gain new metadata keys over time.

What it means in practice

For application developers, pinning becomes meaningful. You can depend on libllama 1.x and expect your C or binding code to compile across minor upgrades.

For people running local models, very little changes on day one. The same GGUF files load, the same quantization choices apply, and the server speaks the same OpenAI dialect. What improves is the default security posture of llama-server, plus FP8 KV caches and the MoE expert cache for fitting larger models or longer contexts into the same memory.

For teams weighing local inference against hosted APIs, a 1.0 C API lowers the risk of building on a fast-moving engine, but the trade-offs remain. llama.cpp shines on single machines, edge devices and mixed hardware. High-throughput multi-user GPU serving is still the home ground of engines like vLLM and SGLang. Our article on self-hosted open-weight coding agents covers where companies draw that line.

What to do now

  1. Pin a version. Move from tracking the main branch or build numbers to a 1.x release, and record it in your build or container file.
  2. Search your code for removed symbols. Check for llama_set_warmup, defrag_thold, llama_set_embd_state_src and includes of clip.h before you upgrade. The release notes list the full set.
  3. Update your bindings. If you use a language wrapper, check that it supports 1.0 before upgrading the shared library underneath it.
  4. Review server exposure. If llama-server listens on anything other than localhost, set --api-key, set --cors-origins explicitly, and leave the new opt-in flags off unless you need them.
  5. Re-measure memory with FP8 KV cache. Try the new f8_e4m3 cache type through --cache-type-k and --cache-type-v, and check output quality on your own prompts before adopting it.
  6. Choose quantization by test, not by habit. Start with Q4_K_M or Q5_K_M, compare against Q8_0 on a small evaluation set of your real tasks, and consider imatrix quantizations for 3-bit and below.
  7. Use prompt caching deliberately. Keep stable system prompts at the start of requests so prefix reuse works, and disable cache_prompt in tests that need deterministic output.
  8. Try the speech endpoint behind a flag. If you already call OpenAI's speech API, point a test client at /v1/audio/speech with a supported TTS model and compare latency and quality.

Risks and open questions

The promise covers the C API, not everything. Server routes, flags and output quality are not explicitly part of the written semver rule, so apps that depend on llama-server behaviour should still test each upgrade.

Minor releases can change numbers. Backend kernel updates are classed as minor. Speed and, rarely, outputs can shift between 1.x versions, so keep a regression set if your product depends on stable answers.

Low-bit quantization has hidden costs. Size tables show bits per weight, not task accuracy. Two-bit and three-bit models can look fine in chat and fail on reasoning or code, so measure on your workload.

Network exposure remains your job. The hardening closes known paths, but a local server with tools, media fetching and MCP enabled is a real attack surface. Treat it like any internal web service.

FP8 KV cache is new. Long-context quality across model families will only become clear with wider use.

Watch next: how fast downstream bindings and desktop apps adopt 1.x, whether the project extends a stability statement to the server's HTTP API, the first 2.0 discussion and what triggers it, and independent quality measurements for FP8 KV caches.

Frequently asked questions

Do I need to re-download or re-convert my GGUF models for llama.cpp 1.0?

The release notes do not announce a GGUF format change, and GGUF carries its own version number, currently 3. Existing files for supported architectures should keep loading.

What is the difference between llama.cpp and ggml?

ggml is the tensor library: it defines operations, data types and quantization formats, and runs them on backends like CUDA or Metal. llama.cpp (libllama) sits on top and knows about model architectures, tokenizers, sampling and the KV cache. Both reached v1.0.0 on the same day.

Which quantization type should I start with?

Q4_K_M is a common starting point because it cuts an 8B model from about 15 GiB to about 4.6 GiB while keeping more precision on sensitive tensors. If you have memory to spare, Q5_K_M or Q8_0 lose less quality. Below 4 bits, prefer IQ types made with an importance matrix, and test on your own tasks.

Is llama-server safe to expose to the internet?

It binds to localhost by default for a reason. The 1.0 release fixes several network attacks and makes risky features opt-in, but you should still add an API key, restrict CORS origins and put it behind a reverse proxy with TLS. Leave tool and private-URL features off unless you need them.

Does semantic versioning mean my outputs will not change between versions?

No. The policy defines breaking changes in terms of the public C API in the include directory. Kernel optimisations and backend updates can ship in minor releases and may change speed or produce slightly different numbers. Keep your own evaluation set to catch changes that matter to you.

Summary

llama.cpp and ggml reached v1.0.0 on 13 November 2026. The release removes deprecated C API functions, adds an OpenAI-compatible /v1/audio/speech endpoint, introduces FP8 KV caches and an MoE expert cache, and closes several security gaps in llama-server. Underneath, a GGUF file holds block-quantized weights and the tokenizer, ggml runs the model graph on CPU, Metal, CUDA, Vulkan or other backends, the KV cache trades memory for speed, and the server exposes it through OpenAI-style routes. The new part is the promise: breaking changes to the public C API will now come with a major version bump. That makes llama.cpp easier to build products on, as long as you remember the promise covers the headers, not server behaviour or model outputs.

Sources

Continue learning

Cite this article

Kelahmet, Ö. C. (2026, November 16). llama.cpp reaches 1.0: how GGUF, quantization, ggml backends and llama-server fit together. Özer Cem KELAHMET. https://www.ozercemkelahmet.com/articles/llama-cpp-1-0-local-inference

BibTeX
@online{kelahmet2026llama,
  author = {Kelahmet, {\"O}zer Cem},
  title  = {{llama.cpp reaches 1.0: how GGUF, quantization, ggml backends and llama-server fit together}},
  year   = {2026},
  month  = {nov},
  url    = {https://www.ozercemkelahmet.com/articles/llama-cpp-1-0-local-inference}
}

How this article connects

12 pages link to or from it. Drag to turn, click a dot to open it.

Version historyfirst version
  1. Published

Factual corrections are listed here with what changed. Wording and formatting edits are not listed.

Last updated Nov 16, 2026
Type to search.
    ↑↓ navigate↵ open