# Small language models

Small language models have a few billion parameters or fewer and run cheaply on modest hardware or devices. Learn where they match large models and where they do not.

## Overview

A small language model, often shortened to SLM, is a language model compact enough to run cheaply on modest hardware: a single consumer GPU, a laptop, a phone or a small cloud instance. There is no official cut-off. In current usage the label usually covers models from a few hundred million to roughly ten billion parameters, compared with frontier models that are far larger. What matters in practice is the result: fast answers, low cost per request, and the option to keep data on your own machine.

Small models are not simply weak versions of large ones. Trained on carefully chosen data, distilled from bigger models and then fine-tuned for one job, they can match much larger models on narrow tasks such as classification, extraction, routing or short summaries. Where they fall behind is breadth: wide general knowledge, long multi-step reasoning and handling unusual requests.

Think of a pocket knife versus a full workshop. The workshop can build anything, but you have to go there and it costs more to run. The pocket knife travels with you, opens instantly and handles most everyday jobs. A small language model is the pocket knife: always at hand, cheap, private, and enough for many tasks, but not the tool for building a house.

## What it is

| Aspect | Typical small model | Typical large frontier model |
|---|---|---|
| Parameters | Hundreds of millions to about ten billion | Tens to hundreds of billions or more |
| Hardware | Laptop, phone, single GPU, CPU | Multiple data-centre GPUs |
| Memory at 4-bit | Under 1 GB to about 6 GB | Tens to hundreds of GB |
| Latency | Low, often milliseconds per token | Higher, depends on serving |
| Best use | Narrow, high-volume or private tasks | Broad, open-ended, complex tasks |
| Customisation | Easy and cheap to fine-tune | Expensive or API-only |

Most small models use the same decoder-only transformer design as large ones; see [GPT architecture](/tools/gpt-architecture). Some encoder models such as [BERT](/tools/bert) are also small, but "SLM" usually refers to generative models.

## Where it came from

Making smaller models from larger ones has a long history. In 2019 Hugging Face released DistilBERT, a distilled version of BERT that kept most of its accuracy with fewer parameters. As generative models grew huge after GPT-3, interest turned to how far small models could go.

A key moment was Microsoft's Phi-1 in June 2023, a 1.3-billion-parameter coding model described in the paper "Textbooks Are All You Need". It showed that very high-quality, partly synthetic training data could make a small model punch above its size. Microsoft's Phi-3-mini followed in April 2024 at 3.8 billion parameters. In February 2024 Google released the first Gemma open models, and in September 2024 Meta added 1-billion and 3-billion-parameter models to the Llama 3.2 family, aimed at phones and edge devices. Other families, such as Qwen from Alibaba, have also released small sizes. For current open families see [Gemma 4](/tools/gemma-4) and [Qwen3.8](/tools/qwen3-8).

Alongside the models, tooling matured. [llama.cpp](/tools/llama-cpp) and the [GGUF](/tools/gguf) file format made it easy to run quantised models on CPUs and laptops, while mobile and browser runtimes brought models onto phones.

## How it works

### Step 1: Train on carefully selected data

Small models have less capacity to memorise, so data quality matters even more. Builders filter web text heavily, add curated educational material and often generate synthetic examples with larger models.

### Step 2: Learn from a larger teacher

[Knowledge distillation](/tools/knowledge-distillation) trains the small model to imitate a larger model's outputs or probability distributions. Recent small reasoning models are often trained on step-by-step traces written by larger [reasoning models](/tools/reasoning-models).

### Step 3: Compress for deployment

Quantisation stores weights in fewer bits, for example 4-bit instead of 16-bit, cutting memory by about four times with modest quality loss. Methods include [AWQ](/tools/awq) and [GPTQ](/tools/gptq). Pruning can remove less useful weights.

### Step 4: Fine-tune for the task

Because small models are cheap to train, teams fine-tune them on their own examples using LoRA or QLoRA. A few thousand good examples can turn a general small model into a strong specialist.

### Step 5: Run where the data is

The model runs on a phone, a laptop, an edge box in a factory or a modest server. Runtimes such as llama.cpp, ONNX Runtime or mobile frameworks handle efficient execution.

### Step 6: Combine with larger models

Many systems use a small model first and escalate hard cases to a large one. A small model can also serve as the draft in speculative decoding.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Parameter | A learned number in the model; more parameters mean more capacity |
| SLM | Small language model, an informal size category |
| On-device inference | Running the model on the user's own hardware |
| Edge deployment | Running near the data source rather than in a central cloud |
| Quantisation | Storing weights with fewer bits to save memory |
| Distillation | Training a small model to copy a larger one |
| Synthetic data | Training data generated by other models |
| Fine-tuning | Further training on task-specific examples |
| Model cascade | Routing easy inputs to a small model, hard ones to a large one |
| Tokens per second | A common measure of generation speed |

## A worked example

An illustrative insurance firm receives 50,000 claim emails a day and needs to extract policy number, claim type and urgency from each.

1. Baseline. A large hosted model does the job at about 96 per cent field accuracy. Each email uses about 800 tokens in and 60 out. At the firm's API price this costs a noticeable amount per month, and data leaves the firm's network.
2. Small model trial. The team takes a 3-billion-parameter open model. Out of the box, with a careful prompt, accuracy is about 85 per cent.
3. Fine-tuning. They label 4,000 emails, partly by having the large model draft labels and staff check them. LoRA fine-tuning on one GPU takes under two hours. Accuracy rises to about 95 per cent.
4. Deployment. Quantised to 4-bit, the model needs about 2 GB of memory. One mid-range GPU server handles the full daily volume with room to spare.
5. Cascade. Emails where the small model's confidence is low, about 4 per cent, are sent to the large model. Combined accuracy is about 96 per cent.

The result in this example is similar accuracy, data kept in-house for most emails, and a large drop in cost. The figures are illustrative; real results depend on the task and data.

## Strengths and limitations

**Strengths:**

- Low cost per request and low latency.
- Can run offline, on-device or on-premises, keeping data private.
- Cheap and fast to fine-tune for a specific job.
- Lower energy use and simpler infrastructure.
- Good as routers, classifiers, extractors and draft models.

**Limitations:**

- Less broad knowledge and more factual errors on open questions.
- Weaker at long, multi-step reasoning and very long contexts.
- More sensitive to prompt wording and unusual inputs.
- Fine-tuned specialists can lose general abilities.
- On-device deployment must fit phone memory, battery and heat limits.

## How it compares with alternatives

| Option | Cost | Privacy | Breadth | Best for |
|---|---|---|---|---|
| Small language model | Low | High if self-hosted | Narrow to moderate | High-volume, focused tasks |
| Large hosted model | High per request | Depends on provider terms | Broad | Complex or varied tasks |
| Large self-hosted open model | High hardware cost | High | Broad | Sensitive data with complex needs |
| Classic ML, for example logistic regression on features | Very low | High | Very narrow | Simple, well-defined classification |
| Encoder models such as [BERT](/tools/bert) | Very low | High | Narrow | Classification and embeddings |

## Practical guidance and best practices

1. Define the task narrowly and build an evaluation set before choosing a model.
2. Try two or three small models of different sizes; the smallest that meets the target usually wins.
3. Use a large model to help create labelled training data, then check it with people.
4. Fine-tune with LoRA or QLoRA before concluding that a small model is not good enough; tools such as Unsloth make this cheap.
5. Quantise for deployment and measure the quality change.
6. Add a confidence check and a fallback route to a larger model.
7. Check the model licence for commercial use and redistribution.
8. Monitor quality in production, since narrow models can fail quietly on new input types.

## Common mistakes

- Judging a small model on broad chat ability when the real task is narrow.
- Skipping fine-tuning and comparing a raw small model with a prompted frontier model.
- Ignoring memory and thermal limits when targeting phones.
- Over-quantising, for example to very low bit widths, without testing.
- Expecting a small model to know rare facts instead of giving it retrieval or tools.
- Forgetting to re-check licence terms when switching model versions.

## Frequently asked questions

### How small is a small language model?

There is no fixed definition. Most people use the term for models up to around ten billion parameters, and especially for models that fit on one consumer device. The boundary keeps moving as hardware improves.

### Can a small model replace a large one?

For narrow, well-defined tasks, often yes, especially after fine-tuning. For open-ended assistance, complex reasoning or wide knowledge, a large model is still usually better.

### Can small models run on a phone?

Yes. Models of one to a few billion parameters, quantised to 4-bit, can run on recent phones. Speed and battery use depend on the device and the runtime.

### Are small models more private?

They can be, because they can run on your own hardware so data never leaves it. A small model called through a third-party API has the same privacy profile as any other API.

### What is the cheapest way to improve a small model?

Usually better task data and LoRA fine-tuning. Clear prompts and retrieval for facts also help a lot.

### Do small models hallucinate more?

On open factual questions, generally yes, because they store less knowledge. Giving them the facts through retrieval, and keeping tasks narrow, reduces this.

## Glossary

- Parameter count: number of learned weights in a model.
- 4-bit quantisation: storing each weight in four bits.
- LoRA: a method that fine-tunes small added matrices instead of all weights.
- Teacher model: the larger model used in distillation.
- Edge device: hardware that runs close to where data is created.
- Runtime: the software that executes a model on hardware.
- Fallback: sending a request to another model when the first is unsure.
- Open weights: model weights published for download and local use.

## Summary

Small language models are compact models, usually up to around ten billion parameters, that run quickly and cheaply on laptops, phones and single GPUs. Milestones such as DistilBERT, Microsoft's Phi series, Google's Gemma and small Llama 3.2 models showed that careful data, distillation and fine-tuning can make them strong at focused tasks. They trade breadth and deep reasoning for speed, cost and privacy. Used with quantisation, task fine-tuning and a fallback to larger models, they handle a large share of real-world workloads well.