Özer Cem KELAHMETKELAHMET
Glossary · Generative AI and LLMs

What is Small language models? Definition and meaning

Language models with relatively few parameters that run on modest hardware or devices, often fine-tuned or distilled for specific tasks.

Definition

Small language models: Language models with relatively few parameters that run on modest hardware or devices, often fine-tuned or distilled for specific tasks.

In short: Compact models for fast, private work.

When it is used

On-device assistants, high-volume classification and private, low-latency workloads.

Where it fits

Small language models belongs to LLM concepts in the Generative AI and LLMs topic. These are the other terms in the same group:

  • Reasoning models: Language models trained to produce long internal reasoning before the final answer, trading extra inference time for accuracy.
  • Context engineering: The discipline of selecting, ordering and compressing instructions, data, tools and memory that enter a model's context window.
  • Prompt engineering: Techniques for phrasing instructions, examples, roles and output formats so a language model produces the intended result.
  • Embeddings: Dense numeric vectors produced by a model so that items with similar meaning are close together in vector space.
  • KV cache: Stored attention keys and values from previous tokens, reused so each new token only needs computation for itself.
  • Speculative decoding: An inference technique where a cheap draft proposes tokens and the main model verifies several at once, preserving its output distribution.
  • Multimodal models: Models that take in or produce several data types, such as text, images, audio and video, within one network.
  • Text-to-SQL: Using a language model to translate a natural-language question into an SQL query against a known schema.
  • BERT: A transformer encoder pre-trained to predict masked words using context from both sides, then fine-tuned for understanding tasks.
  • GPT architecture: A transformer that uses causal self-attention to predict the next token, trained on large text corpora and scaled up.

Learn more

The full guide to Small language models explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.

Previous term: Skills · Next term: Smart city

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open