Gemma 4

Overview

Gemma 4 is a family of open-weight language models from Google, built with research and technology related to the Gemini models. The family is offered in small sizes that run on modest hardware, such as a laptop, a single graphics card or a phone-class device. This makes it a popular choice for on-device and low-cost deployments, and a compact base for fine-tuning to a specific task.

The main idea behind a small open model is simple. Many real jobs do not need the largest model. Classifying a ticket, extracting fields from a document, drafting a short reply or acting as a private assistant can run well on a few billion parameters when the model is tuned for the task. This article explains what Gemma 4 is for, how small models of this kind work, how to run and tune them, and how they support workflows around tabular data and tabular foundation models. Specific figures such as sizes, context length and benchmark results differ by release, so confirm them on the official model card.

What it is

Gemma 4 is a set of decoder-only transformer language models released with open weights under a license set by Google. The weights can be downloaded and run locally, hosted on your own servers, or fine-tuned. The family generally comes in more than one size, and some members may accept images as well as text. Check the model card for the exact list of variants and modalities in this release.

Two ideas define the family.

• Small and efficient. The models are sized to run where large models cannot: on consumer hardware, in browsers and on devices with limited memory and power.
• Derived from Gemini research. Training methods and design choices come from the work behind the larger Gemini models, so the small models benefit from lessons learned at scale.

An analogy: if a frontier model is a large university library, a small open model is a well-chosen personal bookshelf. It holds less, but it is always with you, it is fast to consult and it is yours. For many daily questions, the bookshelf is enough, and for the rare hard one you can call the library.

Typical uses

Base and instruction variants

Open model families commonly ship in two flavors. A pre-trained (base) variant continues text and is the best starting point for heavy customization. An instruction-tuned variant has been trained to follow instructions and chat. For quick use, take the instruction variant. For fine-tuning on a special format or domain, many teams start from the base variant and apply SFT themselves.

Where it came from

Google released the first Gemma models in early 2024 as open models related to Gemini, aimed at developers and researchers. Later generations added improvements in quality, longer context, broader language coverage and, in some versions, image understanding. Across generations the family kept its focus on strong results for the size, and on easy use with common tools such as Hugging Face, PyTorch, JAX, Keras and llama.cpp.

The broader context is the rise of small language models. Research on training smaller models for longer on more and better data, and on distilling knowledge from large teacher models into small students, showed that a well-trained small model can match older, much larger ones on many tasks. Distillation is a key idea: a large model produces outputs or probability distributions, and the small model learns to imitate them, which gives richer training signals than raw text alone.

Gemma sits among several open families, for example Qwen, GLM, DeepSeek and Mistral. Each has its own strengths, sizes and license terms. Gemma 4 is the current generation described here, and details of what changed from earlier ones are in the release notes.

How it works

Step 1: Tokenize

Text is split into tokens using a SentencePiece-style tokenizer with a large vocabulary, see SentencePiece and BPE. A larger vocabulary helps with many languages and with code, and reduces the number of tokens per text.

Step 2: Embed and add position

Tokens become vectors. Position information is added, commonly with rotary embeddings, so the model knows order.

Step 3: Transformer layers

The vectors pass through a stack of decoder layers. Each layer has attention, which lets a token read earlier tokens, and a feed-forward block. Small models save memory and time with several design choices.

• Grouped or multi-query attention. Several attention heads share key and value projections, which shrinks the key-value cache.
• Local and global attention mix. Some layers attend only to a nearby window and others to the full context, which cuts cost for long inputs.
• Efficient normalization and activation functions. Choices that train stably and run fast.

The exact mix in Gemma 4 is on its model card. The principle is to keep quality while shrinking memory and compute.

Step 4: Predict the next token

The last vectors are turned into probabilities over the vocabulary, one token is chosen, and the process repeats. The key-value cache avoids recomputing earlier tokens, see FlashAttention for efficient kernels.

Step 5: How the model was trained

Training happens in stages.

Step 6: Running it in small spaces

To fit on modest hardware, models are quantized: weights are stored in 8, 4 or even fewer bits. A model in 16-bit precision needs about two bytes per parameter. In 4-bit it needs about half a byte. A model of a few billion parameters therefore fits in a few gigabytes. See AWQ, GPTQ, bitsandbytes, GGUF, ExLlamaV2 and TurboQuant.

Step 7: Adapting it

Because the model is small, tuning is cheap. LoRA and QLoRA can adapt it on a single graphics card. Full updates are also feasible for the smaller sizes, see full fine-tuning.

The memory math in practice

The 4 billion figure is an example for the arithmetic and not a statement about a specific Gemma 4 size. Add memory for the key-value cache, which grows with context length, and for activations.

Key concepts and terms

A worked example

This example loads an instruction-tuned Gemma model with Hugging Face Transformers in 4-bit precision, then runs a classification-style prompt. The model name is a placeholder: use the exact id from the official model page.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "example-org/gemma-4-small-it"   # placeholder id

bnb = BitsAndBytesConfig(load_in_4bit=True,
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_quant_type="nf4")

tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb,
                                             device_map="auto")

messages = [
    {"role": "user",
     "content": "Label this support ticket as billing, technical or other. "
                "Answer with one word.\n\nTicket: I was charged twice this month."}
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=5, do_sample=False)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Fine-tuning it with LoRA

A short recipe with PEFT and TRL follows. It trains small adapters on a classification-style dataset.

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

ds = load_dataset("json", data_files="data/tickets.jsonl", split="train")

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  target_modules="all-linear", task_type="CAUSAL_LM")

cfg = SFTConfig(output_dir="out/gemma-tickets", num_train_epochs=2,
                per_device_train_batch_size=8, gradient_accumulation_steps=4,
                learning_rate=2e-4, bf16=True, logging_steps=20,
                max_length=1024)

SFTTrainer(model="example-org/gemma-4-small", args=cfg,
           train_dataset=ds, peft_config=lora).train()

Running it locally with a compact format

For a laptop or a small server, convert to GGUF and run with llama.cpp. The commands below show the shape of the workflow with placeholder paths.

python convert_hf_to_gguf.py /models/gemma-4-small --outfile gemma4.gguf
llama-quantize gemma4.gguf gemma4-q4.gguf Q4_K_M
llama-cli -m gemma4-q4.gguf -p "Summarize: ..." -n 128

A small numeric view of the speed gain

On many devices, token generation speed is limited by how fast weights can be read from memory. Cutting weights from 16-bit to 4-bit reads about a quarter of the bytes per token, so generation can be faster, often by a large factor, though the exact gain depends on the hardware and the software. The quality cost of careful 4-bit quantization is usually small, but you should test it on your task.

Strengths and limitations

Strengths

• Runs on modest hardware, including laptops and single graphics cards.
• Low cost per request, which suits high-volume and always-on jobs.
• Data can stay on the device or inside your network.
• Easy to fine-tune with adapters, even for small teams.
• Works with common tools in the Hugging Face, PyTorch, JAX and llama.cpp ecosystems.
• Good quality for its size, helped by distillation and curated training.
• Predictable latency, since you control the hardware.

Limitations

• Less knowledge and weaker deep reasoning than the largest models.
• Smaller context capacity in practice, and long inputs raise memory use.
• More prone to errors on rare facts, so pair it with retrieval for factual work, see naive RAG.
• Tool use and long agent loops are less reliable than with larger models.
• Quantization can cost accuracy, especially on math and code.
• Open weights come with license terms to read.
• You operate the model yourself, with updates, monitoring and safety review.

How it compares with alternatives

The choice often comes down to where the model must run and how hard the task is. If data must stay on a device or if volume is huge, a small open model wins on cost and privacy. If the task needs deep reasoning, a larger or managed model may be needed. A routing design, where a small model handles most requests and escalates the rest, often gives the best balance, see cost and latency budgets.

Role in tabular foundation models

Gemma 4 is a language model, not a table-native model. Tabular foundation models such as TabPFN, TabICL, TabSTAR and XTab learn from raw tables: they treat rows and columns as structure, handle numeric, categorical and text cells, cope with missing values and missing column names, and are pre-trained on many tables, often synthetic, so that they predict through in-context learning. A compact language model plays a supporting role next to them. See /model/research and /model/architecture for the table-native side.

A compact text encoder inside a table pipeline

Some table models, such as TabSTAR, combine a table architecture with a text encoder so they can use free-text columns and column names. A small language model can serve as that text component, or as a feature extractor that turns text cells into vectors that a table model consumes. Because it is small, it can run on the same server as the table model without a large hardware bill.

Semantic work: column names, descriptions and sectors

Small language models are good at short semantic tasks that sit around a table.

When columns have no names, the language model sees only values. It can still infer a likely meaning from formats and ranges, such as a column of five-digit numbers that look like postal codes. Treat these guesses as suggestions with confidence scores and have a person review high-impact ones.

Missing values, mixed types and anomalies

A language model does not replace a table-native imputer for numeric cells. It can, however, help in the steps around them: spotting sentinel values written as text, such as "n/a" or "unknown"; harmonizing categories like "NY" and "New York"; and proposing rules for flagging anomalies, which a detector then scores, see anomaly detection and imputation networks. Numeric prediction stays with models built for it, such as table foundation models and tree ensembles like LightGBM.

Serializing tables for a language model

If you do give a table to a language model, serialize it with care. Common choices are markdown tables, JSON records, or "column is value" sentences. Keep the table small, round long numbers, and add a short description of units. Large tables cost many tokens and the model will make arithmetic mistakes, so prefer to let it write a query or code and run that instead.

Fine-tuning a small model on table tasks

A small model fine-tuned with LoRA on a narrow table-adjacent task can reach good accuracy at low cost. Examples: column type classification, sector labeling, header generation and anomaly explanation. Build the data from synthetic or masked tables to avoid exposing private records, see synthetic data generation and SFT. Include examples without headers so the model learns to rely on values.

Continual learning and drift

New sectors and new schemas keep arriving. Small models are cheap to retrain, so you can refresh adapters regularly. Keep a replay set of earlier examples to avoid forgetting, see EWC and continued pre-training for related ideas, and track schema drift so you know when to retrain.

Privacy and edge use

Tables from factories, vehicles or medical devices often cannot leave the site. A small open model can run at the edge next to the data, labeling and explaining while a compact table model scores rows, see industry 4.0 and predictive maintenance. Only summaries need to travel to a central system.

Practical guidance and best practices

Choosing a variant

• Pick the smallest size that meets your quality bar on your own evaluation set.
• Use the instruction variant for chat and tool-style tasks, and the base variant when you plan heavy customization.
• Check modalities, context length and license on the model card.
• Compare a quantized and a 16-bit version before committing.

Prompting small models

• Keep prompts short and explicit. State the task, the allowed outputs and the format.
• Show one or two examples, see few-shot learning.
• Ask for constrained output, such as one label from a list or JSON with a schema, see structured outputs.
• Use greedy or low-temperature decoding for classification and extraction.
• Split hard tasks into smaller steps instead of one long request.

Fine-tuning

• Start with prompting. Fine-tune only when a measured gap remains.
• Use LoRA or QLoRA first, and consider full fine-tuning for small sizes with plenty of data.
• Use the exact chat template in training and serving.
• Keep a fixed validation set and a general regression set.
• Libraries that help: PEFT, TRL, Unsloth, Axolotl, LLaMA-Factory and torchtune.

Deployment

• For servers, use an engine such as vLLM or SGLang. For laptops and edge devices, use llama.cpp with GGUF, or ONNX Runtime.
• Set a maximum output length and a timeout.
• Cache repeated prompts.
• Log quality signals and sample outputs for review, see Langfuse.
• Add input and output checks, see guardrails.

Evaluation

• Build a task-specific test set with at least a few hundred examples.
• Measure accuracy, format validity, latency and memory.
• Test with noisy, messy inputs, since real data is not clean.
• Re-test after each quantization or update, see lm-evaluation-harness and evals.

Safety

• Small models can be steered by hostile text in documents. Treat retrieved text as untrusted, see prompt injection defense.
• Run red-team prompts before launch, see red teaming.
• Do not give the model direct write access to important systems.

What "open-weight" means in practice

Open-weight means the trained numbers of the model can be downloaded and run on your own machine. It is not the same as open source in the strict sense, because the training data and the full training recipe are usually not published, and the license can carry terms of use. Think of a recipe book where you receive the finished cake and the right to slice, decorate and serve it, but not the full kitchen notes. Always read the license text that ships with the weights before using them in a product.

Why small models matter

The trade-off is capability. A small model has less room for world knowledge and long reasoning chains, so it shines on focused tasks and struggles on open-ended ones.

Sizing the hardware

A rough rule helps when planning. The memory needed for the weights is the parameter count times the bytes per parameter. At sixteen-bit precision that is two bytes per parameter, at eight-bit it is one byte, and at four-bit it is half a byte. On top of the weights you need memory for the attention cache, which grows with the length of the prompt and the number of simultaneous requests. A plan should therefore leave headroom beyond the raw weight size.

See quantization with bitsandbytes, GPTQ, AWQ and GGUF for the common formats.

A checklist for choosing a small model

1. Write down the task in one sentence and collect fifty real examples.
2. Run those examples through two or three candidate models with the same prompt.
3. Score the answers by hand or with a simple rubric.
4. Check the license for your use case.
5. Measure latency and memory on the target hardware, not on a larger test machine.
6. Check that the tokenizer handles your data, for example numbers, dates and the languages you need.
7. Decide whether prompting is enough or whether fine-tuning is needed.

Serving options

• A local runtime such as llama.cpp for laptops and edge boxes, using a compact file format.
• A high-throughput server such as vLLM or SGLang for shared GPUs.
• A managed endpoint on a cloud platform when you do not want to run servers.
• An on-device runtime for phones and embedded systems, where memory is the hard limit.

Prompt patterns that work well for small models

• Keep instructions short and put the most important rule first.
• Show two or three examples of the exact output format.
• Ask for structured output such as JSON, and validate it in code. See structured outputs.
• Break a hard task into small steps and call the model once per step.
• Provide the needed facts in the prompt instead of expecting the model to recall them.
• Set a low temperature for extraction and classification tasks.

How a small model fits into an agent

A compact model can act as a fast router, a classifier, or a tool-caller in an agent system, while a larger model handles the hardest steps. This split lowers cost, since most steps are simple. See function calling, orchestrator-workers and guardrails for the surrounding patterns.

Evaluating before and after fine-tuning

A drop in general ability after tuning is called forgetting. Mixing some general data into the training set, or using low-rank adapters with a modest rank, reduces it. See LoRA, QLoRA and PEFT.

Typical failure modes of small models

• Confident but wrong answers on facts outside the prompt.
• Drift from the requested format when the prompt is long.
• Arithmetic slips on multi-digit numbers. Use code or a calculator tool for exact math.
• Weaker handling of rare languages or specialized jargon.
• Sensitivity to small wording changes, which is why a regression set matters.

Working with tables through text

When a table is given to a language model, each row is turned into text. Choices matter. A compact delimited format saves tokens. A key-value format makes columns explicit. Rounding numbers reduces noise but loses detail. Row order can change the answer, so shuffle and check stability. For numeric prediction on tables, a dedicated tabular model such as TabPFN or a gradient boosting library usually beats a small text model, and the language model is best used for the semantic layer: understanding column meanings, writing descriptions and explaining results. See /model/architecture for how a table-native model differs from a text model.

A small governance note

Any model that processes personal or regulated data should be covered by a data-handling review. Local deployment helps, but it does not replace logging rules, access control and retention limits. See GDPR for the general frame.

Keeping up with new releases

Model families change quickly. Before relying on a specific variant, read the current model card for its sizes, context length, license and recommended prompt format. Do not assume details from older versions carry over. Build your pipeline so that the model name is a configuration value, and keep a small evaluation set so that swapping in a newer variant is a measurable, low-risk step.

Common mistakes

• Using a small model for everything. Some tasks need more capacity. Measure, and route hard cases to a larger model.
• Skipping the chat template. Wrong formatting is the top cause of poor small-model output.
• Assuming quantization is free. Test the quantized model on your data.
• Expecting reliable facts. Small models forget rare facts. Add retrieval.
• Pasting big tables into prompts. It is expensive and error-prone. Use code or a table-native model.
• Ignoring context memory. Long inputs enlarge the cache and can exhaust memory on small devices.
• Tuning on too little or noisy data. A small dataset with errors teaches errors.
• No regression suite. Fine-tuning can erode general skills without warning.
• Trusting guessed column names. Language-model guesses for unnamed columns need confidence scores and review.
• Forgetting the license. Check terms for your use before shipping.
• Never updating. Data and needs drift. Plan periodic evaluation and refresh.

More pitfalls to avoid

• Assuming a small model knows recent facts. Supply them in the prompt or through retrieval.
• Using the wrong chat template, which quietly lowers quality.
• Judging a quantized model by speed alone, without re-running the evaluation set.
• Fine-tuning on a few hundred noisy examples and expecting broad gains.
• Skipping a license check before shipping weights inside a product.
• Letting prompts grow without limit on a model with a modest context window.
• Trusting numeric answers from the model on tables where exact arithmetic is required.
• Running tests only on a fast desktop GPU and then deploying to a much weaker device.
• Forgetting to pin the exact model version and tokenizer in the deployment configuration.
• Not logging inputs and outputs for later review, which makes regressions hard to explain.
• Mixing up the base and instruction-tuned variants and wondering why chat quality is poor.
• Ignoring the cost of long prompts on memory.
• Assuming one prompt format works across model generations.
• Forgetting to re-test after every quantization change.

Frequently asked questions

What is Gemma 4?

Gemma 4 is a family of open-weight language models from Google, derived from research behind the Gemini models and offered in small sizes. It is meant for on-device and low-cost deployments and as a compact base that teams can fine-tune for their own tasks.

Is Gemma 4 free to use?

The weights are available to download under a license from Google that sets the terms for use, redistribution and fine-tuning. Read that license for your intended use. Running the model still costs hardware and operations effort, even when the weights themselves are open.

Can Gemma 4 run on a laptop or phone?

Small sizes of the family are designed to run on modest hardware, especially when quantized to 8-bit or 4-bit. Whether a particular size fits depends on the device memory and the context length you need, so check the memory arithmetic and test on the target device.

How do I fine-tune Gemma 4?

The usual route is LoRA or QLoRA with libraries such as PEFT and TRL. You prepare prompt and answer pairs in the chat template, train small adapters for a few epochs at a modest learning rate, and evaluate on a held-out set. For the smallest sizes, full fine-tuning is also practical when you have enough data.

How does Gemma 4 compare with larger models?

It is cheaper, faster and easier to run privately, but it has less knowledge and weaker deep reasoning than the largest models. It is a good fit for focused tasks such as classification, extraction and short generation, and less suited to the hardest multi-step reasoning without help.

Can Gemma 4 process tabular data?

It can read small tables serialized as text and help with tasks around tables, such as naming columns, labeling sectors and explaining results. For numeric prediction, imputation and anomaly scoring on real tables, table-native models and tree ensembles are usually more accurate and far cheaper per row.

How does a small language model work with a tabular foundation model?

The small model handles language-heavy parts such as column descriptions, text cells, taxonomy mapping and explanations, while the tabular foundation model handles numeric prediction through in-context learning. A simple orchestration layer or an agent can connect them, with the language model calling the table model as a tool.

What is quantization and should I use it with Gemma 4?

Quantization stores weights with fewer bits, such as 8 or 4, to save memory and speed up generation. It is often the difference between a model fitting on a device or not. Use it, but compare the quantized model with the 16-bit version on your own task before you deploy.

Do I need retrieval with a small model?

For factual or company-specific questions, yes in most cases. Small models store less knowledge and may guess. Retrieval supplies the relevant text at answer time, which improves accuracy and lets you update knowledge without retraining.

How often should I update a Gemma-based system?

Review it on a schedule, for example monthly or quarterly, and whenever a new release, a data change or a quality alert appears. Rerun your fixed evaluation set before any change, and keep the previous version available for rollback.

Glossary

• Adapter. A small trainable module added to a frozen model.
• Base model. A pre-trained model that continues text.
• Chat template. The layout of roles and special tokens a model expects.
• Context window. The maximum number of tokens a model reads at once.
• Distillation. Training a small model to imitate a larger one.
• Edge deployment. Running models close to where data is produced.
• GGUF. A compact file format for running models locally.
• Instruction-tuned. Trained to follow instructions and chat.
• KV cache. Stored attention data that speeds up token generation.
• LoRA. A method that trains low-rank adapters.
• Model card. The official description of a model's sizes, license and limits.
• Multimodal. Able to handle more than one kind of input, such as text and images.
• On-device inference. Running a model on the user's own device.
• Open-weight. Weights are downloadable under license terms.
• Quantization. Storing weights in fewer bits to save memory.

Summary

Gemma 4 is Google's family of open-weight small language models, built from Gemini research and sized for laptops, single graphics cards and edge devices. It shines where cost, privacy and latency matter: on-device assistants, high-volume classification and extraction, and as a compact base that LoRA or full fine-tuning can adapt quickly. Choose the smallest size that passes your own evaluation, use the right chat template, test quantized versions, and pair the model with retrieval for facts.

For tabular work, Gemma 4 is a helper and not the predictor. It can name unnamed columns, label sectors, normalize messy text, explain anomalies and call table-native models such as TabPFN or TabICL as tools, while those models do the numeric imputation, classification and anomaly scoring. Fine-tune it on masked or synthetic tables, keep replay data for continual updates, and review its guesses where mistakes are costly. For the table-native side, see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Open-weight models in my stack, under LLMs, retrieval, and evaluation. Others in the same group: