Qwen 4

Overview

Qwen 4 is the fourth generation of the Qwen family, a set of language models from Alibaba that are published with open weights in a range of sizes. It is a general-purpose family with solid multilingual skill, coding ability and tool use. Because the sizes run from small models that fit on a laptop up to large ones for data centres, Qwen 4 is a common choice as a base for self-hosting and fine-tuning.

This article explains what Qwen 4 is, how the Qwen family grew, and how models of this kind work. It includes a worked example, a comparison with other model families, and a section on how a language model relates to tabular foundation models that learn from raw tables. Specific sizes, context windows and licences vary by variant, so always check the model card of the exact checkpoint you plan to use.

What it is

Qwen 4 is a family of large language models, not a single model. A family means several checkpoints that share a design and training recipe but differ in size and purpose.

A family, not one file

Typical members of a Qwen generation include:

• Small dense models for phones, laptops and edge boxes.
• Mid-sized dense models for single-GPU servers.
• Large dense or mixture-of-experts models for data-centre use.
• Base checkpoints, which are raw pre-trained models good for fine-tuning.
• Instruct or chat checkpoints, which follow directions and hold conversations.
• Specialised variants for code, vision or audio, depending on the release.

The exact lineup for the fourth generation is listed on its official pages.

Why a range of sizes matters

Bigger models usually score higher but cost more to run. A family lets you pick the smallest model that meets your quality bar. It also lets you build a pipeline where a small model handles easy requests and a big one handles hard ones.

Open licences

The family is published under open licences. Open licence does not always mean unrestricted. Terms can differ by size and release, so read the licence for commercial use, redistribution and use of outputs.

What it is good at

• General question answering and writing.
• Multilingual work, including a strong showing in Chinese and English and support for many other languages.
• Code generation and explanation.
• Maths and step-by-step reasoning, in modes that allow longer thinking.
• Tool calling and structured output for agents.
• Serving as a starting point for custom models.

Where it came from

Qwen comes from Alibaba Cloud. The first Qwen models were released in 2023. Since then the family has grown through several numbered generations, each larger in scope than the last.

A short timeline in general terms

• 2023: the first Qwen language models and chat models appear.
• 2024: Qwen1.5 and Qwen2 widen the size range and improve multilingual coverage. Qwen2.5 follows with larger training data, plus specialised code and maths models.
• 2025: Qwen3 introduces both dense and mixture-of-experts models and a hybrid approach in which one model can switch between a fast mode and a thinking mode.
• Later Qwen3 updates add faster variants and refined checkpoints. See Qwen3.8 for a late release in that line.
• The fourth generation, Qwen 4, continues the family.

Companion models

Beyond text models, the Qwen name has also been used for vision-language models, audio models and embedding and reranking models. These share ideas with the main line and are often used together in retrieval pipelines. See advanced RAG and cross-encoder reranking for how such parts fit.

Why the family matters

Qwen is one of the most widely used open-weight families on model hubs. Many community fine-tunes, quantised builds and tools start from Qwen checkpoints. That ecosystem is a practical reason to consider it. Look at Hugging Face for how such models are shared.

How it works

Follow a prompt through a Qwen-style model. Each step has an everyday analogy.

Step 1: Tokenisation

Text is cut into tokens by a byte-level tokenizer built with byte-pair encoding. Byte-level means any text, in any script or with any symbol, can be represented, because the worst case is individual bytes. Analogy: a set of Lego bricks that includes tiny single-stud pieces, so you can always build the shape, even if you have no special brick for it.

Step 2: Embedding and position

Each token becomes a vector. A position scheme, commonly rotary position embeddings, tells the model where each token sits. Analogy: giving each word both a meaning and a seat number, so the model knows who sits next to whom.

Step 3: Attention

Each layer lets tokens attend to earlier tokens. Modern families use grouped-query attention, in which several query heads share a smaller set of key and value heads. This cuts the memory used by the KV cache. Analogy: a team of readers shares a small set of reference books rather than each carrying a full copy.

Step 4: Feed-forward layers

After attention each token passes through a feed-forward block. In dense Qwen models every token uses the whole block. In the mixture-of-experts variants, a router chooses a few expert blocks per token. See mixture of experts.

Step 5: Normalisation and residual paths

Layers use normalisation and residual connections so that very deep stacks train stably. Analogy: handrails and landings on a long staircase keep climbers from falling when the staircase is tall.

Step 6: Next-token prediction

The last layer scores every token in the vocabulary. A sampler picks the next one. The loop repeats until a stop token or a length limit.

Hybrid thinking

Some Qwen generations let one model answer quickly or think first. You control this with a flag or a special instruction. In thinking mode the model writes out its reasoning, then gives the answer. This costs more tokens but helps with hard maths, logic and planning. Check the model card for how Qwen 4 exposes this.

How the model is trained

1. Pre-training on a very large mix of web text, books, code, maths and multilingual data, with next-token prediction. See curriculum scheduling for how data order can be planned.
2. Long-context training so the model can read large inputs.
3. Supervised fine-tuning on instruction data. See SFT.
4. Preference tuning and reinforcement learning to shape helpfulness, safety and reasoning. See DPO and reinforcement learning.
5. Distillation, where larger models help train smaller ones in the family.

Data quality

Large model families invest heavily in filtering, deduplication and synthetic data generation. Clean data often matters more than raw volume.

Key concepts and terms

A worked example

This example loads a mid-sized checkpoint with the Hugging Face libraries, runs a prompt and then sketches a LoRA fine-tune. Names such as the model id are placeholders. Replace them with the real id from the model hub.

Running a prompt

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "example-org/qwen-4-instruct-placeholder"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a concise data assistant."},
    {"role": "user", "content": "Explain what a null-safe join is, in two sentences."},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=200, temperature=0.3, do_sample=True)
reply = tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(reply)

The chat template step matters. Each family has its own format for system, user and assistant turns, and using the wrong one lowers quality.

A LoRA fine-tune outline

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
# Train with your preferred trainer on a small, clean instruction dataset.

Only a tiny share of the weights are trained, so memory use is small. See LoRA, QLoRA, PEFT and TRL.

Sizing walk-through

A model with 8 billion parameters stored in 16-bit form needs about 16 gigabytes for weights, since each parameter takes 2 bytes. In 4-bit form it needs roughly 4 to 5 gigabytes. Add memory for the KV cache and activations. A 32 billion parameter model needs about 64 gigabytes in 16-bit form, so it often needs two GPUs or a quantised build. These are rough figures to guide planning.

Serving it

For many users at once, use a serving engine such as vLLM or SGLang. For local use on modest hardware, use llama.cpp with a GGUF file.

# Example only: start an OpenAI-compatible server on a placeholder path.
vllm serve /models/qwen-4-instruct --max-model-len 32768 --port 8000

Long inputs and the context window

Context window size is the number of tokens the model can read and write in one request. Larger windows let you pass whole reports or code bases, but cost and delay grow with length.

• Memory for the KV cache grows with every token.
• Attention work grows faster than linearly in plain attention, so efficient kernels such as FlashAttention matter.
• Quality often drops for facts buried in the middle of a very long prompt.
• Retrieval usually beats stuffing everything into the window.

Analogy: a desk can hold only so many open books. A bigger desk helps, but a good librarian who fetches the right page is still faster than piling up every book.

Embedding and reranking companions

Search pipelines usually use three kinds of model.

1. An embedding model turns each document chunk into a vector, which is stored in a vector index such as FAISS or pgvector.
2. At query time, the question is embedded and the nearest chunks are fetched.
3. A reranker reorders the chunks by true relevance, and a chat model writes the answer from the best ones.

Using models from one ecosystem for these roles can simplify tokenisation and deployment, but always test each part on its own data.

Sampling settings in practice

For extraction and classification, keep temperature near zero. For ideas and drafts, raise it a little.

Strengths and limitations

Strengths

• A wide range of sizes in one family.
• Open weights under open licences, with a large community.
• Strong multilingual support, including Chinese and English.
• Good code and maths skill for its size.
• Many quantised and fine-tuned variants already available.
• A good base for domain-specific fine-tuning.
• Companion embedding and reranking models help build search pipelines.

Limitations

• Small members are weaker at hard reasoning and long contexts.
• Large members need serious hardware.
• Like all models, it can hallucinate.
• Chat template mistakes are a frequent source of poor results.
• Licence terms can differ by size.
• Safety behaviour should be tested for your use case and region.
• Benchmarks reported by the vendor may not match your data.

Where it is not the best fit

• Predicting a target from a wide table of numeric features.
• Exact calculation over large data.
• Cases needing a vendor service contract, if you cannot run your own infrastructure.

How it compares with alternatives

How to choose a size

1. Define the task and the quality bar with a small test set.
2. Start with a mid-sized model and see if it passes.
3. If it fails on hard cases, try the next size up or a thinking mode.
4. If it passes easily, try the next size down to save cost.
5. Compare quantised and full-precision builds on the same set.

Choosing between base and instruct

• Use the instruct model for chat, extraction and agents.
• Use the base model when you plan heavy fine-tuning on your own format.
• Use an embedding model, not a chat model, for vector search.

Role in tabular foundation models

Qwen 4 is trained on text and code. Tabular foundation models are trained on tables. The two types of model meet in practical pipelines.

What a tabular foundation model does

Models such as TabPFN, TabICL and TabSTAR are pre-trained on many tables, often synthetic tables drawn from random causal graphs. At use time, you supply labelled rows as context and unlabelled rows as queries, and the model predicts in a single forward pass. This is in-context learning on tabular data, and it needs no training on your table. The designs behind such models are discussed at /model/architecture and /model/research.

Important properties of those models:

• They respect row and column structure instead of flattening a table into a token stream.
• They handle mixed types: floats, integers, categories, dates and text.
• They treat missing values explicitly.
• They aim to be robust when column names are missing or uninformative.
• They are designed so that shuffling rows or columns does not change the answer. See axial attention.
• They can support tasks such as imputation, anomaly detection and sector or domain classification from the table itself. See masked cell modeling, imputation networks and semantic sector heads.

Where Qwen 4 fits

• Schema reading: it can read column names, descriptions and sample values and suggest types and meanings.
• Code generation: it can write SQL, pandas or Spark code to prepare and join tables.
• Column and dataset labelling: it can propose a domain or sector label from names and values, which can be compared with what a tabular model infers.
• Explaining outputs: it can turn feature importances, SHAP values and anomaly scores into plain text.
• Agent planning: it can call a tabular model as a tool through function calling or MCP.
• Text columns: when a table has free-text fields, an embedding model from the same ecosystem can turn them into vectors that a tabular model uses as features. Some tabular models, like TabSTAR, are designed to combine text and numbers.

A pipeline sketch

1. A user asks a question about a new dataset.
2. The language model reads the schema and a sample, and writes cleaning code.
3. The code runs in a sandbox and produces a clean table.
4. The agent calls a tabular foundation model to predict, impute or flag anomalies.
5. The language model reads the results and writes a short report with caveats.
6. A person reviews anything high stakes.

Why not just prompt the LLM with the table

• Long tables overflow the context window.
• Numbers are split into odd tokens, hurting precision.
• The cost grows with every row.
• Row order may change the answer, which is a bug for tables.
• Results are hard to reproduce exactly.

Fine-tuning for table-adjacent tasks

A small Qwen model, fine-tuned with LoRA, can be a cheap helper for tasks such as column type detection, unit extraction or data-quality rule writing. Build a labelled set from your own anonymised metadata, keep a hold-out set and track errors. Use synthetic data generation with care and always check samples by hand.

Continual learning

Tabular systems that must adapt to new data can use EWC and experience replay to avoid forgetting older patterns. Language models adapt more often through adapters and retrieval. In both cases a regression set is the guard against silent loss of old skills.

Practical guidance and best practices

Setup

• Use the exact chat template that ships with the checkpoint.
• Pin library versions, so behaviour does not drift.
• Record the checkpoint hash and the licence in your project notes.

Prompting

• State the task, the format and any limits at the top.
• Give a few short examples when the output format is strict.
• Use low temperature for extraction and higher for brainstorming.
• Ask for JSON that matches a schema and validate it. See structured outputs.

Inference

• Use vLLM or SGLang for server workloads.
• Use quantised builds, such as AWQ or GPTQ, when memory is tight.
• Set sensible max token limits.
• Batch requests to raise throughput.
• Monitor time to first token and tokens per second.

Fine-tuning

1. Collect a clean dataset of 500 to a few thousand examples.
2. Split into train, validation and test.
3. Start with LoRA at a small rank.
4. Track validation loss and task metrics.
5. Compare against the untuned model with prompts alone.
6. Keep the adapter small and versioned.

Retrieval

For knowledge that changes, use retrieval instead of fine-tuning. See naive RAG, hybrid search and RAGAS for measuring quality.

Safety

• Add input and output checks. See guardrails.
• Treat retrieved text as untrusted. See prompt injection defense.
• Run generated code in a sandbox.
• Keep personal data out of prompts unless you have a lawful basis and a safe environment.

Evaluation

Build a private test set. Use evals and, where helpful, LLM as judge with spot-checks by people. Re-run after any change of model, prompt or quantisation.

Cost control

• Route simple requests to a small model.
• Cache shared prompt prefixes where the server supports prefix caching.
• Shorten system prompts and remove unused examples.
• Stream tokens so users see progress early.
• Set per-user and per-feature budgets. See cost and latency budgets.

Team workflow

1. Pick a candidate size and write down why.
2. Build the private test set before any tuning.
3. Record baseline scores for prompts alone.
4. Change one thing at a time: prompt, size, precision or adapter.
5. Keep a changelog of model files, prompts and scores.
6. Review a random sample of outputs every week once live.

Data handling for table tasks

• Strip direct identifiers before sending rows to any model.
• Prefer sending summaries and schemas to sending raw records.
• Log only what you need for debugging.
• Keep the model close to the data when privacy rules are strict, which is one reason teams self-host open-weight models.

Common mistakes

• Using the wrong chat template and blaming the model.
• Picking the largest size by default instead of testing smaller ones.
• Fine-tuning a chat model on data in a different format and breaking it.
• Comparing a quantised model to a full model without checking quality.
• Relying on vendor benchmarks only.
• Ignoring the licence for commercial use or redistribution.
• Overfitting a LoRA adapter to a tiny dataset.
• Loading a base model and expecting it to follow instructions.
• Forgetting to cap output length.
• Pasting whole tables into prompts and trusting the arithmetic.
• Letting an agent run tools without limits or review.
• Skipping a regression test after upgrading to a new checkpoint.
• Assuming the tokenizer is identical across sizes or generations without checking.
• Mixing prompts written for one family into another without adjustment.
• Training on test examples by accident, which inflates scores.
• Using a very high temperature for structured output.
• Measuring only accuracy and ignoring cost per correct answer.
• Leaving old adapters in production after the base model has changed.
• Treating a model's confident tone as a sign of correctness.
• Failing to document which checkpoint, prompt and settings produced a result.

Frequently asked questions

What is Qwen 4?

Qwen 4 is the fourth generation of Alibaba's Qwen family of language models. It is published with open weights in a range of sizes, from small models to large ones, and it is used for general chat, multilingual work, coding and as a base for fine-tuning.

Is Qwen 4 open source?

The weights are published under open licences, which makes the family open-weight. That is not always the same as fully open source, because training data and some code may not be released, and licence terms can differ between sizes. Read the licence of the exact checkpoint you plan to use.

Which Qwen 4 size should I choose?

Start by writing a small test set from your real tasks. Try a mid-sized model first. Move up if it fails on hard cases and move down if it passes with room to spare. Compare quantised and full-precision builds too, since they trade a little quality for much lower memory use.

Can I run Qwen 4 on my own hardware?

Smaller members can run on a laptop or a single GPU, especially in quantised form. Larger members need several GPUs. Check the parameter count, multiply by the bytes per parameter for your precision, and add room for the KV cache to estimate memory.

Does Qwen 4 support many languages?

The family is known for strong Chinese and English and for support of many other languages. Quality varies by language and by size, so test with samples from each language you need.

Can I fine-tune Qwen 4?

Yes, where the licence allows. Parameter-efficient methods such as LoRA are common because they need little memory. Try prompts and retrieval first, since they are simpler to maintain, and keep a hold-out set to check for regressions.

How does Qwen 4 relate to tabular foundation models?

They are different kinds of model that work well together. Qwen 4 reads text and code, so it can read schemas, write data-preparation code and explain results. A tabular foundation model is pre-trained on many tables and predicts, imputes or flags anomalies directly from rows and columns using in-context learning. An agent can combine both.

Can Qwen 4 predict values from a spreadsheet?

It can describe a small table and write code to analyse a large one, but it is not the right tool for accurate prediction from many columns. Use a tabular model or gradient boosting for the prediction, and use Qwen 4 for code and explanation.

What is the difference between a base and an instruct model?

A base model is pre-trained to predict the next token and does not reliably follow directions. An instruct model is further tuned on instructions and preferences so it answers questions and follows formats. Use instruct for chat and tools, and base for heavy custom fine-tuning.

How do I keep Qwen 4 outputs reliable?

Use the right chat template, low temperature for extraction, schema validation, retrieval for facts and a private test set. Add human review where the stakes are high, and re-test whenever you change the model or its settings.

Glossary

• Adapter: a small set of added weights trained during fine-tuning.
• Base model: a pre-trained model before instruction tuning.
• Chat template: the text format a model expects for conversation turns.
• Checkpoint: a saved set of weights.
• Context window: the number of tokens a model can read at once.
• Dense model: a model that uses all parameters for each token.
• Embedding: a vector representing text.
• Fine-tuning: more training on task-specific data.
• Grouped-query attention: attention where several query heads share key and value heads.
• Instruct model: a model tuned to follow directions.
• KV cache: stored keys and values for fast generation.
• LoRA: low-rank adapters for efficient fine-tuning.
• MoE: mixture of experts.
• Open-weight: weights published for download under a licence.
• Quantisation: lower-bit storage of weights.
• Reranker: a model that reorders retrieved results.
• Token: a unit of text.
• Tokenizer: the tool that maps text to tokens.
• Prefix caching: reusing computed state for a shared prompt start.
• Retrieval: fetching relevant documents before answering.
• Rotary embedding: a position scheme applied inside attention.
• Sandbox: an isolated place to run untrusted code.
• Thinking mode: a mode that writes reasoning before the answer.
• Vector index: a store that finds nearest embeddings quickly.

Summary

Qwen 4 is the fourth generation of Alibaba's open-weight Qwen family. Its main appeal is the range of sizes, the open licences, solid multilingual and coding skill, and a large ecosystem of tools and community variants. It is a strong general-purpose model and a popular base for fine-tuning.

For tabular work, use it as the language and planning layer. It reads schemas, writes data-preparation code, labels columns, explains outputs and calls other tools. Let tabular foundation models, described at /model/research, /model/architecture and /model/benchmarks, do the cell-level prediction, imputation and anomaly work directly on rows and columns. Choose the smallest size that passes your tests, use the right chat template, read the licence and keep a regression set.

Related

It belongs to Open-weight models in my stack, under LLMs, retrieval, and evaluation. Others in the same group: