lm-evaluation-harness

Overview

The Language Model Evaluation Harness, usually called lm-evaluation-harness or lm-eval, is an open-source framework from EleutherAI for testing language models on standard benchmarks. It implements more than 60 academic benchmarks, with hundreds of subtasks and variants, behind one command-line tool and Python API. You point it at a model, name the tasks, and it returns scores computed in a consistent, documented way.

Consistency is the reason it exists. A benchmark score depends on many small choices: the exact prompt wording, the number of examples shown, how answers are extracted and how the score is normalised. Two papers can report different numbers for the same model on the same benchmark simply because they made different choices. The harness fixes those choices in shared task definitions so results can be reproduced and compared.

An everyday analogy: a standardised exam centre. Every candidate gets the same paper, the same time limit and the same marking scheme, so a score from one centre means the same thing as a score from another. The harness is the exam centre for language models.

What it is

The project is published on PyPI as lm-eval under the MIT licence. Since late 2025, the base package no longer installs transformers or PyTorch; model backends are installed as extras such as lm_eval[hf], lm_eval[vllm] or lm_eval[api].

Where it came from

EleutherAI is a non-profit research collective known for open models such as GPT-J and the Pythia suite. It started the harness around 2020 to evaluate its own models fairly against published results. The software is cited as "A framework for few-shot language model evaluation", by Leo Gao and colleagues.

Key milestones:

• Early versions implemented few-shot evaluation for common benchmarks with Hugging Face models.
• The harness became the backend for Hugging Face's Open LLM Leaderboard, which made its scores a common reference point for open models.
• Version 0.4.0 was a large refactor: config-based task creation in YAML, Jinja2 prompt templates, more configurable answer extraction and multiple generations, and faster backends including vLLM.
• In 2024, EleutherAI authors published "Lessons from the Trenches on Reproducible Evaluation of Language Models", describing the evaluation problems the harness is designed to address.
• Later additions included refactored support for API models with batched and asynchronous requests, SGLang support, experimental multimodal tasks, and an option to strip reasoning traces from models that produce them.
• In December 2025, the command-line interface gained subcommands (run, ls, validate) and YAML config support, and installation became lighter.

How it works

1. Choose tasks

Each task names a dataset, usually from the Hugging Face Hub, and describes how to turn each item into a prompt, what the correct answer is and which metric to compute. You pick tasks by name or group.

2. Choose a model backend

You tell the harness how to run the model: load it locally with transformers, serve it with vLLM, or call an API endpoint. Adapter weights such as LoRA can be evaluated through PEFT, and GGUF files can be loaded with the Hugging Face backend.

3. Build requests

For each example, the task produces one or more requests:

• Log-likelihood: for multiple-choice questions, the harness asks how probable each answer option is as a continuation of the prompt. The highest-scoring option is the model's choice. No text is generated.
• Rolling log-likelihood: for perplexity tasks, the model scores a whole passage.
• Generate until: for open questions, the model generates text until a stop sequence, and the answer is extracted and compared.

Log-likelihood requests need access to token probabilities, which many chat APIs do not expose. Those models can only run generation-based tasks.

4. Run and score

The harness batches requests, runs them, applies the task's metrics and aggregates results by task and group. It reports standard errors where possible, which shows how much a score could move by chance.

lm-eval run --model hf \
    --model_args pretrained=EleutherAI/pythia-160m \
    --tasks hellaswag,arc_easy \
    --num_fewshot 0 \
    --batch_size auto \
    --output_path results/ --log_samples

5. Inspect and report

Results include the full configuration: task versions, few-shot count, model arguments and package version. With sample logging on, you can read every prompt and answer, which is the best way to catch formatting problems.

Key concepts and terms

A worked example

This is an illustrative scenario.

A small team fine-tunes a 7-billion-parameter model for customer-support writing and wants to make sure general reasoning did not degrade.

1. They choose three tasks: a commonsense multiple-choice task, a science question task and a grade-school maths task that uses generation.
2. They run the base model and the fine-tuned model with identical settings: same few-shot count, same batch size, same package version, chat template off for both.
3. The base model scores 0.762 on the commonsense task with a standard error of about 0.004; the fine-tuned model scores 0.758. The gap is within one standard error, so they treat it as no real change.
4. On maths, the fine-tuned model drops from 0.41 to 0.33. Reading logged samples shows that the model now writes polite introductions before the answer, and the extraction rule misses the number in some cases.
5. They re-run with the chat template enabled and a looser extraction filter for both models. The gap shrinks to 0.40 against 0.37, which they record as a modest real regression.

The lesson: read the samples before trusting a score change.

Strengths and limitations

Strengths:

• Large, shared library of standard tasks with documented settings.
• Reproducible configurations and versioned tasks.
• Many model backends, from local weights to served and API models.
• Easy to add custom tasks in YAML.
• Widely used, so results are comparable with published numbers.
• MIT licence.

Limitations:

• Academic benchmarks rarely match a specific product's needs.
• Scores are sensitive to prompt format, few-shot count and chat templates.
• Log-likelihood tasks need token probabilities that many APIs do not provide.
• Public benchmarks may be contaminated by training data.
• Large models and many tasks need substantial GPU time.

How it compares with alternatives

Practical guidance and best practices

1. Fix and record every setting: tasks, few-shot count, model arguments, chat template, package version.
2. Compare models only under identical settings.
3. Turn on sample logging and read a sample of outputs for every new task.
4. Report standard errors, and do not over-interpret differences smaller than them.
5. Use the chat template for instruction-tuned models, and apply the same choice to all models compared.
6. Use --limit for quick smoke tests, never for final numbers.
7. Write custom YAML tasks for your own domain, alongside public benchmarks.
8. Treat benchmark scores as one signal among several, together with evals of real use.

Common mistakes

• Comparing your own numbers with a paper's numbers obtained under different settings.
• Mixing results from different task versions.
• Concluding a model is worse when the real cause is answer extraction.
• Running log-likelihood tasks on a backend that cannot return probabilities.
• Reporting results from a limited subset as if they were full runs.
• Ignoring possible contamination of public benchmarks.

Frequently asked questions

What models can the harness evaluate?

It supports Hugging Face transformers models, including encoder-decoder models, served models through vLLM and SGLang, several API providers and OpenAI-compatible endpoints, adapters through PEFT and GGUF files. You install the backend you need as an extra.

Why do my scores differ from published ones?

Small differences in prompt format, few-shot count, chat template, normalisation, task version or batch settings can change scores. Match every setting, check the task version, and read logged samples to find the cause.

What is the difference between acc and acc_norm?

Both measure accuracy on multiple-choice tasks. acc picks the option with the highest total log-likelihood. acc_norm divides by answer length first, so longer answers are not unfairly penalised.

Can I add my own benchmark?

Yes. Most tasks are defined in YAML files that name a dataset, a prompt template, the answer field and the metrics. You can register a folder of custom tasks and run them like built-in ones.

Is the harness suitable for testing a chatbot product?

Only partly. It measures general capabilities on fixed benchmarks. Product behaviour, safety and user experience need application-level tests with your own data.

Glossary

• Benchmark: a fixed dataset and scoring method used to compare models.
• Log-likelihood: the log probability a model assigns to a piece of text.
• Perplexity: a measure of how well a model predicts text; lower is better.
• Prompt template: the text pattern that wraps each question.
• Stop sequence: text that ends generation.
• Instruction-tuned model: a model trained to follow chat-style instructions.
• Adapter: small trainable weights added to a base model, as in PEFT.

Summary

lm-evaluation-harness is EleutherAI's open-source framework for evaluating language models on a large shared set of benchmarks with reproducible settings. It turns each benchmark into log-likelihood or generation requests, runs them through a chosen backend and reports scores with standard errors and full configuration. Use it to compare base and fine-tuned models fairly, fix and record every setting, read logged samples, and combine benchmark scores with application-level evaluation. See transformers for the most common model backend.

Related

It belongs to LLM observability and evaluation in my stack, under Generative AI and LLMs. Others in the same group: