Özer Cem KELAHMETKELAHMET
Glossary · Generative AI and LLMs

What is lm-evaluation-harness? Definition and meaning

EleutherAI's framework that runs a large library of academic benchmarks against a model with a unified interface.

Definition

lm-evaluation-harness: EleutherAI's framework that runs a large library of academic benchmarks against a model with a unified interface.

In short: Standard framework for benchmarking language models.

When it is used

Reproducible benchmark scores for base and fine-tuned models.

Where it fits

lm-evaluation-harness belongs to LLM observability and evaluation in the Generative AI and LLMs topic. These are the other terms in the same group:

  • LangSmith: A platform from LangChain for tracing runs, building datasets and evaluating prompts and agents.
  • Langfuse: A platform for tracing, prompt management, scoring and cost tracking of LLM applications.
  • Phoenix: A tool for tracing LLM applications with OpenTelemetry and running evaluations on the traces.
  • Braintrust: A platform for running experiments, scoring outputs and tracking quality over time.
  • PromptFoo: A command-line tool that runs test cases and comparisons across prompts and models, including security tests.

Learn more

The full guide to lm-evaluation-harness explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.

Previous term: LLM-as-judge · Next term: Logistic regression

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open