What is lm-evaluation-harness? Definition and meaning
EleutherAI's framework that runs a large library of academic benchmarks against a model with a unified interface.
Definition
lm-evaluation-harness: EleutherAI's framework that runs a large library of academic benchmarks against a model with a unified interface.
In short: Standard framework for benchmarking language models.
When it is used
Reproducible benchmark scores for base and fine-tuned models.
Where it fits
lm-evaluation-harness belongs to LLM observability and evaluation in the Generative AI and LLMs topic. These are the other terms in the same group:
- LangSmith: A platform from LangChain for tracing runs, building datasets and evaluating prompts and agents.
- Langfuse: A platform for tracing, prompt management, scoring and cost tracking of LLM applications.
- Phoenix: A tool for tracing LLM applications with OpenTelemetry and running evaluations on the traces.
- Braintrust: A platform for running experiments, scoring outputs and tracking quality over time.
- PromptFoo: A command-line tool that runs test cases and comparisons across prompts and models, including security tests.
Learn more
The full guide to lm-evaluation-harness explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: LLM-as-judge · Next term: Logistic regression