Learning to rank

Overview

Learning to rank is a family of machine learning methods that train a model to put items in the best order for a given query or context. The model does not only predict a label or a number. It predicts a score for each candidate, and the scores decide the order. Search engines, shops, job boards and feed products use it to decide which result appears first.

The core idea is simple. For each query you have a list of candidate items, and each item has features and, in training data, a relevance judgment. The model learns a scoring function so that more relevant items get higher scores. What makes the field special is the loss function and the metrics. They care about the order of the top of the list, not about the exact score of every item. Learning to rank is closely related to Recommendation and is often built on gradient boosting such as XGBoost and LightGBM.

What it is

Learning to rank, often shortened to LTR, is a supervised learning task. The input is a set of groups. Each group is a query (or a user session, or a context) with a list of candidate documents. Each candidate is a feature vector. The training signal is a relevance grade, a click, a purchase or a pairwise preference.

The output is a function that maps a feature vector to a real-valued score. At serving time you compute the score for every candidate in a group, sort by score, and show the top items.

Think of a librarian helping a visitor. The visitor asks for books about gardening. The librarian does not give each book a grade out of ten in isolation. The librarian cares about which books go on the top shelf near the desk. LTR trains a model to behave like that librarian.

Where ranking sits in a system

Most large systems use a funnel with several stages.

• Candidate generation finds a few hundred or thousand items from millions. This stage is often a keyword index or a vector index such as HNSW or FAISS.
• A first ranker scores those candidates with cheap features.
• A second, richer ranker re-scores the short list, often with a gradient boosted model or a neural network.
• A final stage applies business rules, diversity and freshness constraints.

Learning to rank usually lives in the second and third stages. It is a re-ranking layer on top of retrieval. The same idea appears in retrieval-augmented generation, where Cross-encoder reranking reorders passages before they reach a language model.

Inputs and labels

Features fall into groups.

• Query features: length, type, language, popularity.
• Item features: age, quality score, price, category.
• Query-item features: text match scores such as BM25, embedding similarity, click-through history for this pair.
• User features: past behavior, location, device.

Labels come in several forms.

Where it came from

Early information retrieval ranked documents with hand-made formulas. TF-IDF and later BM25 combined term counts and document length into a single score. These formulas worked well, but each had a few tunable constants. When search engines began to use dozens of signals, tuning them by hand became impractical.

In the late 1990s and early 2000s researchers started to treat ranking as a learning problem. Work on ordinal regression and on learning from preferences showed that a model could learn weights for many signals. A widely cited step was RankNet, a neural pairwise method from the mid 2000s that trained on pairs of documents with a cross-entropy loss over score differences. Ranking SVM, an earlier method, used a similar pairwise idea with support vector machines.

Next came LambdaRank and LambdaMART. LambdaRank noticed that you do not need a smooth loss to train a ranker. You can specify the gradients directly and scale them by how much swapping two items would change a metric such as NDCG. LambdaMART applied that idea to gradient boosted trees. It became a long-standing workhorse in industry and in public ranking challenges.

Listwise methods such as ListNet and ListMLE defined losses over the whole list using permutation probabilities. Later work added neural rankers, unbiased learning to rank to correct click bias, and transformer-based cross-encoders for text.

Public datasets such as the Microsoft learning to rank datasets, Yahoo learning to rank challenge data and Istella helped the field compare methods. The general finding for many years was that boosted trees with good features were very hard to beat on tabular ranking features. That finding still matters today, and it ties the field to tabular modeling.

How it works

This section explains the standard approaches step by step.

Step 1: group the data

Every training row belongs to a group, called a query id. The model is never asked to compare an item from one query with an item from another. A group might be one search, one page view or one user session.

Analogy: a sports league table. You compare teams within the same league and season, not teams across different sports.

Step 2: choose a framing

There are three families.

Pointwise. Treat each item as a regression or classification problem. Predict the relevance grade or the click probability for each item alone. Then sort by prediction. This is simple and works with any regressor. Its weakness is that it ignores the list. A model can have low error on every item and still produce a poor order at the top.

Pairwise. Look at pairs of items in the same group where one is more relevant. Train the model so that the score of the better item is higher. The loss is often a logistic loss on the score difference. This directly targets order. Its weakness is that all pairs count equally, even though a mistake at position 1 matters more than one at position 50.

Listwise. Define the loss over the whole list. Some losses approximate a metric like NDCG. Others use a probability model over permutations. Listwise methods can focus on the top of the list. They are more complex and sometimes harder to optimize.

Step 3: compute the loss

A common pairwise logistic loss for items i and j, where i is more relevant, is:

loss(i, j) = log(1 + exp(-(s_i - s_j)))

Here s_i and s_j are the model scores. The loss is small when s_i is much larger than s_j.

LambdaRank multiplies the gradient of this loss by the absolute change in NDCG that would happen if the two items swapped places. So pairs near the top, or pairs with large grade gaps, push harder on the model.

Step 4: train the model

With gradient boosting, each new tree fits the gradients (the lambdas) of the current model. Trees split on feature thresholds. After many rounds the sum of the trees gives the final score.

With a neural model, you feed the features through layers to get a score and use backpropagation, often in PyTorch.

Step 5: evaluate with ranking metrics

You score a held-out set of groups, sort, and compute metrics on the top k positions. Common metrics are listed in the key terms table below.

Step 6: serve and monitor

At serving time you retrieve candidates, compute features, score and sort. You monitor feature drift, latency and online metrics such as click-through rate. Offline gains do not always carry over, so teams run A/B tests.

Dealing with biased clicks

Click data is biased. People click more on the top positions simply because they see them first. A model trained on raw clicks may learn to keep the old order. Unbiased learning to rank tries to correct this using propensity weights or click models. Randomized swaps in a small slice of traffic give a clean estimate of position effects.

Dealing with ties and missing features

Real ranking data has many ties in labels and many missing feature values. Tree models handle missing values natively. Neural models need imputation or a missing flag. This is the same issue that arises in tabular foundation models, covered below.

Common loss functions in more detail

Pairwise logistic loss. This is the loss used by RankNet. It treats each pair as a tiny binary classification problem. The target is "item i should beat item j." The probability that i beats j is a sigmoid of the score gap. Training pushes that probability toward one.

Hinge loss on pairs. Ranking SVM uses a margin. Two items must differ in score by at least a fixed margin, or the model pays a penalty. This gives a sparse set of active pairs and a robust solution.

Softmax cross-entropy over a list. Turn the scores in a group into a probability distribution with a softmax. Compare it with a target distribution derived from the labels. This is the idea behind ListNet. It is easy to implement in a deep learning framework and works well with neural scorers.

Approximate NDCG losses. Because sorting is not differentiable, some methods replace the rank with a smooth approximation. The loss then follows NDCG more closely. These methods need care with temperature settings and can be noisy.

Lambda gradients. LambdaRank skips the explicit loss and defines the update directly. This was a clever shortcut, and it works well with trees because trees only need gradients and second-order information, not a closed-form loss.

Features that often matter in practice

Features such as popularity need a time window and smoothing. A raw click-through rate on three impressions is noise. A smoothed rate that shrinks toward the category average is much safer.

Sampling negatives and building training lists

A query may have thousands of candidates, but only a handful are judged. Common practice is to build training lists from the judged items plus a sample of unjudged items treated as non-relevant. The sample should look like what the ranker sees at serving time. If the sample is full of easy negatives, the model learns little. If it is full of unjudged but actually relevant items, labels become noisy.

A good habit is to mine hard negatives. These are items that a previous ranker placed high but that users skipped. Training on them sharpens the decision at the top of the list.

Multi-objective ranking

Real products rarely care about one number. A shop wants relevance, margin and delivery speed. A video app wants watch time and satisfaction. There are three common ways to combine goals.

• Train one model per objective and blend scores with tuned weights.
• Train a single model on a combined label.
• Train a multi-task neural model with several heads, then combine.

Weights should be chosen with online tests, because offline metrics for each goal can conflict.

Explaining a ranker

Stakeholders ask why an item was ranked first. For tree models, SHAP values give a per-item breakdown of feature contributions. For auditing, store the top contributing features with each served ranking, at least for a sample of requests. This helps debugging and supports review of fairness concerns.

Key concepts and terms

How NDCG works in plain words

DCG adds up the gain of each result and divides it by a discount that grows with position. Gain is often 2 to the power of the grade, minus one. The discount is often the log of position plus one. A great result at position 1 contributes a lot. The same result at position 10 contributes much less. NDCG divides DCG by the best possible DCG for that list, so the value lies between 0 and 1 and can be compared across queries.

Analogy: a shop window. Items in the center of the window matter much more than items in the corner. NDCG says the same.

A worked example

Below is a small, runnable example using a gradient boosted ranker. It uses synthetic data, so it contains no real records.

import numpy as np
import pandas as pd
import lightgbm as lgb
from sklearn.metrics import ndcg_score

rng = np.random.default_rng(7)

n_queries, per_query = 200, 20
rows = []
for q in range(n_queries):
    for _ in range(per_query):
        text_match = rng.random()
        quality = rng.random()
        price_gap = rng.normal()
        # hidden relevance mixes the features plus noise
        signal = 2.0 * text_match + 1.0 * quality - 0.3 * abs(price_gap)
        grade = int(np.clip(np.round(signal + rng.normal(0, 0.3)), 0, 3))
        rows.append((q, text_match, quality, price_gap, grade))

df = pd.DataFrame(rows, columns=["qid", "text_match", "quality", "price_gap", "grade"])

train = df[df.qid < 160].sort_values("qid")
valid = df[df.qid >= 160].sort_values("qid")

features = ["text_match", "quality", "price_gap"]
group_train = train.groupby("qid").size().to_list()
group_valid = valid.groupby("qid").size().to_list()

ranker = lgb.LGBMRanker(
    objective="lambdarank",
    n_estimators=300,
    learning_rate=0.05,
    num_leaves=15,
    min_child_samples=20,
)
ranker.fit(
    train[features], train["grade"], group=group_train,
    eval_set=[(valid[features], valid["grade"])],
    eval_group=[group_valid],
    eval_at=[5, 10],
)

valid = valid.assign(score=ranker.predict(valid[features]))

scores = []
for qid, g in valid.groupby("qid"):
    scores.append(ndcg_score([g["grade"].values], [g["score"].values], k=10))
print("mean NDCG@10:", float(np.mean(scores)))

Key points in this code:

• Rows must be sorted by query id, and the group sizes must match that order.
• The objective lambdarank uses the lambda gradients described earlier.
• Evaluation uses NDCG at several cutoffs.
• Metrics are computed per query and then averaged.

A numeric walk-through of NDCG

Take one query with four results shown in this order, with grades 3, 2, 0, 1.

Using gain = 2^grade - 1 and discount = log2(position + 1):

DCG is about 9.32. The ideal order is grades 3, 2, 1, 0. Its DCG is 7.00 + 1.89 + 0.50 + 0 = about 9.39. NDCG is about 0.99. If the grade 3 item had been placed last, NDCG would fall sharply. That sensitivity to the top is the reason ranking metrics differ from accuracy.

Using XGBoost instead

XGBoost offers rank:ndcg and rank:pairwise objectives. CatBoost has YetiRank and related losses. The data layout is the same: sorted by group, with group sizes or a query id column.

Strengths and limitations

Strengths

• Targets the order directly, not an average error.
• Handles many heterogeneous features, including counts, flags and scores from other models.
• Boosted tree rankers are fast to train and fast to serve.
• Works well with modest data, because the feature set carries much of the knowledge.
• Easy to combine signals: lexical match, embedding similarity, popularity, freshness.
• Metrics such as NDCG match what product teams care about.

Limitations

• Needs groups. Data without a clear query or context is hard to use.
• Labels are expensive (human raters) or biased (clicks).
• Offline NDCG may not match online behavior.
• Feature engineering matters more than model choice.
• Pairwise and listwise losses can be unstable for deep models on small data.
• Rankers can create feedback loops. If the model decides what users see, it also decides what data it will learn from next.
• Fairness and diversity are not handled by the loss alone. They need extra constraints or post-processing.

How it compares with alternatives

Learning to rank is often confused with classification. A classifier asks, "Is this item relevant?" A ranker asks, "Which of these items should come first?" The second question is the one the user cares about.

Role in tabular foundation models

Learning to rank is a tabular problem at heart. Each candidate is a row. Columns are features of mixed types: numbers, categories, flags, counts, scores. Some values are missing. Rows come in groups. This is exactly the setting that tabular foundation models target, so the two meet in several concrete places.

Rows, columns and mixed types

Ranking features come from many systems, and their names and scales differ from team to team. A tabular foundation model that learns from raw tables must read numeric and categorical columns without being told what each means. Models in the family of TabPFN, TabICL and TabSTAR show that a single pre-trained network can predict from a table it has never seen, using the rows as context. Ranking features fit that picture: a short list of rows with a numeric relevance target.

Group structure as context

In-context learning feeds labeled rows into the model as examples and asks it to predict a new row. A ranking group is a natural context. The labeled items from similar queries serve as examples, and the model scores the new candidates. Today, tabular foundation models mostly handle classification and regression, so a practical route is to use them as a pointwise scorer and then sort. A group-aware or listwise head is an active direction rather than a settled feature.

Missing values

Ranking logs have missing values everywhere: a new item has no click history, a query has no location. Gradient boosting handles missing values by learning a default direction at each split. Pre-trained tabular models handle missing values by seeing masked or absent cells during pre-training. Ideas such as masked cell modeling teach a network to expect holes. See Masked cell modeling and Imputation networks.

No column names

Many ranking tables have anonymous columns, like feature_17 or f203. This is common in public benchmark sets that hide the meaning of each feature. A model that depends on column names cannot use such tables. A tabular foundation model that learns from the values and their joint structure can. This is one reason why tabular pre-training on synthetic tables with random names is attractive.

Pre-training on synthetic or real tables

Models such as TabPFN are pre-trained on a large number of synthetic tables drawn from a prior over data-generating processes. A ranking dataset with a smooth relation between features and grades looks like many of those synthetic tasks. This explains why such models can give a strong pointwise baseline with no tuning on small ranking sets.

Continual learning

Ranking is non-stationary. Items, seasons and user taste change. Retraining from scratch each week is costly. Continual learning methods such as experience replay and regularization keep old knowledge while adapting to new data. See Experience replay and EWC. A tabular foundation model with a continual learning path can adapt its scoring to drift without forgetting older patterns.

Imputation

Features for ranking are often filled in by imputation: a median for price, a zero for clicks. Poor imputation hurts the ranker silently. A model that imputes from the joint distribution of the row gives better inputs. Learned imputers serve as a preprocessing stage in front of the ranker.

Anomaly detection

Ranking logs contain bots, spam and click fraud. These corrupt labels. Anomaly detection with Isolation forest or learned density models flags suspicious sessions so that they can be removed or down-weighted before training the ranker.

Sector and semantic classification

When items or queries carry text, a semantic classification step can assign a category or sector to each. That category then becomes a feature or a routing key: use one ranker for electronics and another for travel. A tabular model with semantic heads can read a row and predict such a label even when the column names are unknown. See Semantic sector heads.

Where tree ensembles still win

Honest comparison matters. On large ranking sets with rich features, LambdaMART style boosted trees remain very strong and cheap to serve. Foundation models give the most value when data is small, features are anonymous, or you need a quick baseline. For deeper discussion of how such models are built and evaluated, see /model/architecture and /model/benchmarks, and for background reading see /model/research.

Practical guidance and best practices

Data

• Split by query or by time, never by row. Rows from the same query must stay in the same split, or you leak information.
• Prefer a time-based split when the system will predict the future.
• Keep a clean validation slice of fully judged queries for model selection.
• Log the position at which each item was shown. You need it to correct bias.
• Log the features used at serving time, not features recomputed later. This avoids train-serve skew.

Features

• Start with a few strong signals: lexical score, embedding similarity, popularity, freshness, quality.
• Add interaction features, such as a ratio of price to category median.
• Normalize features within a query when absolute values vary. Rank-based or z-score features per group often help.
• Keep a feature importance view. SHAP values show which features move scores.

Modeling

• Begin with a pointwise baseline, then try LambdaMART.
• Tune with Optuna or a similar search: number of leaves, learning rate, minimum samples per leaf, number of trees.
• Use early stopping on NDCG at the cutoff that matches the product, such as 5 or 10.
• Compare with a simple heuristic baseline to make sure the model adds value.

Evaluation

• Report NDCG at several cutoffs and also MRR or MAP.
• Report per-segment results, such as head and tail queries.
• Check calibration if scores feed other systems.
• Run interleaving or A/B tests. Interleaving needs less traffic to detect a difference.

Serving

• Keep feature computation fast. Often features, not the model, dominate latency.
• Cache stable features.
• Score only the short list.
• Version models and features together.

Fairness and diversity

• Add a diversity step such as maximal marginal relevance after scoring.
• Audit exposure across groups of items or providers.
• Consider exploration, such as small random swaps, to keep collecting unbiased data.

A short end-to-end checklist

Use this list when you build a ranker for the first time.

1. Define the unit of ranking: a search, a page view or a session.
2. Define the label: graded judgments, clicks or purchases.
3. Log the shown position and the features at serving time.
4. Build a baseline from a simple score such as BM25 or popularity.
5. Split data by time or by query.
6. Train a pointwise model for a quick reference.
7. Train a LambdaMART model with group information.
8. Compare NDCG at the cutoffs that match the product.
9. Slice results by query type, device and segment.
10. Inspect feature importance and a few failure cases by hand.
11. Run an interleaving or A/B test.
12. Set up monitoring for feature drift and metric drift.
13. Plan a retraining schedule and a rollback path.

Reading failure cases by hand

Numbers hide problems. Pick twenty queries at random and look at the top five results from the new model and the old one. Ask whether a person would agree with the new order. Often you will find a feature bug, such as a unit mismatch, or a label problem, such as a category that raters judged differently. Ten minutes of manual review often saves days of tuning.

Cost and latency budgets

A ranker must answer within a strict time limit, often tens of milliseconds for the whole page. Costs come from three places: fetching features, running the model and sorting. Reduce cost by limiting the number of candidates, pre-computing heavy features offline, and using smaller trees for the first re-ranking stage with a richer model only on the top slice.

Common mistakes

1. Random row split. Items from the same query land in train and test. The metric looks great and fails in production.
2. Unsorted groups. Most libraries expect rows sorted by group. A mismatch silently trains on wrong groups.
3. Training on clicks without bias handling. The model learns the old ranking, not relevance.
4. Optimizing accuracy or AUC. These do not reflect top-of-list quality.
5. Ignoring ties. Many items share the same label. Metrics need a consistent tie-breaking rule.
6. Leaky features. Using a feature computed after the click, such as final dwell time, as an input.
7. Too many items per group. Very long lists dilute the loss. Truncate or sample.
8. Different candidate sets in training and serving. The ranker sees easy negatives in training and hard ones in production.
9. Not monitoring drift. Features such as popularity change quickly.
10. Tuning on the test set. Always keep a final untouched slice.
11. Comparing NDCG across different cutoffs or label scales. Use the same definition of gain and discount.
12. Skipping a simple baseline. A BM25 or popularity baseline tells you whether the learned model is worth its cost.

Frequently asked questions

What is learning to rank in simple terms?

Learning to rank is a way of training a model to put items in a good order for a query. Instead of predicting one label per item, the model learns scores so that more relevant items rank higher. The best-known uses are web search, product search and feeds, where the top few positions get most of the attention.

What is the difference between pointwise, pairwise and listwise ranking?

Pointwise methods score each item alone, as in regression. Pairwise methods learn from pairs of items and try to put the better one first. Listwise methods define the loss over the whole list and often relate to a metric such as NDCG. Listwise and pairwise methods focus more on order, while pointwise methods are simpler.

What is LambdaMART?

LambdaMART is a gradient boosted tree method for ranking. It uses lambda gradients, which are pairwise gradients scaled by how much a swap would change a ranking metric such as NDCG. It has been a strong, fast and practical choice for many years, and it is available in libraries such as XGBoost, LightGBM and CatBoost.

Which metric should I use to evaluate a ranker?

NDCG at a cutoff such as 5 or 10 is the most common choice when you have graded relevance. MRR suits tasks with one right answer, and MAP suits binary relevance with many relevant items. Pick the metric that matches how users look at the list, and report more than one cutoff.

How is learning to rank different from recommendation?

A ranking system usually starts with an explicit query and orders documents for it. A recommender often has no query and predicts what a user may like from past behavior. In practice the two overlap, because recommenders use ranking models at their final stage. See the Recommendation page for the user-item view.

Can I learn to rank from clicks?

Yes, but clicks are biased by position and by presentation. Users click what they see first. To use clicks well you can model position bias with propensities, use click models, or collect a small randomized slice. Without a correction the model tends to reproduce the existing order.

Do tabular foundation models support ranking?

Pre-trained tabular models mainly support classification and regression today. You can use them as a pointwise scorer and sort the outputs, which gives a quick baseline on small ranking sets. Native group-aware ranking heads are an area of ongoing work, so check the model documentation before relying on them.

How much data do I need?

It depends on the number of features and the noise in labels. Boosted rankers can work with a few thousand judged queries when features are strong. Neural rankers on raw text need far more data. When data is small, a pre-trained model or a simple baseline is often the better start.

Should I use a neural network or gradient boosting?

For dense tabular ranking features, gradient boosting is a strong default and often wins. Neural rankers shine when you feed them raw text, images or very large click logs. Many production stacks combine both: neural models produce scores or embeddings, and a boosted ranker combines them with other features.

How do I handle new items with no history?

This is the cold start problem. Use content features such as category and text embeddings, add exploration so new items get some exposure, and use priors that shrink toward the category average. Tree models handle missing history features natively, and tabular foundation models are built to cope with missing cells.

Glossary

• A/B test: An online experiment comparing two versions on live traffic.
• BM25: A classic text scoring function based on term frequency and document length.
• Candidate generation: The first stage that finds items worth scoring.
• Click model: A model of how users examine and click results.
• Cold start: The lack of history for new users or items.
• Cross-encoder: A model that reads a query and a document together to score relevance.
• DCG: Discounted cumulative gain, a position-weighted sum of gains.
• Feature: An input variable describing a query, an item or their pair.
• Gain: The reward for a relevance grade, often 2^grade minus 1.
• Gradient boosting: An ensemble method that adds trees to fit the errors or gradients of the current model.
• Group: The set of candidates for one query.
• Interleaving: An online test that mixes results from two rankers in one list.
• Lambda gradient: A pairwise gradient scaled by the change in a metric.
• MAP: Mean average precision.
• MRR: Mean reciprocal rank.
• NDCG: Normalized DCG, DCG divided by the ideal DCG.
• Pairwise loss: A loss over pairs of items.
• Listwise loss: A loss over a full ranked list.
• Pointwise loss: A loss over single items.
• Position bias: The tendency to click higher positions more.
• Propensity: The probability that a user examines a position.
• Re-ranking: Reordering a short list with a stronger model.
• Train-serve skew: A mismatch between training features and serving features.

Summary

Learning to rank trains a model to order candidates so that the best items appear first. The three main framings are pointwise, pairwise and listwise, and the most proven practical method is LambdaMART with gradient boosted trees. Success depends on correct grouping, honest splits, bias-aware labels and metrics such as NDCG that reward the top of the list.

Because ranking data is made of rows with mixed types, missing values and often anonymous columns, it connects directly to tabular foundation models. Those models can serve as a quick pointwise scorer, help with imputation, flag anomalies in logs and classify items into semantic groups. For large, feature-rich ranking problems, boosted trees remain a strong baseline. Start simple, split by query or time, measure NDCG, and validate online before you trust the numbers. For more background see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Forecasting and tasks in my stack, under Foundation models and applied ML. Others in the same group: