Logistic regression

Overview

Logistic regression predicts the probability that an example belongs to a class. Will this customer cancel next month? Is this payment fraudulent? Will this patient be readmitted? The answer comes out as a number between 0 and 1, which you can then turn into a decision by choosing a threshold.

Despite its name, logistic regression is a classification method. It combines the features linearly, exactly as linear regression does, and then passes the result through the logistic function, also called the sigmoid. The sigmoid squeezes any number, however large or small, into the range from 0 to 1. Large positive scores become probabilities close to 1, large negative scores become probabilities close to 0, and a score of zero becomes exactly 0.5.

An everyday picture is a points scorecard. A lender adds points for a steady income, subtracts points for missed payments and adds a few for a long customer history. The total is then converted into a likelihood of repayment. Logistic regression learns how many points each factor should be worth, and how to convert the total into a probability, from past outcomes.

What it is

The key idea is that the model is linear in the log-odds. A coefficient of 0.7 on a feature means each extra unit multiplies the odds by e^0.7, about 2. That is the language doctors and credit analysts often use: "each missed payment roughly doubles the odds of default."

For more than two classes, the multinomial version (also called softmax regression) gives each class its own linear score and turns the set of scores into probabilities that add up to 1.

Where it came from

The logistic function itself is older than the model. Pierre François Verhulst published it in 1838, and in more detail in 1845, to describe how a population grows quickly at first and then levels off. In the twentieth century it was adopted for bioassay, the study of how organisms respond to doses of a substance. Joseph Berkson promoted the logistic model in the 1940s and named it the "logit", by analogy with the existing probit model.

During the following decades statisticians, including David Cox, developed logistic regression into a general method for binary outcomes, and Cox introduced the multinomial logit in 1966. Daniel McFadden linked the multinomial logit to economic theories of choice in 1973, work recognised by the Nobel memorial prize in economics in 2000. The model then became standard in epidemiology, credit scoring and social science, and later the default classification baseline in machine learning libraries such as scikit-learn.

How it works

Step 1: compute a score

For each example, multiply each feature by its weight and add the intercept. The result can be any real number.

Step 2: convert to a probability

Apply the sigmoid. A score of 2 becomes about 0.88, a score of 0 becomes 0.5, and a score of −2 becomes about 0.12.

Step 3: measure the error with log loss

Training uses log loss, also called binary cross-entropy. For a positive example the loss is −log(p); for a negative example it is −log(1 − p). Confident correct predictions cost almost nothing; confident wrong ones cost a great deal. Minimising log loss is the same as maximum likelihood estimation, a link to probability theory.

Step 4: optimise the weights

There is no closed-form solution, so the weights are found iteratively. Common solvers include L-BFGS, Newton-type methods and stochastic gradient descent. Because log loss for this model is convex, these methods reach the single best solution.

Step 5: regularise

Without a penalty, weights can grow very large, especially if a feature perfectly separates the classes. L2 regularisation shrinks weights smoothly, L1 can set some to zero, and elastic net mixes the two. In scikit-learn the strength is controlled by C, where smaller C means a stronger penalty.

Step 6: choose a threshold and evaluate

The default 0.5 threshold is rarely the right business choice. Pick the threshold from the cost of false positives and false negatives. Evaluate ranking quality with ROC AUC or precision-recall AUC, and probability quality with log loss, the Brier score and a calibration plot.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

clf = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000))
clf.fit(X_train, y_train)
probs = clf.predict_proba(X_test)[:, 1]

Key concepts and terms

A worked example

This scenario is illustrative. A subscription service wants to flag customers likely to cancel within 30 days. It has 50,000 customer records, of which 6 per cent cancelled. Features include months since joining, support tickets in the last 90 days, whether the customer is on a monthly or annual plan, and the change in weekly usage.

After scaling, an L2-regularised logistic regression is fitted with five-fold stratified cross-validation to choose C. Some coefficients, converted to odds ratios, are:

• Monthly plan: odds ratio 2.6, so monthly customers have about 2.6 times the odds of cancelling.
• Each extra support ticket: odds ratio 1.3.
• Each 10 per cent drop in usage: odds ratio 1.5.

On the held-out set, ROC AUC is 0.81. The retention team can call 1,000 customers a month. Instead of using 0.5, the team ranks customers by probability and calls the top 1,000; among those, precision is 31 per cent, five times the base rate. A calibration plot shows predicted probabilities slightly too high in the top decile, so the team applies isotonic calibration before using the numbers in a revenue forecast.

Strengths and limitations

Strengths:

• Outputs probabilities, which support threshold choice and expected-value decisions.
• Coefficients and odds ratios are easy to explain to non-specialists and regulators.
• Fast, convex training with a unique solution.
• Works well with many sparse features, such as words in text.
• Often reasonably calibrated out of the box.

Limitations:

• The decision boundary is linear in the features unless you add interactions or transformations.
• Needs careful encoding and scaling of features.
• Sensitive to strongly correlated features, which make coefficients unstable.
• Usually less accurate than gradient boosting on complex tabular data.
• Coefficients describe associations, not causes.

How it compares with alternatives

Practical guidance and best practices

1. Use logistic regression as the first baseline for any classification problem.
2. Scale numeric features so the regularisation penalty treats them equally.
3. Choose the regularisation strength by cross-validation, stratified by class.
4. Set the decision threshold from business costs, not by default at 0.5.
5. Report precision-recall metrics when the positive class is rare.
6. Check calibration before using probabilities in forecasts or pricing.
7. Add interaction terms or splines when residual patterns show non-linearity.
8. Fit encoders and scalers inside the training folds to avoid leakage.

Common mistakes

• Reporting accuracy on imbalanced data, where predicting "no" for everyone looks good.
• Interpreting raw coefficients without remembering they act on log-odds.
• Forgetting regularisation, then seeing huge weights from perfectly separating features.
• Rebalancing classes by oversampling and then trusting the shifted probabilities.
• Including features that are only known after the outcome, such as a cancellation reason code.

Frequently asked questions

Why is it called regression if it classifies?

Because it is a regression model for the log-odds of a class. The model fits a continuous quantity, the probability, and classification only happens when you apply a threshold. The name comes from its statistical roots in generalised linear models.

What is the difference between logistic regression and linear regression?

Linear regression predicts an unbounded number and is trained with squared error. Logistic regression predicts a probability through the sigmoid and is trained with log loss. Using linear regression for a yes or no target can produce predictions below 0 or above 1, which makes no sense as probabilities.

How do I handle a rare positive class?

Keep the natural class balance if you need calibrated probabilities, and use stratified splits and precision-recall metrics. Class weights can help the model pay attention to the rare class, but they shift probabilities, so recalibrate afterwards if the numbers will be used directly.

Can logistic regression handle many classes?

Yes. The multinomial form gives each class its own set of weights and uses the softmax function to produce probabilities that sum to one. An alternative is one-versus-rest, which trains one binary model per class.

Is logistic regression good for text?

It is a strong text baseline when combined with bag-of-words or TF-IDF features, because it copes well with very many sparse features. For deeper understanding of meaning, models built on embeddings or transformers usually do better.

Glossary

• Sigmoid: a function mapping any real number to the range 0 to 1.
• Odds: probability of an event divided by probability of it not happening.
• Log-odds: the natural logarithm of the odds.
• Log loss: the cross-entropy loss used to train the model.
• Threshold: the probability cut-off for predicting the positive class.
• Calibration: agreement between predicted probabilities and observed rates.
• Softmax: the multi-class generalisation of the sigmoid.
• Regularisation strength: how strongly weights are penalised; C in scikit-learn is its inverse.

Summary

Logistic regression turns a weighted sum of features into a probability with the sigmoid function and is trained by minimising log loss. Its roots go back to Verhulst's logistic curve of 1838 and to Berkson's logit of the 1940s, and it remains the standard baseline for classification. It is fast, convex, interpretable through odds ratios and usually reasonably calibrated. Its boundary is linear, so complex patterns need engineered features or a more flexible model. Regularise it, pick thresholds from real costs, check calibration and evaluate with metrics suited to the class balance.

Related

It belongs to ML foundations in my stack, under Machine learning. Others in the same group: