Naive Bayes
Naive Bayes classifies by applying Bayes' theorem with the simplifying assumption that features are independent. Learn why it works well for text and spam filtering.
Overview
Naive Bayes is a family of classifiers built on Bayes' theorem, the rule for updating a belief when new evidence arrives. Given an example, it asks: how likely is each class to have produced these features, and how common is each class to begin with? It combines the two and picks the class with the highest result.
The "naive" part is a simplifying assumption: once you know the class, each feature is treated as independent of the others. In real data this is almost never true. The word "free" and the word "winner" tend to appear together in spam. Yet the assumption makes the method extremely fast and needs very little data, and in practice the ranking of classes is often right even when the probabilities themselves are off.
An everyday picture is a spam filter reading an email. It notices "free", "winner" and "click now". Each word on its own makes spam a little more likely. The filter multiplies the evidence from each word, as if each were an independent clue, and concludes that the message is probably spam.
What it is
| Variant | Feature type | Typical data |
|---|---|---|
| Multinomial | Counts | Word counts in documents |
| Bernoulli | Binary presence or absence | Whether each word appears |
| Gaussian | Continuous values, assumed normal per class | Sensor readings, measurements |
| Complement | Counts, with class-complement statistics | Imbalanced text classes |
| Categorical | Discrete categories | Survey answers, codes |
All variants share three ingredients:
- Prior: how common each class is before looking at the features.
- Likelihood: how probable the observed features are within each class, estimated feature by feature.
- Posterior: the updated probability of each class after seeing the features, proportional to prior times likelihood.
Because each feature's likelihood is estimated separately, training amounts to counting. For text, that means counting how often each word appears in each class. There are no iterations and no gradients.
Where it came from
Bayes' theorem is named after Thomas Bayes, whose essay on the subject was published after his death, in 1763. Pierre-Simon Laplace developed the ideas further and used them widely. The independence simplification that makes classification practical was studied in pattern recognition and information retrieval from the early 1960s, where it was used to sort documents by subject.
The method's best-known success came with email. Bayesian approaches to filtering mail were in use by the mid-1990s, and in 1998 Mehran Sahami and colleagues published the first scholarly paper on Bayesian spam filtering. In 2002 Paul Graham's essay "A Plan for Spam" popularised the idea among programmers, and naive Bayes filters spread into many mail systems. Later research explained why the method works well despite its false assumption, and proposed fixes such as Complement Naive Bayes for imbalanced text. Today it is a standard baseline in scikit-learn and in text-classification courses.
How it works
Step 1: estimate the priors
Count the share of training examples in each class. If 30 per cent of emails are spam, the spam prior is 0.3.
Step 2: estimate the likelihoods
For multinomial naive Bayes, count how often each word appears in each class and divide by the total number of words in that class. For Gaussian naive Bayes, compute the mean and variance of each feature within each class.
Step 3: smooth the counts
If a word never appeared in spam during training, its estimated probability is zero, and one zero would wipe out the whole product. Laplace or additive smoothing adds a small count, often 1, to every word so no probability is exactly zero. In scikit-learn this is the alpha parameter.
Step 4: score a new example
For each class, multiply the prior by the likelihood of each observed feature. Multiplying many small numbers underflows, so implementations add logarithms instead of multiplying probabilities.
Step 5: pick the class
Choose the class with the highest score. Normalising the scores gives posterior probabilities, but treat them with caution, because the independence assumption tends to push them towards 0 or 1.
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
spam = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0))
spam.fit(train_texts, train_labels)
Key concepts and terms
| Term | Meaning |
|---|---|
| Bayes' theorem | Posterior is proportional to likelihood times prior |
| Prior | Probability of a class before seeing the features |
| Likelihood | Probability of the features given a class |
| Posterior | Probability of a class after seeing the features |
| Conditional independence | Features treated as unrelated once the class is known |
| Laplace smoothing | Adding a small count so no probability is zero |
| Log probabilities | Adding logs instead of multiplying to avoid underflow |
| Generative classifier | Models how each class produces data |
| Discriminative classifier | Models the boundary between classes directly |
| Bag of words | Representing text by word counts, ignoring order |
A worked example
This scenario is illustrative. A small company wants to route incoming support emails into three queues: billing, technical and account. It has 6,000 labelled past emails: 2,400 billing, 2,700 technical and 900 account.
The team converts each email to word counts, removes very rare words, and trains multinomial naive Bayes with alpha = 1. Training takes under a second. Consider a new email: "I was charged twice on my invoice this month." The words "charged" and "invoice" are many times more frequent in billing emails than in the other queues, so the billing score rises sharply. "Month" is common everywhere and barely matters. The model routes the email to billing.
On a held-out set of 1,200 emails, accuracy is 89 per cent. The account queue, the smallest class, has the lowest recall at 74 per cent. Switching to Complement Naive Bayes raises account recall to 81 per cent with little loss elsewhere. The team keeps this as the production baseline and later compares it with logistic regression on TF-IDF features and with a classifier built on embeddings.
Strengths and limitations
Strengths:
- Extremely fast to train and to predict.
- Works with small training sets.
- Scales to very many features, such as large vocabularies.
- Easy to update incrementally as new labelled data arrive.
- Simple to understand and debug through word or feature counts.
Limitations:
- The independence assumption ignores correlations and word order.
- Predicted probabilities are often poorly calibrated, too close to 0 or 1.
- Usually less accurate than discriminative models when data are plentiful.
- Gaussian naive Bayes assumes roughly normal features within each class.
- Cannot learn interactions between features.
How it compares with alternatives
| Method | Speed | Accuracy on text | Probability quality |
|---|---|---|---|
| Naive Bayes | Very fast | Good baseline | Often poor |
| Logistic regression | Fast | Often better with enough data | Usually good |
| Support vector machines | Medium | Strong on sparse text | Not native |
| BERT-style models | Slow | Much stronger on meaning | Good after calibration |
| k-nearest neighbors | Slow at prediction | Moderate | Coarse |
Practical guidance and best practices
- Match the variant to the data: multinomial for counts, Bernoulli for presence, Gaussian for continuous values.
- Always use smoothing, and tune alpha with cross-validation.
- Try Complement Naive Bayes when text classes are imbalanced.
- Remove extremely rare and extremely common tokens to reduce noise.
- Use it as a fast first baseline before heavier models.
- Calibrate probabilities if you need to use them as real likelihoods.
- Retrain or update counts regularly as language and topics drift.
Common mistakes
- Forgetting smoothing, so unseen words produce zero probabilities.
- Feeding negative values, such as standardised features, to multinomial naive Bayes, which expects counts.
- Trusting the raw posterior probabilities for decisions that need calibrated risk.
- Using Gaussian naive Bayes on heavily skewed or multi-modal features without transformation.
- Concluding that because features are correlated, the method cannot work, without testing it.
Frequently asked questions
Why does naive Bayes work if its assumption is wrong?
Classification only needs the right class to get the highest score, not exact probabilities. Errors from ignoring correlations often affect all classes in similar ways, so the ranking survives. Research has shown it can be competitive even when features are clearly dependent.
Is naive Bayes still used?
Yes, as a fast baseline, in resource-constrained systems and where training data are small. For high-stakes text understanding, transformer-based models usually perform better, but naive Bayes remains a useful benchmark and a sensible first model.
What is the alpha parameter?
It is the smoothing amount added to every feature count. A value of 1 is classic Laplace smoothing. Smaller values trust the observed counts more; larger values pull estimates towards uniform. Choose it by validation.
Can naive Bayes handle continuous features?
Yes. Gaussian naive Bayes models each feature within each class as a normal distribution. If features are far from normal, transforming them or discretising them into bins can help.
Is it a generative or discriminative model?
It is generative: it models how each class produces features and then uses Bayes' theorem to classify. Logistic regression is its discriminative counterpart, modelling the class probability directly from the features.
Glossary
- Bayes' theorem: the rule for updating probabilities with new evidence.
- Prior probability: belief in a class before seeing the data.
- Posterior probability: belief in a class after seeing the data.
- Likelihood: probability of the observed features under a class.
- Smoothing: adding pseudo-counts to avoid zero probabilities.
- Token: a unit of text, such as a word, used as a feature.
- Underflow: numbers too small for the computer to represent, avoided with logarithms.
- Spam filter: a classifier that separates unwanted email from wanted email.
Summary
Naive Bayes applies Bayes' theorem with the assumption that features are independent given the class, which turns training into simple counting. It became famous through spam filtering, with the first scholarly paper in 1998 and wide adoption after 2002. It is fast, needs little data and handles huge vocabularies, but its probabilities are poorly calibrated and it cannot model feature interactions. Use the right variant, always smooth, treat it as a strong first baseline, and compare it with discriminative and embedding-based models.
Related
It belongs to ML foundations in my stack, under Machine learning. Others in the same group: