Whisper

Overview

Whisper is a speech recognition model that turns audio into text. It was trained on a very large collection of audio paired with transcripts gathered from the web, an approach called large-scale weak supervision. The result is a single model that handles many languages, accents, background noise levels and recording conditions with no special tuning.

Whisper is an encoder-decoder transformer. The encoder reads a short window of audio, shown as a spectrogram, and the decoder writes text token by token. Special tokens let the same model transcribe, translate speech to English, detect the spoken language and mark timestamps. Because it is robust and easy to run, it is widely used for transcription, subtitles, meeting notes and speech analytics. Those outputs are text and time-stamped segments, which are naturally tabular, and they feed pipelines built on tabular foundation models.

What it is

Whisper is an open, general-purpose automatic speech recognition model, often shortened to ASR. It is released in several sizes, from a very small model that runs on a laptop to large ones that need a GPU. Larger sizes are more accurate and slower.

Main abilities

• Speech-to-text transcription in many languages.
• Language identification.
• Translation of speech from many languages into English text.
• Segment-level timestamps, and with some tooling word-level timestamps.
• Robustness to noise, accents and technical vocabulary better than many older systems.

Everyday analogy

Think of a court reporter who has listened to a huge amount of recorded speech from all over the world. They have heard soft voices, loud rooms and every kind of accent. Given a short clip, they write down what was said, note the language, and can even give an English version. Whisper is that reporter, learned from data.

What it is not

• It is not a speaker identification system. It does not say who spoke, although it can be paired with a tool that does.
• It is not a live streaming engine by default. It works on windows of audio, so real-time use needs extra engineering.
• It is not perfect on rare words, names and overlapping speech.
• It is not a text-to-speech system. It listens, and it does not talk.

Where it came from

Whisper was released in 2022 as an open model with a paper titled "Robust Speech Recognition via Large-Scale Weak Supervision". It built on a decade of progress in speech recognition.

A short history of speech recognition

• Early systems used hidden Markov models and hand-built pronunciation dictionaries.
• Deep neural networks replaced the acoustic models, then end-to-end models arrived.
• Connectionist temporal classification and attention-based sequence models learned to map audio to text directly.
• Self-supervised models such as wav2vec 2.0 learned from unlabelled audio, then fine-tuned on a small amount of labelled data.
• Transformers became the common backbone, as in text and vision. See Transformers.

The weak supervision idea

Most earlier high-quality systems used carefully labelled datasets, which are small and costly. Whisper took the opposite path. It used a very large and diverse set of audio with transcripts that were already available online. These labels are noisy, so they are called weak. The paper's bet was that scale and diversity would beat small and clean, and that the model would then work well without fine-tuning on each new benchmark.

Zero-shot robustness

A key claim was robustness. Models tuned to one benchmark often fail on another with different accents or noise. Whisper was tested without fine-tuning on many datasets and tended to make fewer mistakes across them than models tuned for one. This is the same effect seen in other foundation models: wide pre-training gives broad, if not always best-in-class, performance on any single set.

Multilingual and multitask training

The training mix included many languages, and a task format that covers transcription, translation and other jobs in one model. The special tokens that select the task and language make the model flexible. A single set of weights replaces several separate systems.

Later developments

The community produced faster implementations, quantised versions, versions with better timestamps and models fine-tuned for particular languages. Newer releases of the model family also appeared. Check the model card of the specific release for facts about sizes and languages, since details change.

How it works

The pipeline has a clear sequence. Each step is explained simply.

Step 1: resample and chunk the audio

The audio is converted to a fixed sample rate and mono. It is cut into windows of 30 seconds. Shorter clips are padded with silence, and longer ones are processed as a series of windows. A fixed window keeps the model input size constant.

Step 2: turn sound into a picture

The raw waveform is a long list of air-pressure values. The model does not read it directly. Instead, a short-time Fourier transform turns each small slice of time into a set of frequency strengths, and these are mapped to a perceptual scale called the Mel scale. The result is a log-Mel spectrogram: an image with time on one axis and frequency bands on the other, where brightness shows energy. It is like a musical score that shows which pitches are active at each moment.

Step 3: encode

Small convolutional layers first reduce the time resolution and project the features. Then a stack of transformer encoder blocks, with sinusoidal position encodings, builds a rich representation. Self-attention lets each time frame see the whole window, so the model can use context. A sound that is ambiguous alone becomes clear given the words around it.

Step 4: decode into text

A transformer decoder generates text tokens one at a time. At each step, it attends to the encoder output through cross-attention and to the tokens written so far. It uses learned position embeddings. The vocabulary is a byte-pair encoding that covers many languages.

Step 5: control with special tokens

The decoder starts with a prompt of special tokens.

By changing these tokens, you change the behaviour without changing the model. This is multitask learning through a shared format.

Step 6: decode with search

The model can decode greedily or use beam search. A temperature fallback is often used: if the output looks bad, for example too repetitive or low in confidence, decoding is retried at a higher temperature. Quality checks use compression ratio and average log probability to detect loops and failures.

Step 7: stitch windows

For long audio, the next window starts where the previous text ended, using timestamps to find the boundary. Passing the previous text as context can help consistency but can also spread mistakes forward. Some pipelines turn this off for stability.

Why timestamps work

Timestamps are predicted as special tokens that mark times in steps of a fraction of a second. The decoder interleaves them with text. This gives segment-level alignment without a separate aligner. Word-level timing usually needs extra methods, such as using cross-attention patterns or a forced alignment tool.

Training in brief

1. Collect audio and transcripts at scale, and filter out low-quality or machine-generated transcripts.
2. Cut the audio into 30-second windows paired with the matching text.
3. Format the target as a sequence of special tokens, optional timestamps and text.
4. Train the encoder-decoder with a standard next-token loss.
5. Train several sizes, from small to large.

Model sizes

Distilled and optimised variants exist that keep much of the accuracy with less compute.

The audio front end in more detail

Sound is a stream of pressure values. Speech sits mostly in a band of frequencies that the ear is most sensitive to. The Mel scale spaces frequency bands closely at low frequencies and widely at high ones, matching human hearing. Taking the log of the energy mimics how loudness is perceived. Together, these steps compress the signal into a form where speech sounds form clear patterns, like letters printed in a font.

• Window length is short, a few tens of milliseconds, so each frame captures one tiny slice of a sound.
• Frames overlap, so no sound falls between two frames.
• The number of Mel bands is fixed, so the encoder input has a constant height.

Why the decoder behaves like a language model

The decoder predicts the next token given previous tokens and the audio. This makes it good at fixing ambiguous sounds, since "their" and "there" sound the same but differ by context. The same property explains hallucination: when the audio is unclear, the language side fills in what is likely.

Multitask format at a glance

One sequence format covers several jobs.

1. Transcribe speech in the same language, with or without timestamps.
2. Translate speech from another language into English text.
3. Identify the language, by predicting the language token.
4. Mark windows with no speech.

The model learns that the same audio can have different correct outputs depending on the leading tokens. It is an early, clean example of a task-conditioned foundation model.

Data filtering and balance

Training data came from many sources and languages. Filters aimed to remove transcripts that looked machine-generated and audio-text pairs where the language did not match. Language coverage was not even, with far more English than many other languages, and this explains the gap in accuracy between languages.

Speed-ups used in practice

• Lower-precision weights reduce memory and often speed up GPU inference.
• Batched decoding handles many windows at once.
• Key-value caching in the decoder avoids recomputing past steps.
• Distilled models use a smaller decoder with similar accuracy.
• Voice activity detection avoids spending compute on silence.
• Native ports run on laptops and phones.

Key concepts and terms

A worked example

The example uses the Hugging Face Transformers pipeline. Names are placeholders.

import torch
from transformers import pipeline

device = "cuda:0" if torch.cuda.is_available() else "cpu"
asr = pipeline(
    "automatic-speech-recognition",
    model="example-org/whisper-checkpoint",  # placeholder name
    torch_dtype=torch.float16 if device != "cpu" else torch.float32,
    device=device,
    chunk_length_s=30,
)

result = asr(
    "example_meeting.wav",
    return_timestamps=True,
    generate_kwargs={"language": "en", "task": "transcribe"},
)

print(result["text"])
for seg in result["chunks"]:
    print(seg["timestamp"], seg["text"])

Reading the example

• The pipeline loads audio, makes the spectrogram, runs the model and decodes.
• Setting the language skips language detection and avoids errors on short clips.
• return_timestamps gives a list of segments with start and end times.
• Chunking handles audio longer than 30 seconds.

From transcript to a table

Speech analytics usually needs a table. Each row is a segment, with columns such as start, end, text, speaker and confidence.

import pandas as pd

rows = []
for i, seg in enumerate(result["chunks"]):
    start, end = seg["timestamp"]
    rows.append(
        {
            "call_id": "call_0001",
            "segment": i,
            "start_s": start,
            "end_s": end,
            "text": seg["text"].strip(),
            "word_count": len(seg["text"].split()),
        }
    )

df = pd.DataFrame(rows)
df["duration_s"] = df["end_s"] - df["start_s"]
df["words_per_min"] = df["word_count"] / (df["duration_s"] / 60.0)
print(df.head())

A numeric walk-through of the input size

At a sampling rate of 16,000 samples per second, a 30-second window holds 480,000 samples. With a hop of 160 samples, the spectrogram has 3,000 frames. The encoder's initial convolution with a stride of two halves this to 1,500 positions. The encoder then processes 1,500 vectors. The decoder produces text, typically a few dozen to a couple of hundred tokens for a 30-second window of speech. These numbers describe the standard setup and are shown for intuition.

A numeric walk-through of word error rate

If a reference has 50 words and the transcript has 3 substitutions, 1 deletion and 2 insertions, the word error rate is the total errors divided by the reference length: 6 divided by 50, which is 0.12 or 12 percent. The metric can exceed 100 percent if there are many insertions.

Simple quality checks

def flag_segments(df, min_wpm=20, max_wpm=300):
    bad = (df["words_per_min"] < min_wpm) | (df["words_per_min"] > max_wpm)
    repeated = df["text"].duplicated(keep=False) & (df["word_count"] > 3)
    return df[bad | repeated]

suspicious = flag_segments(df)
print(len(suspicious), "segments need review")

Strengths and limitations

Strengths

• Strong accuracy across many languages and accents with no tuning.
• Robust to background noise and varied recording quality.
• One model for transcription, translation and language detection.
• Open weights and a wide ecosystem of fast and quantised implementations.
• Handles punctuation and casing, so output is readable.
• Several sizes allow a trade-off between speed and accuracy.

Limitations

• It can hallucinate text, especially during silence or music, by producing fluent phrases that were never spoken.
• Repetition loops can occur on long or unclear audio.
• Accuracy is lower for low-resource languages than for common ones.
• Proper nouns, rare terms and acronyms are often misspelled.
• No built-in speaker labels, and overlapping speech is handled poorly.
• Fixed 30-second windows make true real-time use harder.
• Timestamps can drift, and word-level timing needs extra tools.
• Large sizes need a GPU for reasonable speed.
• Accuracy varies by language and the training mix was not balanced.

Privacy and fairness

• Audio often contains personal data. Process it under clear consent and retention rules.
• Error rates differ across accents and dialects, which can disadvantage some speakers.
• Test on your own population before relying on it for decisions.

How it compares with alternatives

How to choose

• Pick Whisper for broad, robust offline transcription and translation to English.
• Pick a streaming model for live captions with low delay.
• Pick a fine-tuned self-supervised model for a narrow domain with in-domain labelled data.
• Pick a hosted service when you need an operational guarantee and accept the trade-offs.
• Combine Whisper with diarisation and voice activity detection for meetings and calls.

Role in tabular foundation models

Whisper handles audio, not tables. Its role in the tabular world is as a data source and as a design precedent.

Speech becomes rows and columns

Transcription turns a call or meeting into a structured table: one row per segment or per utterance, with columns for time, text, language, confidence, speaker and call identifier. Many call-centre, healthcare, media and field-service systems then join these rows with customer, product or case tables. The combined table is the raw material for a model that learns from tables, such as TabPFN or a larger tabular foundation model.

Mixed types and missing values

A speech-derived table mixes numbers (duration, word rate, sentiment score), categories (language, channel, topic label), timestamps and free text. Some fields are missing, for example when speaker labels fail or when audio is silent. A tabular model needs to handle these gaps, and imputation methods fill them in from related columns. Whisper's confidence scores are useful features and also useful flags for missing or unreliable text.

No column names and semantic labelling

When transcripts are joined with other exports, headers are often unclear. A model that infers column meaning from values can label them, for example text columns that hold dates spoken aloud, amounts, or product names. Semantic classification can also tag whole transcripts by domain or sector, such as logistics, retail or healthcare, from vocabulary alone. Whisper itself shows a related lesson: one model with task tokens replaces many specialised systems, much as a foundation model for tables hopes to replace per-dataset models.

Pre-training on synthetic and real data

Whisper's recipe, scale and diversity over small and clean, matches the case for pre-training tabular models on very many tables. The labels in Whisper's data were noisy but plentiful. Tabular pre-training uses synthetic tables, which are perfectly labelled by construction, plus real tables, which are realistic but limited. The shared idea is that broad pre-training gives broad robustness.

In-context learning

For text, a short previous passage can be passed to Whisper as a prompt to bias spelling, such as a list of product names. This is a light form of in-context adaptation. Tabular foundation models take the idea further: example rows placed in the input let the model fit a new task with no weight update. Both reduce the need for per-task training.

Continual learning

Adding a new language or domain by retraining the full model is costly. Teams often fine-tune small adapters, in the spirit of LoRA, so that each domain has its own small add-on. This protects earlier skills and is a pattern shared with continual learning for tabular models. See /model/architecture.

Anomaly detection

Speech analytics tables lend themselves to anomaly checks. Unusual call durations, very low word rates, sudden language switches or segments with low confidence can mark problems in either the audio pipeline or the underlying events. A tabular anomaly detector can score each call from these columns and surface the strange ones for review.

Putting it together

1. Run Whisper on audio and store segments in a table.
2. Add confidence, language and timing columns.
3. Join with business tables by call or case identifier.
4. Impute missing fields and label ambiguous columns.
5. Train or apply a tabular model for classification, regression or anomaly scoring.
6. Review a sample by hand and feed corrections back.

For measured results on tabular tasks, see /model/benchmarks. For research context, see /model/research.

Practical guidance and best practices

Preparing audio

• Use a clean sample rate and mono audio. Convert before processing.
• Trim long silences, or use voice activity detection to cut non-speech.
• Normalise volume, but avoid heavy noise reduction that damages speech.
• Keep a lossless copy of the original.
• Split very long files into logical sections, such as one per call.

Choosing a size and implementation

• Start with a mid-sized model and measure on your own audio.
• Move up in size only if errors matter and hardware allows.
• Use an optimised implementation for speed. Faster runtimes and lower-precision weights can cut time and memory a lot.
• Consider ONNX Runtime or llama.cpp-style native ports for edge deployment, where such ports exist for your model.
• Batch files on a GPU for throughput.

Setting options

Reducing hallucination

1. Remove silence and music with voice activity detection.
2. Set a no-speech threshold and check confidence.
3. Use shorter windows when the audio is bad.
4. Detect repeated phrases and re-run those segments.
5. Review a sample of low-confidence segments by hand.

Adding speakers

Whisper does not label speakers. Pair it with a diarisation tool, then merge by timestamps. Check the merge for words that straddle a speaker change.

Fine-tuning

• Collect in-domain audio with verified transcripts.
• Use low-rank adapters or a small learning rate to avoid forgetting.
• Hold out a test set by speaker, not only by file, to measure real generalisation.
• Measure word error rate and also error rate on key terms.
• Keep a log of data sources and consent.

Operations

• Track word error rate on a regularly refreshed sample.
• Store model version, options and audio hash with every transcript.
• Set retention and deletion rules for audio and text.
• Encrypt audio in transit and at rest.
• Provide a way for users to correct transcripts, and feed corrections into evaluation.

Evaluation tips

• Normalise text before scoring, such as casing and punctuation, so formatting does not count as an error.
• Report results by language, accent, noise level and domain.
• Measure the share of segments with hallucinated text on silence.
• Compare more than one metric, because word error rate hides which words matter.

Common mistakes

• Assuming the transcript is a perfect record. Treat it as a draft for important uses.
• Leaving language detection on for very short clips and getting the wrong language.
• Feeding hours of audio with silence and music, and getting invented text.
• Comparing model sizes on clean audio only, then deploying on noisy calls.
• Ignoring proper nouns and domain terms. Use a prompt or post-correction list.
• Scoring without text normalisation, which inflates error rates.
• Splitting audio mid-word and creating errors at boundaries.
• Using the translate task when the same-language transcript was wanted.
• Trusting timestamps to the word without checking drift.
• Storing audio and transcripts without a retention and consent plan.
• Training on test-speaker audio and overstating improvement.

Troubleshooting checklist

1. Wrong language? Set the language explicitly.
2. Repeating text? Raise the temperature fallback, shorten the window or cut silence.
3. Missing words at the start of a window? Add a small overlap between chunks.
4. Slow on CPU? Use a smaller size, a faster runtime or lower precision.
5. Out of memory on GPU? Use half precision, a smaller size or smaller batches.
6. Names misspelt? Provide an initial prompt or apply a correction dictionary.

Typical applications

Designing a speech analytics table

A good schema makes later modelling easier.

• One row per segment, with a stable call identifier.
• Start and end times in seconds.
• The text, the detected language and a confidence value.
• A speaker label if diarisation is used.
• A flag for suspected hallucination or silence.
• The model name, version and options used.

With such a table, you can aggregate to one row per call: total duration, share of silence, words per minute, number of speaker turns and topic counts. That call-level table is ready for a tabular model.

Cost and capacity planning

• Estimate audio hours per day, then multiply by the processing speed of your chosen setup.
• Faster-than-real-time processing on a GPU is common for mid-sized models.
• Plan for peak load, since audio often arrives in bursts.
• Keep a queue with retries and dead-letter handling.

Frequently asked questions

What is Whisper?

Whisper is an open speech recognition model that turns audio into text. It is an encoder-decoder transformer trained on a very large and varied set of audio with transcripts, and it can also detect languages, translate speech into English and produce timestamps.

What does weak supervision mean here?

Weak supervision means the training labels are plentiful but not perfectly clean, such as transcripts found alongside web audio. The bet is that the huge scale and diversity of the data outweigh the noise in the labels, and that the model becomes robust across many conditions.

Why does it use 30-second windows?

A fixed window keeps the model input a constant size, which simplifies the encoder. Longer audio is processed as a series of windows, and the text is stitched together using timestamps. The downside is that real-time use needs extra engineering.

Can Whisper identify who is speaking?

No. It transcribes what was said, not who said it. To label speakers, you combine it with a diarisation tool and merge the results by time. Check the boundaries carefully, since speaker changes can fall inside a segment.

Why does Whisper sometimes write text for silence?

The decoder is a language model, so given an unclear or empty window it may produce a fluent phrase that is plausible but was not spoken. Voice activity detection, a no-speech threshold and confidence checks reduce this problem.

How accurate is Whisper?

Accuracy depends on the language, the model size and the audio quality. It is often strong on common languages with clear audio, and weaker on low-resource languages, heavy accents, overlapping speech and rare terms. Measure it on your own audio before relying on it.

Can it run in real time?

The standard setup processes windows of audio, so it is not a streaming model by nature. Systems that approximate real time use short chunks, voice activity detection and fast implementations, with some trade-offs in accuracy and delay.

Is Whisper useful for tabular data work?

Yes, as a source. Transcripts and their metadata form tables with one row per segment or call. Those tables can be joined with other data and used by tabular models for classification, imputation and anomaly detection. Whisper itself does not process tables.

Can I fine-tune it for my domain?

Yes. Collect in-domain audio with verified transcripts, then fine-tune with a small learning rate or low-rank adapters to avoid forgetting. Hold out test speakers and measure both overall error and the error on key domain terms.

Is my audio sent anywhere when I run it?

When you run an open implementation locally, the audio stays on your machine. Hosted services work differently, so read their terms. In either case, follow your own consent, retention and security rules.

Glossary

• ASR: Automatic speech recognition.
• Beam search: A decoding method that tracks several candidate outputs.
• BPE: Byte-pair encoding, a way to split text into tokens.
• Cross-attention: Attention from the decoder to the encoder output.
• Diarisation: Labelling who spoke when.
• Encoder-decoder: A model with a reader network and a writer network.
• Fine-tuning: Continued training on a smaller dataset.
• Hallucination: Output that is fluent but not supported by the input.
• Mel scale: A frequency scale that follows human hearing.
• Sampling rate: The number of audio samples per second.
• Spectrogram: A time-frequency picture of sound.
• Timestamp token: A special token that marks a time in the audio.
• VAD: Voice activity detection.
• Weak supervision: Learning from large amounts of noisy labels.
• Word error rate: The share of words that differ from a reference.

Summary

Whisper is a robust, multilingual speech recognition model built as an encoder-decoder transformer and trained on a very large collection of web audio with transcripts. It converts audio into a log-Mel spectrogram, encodes it, and decodes text with special tokens that choose the language, the task and timestamps.

It transcribes, translates to English and aligns text at the segment level, but it can hallucinate during silence, has no speaker labels and works on fixed windows. For tabular work, it is a source of structured speech-derived tables and a precedent for broad pre-training and task tokens. Those tables then feed imputation, semantic labelling and anomaly detection with models such as TabPFN. Related reading includes Transformers, LoRA, Hugging Face and /model/benchmarks.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: