Gemini 4

Overview

Gemini 4 is the fourth generation of Google's Gemini family of large models. It is natively multimodal, which means one model is trained to handle text, images, audio and video together rather than bolting separate systems side by side. It is aimed at long-context and multimodal reasoning, and at tasks that mix documents, images and code.

For tabular data, Gemini 4 is useful in two ways. It can read tables that appear as text, as spreadsheets exported to text, or as images of tables inside scanned documents. It can also write queries and explain results. It is not a model that learns from millions of raw rows. For that, models pre-trained on tables, described at /model/research, are the better tool. The two are partners, not rivals.

What it is

Gemini 4 is a transformer-based model family. "Natively multimodal" means that during training the model sees several kinds of data in one network, so it can relate a chart in an image to a paragraph of text and to the code that drew it.

Like other recent families, it comes in sizes. Larger members give stronger reasoning. Smaller members, such as the Flash line described at Gemini 3.8 Flash, favor speed and cost. Exact internals and training data are not fully public, so this article stays with general, well-established ideas.

Typical capabilities

• Reading and reasoning over long documents.
• Understanding images, charts, diagrams and screenshots.
• Handling audio and video input in supported products.
• Writing and reviewing code.
• Calling tools and returning structured output.
• Summarizing, translating, extracting and classifying.

Typical limits

• It can produce fluent but false statements.
• Exact arithmetic over long columns needs tools.
• Very long inputs raise cost and latency.
• Reading dense tables from images can drop or swap cells.
• Hosted use sends your data to a provider.

Why multimodality matters for data work

Business data often arrives as files that are not clean tables: a PDF invoice, a photo of a form, a slide with a chart, a screenshot of a dashboard. A multimodal model can read these directly and turn them into rows. That makes it a front door to tabular pipelines.

Where it came from

Google's Gemini family was introduced as a successor to earlier language models, with the goal of one model that handles many data types. It builds on the transformer from the 2017 paper "Attention Is All You Need" and on years of work in language, vision and speech research. Earlier generations raised context length, added tool use and improved reasoning. The family then split into tiers so one name covers both large and fast models.

Ideas behind the line

• Joint training across modalities rather than separate encoders only.
• Long context, so whole reports or codebases fit in one request.
• Tool use, so the model can search, run code and call APIs.
• Reasoning modes that spend extra compute on hard problems.
• Tiers that match different cost and speed needs.

For other families that share these trends, see GPT-6 and Claude Opus 5.5.

How it works

The steps below describe the general design of multimodal transformers. Each has an everyday picture.

Step 1: Turn every input into tokens

Text is split into tokens. Images are cut into small patches and each patch becomes a token-like vector. Audio is cut into short frames. Video is a sequence of image patches over time. After this step, everything is a sequence of vectors.

Picture a mail room that opens every kind of package and puts the contents into identical trays so one team can handle them.

Step 2: Add position and type information

The model must know where each piece sits. Text has order. Images have rows and columns of patches. Video has time. Position signals are added to each vector so the layout is not lost.

Step 3: Attend across everything

Transformer layers let every vector look at every other vector in the context. A word in a caption can attend to the patch of the image it describes. A question can attend to a cell in a table image.

Picture a detective with a corkboard, linking a photo, a note and a receipt with string.

Step 4: Generate output

The model predicts the next output token, one at a time. Output is usually text, including code and structured data. Some products also produce images or audio through related components.

Step 5: Pre-training

The model learns from very large mixed collections of data by predicting missing or next pieces. This gives broad knowledge of language, code and visual patterns.

Step 6: Post-training

Fine-tuning on good examples, preference learning and reinforcement learning make the model follow instructions and behave safely. See SFT and RLHF.

Step 7: Long context

Handling very long inputs needs efficient attention and memory use. Techniques such as FlashAttention reduce cost. Even so, longer inputs cost more and may be recalled less reliably in the middle.

Step 8: Tools and agents

At run time the model can call tools. Your code runs them and returns results. Chains of such calls form agents. See Function calling and ReAct loop.

Key concepts and terms

A worked example

This example turns a scanned invoice into rows, then checks them with code. It shows how a multimodal model feeds a tabular pipeline.

The task

An image of an invoice has a header and a table of line items. The goal is a clean list with item, quantity and unit_price, plus a total check.

The request

import json

SCHEMA = {
    "type": "object",
    "properties": {
        "lines": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "item": {"type": "string"},
                    "quantity": {"type": "number"},
                    "unit_price": {"type": "number"},
                },
                "required": ["item", "quantity", "unit_price"],
            },
        },
        "stated_total": {"type": "number"},
    },
    "required": ["lines", "stated_total"],
}

def extract(client, image_bytes):
    reply = client.generate(
        model="gemini-4",
        input=[
            {"image": image_bytes},
            {"text": "Extract the line items and the stated total."},
        ],
        response_schema=SCHEMA,
        temperature=0,
    )
    return json.loads(reply.text)

The check

def verify(data, tol=0.01):
    computed = sum(l["quantity"] * l["unit_price"] for l in data["lines"])
    return abs(computed - data["stated_total"]) <= tol, computed

data = extract(client, open("invoice.png", "rb").read())
ok, computed = verify(data)
print("match" if ok else f"mismatch: computed {computed}")

Reading the numbers

Say the lines are 2 items at 10.00, and 1 item at 5.50. The computed total is 2 x 10.00 + 1 x 5.50 = 25.50. If the invoice states 25.50, the extraction is consistent. If it states 28.50, then a quantity, a price or the total was misread, and the row goes to a human. This cross-check is cheap and catches the most common image-reading slips.

A second example: asking questions of a chart

[image: bar chart of monthly orders]
Question: Which month had the highest orders, and roughly how many?
Rule: if the axis is unreadable, say "unclear" rather than guessing.

The rule line matters. Without it, a model may invent a value for a blurry axis.

Where a tabular model comes in

After many invoices are turned into rows, a table exists. A tabular model such as TabICL can then predict late payment from vendor, amount and terms. Gemini 4 produced the table, and the tabular model learns from it.

A third example: a batch pipeline with a review queue

The sketch below processes many scans, validates each, and sends mismatches to a queue.

from pathlib import Path

accepted, review = [], []

for path in Path("scans").glob("*.png"):
    data = extract(client, path.read_bytes())
    ok, computed = verify(data)
    record = {"file": path.name, "data": data, "computed": computed}
    (accepted if ok else review).append(record)

print(f"accepted={len(accepted)} review={len(review)}")

Measuring the pipeline

If 1,000 scans produce 940 accepted and 60 reviewed, the review rate is 6 percent. Check a random slice of the 940 by hand to estimate the escape rate. If 3 of 100 sampled accepted rows are wrong, the estimate is 3 percent, and you may add a second check such as a vendor-name lookup.

Everyday analogy

Think of a mail clerk who sorts letters quickly. Most go to the right slot. A supervisor spot-checks a few bundles, and anything with a smudged address goes to a separate tray. The clerk is the model, the supervisor is the sampling check, and the tray is the review queue.

Strengths and limitations

Strengths

• Handles documents, images and text in one request.
• Long context suits large reports and codebases.
• Good fit for extraction from messy files.
• Tool use and structured output support pipelines.
• Tiers let teams control cost.

Limitations

• Dense tables in images can be misread.
• Long inputs cost more and may be recalled unevenly.
• Numbers need verification by code.
• Behavior can change between versions.
• Privacy depends on how data is sent and stored.

Table-focused view

How it compares with alternatives

Comparison should use your own tasks. A model that reads scans well may be average at code, and the reverse is also common.

Role in tabular foundation models

A tabular foundation model is pre-trained on many tables so it can predict, impute or flag anomalies on a new table with little or no training. Its structure is described at /model/architecture, and its results at /model/benchmarks. Gemini 4 is a different kind of model, but it connects to this field in practical ways.

The front door: from files to rows

Most tabular models need clean rows. Real data often lives in PDFs, images and forms. A multimodal model can extract fields into a schema, which creates the table a tabular model needs. This is probably the strongest link: Gemini 4 builds tables, and a tabular foundation model learns from them.

Row and column structure

A tabular foundation model keeps the grid and attends across rows and columns; see Axial attention. Gemini 4 treats a table as a sequence of tokens or as image patches. It can recover structure, but it does so each time, and it can slip on wide tables.

Mixed types

Tables mix numbers, categories, dates and text. A tabular model encodes each type suitably. A multimodal language model sees them as tokens. For free-text columns, however, Gemini 4 is strong: it can summarize, classify or embed the text into features that a tabular model can use. See Contrastive learning for how embeddings are trained.

Missing values

Tabular models treat missing cells as something to predict; see Masked cell modeling and Imputation networks. Gemini 4 can help by reading context, such as filling a missing country from a city name, but for numeric gaps a statistical or tabular model is more faithful to the data.

No column names

A table with names like c1 gives a language model little to work with. A tabular foundation model is trained to cope by using value patterns. Gemini 4 can propose readable names if given sample values, and a person should confirm them.

Pre-training on synthetic or real tables

Tabular foundation models are often pre-trained on large sets of synthetic tables, sometimes mixed with real ones, to learn a general fitting skill. See Synthetic data generation. Gemini 4's pre-training is on mixed media. Its table skill is a by-product.

In-context learning

Gemini 4 learns formats from a few examples in the prompt. A tabular foundation model reads the entire training table as context and predicts test rows in one pass. See TabPFN.

Continual learning

Hosted models are not updated per user. Freshness comes from retrieval; see Agentic RAG. Tabular models can be updated with replay or regularization; see Experience replay.

Anomaly detection

A tabular model scores row outliers from the numbers; see Anomaly detection. Gemini 4 can review flagged rows alongside the source document and say whether the document explains the oddity.

Sector and semantic classification

When a table comes with a data dictionary, report or screenshot, Gemini 4 can read it and infer the domain. For raw tables without such help, a model with Semantic sector heads works from values. Using both and flagging disagreement is a good control.

A combined pipeline

Practical guidance and best practices

Inputs

• Use text tables when you can. Use images only when the source is an image.
• Send images at a clear resolution and crop to the area of interest.
• Ask for a schema, not free text, when rows are the goal.
• Tell the model what to do when a cell is unreadable.

Validation

• Recompute totals and ratios in code.
• Check types: numbers parse, dates parse, categories are known.
• Route low-confidence or mismatched rows to a person.
• Sample outputs weekly and read them.

Long context

• Put the question after the material, and repeat key constraints at the end.
• Use retrieval to cut input to what matters.
• Test recall on facts placed in the middle of long inputs.
• Watch cost per request as inputs grow.

Cost and speed

• Use a lighter tier for easy extraction. See Gemini 3.8 Flash.
• Batch requests where latency allows.
• Cache repeated instructions if the service supports it.
• Set budgets. See Cost and latency budgets.

Privacy and safety

• Mask personal fields before sending files. See Data tokenization.
• Treat text inside documents as untrusted. See Prompt injection defense.
• Apply Guardrails to outputs.
• Log inputs and outputs under your retention rules.

Evaluation

• Build a labeled set of real documents.
• Score field-level accuracy, not only whole-document accuracy.
• Track the rate of schema failures and the rate of mismatched totals.
• Re-run on every model or prompt change.

Designing prompts for multimodal inputs

• Name each input: "Image 1 is the invoice. Image 2 is the purchase order."
• Say what to extract and what to ignore.
• Give the output schema and one short example.
• State the language and number format, such as comma as decimal separator.
• Ask for a confidence flag per field when your pipeline can use it.

Example of a number-format rule

Numbers in this document use a comma as the decimal separator and a dot as
the thousands separator. Return numbers using a dot as the decimal separator
and no thousands separator. If a value is unreadable, return null.

Roll-out checklist

• Start with one document type.
• Build a labeled set of at least a few hundred documents.
• Set pass thresholds for field accuracy and escape rate.
• Run in shadow mode, where people still do the work and you compare.
• Move to review-by-exception once the thresholds hold.
• Keep sampling after launch, since new templates appear over time.

Operating notes

• Keep prompts and schemas in version control.
• Store the model version with every extracted record.
• Plan for template drift, where a vendor changes its invoice layout.
• Budget for re-labeling when drift appears.
• Keep a path for people to correct records, and feed corrections back into the test set.

Handling long documents step by step

1. Split the file into logical parts, such as chapters or pages.
2. Ask the model to summarize each part into a short note with page references.
3. Combine the notes and ask the final question over the notes.
4. Return to the original pages for any answer that needs exact wording.

This map-then-reduce pattern keeps each request small and gives you an audit trail. It is like reading a book by making index cards first, then answering from the cards while keeping the book open for quotes.

Choosing between one big request and many small ones

Sizing a pilot

• Pick 200 documents across the main templates.
• Label key fields by hand once.
• Run the pipeline and compute the metrics above.
• Decide which fields can be automated and which need review.
• Present the results with the cost per document, so owners can decide on scale-up.

Reporting results

Share field-level accuracy in a table, list the five most common errors with examples, and state the review rate and escape rate. Honest reporting builds trust and shows where the next prompt change should aim.

Common mistakes

• Trusting numbers read from images without a cross-check.
• Sending a whole document when one page would do.
• Asking for free text, then parsing it with fragile rules.
• Leaving out a rule for unreadable cells, so the model guesses.
• Using a multimodal model to predict on a large table instead of a trained predictor.
• Forgetting that a chart image gives approximate values only.
• Not masking personal data in scans.
• Comparing models on a single sample document.
• Ignoring version changes that shift output format.
• Skipping human review for high-impact fields such as amounts and dates.

Frequently asked questions

What is Gemini 4?

Gemini 4 is Google's fourth-generation family of large models. It is natively multimodal, so it reads text, images, audio and video in one system, and it is used for long-context and mixed-media reasoning. Its internals are not fully public, so judge it by testing on your own tasks.

What does natively multimodal mean?

It means the model was trained on several data types together in one network, instead of joining separate text and image systems after the fact. The practical effect is that it can relate parts of an image to parts of a text, such as a chart to its caption.

Can Gemini 4 read tables in PDFs and images?

Yes, it can extract tables from many documents and images, but accuracy varies with image quality and table density. Ask for a schema, add a rule for unreadable cells and verify totals with code before trusting the rows.

Is it a tabular foundation model?

No. It is a general multimodal model. A tabular foundation model is pre-trained on tables and works on the grid directly. Gemini 4 often builds the tables that such a model then learns from.

How is it different from Gemini 3.8 Flash?

Flash is a lighter, faster and cheaper member aimed at high-volume work, while Gemini 4 is the larger flagship family aimed at harder reasoning and mixed-media tasks. Many teams route easy requests to Flash and hard ones to the larger model.

How long an input can it take?

It supports long contexts, but the exact limit depends on the product and tier, so check the current documentation. Long inputs cost more and may be recalled unevenly, so retrieval and trimming are still wise.

Can it do exact math on a spreadsheet?

It can write code to do so, and that is the safer route. Its own arithmetic over long columns can slip, so let a database or script compute totals and let the model explain them.

How do I keep private data safe?

Mask or tokenize sensitive fields before sending, send only what the task needs, review the provider's terms on storage and retention, and keep very sensitive work on systems you host yourself.

Should I fine-tune it on my documents?

Start with a good schema, clear instructions and a few examples. Fine-tuning may help for a fixed document type at high volume, but it needs a labeled set and a plan to re-test when versions change.

How does it compare with GPT-6?

Both are broad, strong families with different emphases, and results depend on the task. Test them on your own documents and queries, scoring quality, cost and latency together. See GPT-6 for the other side of the comparison.

Glossary

• Context window: the maximum input size per request.
• Grounding: tying answers to supplied sources.
• Hallucination: fluent but false output.
• Multimodal: handling more than one data type.
• OCR: reading text from images.
• Patch: a small square cut from an image.
• Retrieval: fetching relevant passages for the prompt.
• Schema: a description of the required output fields.
• Structured output: output that matches a schema.
• Temperature: a randomness setting.
• Tier: a size and cost level in a family.
• Token: a small unit of input or output.
• Tool call: a structured request for external code.
• Tabular foundation model: a model pre-trained on tables.

Summary

Gemini 4 is Google's fourth-generation, natively multimodal model family. It reads text, images, audio and video, handles long inputs and calls tools, which makes it a strong front door for turning documents and screenshots into tables.

It is not a model that learns from raw rows. Use it to extract, explain and route, use code to verify numbers, and use tabular foundation models, covered at /model/research, /model/architecture and /model/benchmarks, to predict, impute and flag anomalies. Test on your own data, mask private fields and re-check after each version change.

Related

It belongs to Frontier models in my stack, under LLMs, retrieval, and evaluation. Others in the same group: