Computer use

Overview

Computer use is the ability of an AI model to operate a computer the way a person does. It looks at the screen, moves the pointer, clicks, types, presses shortcuts and scrolls. It does not need an API for the application in front of it. If a person can do the task through the graphical interface, a computer-use agent can, in principle, attempt it too.

Think of helping a colleague over a screen share. You watch their screen and say "click the blue button at the top right, then type the order number into the search box". Computer use puts the model in both seats at once: it sees the screen through screenshots and it moves the mouse and keyboard itself.

This ability fills a real gap. Many business systems are old desktop programs, internal tools or web apps with no usable API. Computer use can reach them. The price is speed, cost and risk: every step needs a screenshot and a model call, screens change, and anything visible on the screen can try to steer the agent. This page explains how it works and how to use it responsibly.

What it is

A computer-use system has three parts: a model that can read images and choose actions, a small set of actions it may request, and an environment that performs those actions and returns what happened.

Some providers also pair the computer tool with a shell tool and a text-editing tool, so the agent can run commands or edit files directly when that is easier than clicking.

Where it came from

Research on agents that operate graphical interfaces is older than today's large models, but it became practical when vision-capable language models could read screens reliably. In April 2024 researchers released OSWorld, a benchmark of 369 tasks in real operating systems and applications, which became a common way to measure progress. Early scores were far below human performance.

Anthropic released computer use as a public beta in October 2024 with an upgraded Claude 3.5 Sonnet, the first time a frontier model provider offered the capability through a public API. It shipped a reference implementation that runs a desktop inside a Docker container with a virtual display. In January 2025 OpenAI introduced Operator, powered by a model it called the Computer-Using Agent, and later folded that ability into its ChatGPT agent. Google released a Gemini model specialised for computer use in October 2025. Through 2025 and 2026, model versions added richer actions, such as more click types, drag and zoom, and scores on benchmarks like OSWorld rose sharply, although real-world reliability still varies by task.

How it works

The perception and action loop

1. The harness gives the model a goal and a description of the display size.
2. The model asks for a screenshot.
3. The environment captures the screen and returns the image.
4. The model reads it, decides the next action and returns it as a structured tool call, for example a left click at given coordinates.
5. The environment performs the action, waits briefly and returns a new screenshot.
6. Steps 3 to 5 repeat until the goal is reached, the agent asks for help, or a limit is hit.

Grounding

The hard part is grounding: turning "the Save button" into exact pixel coordinates. Models are trained on many screenshots to locate interface elements. Accuracy drops with very high resolutions, tiny text, crowded layouts and unusual themes. Images are often scaled down before the model sees them, so the harness must scale coordinates back to the real screen. A modest, fixed resolution usually works best.

The environment

Most deployments run a virtual desktop in a container or virtual machine: a virtual display server, a lightweight window manager and the needed applications. This keeps the agent away from the user's own computer, files and accounts. Agents that do operate a person's own machine need tighter approval rules.

Control and safety

The harness, not the model, decides what is allowed. It can block certain actions, require human confirmation before submitting forms or making payments, cap the number of steps, and save every screenshot and action for review. See sandboxing and human in the loop.

Key concepts and terms

A worked example

This is an illustrative scenario. An operations team must copy shipment statuses for about 60 orders each morning from a legacy desktop logistics program, which has no API, into a shared spreadsheet.

1. The harness starts a fresh virtual machine with the logistics client and a spreadsheet application installed. It has a dedicated read-only account, no email and no browser access beyond the spreadsheet service.
2. The agent opens the logistics client, enters the order number in the search field and presses Enter.
3. It reads the status and the expected delivery date from the result screen, using zoom on the small status text.
4. It switches to the spreadsheet, finds the row for that order and types the two values.
5. Each order takes about eight steps. With roughly three to five seconds per step, the run takes 25 to 40 minutes, which is acceptable for an overnight job.
6. When an unexpected dialog appears, such as a licence warning, the agent stops and sends a message to the on-call person with the last screenshot, rather than guessing.
7. At the end, the harness stores the trajectory and a short report: 58 orders updated, two skipped because the order was not found.

The team keeps a weekly check of ten random rows against the source system.

Strengths and limitations

Strengths:

• Works with almost any application, including legacy and internal tools.
• Needs no integration project or API access.
• Handles variation better than fixed scripts, such as a moved button.
• Can combine several applications in one task.

Limitations:

• Slow and costly, because each step needs an image and a model call.
• Less reliable than APIs; small layout changes or pop-ups can derail a run.
• Vulnerable to prompt injection from anything visible on screen.
• Dangerous on a machine with real credentials, payment methods or personal data.
• Harder to test and audit than code, because behaviour is not fully deterministic.

How it compares with alternatives

Practical guidance and best practices

1. Use APIs, function calling or MCP tools first; reach for computer use only where they do not exist.
2. Run every agent in an isolated virtual machine or container, never on a personal computer with saved logins.
3. Give it dedicated accounts with the least access the task needs.
4. Require confirmation before purchases, sending messages, deleting records, accepting terms or signing in.
5. Treat all on-screen text as untrusted data; see prompt injection defence.
6. Use a fixed, modest screen resolution and a clean, consistent desktop.
7. Cap steps and time per run, and stop on unexpected screens instead of improvising.
8. Log screenshots and actions for every run, and review samples regularly.
9. Measure success rate on a fixed set of tasks before trusting it with real work.

Common mistakes

• Using computer use for a system that has a perfectly good API.
• Letting the agent run on a laptop where the browser is signed in to email and banking.
• Assuming one successful demo means the task is reliable.
• Using a very high screen resolution, which hurts click accuracy.
• Allowing the agent to click through warnings and dialogs it does not understand.
• Keeping no record of what the agent saw and did.

Frequently asked questions

What is computer use in AI?

It is the ability of a model to operate a computer through its graphical interface. The model receives screenshots, decides what to do, and requests mouse and keyboard actions that a harness performs, repeating the cycle until the task is done.

Is computer use the same as RPA?

No. Classic robotic process automation follows recorded or scripted steps and fails when the screen differs from what was expected. Computer use relies on a model that reads the screen and decides each step, which makes it more flexible but also slower and less predictable.

Why is computer use slow?

Every step involves capturing a screenshot, sending it to a model, waiting for the decision and performing the action. A task that a person finishes with twenty clicks may take twenty or more model calls, each with an image.

Is computer use safe?

It can be made reasonably safe with isolation, limited accounts, confirmation gates for consequential actions, step limits and logging. Without these, an agent can make costly mistakes or follow instructions planted in on-screen content.

Which models support computer use?

Several frontier providers offer it, including Anthropic, OpenAI and Google, each with its own tool definitions and action sets. Supported actions differ between model versions, so check the provider's current documentation before building.

How is success measured?

On benchmarks, by the share of tasks completed correctly in real environments, as in OSWorld. In production, by your own fixed task set: completion rate, error rate, steps per task, time and cost.

Glossary

• Action: a single mouse, keyboard, scroll or wait instruction.
• Screenshot: an image of the current screen sent to the model.
• Grounding: locating an interface element in pixel coordinates.
• Harness: the program that runs the loop and enforces rules.
• Virtual machine: an isolated computer that runs inside another computer.
• Confirmation gate: a required human approval before a sensitive action.
• Trajectory: the recorded sequence of screenshots and actions.
• Prompt injection: text that tries to override the agent's instructions.
• Step limit: the maximum number of actions allowed in one run.

Summary

Computer use lets a model operate any application by reading screenshots and sending mouse and keyboard actions, which fills the gap where no API exists. It grew from GUI-agent research into a product capability offered by several providers from late 2024. It is flexible but slow, costly and exposed to screen-borne prompt injection. Use APIs first, run agents in isolated machines with limited accounts, gate consequential actions behind human approval, cap steps and log everything.

Related

It belongs to Coding and computer agents in my stack, under Agentic AI. Others in the same group: