← back to the archiveCover illustration for “Computer use: AI that operates a screen like a person”
TERM6mo ago·Published · Updated ·by Andy Padia

Computer use: AI that operates a screen like a person

In short: Computer use lets an AI model read screenshots and drive the mouse and keyboard, so an agent can operate apps that have no API, slowly and with care.

Computer use is a capability where an AI model operates software through its visual interface: it receives screenshots, decides where to click or what to type, and the harness executes those mouse and keyboard actions in an environment you control. It lets an agent work with applications that have no API. Anthropic's computer use tool documentation, read in November 2026, describes this screenshot, mouse and keyboard loop.

Computer use in plain words

Instead of plugging into the app's back door, the AI sits in front of the screen and uses it the way you would, one click at a time.

How it works

The loop is screenshot, decide, act, screenshot again. In OpenAI's computer use guide, the model returns a computer_call with an ordered list of actions, such as clicks at coordinates and typing. Your application executes them and sends back a screenshot, and the loop repeats until the model stops asking for actions. Every step needs a model call and a fresh image, so long tasks add up in time and cost.

Both vendors tell you to treat what is on screen as untrusted, because text in a page or image can try to override your instructions. Anthropic recommends a dedicated virtual machine or container with minimal privileges, an allowlist of domains and human confirmation for consequential actions. OpenAI adds step limits and a check of the actual outcome.

Example

As an illustration, take an expense report in a legacy web portal with no API: open the portal, log in through a pre-authenticated session, open "new claim", type each field from a receipt, attach the PDF, stop before "submit" and ask a person to confirm.

Where you'll meet it

  • Anthropic's and OpenAI's computer-use tools
  • Desktop automation in agent apps
  • Testing and QA of graphical interfaces
  • RPA replacement projects for legacy systems

What it changes for you

For RPA teams, my view is that it trades brittle selectors for a model that reads the screen, and gives up some speed and repeatability in exchange. For developers, the rule is simple: use an API or MCP server when one exists, and screen control only where none does. For security teams, a screen-driving agent sees everything on that screen.

Computer use vs browser agent

A browser agent is limited to web pages and can often read the page structure directly. Computer use controls the whole desktop, any application, through pixels.

Common mistake

My view: running computer-use agents on a real logged-in desktop. Use a dedicated VM with only the accounts the task needs, because the agent can be misled by anything it reads on screen.

Browser agent · Agent sandbox · Prompt injection · Tool calling · AI agent

Give a computer-use agent a dedicated VM that holds only the accounts the task needs.

#field-guide#agents#computer-use#automation

Sources

  1. platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool
  2. developers.openai.com/api/docs/guides/tools-computer-use
← older drop
AI ROI: measuring whether AI is worth the money
newer drop →
Headless mode: running a coding agent from scripts and CI

related drops

explore all 557 drops →
← back to the archiveday 175