Computer use

Anthropic describes computer use as letting a model operate ordinary software from the screen, rather than only through a tool built for that program. In its October 2024 account, Claude looks at screenshots, counts pixels to place a cursor, clicks, and types through a virtual keyboard¹. Google's Computer Use tool runs the same kind of loop for a browser, a mobile device or a desktop. The application sends a screenshot, and the model returns a user-interface action such as a click, a scroll or a keystroke².

The application then carries out the action, takes a new screenshot and sends that back. The loop repeats until the task is finished or stopped². Google's response can also include a safety decision: the action may be allowed, blocked, or marked as needing user confirmation².

This is not a model left alone on a machine. Google says the developer runs the loop, recommends a sandboxed virtual machine or container, and says the preview can contain errors, so important tasks should be supervised². Anthropic's write-up says Claude's computer use remains slow and often error-prone, and that seeing the screen as a series of screenshots can miss short-lived actions or notifications¹.

OpenAI's Operator system card describes a research preview that does browser tasks by reading screenshots and using a cursor and keyboard. The examples include ordering groceries, booking a reservation and buying tickets. The card says the model should ask the user to confirm before an action that changes the world, such as completing a purchase or sending an email³.