**Rephrased Blog Content**
**TL;DR:** Computer-use agents, or GUI agents, are vision-language models that mimic human users on unmodified software. Initial benchmarks on OSWorld showed human performance at 72.36% and the best model at 12.24%; Anthropic’s Claude Sonnet 4.5 now reports 61.4%. Gemini 2.5 Computer Use leads several web benchmarks but isn’t yet optimized for operating systems. Future work focuses on OS-level robustness, sub-second action loops, and enhanced safety policies, with open community recipes for training and evaluation.
**Definition:** Computer-use agents, also known as GUI agents, are AI models that observe the screen, identify UI elements, and execute bounded UI actions (click, type, scroll, key-combos) to complete tasks in unmodified applications and browsers. Key implementations include Anthropic’s Computer Use, Google’s Gemini 2.5 Computer Use, and OpenAI’s Computer-Using Agent powering Operator.
**Control Loop:** The typical runtime loop involves capturing screenshots and state, planning the next action with spatial and semantic grounding, executing the action via a constrained action schema, and verifying and retrying on failure. Vendors document standardized action sets and guardrails, while audited harnesses normalize comparisons.
**Benchmark Landscape:**
– **OSWorld (HKU, Apr 2024):** This benchmark covers 369 real desktop/web tasks, spanning OS file I/O and multi-app workflows. At launch, humans achieved 72.36%, with the best model at 12.24%.
– **State of play (2025):** Anthropic’s Claude Sonnet 4.5 reports 61.4% on OSWorld, a significant jump from its previous 42.2%.
– **Live-web benchmarks:** Google’s Gemini 2.5 Computer Use reports 69.0% on Online-Mind2Web, 88.9% on WebVoyager, and 69.7% on AndroidWorld. Notably, the current model is browser-optimized and not yet optimized for OS-level control.
**Architecture Components:**
1. **Perception & Grounding:** Periodic screenshots, OCR/text extraction, element localization, and coordinate inference.
2. **Planning:** Multi-step policy with recovery, often post-trained or RL-tuned for UI control.
3. **Action Schema:** Bounded verbs (click_at, type, key_combo, open_app), with benchmark-specific exclusions to prevent tool shortcuts.
4. **Evaluation Harness:** Live-web/VM sandboxes with third-party auditing and reproducible execution scripts.
**Enterprise Snapshot:**
– **Anthropic:** Offers a Computer Use API with Sonnet 4.5 at 61.4% on OSWorld, emphasizing pixel-accurate grounding, retries, and safety confirmations.
– **Google DeepMind:** Provides a Gemini 2.5 Computer Use API with model card, reporting Online-Mind2Web 69.0%, WebVoyager 88.9%, and AndroidWorld 69.7%, along with latency measurements and safety mitigations.
– **OpenAI:** Offers Operator, a research preview for U.S. Pro users, powered by a Computer-Using Agent, with separate system card and developer surface via the Responses API, but limited availability.
**Where They’re Headed: Web → OS**
– **Few-/one-shot workflow cloning:** Near-term focus is on robust task imitation from a single demonstration (screen capture + narration).
– **Latency budgets for collaboration:** To preserve direct manipulation, actions should land within 0.1–1 s HCI thresholds, requiring engineering on incremental vision, cache-aware OCR, and action batching.
– **OS-level breadth:** File dialogs, multi-window focus, non-DOM UIs, and system policies add failure modes absent from browser-only agents, making OS-level breadth the next step.
– **Safety:** Prompt-injection from web content, dangerous actions, and data exfiltration are key concerns, with model cards describing allow/deny lists, confirmations, and blocked domains, and typed action contracts and “consent gates” for irreversible steps expected.
**Practical Build Notes:**
– Start with a browser-first agent using a documented action schema and a verified harness (e.g., Online-Mind2Web).
– Add recoverability: explicit post-conditions, on-screen verification, and rollback plans for long workflows.
– Treat metrics with skepticism: prefer audited leaderboards or third-party harnesses over self-reported scripts; OSWorld uses execution-based evaluation for reproducibility.
**Open Research & Tooling:**
– Hugging Face’s Smol2Operator provides an open post-training recipe that upgrades a small VLM into a GUI-grounded operator, useful for labs/startups prioritizing reproducible training over leaderboard records.
**Key Takeaways:**
– Computer-use (GUI) agents are VLM-driven systems that perceive screens and emit bounded UI actions (click/type/scroll) to operate unmodified apps, with key implementations including Anthropic Computer Use, Google Gemini 2.5 Computer Use, and OpenAI’s Computer-Using Agent.
– OSWorld benchmarks 369 real desktop/web tasks, with humans achieving 72.36% and the best model at 12.24%, highlighting grounding and procedural gaps.
– Anthropic Claude Sonnet 4.5 reports 61.4% on OSWorld, a significant jump from prior Sonnet 4 results.
– Gemini 2.5 Computer Use leads several live-web benchmarks but isn’t yet optimized for OS-level control.
– OpenAI Operator is a research preview powered by the Computer-Using Agent (CUA) model, using screenshots to interact with GUIs, with limited availability.
– Open-source trajectory: Hugging Face’s Smol2Operator provides a reproducible post-training pipeline that turns a small VLM into a GUI-grounded operator, standardizing action schemas and datasets.
