Skip to content
Road to Intelligence

Concept · Chapter 13: Agents

Computer-Use Agents

Should knowUnderstand7 minDifficulty

Computer-use agents act through interface controls and inspect fresh observations to determine the resulting state.

The problem

Many tasks are exposed through applications rather than a stable dedicated API.

The solution

Use an observe-act-observe loop over the page, accessibility tree or screen.

The consequence

Dynamic interfaces make stale observations and ambiguous outcomes costly.

You should understand first

  1. Text as Data
  2. Probability and Distributions
  3. Conditional Probability and Bayes' Theorem
  4. Probability of Sequences
  5. Language Modeling
  6. Vectors
  7. Dot Product
  8. Embeddings
  9. Attention
  10. Softmax
  11. Self-Attention
  12. Causal Masking
  13. Entropy
  14. Loss Functions
  15. Cross-Entropy Loss
  16. Autoregressive Next-Token Prediction
  17. Pretraining at Scale
  18. GPT-1 → GPT-2 → GPT-3
  19. In-Context Learning
  20. The Turing Test
  21. Symbolic AI
  22. Logic and Rules
  23. Expert Systems
  24. Knowledge Representation
  25. The Knowledge-Acquisition Bottleneck
  26. From Rules to Learning
  27. Supervised, Unsupervised and Self-Supervised Learning
  28. Expected Value and Variance
  29. Reinforcement Learning
  30. MDPs, Policies and Value
  31. Decoding: Greedy, Temperature, Top-k, Top-p
  32. One-Hot Encoding
  33. Tokenization
  34. Chat Templates
  35. System Prompts and Instructions
  36. Structured Outputs and Constrained Decoding
  37. Tool Calling
  38. LLM Agents
  39. Agent Runtime
  40. Tool Execution in Agents
  41. Computer-Use Agents

Clicking is not completion

An agent can select Save and still fail to save: validation may block it, the network may time out, or the click may hit a different control after layout changes. Inspect the resulting state before declaring success.

WebArena provides reproducible web environments and evaluates functional task completion. It motivates checking what changed, not merely whether the action trace looks plausible.

Prefer observable actions

Use stable semantic controls when available. After navigation, obtain a fresh observation. A screenshot supplies visual evidence but not an omniscient view of application state; hidden records and pending requests can matter.

Side-effecting UI operations need the same permission boundaries as API calls. A purchase confirmation screen is an opportunity to review a specific operation, not proof that the purchase has already happened. A timeout may require reconciling state rather than clicking again.

What to remember

  • Computer-use agents act through interface controls and inspect fresh observations to determine the resulting state.
  • Use an observe-act-observe loop over the page, accessibility tree or screen.
  • Dynamic interfaces make stale observations and ambiguous outcomes costly.

Key papers