How Computer-Using Agents Work

▶ Watch (00:21)

A computer-using agent takes a text prompt, connects to a machine, and operates it with mouse and keyboard. Each action follows the same loop: take a screenshot, analyze what’s on screen, build a plan, act. That loop repeats until the task is done. No API required.

“because computer use is always analyzing the content of the screen” — Unknown (introduced by Mona)

Copilot Studio now ships this as a built-in capability. Previously, agents in the platform could call thousands of API-backed actions. Computer use adds everything else: any website or desktop application that never published an API.

Configuring a Computer Use Tool in Copilot Studio

▶ Watch (03:11)

Setting up the tool takes three inputs: a target machine (either a VM on your own infrastructure or a hosted machine provisioned inside Power Platform), a description the agent reads to decide when to call the tool, and the task steps written in plain language. For the invoice demo, that meant “navigate to the website, fill in the invoice, submit it.”

Dynamic inputs let the tool accept data at runtime. One input named “invoice” passes the extracted PDF content into each execution. No scripts, no formulas.

“All it takes is natural language.” — Unknown (introduced by Mona)

Live Execution: Adaptive Form-Filling

▶ Watch (05:04)

Watch the invoice portal automation live demo (05:04) to see two things the instructions never mentioned. First, the portal opened with a maintenance notification popup. The agent read it, judged it non-critical, and dismissed it. Second, the invoice listed the month as a word (“January”) while the form required DD/MM/YYYY digits. The agent converted the format without being told.

Both recoveries happened because the model analyzes the actual screen state at each step, not a static script. Errors in the instructions can be fixed in natural language after testing.

Assembling the End-to-End Agent and Use Cases

▶ Watch (07:40)

The full invoice agent connects three pieces: a computer use tool for portal entry, an email trigger that fires on each new invoice, and an agent flow that emails the vendor and posts a Teams notification. The activity view logs every run with screenshots and reasoning steps.

“the possibilities of combining computer use with agents are really endless” — Unknown (introduced by Mona)

Three patterns are emerging: data entry into API-less systems, data extraction for reports, and cross-application sessions that span a desktop app plus multiple websites. Access requires the Frontier program (500,000 component messages purchased minimum).

Notable Quotes

All it takes is natural language. Unknown (introduced by Mona) · ▶ 11:02

because computer use is always analyzing the content of the screen Unknown (introduced by Mona) · ▶ 5:53

the possibilities of combining computer use with agents are really endless Unknown (introduced by Mona) · ▶ 11:32

Key Takeaways

  • CUAs operate any app by taking a screenshot, building a plan, acting, and repeating, requiring no API.
  • Copilot Studio’s computer use tool accepts natural-language task steps and dynamic inputs defined by the agent.
  • A single session can span desktop apps and multiple websites, enabling cross-application automation without code.