Agent Desktop Harness ADH Project
Use this command to install Agent Desktop Harness with WinGet:
winget install --id=ADHProject.ADH -e Trusted Python computer-use harness for Windows desktop agents.
Loading...
README
Agent Desktop Harness (ADH) — Windows Computer Use and RPA for AI Agents
> Reliable Windows computer use, desktop automation, and reusable RPA through MCP and CLI.
Agent Desktop Harness is an open-source Windows computer-use and RPA runtime for AI agents. It lets agents and command-line users observe and control real desktop applications, automate visible Windows interfaces, and turn successful interactions into reusable Robot Framework workflows. ADH also validates those workflows before execution and explains failures through concrete UI evidence.
ADH combines native Windows UI automation, accessibility inspection, screenshots, a first-class CLI, MCP tools, trusted Python, and Robot Framework in one execution model. It is built for coding agents and automation developers who need to control visible Windows applications and web UIs without leaving behind an opaque recording or a pile of unexplained coordinates.
Turn an AI agent's exploration into reusable RPA
Interactive computer use is useful for discovering how an application behaves. Production automation needs more: readable source, controlled inputs, repeatable execution, explicit assertions, and enough evidence to explain a failure.
ADH gives an agent a complete workflow:
- Explore the exact Windows application through CLI, MCP, or the Python Computer SDK.
- Author a readable Robot Framework task with semantic UI selectors.
- Validate Robot syntax, imports, keyword signatures, variables, output scope, and source-health rules before touching the desktop.
- Run one bounded task through ADH's fixed runtime and native Windows backend.
- Diagnose failures from canonical events, the failed keyword, the last useful observation, screenshots, UI trees, and run artifacts.
- Refine and reuse the workflow with new JSON inputs instead of repeating an interactive session.
flowchart LR
Agent["AI agent or developer via CLI / MCP"] --> Explore["Explore the real desktop"]
Explore --> Source["Author Robot Framework RPA"]
Source --> Validate["Validate source and inputs"]
Validate --> Run["Run through ADH"]
Run --> Apps["Windows apps and visible web UIs"]
Run --> Evidence["Events, screenshots, UI tree, reports"]
Evidence --> Source
Computer-use and RPA showcases
These public RPA challenges were completed by agent-authored Robot Framework workflows running through ADH's native Windows automation backend. Although the challenge pages are rendered in a browser, ADH drives the visible application and the Windows accessibility surface directly—without a browser extension, Selenium, Playwright, or a challenge-specific API.
OCR invoice challenge: browser, system OCR, CSV, and file dialog
In the public RPA Challenge OCR, one ADH workflow moved across multiple operating-system surfaces:
Microsoft Edge → invoice images → Windows Text Extractor → clipboard → CSV → native file picker → success page
It inspected the invoice table, opened the source images, selected image regions through the Windows Text Extractor overlay, parsed the captured text, built the required CSV, uploaded it through the native Windows file dialog, and verified the terminal success state.
Result: CONGRATS in 55.2 seconds · 6 eligible invoices · 105
completed ADH actions · 1/1 Robot task passed · no partial effects.
Explore or run the agent-authored source in the OCR invoice challenge example, which also contains the unedited completion evidence.
Lastest Playground: 18 UI automation scenarios
ADH also completed all 4 Easy, 10 Medium, and 4 Hard scenarios exercised in the Lastest RPA Playground. The suite covers forms, dialogs, dynamic content, multiple windows and tabs, uploads and downloads, drag-and-drop, iframes, Shadow DOM, and deliberately tricky selectors—all through the same OS-level automation model used for native Windows applications.
<a href="docs/assets/rpa-showcases/file-transfer-workflow.png"><img src="docs/assets/rpa-showcases/file-transfer-workflow.png" /></a>
<a href="docs/assets/rpa-showcases/windows-tabs-workflow.png"><img src="docs/assets/rpa-showcases/windows-tabs-workflow.png" /></a>
<strong>File upload & download</strong>Use the native picker and drop zone, reject an oversized file, download a report, and verify it.
<strong>Windows & tabs</strong>Switch tabs and popups, recover a secret code, handle a delayed window, and verify closure.
<a href="docs/assets/rpa-showcases/drag-drop-workflow.png"><img src="docs/assets/rpa-showcases/drag-drop-workflow.png" /></a>
<strong>Drag & drop (Hard)</strong>Match three targets, reorder a priority list, set a precision slider to exactly 75, and assert completion.
The illustrations above explain what each task does. Unedited completion screenshots from the real ADH runs are retained with the examples as verification evidence. The challenge sites are independent projects and are not affiliated with ADH.
The complete Easy, Medium, and Hard workflows are available in the Lastest RPA Playground example.
Why use ADH for computer use and RPA?
| Capability | What it gives you |
|---|---|
| Agent-authored RPA as code | Reviewable .robot tasks, resources, JSON Schema inputs, and test data instead of an opaque macro. |
| Real Windows application control | Enumerate apps and windows, launch and focus targets, inspect accessibility state, click, type, scroll, drag, and use guarded global keys. |
| Semantic UI selectors | Locate controls by role, name, value, automation ID, class, and ancestor relationships; use coordinates only behind explicit observation guards. |
| Validate before execution | Catch invalid Robot source, imports, keyword calls, variables, unsafe output paths, encoding damage, and stale UI references before a workflow runs. |
| Evidence-first debugging | Correlate Robot keywords with native desktop actions in one event chain and retain bounded failure evidence. |
| One model across MCP, Python, and Robot | Explore interactively, author reusable automation, and debug it without switching to an unrelated desktop driver. |
| Controlled artifacts | Keep reports, logs, screenshots, observation fixtures, and per-run summaries inside an authorized output root. |
ADH is not a session recorder. adh rpa init creates a neutral project scaffold;
the agent writes the business workflow explicitly and proves it through validation
and execution.
RPA quick start
Create a new automation workspace:
adh rpa init --out .\rpa\search-workflow --name search-workflow
The scaffold contains:
search-workflow/
├── task.robot
├── variables.schema.json
├── defaults.yaml
├── README.md
└── testdata/
└── sample-input.json
Write the workflow, validate one selected task and its inputs, then run it:
adh rpa validate .\rpa\search-workflow `
--vars .\rpa\search-workflow\testdata\sample-input.json `
--task "Search In A Verified Window"
adh rpa run .\rpa\search-workflow `
--vars .\rpa\search-workflow\testdata\sample-input.json `
--task "Search In A Verified Window" `
--output-root .\rpa-output
A reusable Robot keyword can express the visible UI boundary with semantic selectors and an independent success assertion:
*** Keywords ***
Search In Verified Window
[Arguments] ${window} ${query}
Focus Window ${window}
${search}= Wait For Element ${window} role=textbox timeout=15s
Set Field Value ${window} ${query} ${search} verify=none
${submit}= Wait For Element ${window} role=button name=Search timeout=15s
Click Element ${window} ${submit} verify=none
Text Should Be Visible ${window} ${query} timeout=15s
Each run receives its own UTC-partitioned output directory. Depending on the workflow, the evidence set can include:
output.xml,log.html, andreport.htmlfrom Robot Framework;adh-rpa-events.jsonl, the portable canonical event chain;adh-rpa-run.json, a compact derived run summary;- failure-time observations, UI trees, screenshots, and explicitly saved artifacts.
Installation
Public release bootstrap
The following command will become active when this repository is public and the
next release has passed its clean-machine gates and been promoted to a normal
GitHub latest release:
irm https://github.com/xuyw1997/agent-desktop-harness/releases/latest/download/install.ps1 | iex
The bootstrap downloads the fixed Windows x64 installer and its SHA-256 sidecar,
verifies the installer bytes, and performs a per-user installation under
%LOCALAPPDATA%\Programs\ADH. The packaged runtime includes its own pinned
CPython 3.12 environment; a system Python installation is not required.
After installation:
adh start
adh doctor --json
adh install agent --client codex --scope user
Restart the agent host after installing its MCP connection and shared ADH skill.
Claude can be selected with --client claude.
Updating ADH
ADH does not currently provide an adh update subcommand. To upgrade an
existing installation to the latest stable release, stop ADH and rerun the
verified release installer:
adh stop; $p="$env:TEMP\adh-update.ps1"; irm https://github.com/xuyw1997/agent-desktop-harness/releases/latest/download/install.ps1 -OutFile $p; powershell.exe -NoProfile -ExecutionPolicy Bypass -File $p
Open a new PowerShell window after the upgrade, then start ADH and verify the installed version and runtime health:
adh start
adh doctor
The installer recognizes the existing per-user installation as an upgrade, verifies the downloaded installer against its SHA-256 sidecar, and keeps the same install path. Existing MCP configurations therefore do not normally need to be changed. If you use the bundled agent skill, refresh that integration and restart the agent host:
adh install agent --client codex --scope user
Use --client claude instead when appropriate.
WinGet status
ADHProject.ADH has not yet been published to the WinGet community source.
The repository can generate and validate WinGet manifests, but this command will
only work after the first public release is submitted and accepted upstream:
winget install --id ADHProject.ADH --exact
Do not advertise WinGet as an available installation channel until that package
is visible through winget show --id ADHProject.ADH --exact.
Repository development build
Maintainers with the pinned Python, Rust, Visual Studio, Windows SDK, and runtime inputs can build and refresh the local installation with:
scripts\windows\refresh.cmd
adh doctor --json
See scripts/windows/README.md for the development wrappers and the Windows release runbook for the packaging gates.
CLI or MCP: which should you choose?
For most modern AI agents, use MCP. Agent shells commonly run inside a security sandbox that restricts Windows named pipes, native processes, GUI access, or child process visibility. ADH's host-managed MCP connection reaches the desktop runtime through the supported integration boundary, so computer use and RPA generally work without fighting the agent's shell sandbox.
Use the CLI when you are working directly in PowerShell, building scripts or CI, or when you have deliberately granted the agent full host access without a sandbox. Do not weaken an agent's sandbox merely to make the CLI path work; configure MCP instead.
| Situation | Recommended interface |
|---|---|
| Sandboxed coding or AI agent | MCP — the smoothest and safest default. |
| Agent with explicit, unrestricted host access | CLI or MCP — choose the interface that best fits the workflow. |
| Human-operated terminal, PowerShell script, or CI diagnostics | CLI — direct and easy to compose. |
CLI and MCP are functionally equivalent. Both expose the same desktop exploration, trusted Python, RPA initialization, validation, execution, and diagnostic capabilities through the same installed runtime and contracts. Their command shapes differ, but an automation achievable through one interface is achievable through the other, and workflows do not need to be rewritten when switching channels.
ADH CLI
Choose the CLI for direct terminal use or for an agent that has unrestricted host access. For a sandboxed agent, prefer MCP.
The native adh command is a first-class interface, not just an installer shim.
It supports local terminal workflows, PowerShell automation, CI diagnostics,
independent RPA execution, and debugging when an MCP host is unavailable.
Runtime and health commands
adh --version
adh start
adh status
adh doctor --json
adh stop
adh doctor --json reports installation identity, daemon and driver readiness,
process integrity, runtime compatibility, and actionable recovery information.
Persistent trusted Python sessions
Separate CLI invocations reuse the current daemon-owned session, including Python globals and worker generation:
adh start_session
adh run_python --code "value = 6 * 7; print(value)"
adh run_python --file .\explore_desktop.py
adh reset_python
adh close_session
run_python returns structured JSON and includes bounded diagnostics when an
execution fails, times out, or produces partial desktop effects. The trusted
Python runtime exposes the ADH Computer SDK for application discovery, window
selection, observation, semantic element lookup, actions, assertions, and
artifact capture.
RPA, agent integration, and artifacts
# Reusable Robot Framework RPA
adh rpa init --out .\rpa\my-workflow --name my-workflow
adh rpa validate .\rpa\my-workflow --vars .\inputs.json --task "My Task"
adh rpa run .\rpa\my-workflow --vars .\inputs.json --task "My Task" --output-root .\out
# Codex or Claude integration
adh install agent --client codex --scope user
adh mcp status --client codex --scope user
# Daemon-owned artifacts
adh artifact get
adh artifact export --output .\artifact.png
| CLI area | Purpose |
|---|---|
adh start, status, doctor, stop | Manage and diagnose the native Windows runtime. |
adh start_session, run_python, reset_python, close_session | Explore and automate through persistent trusted Python. |
adh rpa init, validate, run | Author, check, and execute reusable Robot Framework automation. |
adh install agent, mcp status | Configure and verify Codex or Claude MCP integration. |
adh artifact get, artifact export | Inspect or export artifacts retained by the daemon. |
CLI and MCP use the same installed runtime and execution contracts. A workflow debugged through the CLI does not need to be rewritten to run through MCP.
MCP and AI agent integration
Choose MCP for sandboxed agents. It provides the same ADH automation capabilities as the CLI without requiring the agent's shell to access native Windows IPC and desktop resources directly. See CLI or MCP: which should you choose? for the full decision guide.
ADH exposes eight focused MCP tools:
- persistent Python session lifecycle:
start_session,run_python,reset_python, andclose_session; - reusable RPA lifecycle:
init_rpa,validate_rpa, andrun_rpa; - bounded canonical diagnostics:
read_log_events.
The MCP path is designed for coding agents such as Codex and Claude. An agent can inspect the current UI, test a selector, write or edit Robot Framework source, validate the project, run the smallest meaningful task, and use returned evidence to repair the exact failure.
The core execution names intentionally match across CLI and MCP:
| CLI | MCP tool | Purpose |
|---|---|---|
adh start_session | start_session | Start or select a persistent Python session. |
adh run_python | run_python | Execute trusted Python against the real desktop. |
adh reset_python | reset_python | Reset session globals without replacing the ADH installation. |
adh close_session | close_session | Close an idle execution session. |
adh rpa init | init_rpa | Create a neutral agent-authored RPA workspace. |
adh rpa validate | validate_rpa | Validate source, selection, inputs, and output scope. |
adh rpa run | run_rpa | Run a validated Robot Framework task. |
MCP additionally exposes read_log_events for bounded paging of canonical event
chains. CLI-only workflows can read the same referenced JSONL segments directly.
The trusted Python environment is intentionally full Python, not a security sandbox. Read the security policy before connecting an autonomous agent or running RPA source from another party.
Windows computer-use capabilities
The current Windows backend supports:
- application, window, and display discovery;
- application launch, window focus, and owned-window cleanup;
- window and desktop screenshots plus accessibility-tree observations;
- semantic element lookup and descendant selection;
- click, double-click, secondary actions, text entry, field values, keys, global keys, scrolling, and drag operations;
- window-title, visible-text, tree, element existence, enabled, and focus assertions;
- observation fixtures for debugging selector and parsing logic without repeating desktop side effects;
- daemon-owned artifacts and canonical JSONL event logs.
Architecture
ADH uses one native Windows runtime rather than separate implementations for interactive agent control and reusable automation:
CLI scripts / PowerShell Codex / Claude
| |
+---------- CLI / MCP -----------+
|
trusted Python sessions + RPA runner
|
ADH Computer SDK
|
native daemon + UI driver
|
Windows UI Automation + input + capture
Robot Framework calls the Computer SDK directly. MCP and CLI Python sessions use daemon-owned persistent workers. Both paths share the same native protocol, artifact model, Windows target identities, and canonical action events.
Platform support
| Platform | Status | Notes |
|---|---|---|
| Windows 11 24H2, build 26100+, x64 | Alpha, supported target | Requires an interactive, unlocked user desktop. |
| Windows on ARM64 | Not available | No ARM64 package is produced by the current alpha. |
| Linux | Not supported | ADH currently targets native Windows desktop automation. |
ADH cannot automate a locked or disconnected interactive desktop as though it were still visible. Elevated targets, secure desktop surfaces, application-specific accessibility limitations, and system policy may also restrict automation.
How ADH differs from browser automation
Selenium and Playwright are excellent when a browser DOM is the correct automation boundary. ADH targets the visible Windows session: native desktop software, system overlays, accessibility surfaces, and browser windows operated as real desktop applications. A workflow can still use normal HTTP, filesystem, or data-processing libraries for non-UI work and cross into ADH only where visible interaction is required.
Project status
ADH is alpha software. The Windows runtime, Computer SDK, MCP integration, Robot Framework library, RPA validator and runner, installer, checksummed release bootstrap, packaging manifests, SBOM generation, and clean-machine release gates are implemented. Public distribution, Authenticode signing, and WinGet community publication are separate launch steps and should not be treated as complete yet.
Expect interfaces and packaging details to evolve during the 0.x series. Bug
reports with minimal reproduction steps and redacted diagnostics will be welcome
after the repository becomes public. Report security issues privately as described
in SECURITY.md.
Frequently asked questions
What is Agent Desktop Harness?
Agent Desktop Harness is a local Windows computer-use and RPA runtime for AI agents. It connects MCP and trusted Python exploration with reusable Robot Framework desktop automation.
What is computer use for AI agents?
Computer use lets an AI agent observe and operate the same visible desktop applications as a human. ADH provides Windows computer use through MCP, CLI, trusted Python, accessibility inspection, screenshots, native input, assertions, and reusable Robot Framework RPA.
Can an AI agent create an RPA workflow with ADH?
Yes. An agent can inspect the real application, author .robot source and input
schemas, validate them through ADH, run one selected task, and debug it from
failure-time UI evidence. ADH creates the execution scaffold and safety rails;
it does not silently invent business steps from a recording.
Can I use ADH without MCP or an AI agent?
Yes. The adh CLI can manage the runtime, execute persistent Python sessions,
validate and run Robot Framework RPA projects, inspect artifacts, and report
diagnostics independently. MCP exposes the same core execution model to Codex,
Claude, and other compatible agent hosts.
Does ADH automate native Windows applications or only websites?
Both visible native applications and visible browser windows can be automated. ADH operates through Windows application, window, accessibility, input, and capture capabilities rather than a browser-only DOM driver.
Is ADH a replacement for Selenium or Playwright?
Not usually. Use a browser automation API when the DOM is the correct boundary. Use ADH when the workflow must interact with the real Windows desktop, a native application, a system overlay, or browser UI that must behave like a user-visible desktop session.
Does ADH require a system Python installation?
The packaged Windows release includes a pinned CPython 3.12 runtime. Repository development has additional Python, Rust, Visual Studio, and Windows SDK requirements.
Is WinGet installation available?
Not yet. The WinGet manifests are generated and tested, but ADHProject.ADH must
be submitted to and accepted by the WinGet community repository after the first
public GitHub release.
Related project
Accio Computer Use (accio-cu) is ADH's sibling project for macOS; ADH focuses on Windows.
Citation
If you use Agent Desktop Harness in your work, please cite this repository:
@misc{Xu2026AgentDesktopHarness,
author={Yuanwu Xu and Li Cai and Lanbo Lin and Tianyuan Yang and Yu Gu and Sicong Xie},
title={Agent Desktop Harness},
note={GitHub repository},
howpublished={\url{https://github.com/xuyw1997/agent-desktop-harness}},
year={2026}
}
If your work also uses Accio Computer Use, please cite it separately:
@misc{Yang2026Accio,
author={Tianyuan Yang and Yu Gu and Lanbo Lin and Yuanwu Xu and Li Cai and Sicong Xie},
title={Accio Computer Use},
note={GitHub repository},
howpublished={\url{https://github.com/Accio-org/accio-cu}},
year={2026}
}
License
Agent Desktop Harness is licensed under the Apache License 2.0. Third-party attributions are provided in packaging/THIRD_PARTY_NOTICES.txt and in the SBOM shipped with each release.