Your computer: QuickAgents that use your applications¶
A QuickAgent can use an application on your own computer: any application you add to Agentry Desktop and approve, through its local MCP server, its windows, or both. The run itself stays in its sandbox. Each time the agent uses the application, the call goes out through the desktop link to Agentry Desktop on your computer, which runs the application's MCP server there as you, asks you before a call that runs code, and answers.
With computer use the agent sees and uses the windows of the applications you approved for it, through Agentry Desktop, which shows when it does and stops it on your key.
Such a QuickAgent runs on your own computer with Agentry Desktop running (from Agentry Desktop, or from the CLI on the same computer), or on your own runhost while your computer follows the run. It never runs in Agentry: a hosted run never reaches anyone's computer. Any application works with Agentry Desktop 0.7.29 or newer and the CLI v0.7.35 or newer; earlier versions use only Blender and Unreal Engine.
How you spot one¶
The QuickAgent's page lists what it uses under desktop, in the words its install asks you. An application is named by the short id the release gives it:
- "the application blender on your computer, through its MCP server (every tool it lists)", or "(tools get_scene_info, look)" when it narrows them
- "the application kdenlive on your computer, through its windows only"
- "an application you choose on your computer, through its MCP server (every tool it lists)", when the release asks you to pick one of yours
- for computer use, "can see and use the windows of blender on your computer", or "can see and use your whole screen"
Agentry Desktop adds what it knows of your own applications when you install
and run: which of yours the id is, its tools, and whether it came from Agentry's
library or you set it up. The directory's Capability filter has desktop to
find these QuickAgents.
Your applications¶
In Agentry Desktop, open Plugins → Your computer (Settings → Your computer opens it too). It lists the applications agents may use on this computer. Each card has two switches, both off when you add it:
- Agents may use its MCP server: a QuickAgent that names it may call its tools, through Agentry Desktop.
- Agents may see and use its windows: computer use may reach its windows.
Turning either on asks you again in a window of its own. The card shows the server's command or address, its variables, and its tools, each with its class: read (reads the application), act (changes it or works inside it) or exec (runs code or commands inside it, asked first). You can class a tool, which asks you again when it means fewer questions, or set it never given. A tool nobody has classed asks you before each call, like exec.
Add an application in two ways:
- Add from the library. Agentry's library sets up popular applications in
one click: Blender (
blender, itsmcp-for-blenderserver at the version Agentry pins, with its usage reporting off and its script screening on, both ordinary variables you can change) and Unreal Engine 5.8 (unreal, Epic's Unreal MCP plugin on its own port). Each lists the steps to take once, such as installing Blender's add-on and starting its server from the BlenderMCP tab of its sidebar, or enabling the Unreal MCP plugin. - Add another application. Give its name and id, the identifiers its windows
carry (Pick a running window reads them off an open window where your desktop
lets it, on X11), and its MCP server: a command Agentry Desktop starts (an
npxoruvxline, or the application's own program) or an address on your computer's loopback the application serves. Test starts it, after asking you, and lists its tools. An application you set up is not vetted by Agentry, and the run's receipt says so.
When you first open this version of Agentry Desktop, it adds the library's applications it finds ready on this computer, with nothing approved; remove any you do not want.
Computer use is a switch of its own on the same page. On macOS, allow Agentry Desktop under Screen & System Audio Recording and Accessibility, then open it again; on Linux under Wayland your desktop asks which window to share.
When it runs¶
- The run setup shows a card for each application the QuickAgent names: which of yours it is, whether it is ready, and the step to take when it is not. For an application you choose, or an id none of yours has, it lets you pick one of your approved applications and remembers your choice for that QuickAgent. Run stays off until each one is ready.
- Agentry Desktop asks you, in a window of its own, before a call that runs code or that nobody has classed, quoting what would run. You can run it once, allow every such call for that run, or say no; a no is the agent's answer.
- While the agent uses your screen, Agentry Desktop shows a banner and a tray item with Pause and Stop. The stop key is Ctrl+Shift+Escape (Ctrl+Shift+F12 on Windows). Stop refuses every later call, and the run goes on without your computer.
- Screenshots are kept nowhere, unless you switch on Keep what the agent saw, which saves them in the run's own folder on your computer.
- When an agent starts the run for you (the Assistant or Agentry Helper running a QuickAgent), the question it asks you before its first run of that version names each application and the computer use, and the run is held to what you approved here, like any other. This needs Agentry Desktop 0.7.42 or newer and the CLI v0.7.53 or newer.
What a computer's policy can say¶
The runner policy on a computer can keep runs to the library's applications as
they come, with desktop.allowCustomServers: false: a run that would use a
server you set up, or a library application whose server you changed, is
refused before it starts, naming the policy. Unsafe Mode, where it is on,
overrides that, and the receipt says so.
What the receipt says¶
The run's receipt has a line for each application: where its set-up came from (Agentry's library, the library's changed by you, or set up by you), the tools it was given, its calls by class, what was refused and why, and that your computer is outside the run's network and file rules, which Agentry cannot enforce there. The server runs as you, with your computer's network and files: Blender's asset tools, for instance, reach Poly Haven, Sketchfab and 3D generation services from your computer. Each call is a line of its own log, with its arguments.
For computer use, the receipt says what your computer's own system held (on macOS and Windows the capture of one window; under Wayland the input too) and what only Agentry Desktop's check before each input held.
On your runhost¶
A run sent to your own runhost with --on, or from the run setup, binds to
Agentry Desktop on this computer first, and the computer keeps a link to the
runhost open while the run lasts. If this computer goes away, the run's later
calls are refused and the run goes on; agentry runhost attach links it again.
A run that uses your computer is not detached and not scheduled on a runhost.
Persistent agents¶
A persistent agent that declares it (the Assistant and the Coding agent do) lists your applications approved for their MCP server, and computer use, on its page, each off until you switch it on. The Coding agent has them from Agentry Desktop 0.7.42 and the CLI v0.7.53: with computer use it can see and click the application it is building, or a window you point it at. It also runs programs with open internet access, so what it sees there can leave your computer through them. Switch computer use on only for windows you are happy for it to read. A switch takes effect at the agent's next turn; nothing asks you to stop the agent. Each application has a second switch, "code without asking": off, each call that runs code asks you first. Whether the application is open is read at each call, so opening or closing it restarts nothing. An agent on your runhost reaches this computer while Agentry Desktop runs here and is linked to it (Settings → Runs → Runhosts says "Linked to this computer").