Add a persistent isolated JavaScript worker for native app actions and batch
known operations without a model round trip between each input. Preserve
per-cell context, native errors, screenshot coordinates, and image types.
Align macOS gesture, key, inventory, capture, scroll, and clipboard behavior;
include a signed native receiver fixture and compiled sidecar regression tests.
Keep the Windows pixel route and fix cancellation with session-owned mouse
cleanup, lock revalidation, and portable signing-fixture tests.
Rebuilt against main so the branch carries the Computer Use work and no other
divergence. Three unrelated efforts had been sitting uncommitted in this
worktree and were swept into an earlier commit; they are preserved on
cu-worktree-full-backup and belong on their own branches — adapter control
credentials, Electron asar sealing, and the sidecar code-loading audit. Every
file outside Computer Use now matches main exactly.
The engine
A Swift helper drives apps through the accessibility tree, with coordinate
actuation for the Chromium and Electron apps whose tree is a bare window
frame. Ten primitives matching the shape Codex uses, so an app's guidance and
the model's habits transfer.
Coordinate actions resolve their target window once and refuse when none can
be named. The unbound event they used to fall back to is discarded by custom
renderers, so a minimized target produced a whole session of "Action
completed" with nothing behind it.
Input acceptance is established for typing and key presses as well as clicks:
each MCP call is seconds apart, so the keyboard cannot inherit the focus a
click established. The synthetic focus notification is gated on the target
not already being active — sent unconditionally it names window 0 at an app
that already owns a key window, and nine window-bound clicks were discarded
with the traffic lights fully lit.
State the model can trust
An off-screen target says so, and says which tools still reach it: element
actions need no on-screen geometry, so an app with a real tree can still be
driven from the Dock. A fully covered window is recovered once, then left
alone — burying it again is the user wanting their screen back. A repeated
capture is reported with the cause that actually applies rather than both,
because coverage is something we compute.
Signing
The helper is signed under a stable identity before electron-builder sees it,
and excluded from re-signing: macOS ties Accessibility and Screen Recording
grants to the signing identity, so rotating it drops both on every update.
Discoverability
The desktop slash menu falls back to a directory scan while a session's CLI
has not started, which is when the menu is first opened. Built-ins and
bundled skills live in the binary, so /computer-use was absent until after
the first message.
Computer Use is useful when explicitly needed, but exposing its MCP tools by default creates unnecessary desktop-control surface for users who want coding-only sessions. This adds a shared disable path for CLI flags, environment, and desktop settings while keeping preauthorized app state in one config file.
The same change also preserves Windows and WSL shell startup behavior by applying the MSYS argument-conversion guard only on WSL-bound launches.
Constraint: Computer Use MCP must not be exposed to the Coding Agent when disabled
Constraint: Desktop settings and CLI sessions need to read the same persisted Computer Use config
Rejected: Environment-only disable switch | desktop users need a persistent Settings control
Rejected: Remove Computer Use setup entirely | enabled sessions still need the existing built-in MCP path
Confidence: high
Scope-risk: moderate
Directive: Keep every new Computer Use entrypoint wired through loadStoredComputerUseConfig or the CLI disable flag before adding MCP tools
Tested: bun test src/utils/computerUse/gates.test.ts src/utils/computerUse/preauthorizedConfig.test.ts src/server/__tests__/computer-use-api.test.ts src/utils/shell/wslInterop.test.ts desktop/src/pages/ComputerUseSettings.test.tsx
Tested: bun run check:server; bun run check:desktop; bun run check:docs; bun run check:policy; bun run check:native; git diff --check
Tested: SKIP_INSTALL=1 ./desktop/scripts/build-macos-arm64.sh; codesign verify; hdiutil verify; built CLI Computer Use E2E exposure and disable checks
Not-tested: Full screenshot/control action after granting macOS Screen Recording permission on this machine
Desktop sessions were missing a visible request_access approval path and could
mis-detect their own app window as an unapproved frontmost target, which caused
Computer Use clicks to fail even after opening the intended app. On macOS, text
entry was also split across inconsistent clipboard and keystroke paths, making
Electron inputs unreliable for Chinese and short strings.
This change adds a desktop approval bridge over the existing session websocket,
renders a dedicated desktop approval modal, threads the real desktop bundle id
into the Computer Use executor, and switches macOS clipboard typing onto the
native pasteboard plus system paste shortcut path. It also makes tool error
results expandable in the desktop chat UI so frontmost-gate failures are fully
visible during debugging.
Constraint: Desktop sessions run the CLI over the SDK websocket path, so Ink tool JSX dialogs are not visible there
Constraint: macOS IME and Electron text inputs are unreliable with pyautogui.write and generic hotkey synthesis
Rejected: Reuse CLI setToolJSX dialogs in desktop mode | no transport for mid-call Ink UI over the SDK bridge
Rejected: Keep shell pbcopy/pbpaste for clipboard typing | inconsistent with NSPasteboard path and less reliable for Chinese text
Confidence: medium
Scope-risk: moderate
Reversibility: clean
Directive: Keep desktop Computer Use approvals and macOS text-entry behavior on a single bridge/path; avoid reintroducing separate CLI-only and desktop-only codepaths for the same action
Tested: python3 -m unittest runtime/test_helpers.py
Tested: bun test src/utils/computerUse/permissions.test.ts src/server/__tests__/conversation-service.test.ts
Tested: cd desktop && bun run test ComputerUsePermissionModal chatStore
Tested: cd desktop && bun run test chatBlocks
Tested: cd desktop && bun run lint
Not-tested: End-to-end manual Computer Use interaction against a live Electron target app on macOS
Bypass all three gating layers (compile-time feature flag, subscription
check, GrowthBook remote config) and replace Anthropic's private native
modules (@ant/computer-use-swift, @ant/computer-use-input) with a Python
bridge using pyautogui + mss + pyobjc. Works on any macOS with any
Anthropic-protocol compatible model.
程序员阿江(Relakkes)