Route native-only changes through macOS checks and verify relocated cursor resources in final packages. Reject resources that escape the app and exercise visible click feedback against a disposable native receiver.
Connect the regressions to required checks and capture shell fixture output through temporary files.
Resolve optional click animation assets from the running helper bundle.
Avoid SwiftPM Bundle.module's fatal fallback to a build-machine path so
visible clicks can return successfully and keep the daemon alive.
Keep procedural feedback when optional frames are absent or unreadable.
Add an input-free resource probe and run it against relocated release and
staged helper apps. Cover standard Resources copying and reject probes
that load from an external build directory.
Validation: 524 XCTest and 15 Swift Testing cases passed, including eight
new resource regressions. Build fixtures, compiled sidecar smoke, Electron
checks, local packaging and final relocated-package probes passed.
Rebuilt against main so the branch carries the Computer Use work and no other
divergence. Three unrelated efforts had been sitting uncommitted in this
worktree and were swept into an earlier commit; they are preserved on
cu-worktree-full-backup and belong on their own branches — adapter control
credentials, Electron asar sealing, and the sidecar code-loading audit. Every
file outside Computer Use now matches main exactly.
The engine
A Swift helper drives apps through the accessibility tree, with coordinate
actuation for the Chromium and Electron apps whose tree is a bare window
frame. Ten primitives matching the shape Codex uses, so an app's guidance and
the model's habits transfer.
Coordinate actions resolve their target window once and refuse when none can
be named. The unbound event they used to fall back to is discarded by custom
renderers, so a minimized target produced a whole session of "Action
completed" with nothing behind it.
Input acceptance is established for typing and key presses as well as clicks:
each MCP call is seconds apart, so the keyboard cannot inherit the focus a
click established. The synthetic focus notification is gated on the target
not already being active — sent unconditionally it names window 0 at an app
that already owns a key window, and nine window-bound clicks were discarded
with the traffic lights fully lit.
State the model can trust
An off-screen target says so, and says which tools still reach it: element
actions need no on-screen geometry, so an app with a real tree can still be
driven from the Dock. A fully covered window is recovered once, then left
alone — burying it again is the user wanting their screen back. A repeated
capture is reported with the cause that actually applies rather than both,
because coverage is something we compute.
Signing
The helper is signed under a stable identity before electron-builder sees it,
and excluded from re-signing: macOS ties Accessibility and Screen Recording
grants to the signing identity, so rotating it drops both on every update.
Discoverability
The desktop slash menu falls back to a directory scan while a session's CLI
has not started, which is when the menu is first opened. Built-ins and
bundled skills live in the binary, so /computer-use was absent until after
the first message.
程序员阿江(Relakkes)