Resolve optional click animation assets from the running helper bundle.
Avoid SwiftPM Bundle.module's fatal fallback to a build-machine path so
visible clicks can return successfully and keep the daemon alive.
Keep procedural feedback when optional frames are absent or unreadable.
Add an input-free resource probe and run it against relocated release and
staged helper apps. Cover standard Resources copying and reject probes
that load from an external build directory.
Validation: 524 XCTest and 15 Swift Testing cases passed, including eight
new resource regressions. Build fixtures, compiled sidecar smoke, Electron
checks, local packaging and final relocated-package probes passed.
Route coordinate clicks through the exact AX hit, reuse render-local
attributes and keep keyboard macros within validated native boundaries.
Preserve lossless capture evidence while returning bounded model images,
and propagate cancellation through daemon and lock acquisition.
Remove legacy per-app restrictions after feature-wide consent on both
platforms while retaining global disablement and target validation.
Include native helpers, regression tests and a stable receiver fixture.
Validation: 224 focused TypeScript tests, 516 XCTest cases and 15 Swift
Testing cases passed. Local sidecar and Electron builds, development
package creation and package smoke checks passed.
Add a persistent isolated JavaScript worker for native app actions and batch
known operations without a model round trip between each input. Preserve
per-cell context, native errors, screenshot coordinates, and image types.
Align macOS gesture, key, inventory, capture, scroll, and clipboard behavior;
include a signed native receiver fixture and compiled sidecar regression tests.
Keep the Windows pixel route and fix cancellation with session-owned mouse
cleanup, lock revalidation, and portable signing-fixture tests.
Keep occluded renderers live with display-filtered window streams, refresh
capture regions after geometry changes, and preserve the paste read window.
Allow application shutdown to finish even when resource cleanup throws.
Validated with check:native and an isolated desktop MCP workflow covering
background search, playback, lyrics, and repeated text entry.
Track real and synthetic focus with process-bound monitor receipts.
Bind keyboard activation to the snapshot window and align targeted
pointer and keyboard bursts with the reference event sequence.
Wait for pasteboard data delivery before restoring the clipboard,
and add focus/stream diagnostics with transition regression coverage.
Validated with check:native and six covered NetEase focus cycles.
Three failures shared one root cause: the engine reported success for input it
had no way to deliver or verify.
A fully covered Chromium/CEF window stops drawing, so every screenshot returns
the last frame it painted while each action still answers "Action completed".
One session read a frozen image for three and a half minutes, pressed play four
times, and reported a song playing that the final capture showed paused.
Mutating commands now fail closed with `window_occluded` instead.
The physical-input monitor counted `.mouseMoved`, so moving the cursor anywhere
on screen aborted background automation — the one thing the feature exists to
allow. Movement is not an interaction with any app; presses, drags, keys and
scroll still are.
Focus notifications went out on NSEvent type 13 alone. The target accepted them,
ignored them, and reported nothing: 24 actions, 1 effective. Key-focus subtypes
travel on type 21, and `keyFocusReturned` (0x8000) needs a sign-preserving
truncation or it collapses to subtype 0 — a notification the target accepts and
discards.
The stale-capture notice claimed input does not depend on visibility. Measured
behaviour contradicts it, so it now says actions are refused while covered, and
says it without asking the model to consult the user about window management —
an earlier wording did, and the model handed back a three-step task after one
action.
Claude-Session: https://claude.ai/code/session_015j1yxxaoonyAS2iZ7qGnTS
A coordinate click on a list row or a button did nothing. It moved the pointer
(the row lit up on hover), the events reached the window, the tool reported
"Action completed" — and the control never activated. Clicking a text field
worked. Clicking the same row twice, as one `click_count: 2`, worked.
Measured on one real session against an Electron target: single clicks 1/7,
double clicks 3/3. The cleanest pair is two clicks 2px apart on the same
suggestion row, with the window active both times — the only variable was the
repeat count.
Two fields were wrong, both since the engine's first commit:
**Movement claimed to be part of a click.** The leading `mouseMoved` carried
`clickCount` = the repeat count (1 for a single click, 2 for a double) instead
of 0, so the target was told "a pointer belonging to click N arrived" before any
button went down. The June comment that introduced it explains the 1…N cadence
of the press/release pairs and says nothing about why the move takes it — a
field reused, not a decision. August then widened it: the same value started
being passed to `NSEvent.mouseEvent(clickCount:)` as well, under a `max(1, …)`
that made 0 unreachable.
**A press and its release were not tied together.** Each event took a fresh
`eventNumber`, and AppKit uses that number to read two events as one click.
A text field focuses on the press alone, so it worked; anything needing a
complete click did not.
Both now follow the reference implementation, read out of its disassembly: its
drag-movement events carry `clickCount = 0` while the surrounding down/up carry
1, and its press/release pairs share one event number (a drag's down and up
share one, the dragged steps in between share another).
Verified against the same task that failed: single click on the suggestion row
lands first try, 7 tool calls and 55s where the failing run took 31 calls and
402s, no retries and no double click.
The rule is now stated once in `mouseClickState(for:click:)` instead of spelled
out per event, and nine tests cover it — there were none. `pressure` and
`mouseEventClickState` appeared nowhere in the suite, and no test had ever
called `clickPoint`, which is how these values survived from the first commit
through several rounds of "fix the clicks".
The glowing border framed the app being driven. It was purely decorative and
the user asked for it to go — the animated cursor already shows what Claude is
acting on.
It was not, as suspected, interfering with clicks: the overlay is
`ignoresMouseEvents`, sits at `CGShieldingWindowLevel()`, and every window
enumeration in `WindowGeometry` filters to `layer == 0`, so it could not enter
any hit-test or occlusion decision. It went because it earns nothing, not
because it broke anything.
`WindowFrameTracker` goes with it: its own header says it exists to keep the
glow glued to a moving window, and nothing else referenced it. That also
resolves the "30Hz CGWindowList polling" entry on the redesign doc's cut list —
the polling was the tracker's fallback path, so it is gone rather than reduced.
`NonAnimatedGradientLayer` moved into `VirtualCursor.swift`. It lived in the
glow file but the cursor's orb needs it too (without it Core Animation's
implicit 0.25s actions smear every frame change), and the compiler caught the
dangling reference.
`overlay_show` now only reveals the cursor, so `playSound` and
`stopOverlaySession(animated:)` lost their only consumers. The TypeScript side
had eleven comments describing a border that no longer exists; they now describe
what the code does.
Rebuilt against main so the branch carries the Computer Use work and no other
divergence. Three unrelated efforts had been sitting uncommitted in this
worktree and were swept into an earlier commit; they are preserved on
cu-worktree-full-backup and belong on their own branches — adapter control
credentials, Electron asar sealing, and the sidecar code-loading audit. Every
file outside Computer Use now matches main exactly.
The engine
A Swift helper drives apps through the accessibility tree, with coordinate
actuation for the Chromium and Electron apps whose tree is a bare window
frame. Ten primitives matching the shape Codex uses, so an app's guidance and
the model's habits transfer.
Coordinate actions resolve their target window once and refuse when none can
be named. The unbound event they used to fall back to is discarded by custom
renderers, so a minimized target produced a whole session of "Action
completed" with nothing behind it.
Input acceptance is established for typing and key presses as well as clicks:
each MCP call is seconds apart, so the keyboard cannot inherit the focus a
click established. The synthetic focus notification is gated on the target
not already being active — sent unconditionally it names window 0 at an app
that already owns a key window, and nine window-bound clicks were discarded
with the traffic lights fully lit.
State the model can trust
An off-screen target says so, and says which tools still reach it: element
actions need no on-screen geometry, so an app with a real tree can still be
driven from the Dock. A fully covered window is recovered once, then left
alone — burying it again is the user wanting their screen back. A repeated
capture is reported with the cause that actually applies rather than both,
because coverage is something we compute.
Signing
The helper is signed under a stable identity before electron-builder sees it,
and excluded from re-signing: macOS ties Accessibility and Screen Recording
grants to the signing identity, so rotating it drops both on every update.
Discoverability
The desktop slash menu falls back to a directory scan while a session's CLI
has not started, which is when the menu is first opened. Built-ins and
bundled skills live in the binary, so /computer-use was absent until after
the first message.
程序员阿江(Relakkes)