Windows has no equivalent of `CGEvent.postToPid`. `pyautogui` bottoms out in
`SendInput`, which injects into the one system-wide input stream and warps the
one real cursor — so on Windows the agent shares the mouse and keyboard with
the user, and `SendInput` reports success unconditionally whether or not
anything acted on the events.
That combination produced the same lie the macOS engine was just fixed for:
click a point behind another window and it lands on that window; click one
off-screen and it lands nowhere; either way the helper answered "Action
completed".
So the helper now refuses instead of guessing:
* `ForegroundLease` samples `GetLastInputInfo` around every mutating command.
Interference before the action is `user_interference` — nothing ran, retry
is safe. Interference during it is `user_interference_result_unknown`,
because injection already went out and a retry could double-apply it. On a
play/pause toggle those two differ by exactly one wrong outcome.
* `ensure_point_on_screen` and `ensure_target_window_reachable` reject
coordinates outside every display and targets whose windows are minimized
or hidden, before anything is sent.
`GetLastInputInfo` is the signal because it needs no privileges and is not
advanced by `SendInput`, so the agent cannot trip its own detector. Both guards
fail open on an unreadable reading: a safety layer that turns the feature off
is not safety.
The guard set lives in one place rather than in each dispatcher branch — an
eleventh verb wired like the ten before it would otherwise be silently
unguarded, which is the bug class this pass removes. All ten mutating branches
now return through `_finish`, so no branch can write its own success response
and skip the post-action check.
Also adds a Windows cursor badge. It deliberately does NOT mirror the macOS
virtual cursor: there the real pointer never moves, so the drawn one is the
only cursor and replaces it. Here the real pointer does move, and a second
fake pointer would just be two cursors with one of them lying about where the
click lands. The badge annotates instead — it answers "is this me or the
agent?", which matters because grabbing the mouse mid-action is what makes the
two input streams interleave.
Retires `runtime/mac_helper.py` and its pyobjc requirements: macOS routes every
command to the signed native daemon and `helperBridge` refuses to fall back, so
both were unreachable. Renames the `callPythonHelper` alias to `callHelper`,
which is what it has actually imported since the native engine landed.
Verified with mutation testing — 13 injected regressions across both languages
(dropped lease, bypassed `_finish`, collapsed interference codes, guard reusing
the filtered `list_windows`, badge losing click-through), all caught.
Claude-Session: https://claude.ai/code/session_015j1yxxaoonyAS2iZ7qGnTS
Computer Use now rejects Python runtimes below 3.9 before running pip,
keeps Pillow on the Python 3.9-compatible 11.x line, and falls back from
the configured mirror to the default PyPI index when dependency installs
fail.
Constraint: Pillow 11.3 requires Python 3.9+, while user machines may resolve a different Python than expected
Rejected: Downgrade Pillow to the Python 3.8-compatible 10.x line | current product direction assumes modern Python installs
Confidence: high
Scope-risk: narrow
Directive: Keep runtime package ranges aligned with the setup Python minimum before changing either side
Tested: bun test src/server/__tests__/computer-use-api.test.ts src/server/__tests__/computer-use-requirements.test.ts src/server/__tests__/computer-use-python.test.ts src/utils/computerUse/pipInstall.test.ts
Tested: bun run check:server
Tested: bun run check:coverage
Tested: bun run check:policy
Not-tested: Real Windows 10 machine dependency install against the reported user environment
Desktop sessions were missing a visible request_access approval path and could
mis-detect their own app window as an unapproved frontmost target, which caused
Computer Use clicks to fail even after opening the intended app. On macOS, text
entry was also split across inconsistent clipboard and keystroke paths, making
Electron inputs unreliable for Chinese and short strings.
This change adds a desktop approval bridge over the existing session websocket,
renders a dedicated desktop approval modal, threads the real desktop bundle id
into the Computer Use executor, and switches macOS clipboard typing onto the
native pasteboard plus system paste shortcut path. It also makes tool error
results expandable in the desktop chat UI so frontmost-gate failures are fully
visible during debugging.
Constraint: Desktop sessions run the CLI over the SDK websocket path, so Ink tool JSX dialogs are not visible there
Constraint: macOS IME and Electron text inputs are unreliable with pyautogui.write and generic hotkey synthesis
Rejected: Reuse CLI setToolJSX dialogs in desktop mode | no transport for mid-call Ink UI over the SDK bridge
Rejected: Keep shell pbcopy/pbpaste for clipboard typing | inconsistent with NSPasteboard path and less reliable for Chinese text
Confidence: medium
Scope-risk: moderate
Reversibility: clean
Directive: Keep desktop Computer Use approvals and macOS text-entry behavior on a single bridge/path; avoid reintroducing separate CLI-only and desktop-only codepaths for the same action
Tested: python3 -m unittest runtime/test_helpers.py
Tested: bun test src/utils/computerUse/permissions.test.ts src/server/__tests__/conversation-service.test.ts
Tested: cd desktop && bun run test ComputerUsePermissionModal chatStore
Tested: cd desktop && bun run test chatBlocks
Tested: cd desktop && bun run lint
Not-tested: End-to-end manual Computer Use interaction against a live Electron target app on macOS
Bypass all three gating layers (compile-time feature flag, subscription
check, GrowthBook remote config) and replace Anthropic's private native
modules (@ant/computer-use-swift, @ant/computer-use-input) with a Python
bridge using pyautogui + mss + pyobjc. Works on any macOS with any
Anthropic-protocol compatible model.
Relakkes Yang