unix_computer_use

Files (2)

• SKILL.md
• plugin.py

SKILL.md (first lines)

# Unix Computer Use

This bundled extension exposes a reviewable computer-use substrate for local
macOS/Linux desktops and explicitly configured remote targets. **All actions are
for supervised, low-risk workflows under human or benchmark observation.** It is
offered and on by default on macOS and Linux (`os`); Windows desktops use
`windows_computer_use`. A previously enabled Unix remote skill stays available
on Windows through later OFF/ON. Actor-less legacy ON is retained too: manual
and early automatic choices cannot be distinguished. This adds no local Windows backend.

Tools:

- `capabilities` reports the platform, display-session type, who is in front
  (local and remote), the OS grants the local process holds (prompt-free
  preflights on macOS: Accessibility, Screen Recording, input events;
  `permission_state_verified` is true only when they ran; a remote machine's
  grants are not probed), whether `app=` targeting works in this session and
  what it is resolved from, and the detected screenshot/input backends plus
  the active connection.
- `list_connections`, `add_connection`, `test_connection`, `activate_connection`,
  and `use_local` manage the local/remote connection registry in skill state.
- `screenshot` captures the main display (or, on macOS with `app=`, the display
  holding that application's front window), downscales the image to fit WXGA
  (1280x800, configurable) and persists the exact image-to-input coordinate
  transform, including the display's origin. `display.selected_by` says how the
  display was chosen: `default`, `app`, or `fallback` when the named window
  could not be located (`app_window_unread` says why) and the main display was
  captured instead; the result then carries no `app`. Linux and the remotes
  capture their one screen and ignore `app=` (the result says so). Pass
  coordinates read off the returned image directly to the input tools — they
  are remapped automatically. The result carries the typed `auto_attach_image`
  field, so the host attaches the image to the conversation in the same round;
  an engine that cannot take images receives the path instead (host behaviour),
  so the text facts in every other tool are the way to check without a picture.
- `click` (left/right/middle, double/triple), the thin aliases
  `double_click`/`triple_click` (models call these names unprompted), `move`,
  `left_click_drag`, `mouse_down`/`mouse_up`, `cursor_position`, `type_text`,
  `key`, `hold_key`, `scroll`, and `wait` execute input through platform tools
  when available. Pointer coordinates pass through one normalizer that accepts
  the malformations models actually emit (the pair packed into `x` with `y`
  absent or duplicated) and fails loudly on anything ambiguous.
- `activate_app` brings a running application to the front (name, bundle id /
  WM_CLASS, or `pid:<n>` as `window_list` prints it; resolved from the running
  list, never launched) and reports who is in front afterwards.
- `window_list` lists running applications with their windows: who is in
  front, titles, bounds, pid, and on macOS whether the application has its own
  scripting commands (`scriptable`); `app=` filters, `limit` caps the rows.
- `ax_tree` reads one window as text through the Accessibility API, bounded:
  the frontmost application's window, or `app=`'s without bringing it forward.
- `remote_exec` runs a shell command on the active remote backend only; it
  refuses on local backend. Use ordinary Ouroboros shell/file tools for local
  work.

Acting in a named application (`app=`):

- `click`, `double_click`, `triple_click`, `left_click_drag`, `mouse_down`,
  `type_text`, `key`, `hold_key` and `scroll` take an optional `app`: a running
  application's name (case-insensitive), its bundle id or WM_CLASS, or
  `pid:<n>` copied from `window_list`; on X11 and OSWorld also a window's exact
  title or its `0x…` id. Resolution is exact: no prefix or substring guessing
  (on X11 and OSWorld a substring hit is returned as a candidate, so the exact
  name can be given; macOS and SSH answer with the running list; the one
  looser form is the WM_CLASS rule in the X11 row below, where `wmctrl` joins
  instance and class with a dot).
  On X11 and OSWorld the target is a set of windows: the one window for a
  title or id, every window of that WM_CLASS for a class (a process may host
  several classes, as LibreOffice's Writer and Calc do), every window of the
  process for `pid:`; "in front" means the active window is one of them. The
  tool resolves the name against the running list without launching anything,
  brings the application forward when it is not already in front, reads who is
  in front again, and only then sends the input. Two matches are
  a refusal that lists the candidates; an application that is not running is
  a refusal that lists the running applications; one that did not come to the
  front (macOS 14+ may refuse activation from a background process) is a
  refusal that names who is in front; a front that could not be read after
  the switch is the refusal `front_unread`.
  A refusal sends nothing and carries no effect mark (a switch it observed
  first keeps its own, `activation_done`). An empty `app` (what a voice model
  sends for an unused optional field) means "untargeted": the input goes to
  whatever is in front, as before.
- A `key` sequence (`'down down return'`) brings the target forward once and,
  before every later step, reads the front again: when the named application
  (or window) has left the front (a `cmd+tab` in the sequence, the owner
  switching away, a sibling window of the same process coming forward for a
  title-named target, a window of the class that was not listed with a
  class-named target, such as a dialog a step opened) the remaining steps are
  not sent (`target_lost`,
  `steps_done` says how many ran); a switch that ran before a step that could
  not be sent keeps its mark (`activation_done`). Without `app=` every step
  goes to whatever is in front at that moment, and the receipt names each
  step's recipient when they differ (`steps_delivered_to`, `"unknown"` for a
  step whose front could not be read). On a sequence that stopped, `dispatched`
  is the fact for the step that failed (`steps_done` counts the steps that ran
  before it; a step whose remote call did not answer is `null`, and the note
  says so, never "not sent"). A tool's registered budget covers one
  input's worst case (`type_text` 180 s, the other inputs 90 s, `move` and
  `mouse_up` 60 s), not a long
  sequence: when the host's budget ends a call, no receipt comes back.
- `move` and `mouse_up` never change which application is in front: a drag
  must end where it started.
- Keyboard focus follows the switch and stays there, as if the owner had
  clicked the application; nothing moves it back. Do this only when the owner
  asked for an action in that application (owner decision 27.10.2026).
- Where an application has its own commands (macOS: `scriptable: true` in
  `window_list`; AppleScript/JXA through `run_command`/`run_script`, e.g.
  `osascript -e 'tell application id "com.apple.iWork.Keynote" to show next'`
  while a presentation plays, or `set current slide of front document to
  slide 2` in the editor), prefer them: they do not depend on which window is
  in front and they can read the state back (`slide number of current slide`).
  A keystroke is the right instrument only where no such command exists, and
  the key that means "next" differs per application and per mode (Keynote
  advances on Right Arrow while presenting, Down Arrow in the editor's
  navigator).

Coordinate contract:

- Input tools accept coordinates in the LAST screenshot's image space and
  remap them through the stored transform (scale and display origin);
  `raw=true` bypasses remapping.

From razzant/ouroboros, MIT.

More from razzant/ouroboros