Computer Use

Computer use lets a TF Code agent control the real machine the backend runs on — moving the mouse, clicking, typing, and reading the screen. The agent sees screenshots and emits pixel coordinates to drive a screenshot → act → screenshot loop, exactly like a human watching the screen and reacting.

It works across every frontend: the TUI, the web/desktop app, the CLI (tfcode run), and headless server mode.

note

Computer use controls the actual OS GUI on the host running the tfcode backend. It is not a sandboxed browser VM — the agent drives your real mouse, keyboard, and applications. See the safety model for the guardrails.

Requirements

The backend shells out to OS-native tools (no native npm modules), so requirements differ per platform:

Platform	Input tool	Screenshot tool	OS permissions
macOS	`cliclick` (`brew install cliclick`, or bundled in some installs)	built-in `screencapture`	Accessibility + Screen Recording granted to the terminal/app running tfcode
Windows	PowerShell + a compiled `SendInput` helper (auto-compiled & cached)	`System.Drawing.CopyFromScreen`	None at normal integrity
Linux (X11)	`xdotool`	`maim` / `scrot` / `import` (ImageMagick)	X11 access
Linux (Wayland)	`ydotool` / `wtype`	`grim` / `gnome-screenshot` / `spectacle`	Wayland session + `ydotool` uinput access

Screenshots are resampled to a longest edge of 1280px before being sent to the model, so the dimensions the model is told always match the image it actually sees. Override with OPENCODE_COMPUTER_USE_IMAGE_MAX.

First-time setup

Prepare the machine in one step — this probes OS tools and permissions, performs safe side effects (like pre-compiling the Windows input helper), and tells you exactly what to install if anything is missing:

tfcode computer setup

setup detects your platform and (on Linux) your distro's package manager, then prints the exact install command for any missing tools. Pass --install to run it automatically — sudo will prompt for your password on Linux:

tfcode computer setup --install

tip

On macOS, tfcode computer setup probes Accessibility and Screen Recording, which triggers the OS permission dialogs so you can click Allow right away. Those two permissions can't be pre-installed or auto-granted — they always require a one-time click in System Settings → Privacy & Security.

Then verify everything is ready:

tfcode computer doctor

Capture a single screenshot to confirm end-to-end capture works:

tfcode computer shot              # writes ./tfcode-shot.png
tfcode computer shot --out ~/Desktop/check.png

Enabling

Computer use is off by default. Enable it per session or process-wide — the computer tool and the system-prompt guidance only appear when it is on.

Web & Desktop app

Click the computer-use icon in the prompt toolbar, or press Ctrl+Alt+C. The toggle is always available — no environment variable needed. When on, a computer chip appears in the prompt footer and computerUse is sent with every prompt in that session. The setting persists across reloads.

TUI

Run the slash command:

/computer-use

(Alias: /toggle-computer-use.) It pre-flights the machine first — if OS tools or permissions are missing, it refuses to enable and points you to tfcode computer setup. When ready, it flips computer use for the current session and shows a toast. A computer chip appears in the prompt footer when active.

You can also bind a key for it. Add the keybind in tui.json under keybinds (it defaults to none, so it is slash-only out of the box):

{
  "keybinds": {
    "computer_use_toggle": "ctrl+alt+c"
  }
}

See Commands for all slash commands and shortcuts.

CLI

Pass --computer to a single run (works with both a free-form prompt and a named command):

tfcode run --computer "Open System Settings and turn on Dark Mode"
tfcode run --computer --command deploy "Run the deploy script"

This wires computerUse into the prompt/command call for that run only.

note

--computer pre-flights the local machine before starting. If OS tools or permissions are missing, it prints the exact items to fix and exits without starting the session — run tfcode computer setup first. Remote --attach runs control the remote machine, so the local pre-flight is skipped.

Server mode & non-interactive runs

For headless server mode and scheduled jobs, set the environment variable so computer use is on for the whole process:

OPENCODE_ENABLE_COMPUTER_USE=true tfcode serve

See the environment variable section below.

How it works

Once enabled, the model gets a computer tool. It calls it with an action and (for most actions) a target coordinate. After every action the tool returns a fresh screenshot as an image attachment, so a vision model observes the result and continues:

The agent calls computer with action: "screenshot" to see the screen.
It reads the screen, picks a target pixel, and emits an action (e.g. left_click at [420, 180]).
The tool performs the action on the real OS and returns a new screenshot.
Repeat until the task is done.

Coordinates are absolute screen pixels measured from the top-left corner (0, 0). The screenshot you receive and the Screen is WxH line share the same pixel dimensions — that is your coordinate space. Do not apply a DPI/Retina multiplier or convert between "image space" and "screen space"; the backend handles scaling transparently.

Each screenshot result also reports key UI landmarks (menu bar, front window, dock on macOS) as pixel rects to help the model orient.

Actions

Action	Description	Needs coordinate
`screenshot`	Capture the screen	no
`cursor_position`	Return the current mouse position	no
`mouse_move`	Move the pointer to a pixel	yes
`left_click`	Single left click	yes
`right_click`	Single right click	yes
`double_click`	Double click	yes
`middle_click`	Middle click	yes
`left_click_drag`	Drag from `start_coordinate` to `coordinate`	both
`scroll`	Scroll the wheel	yes + `scroll_direction` / `scroll_amount`
`type`	Type a string of text	no (`text`)
`key`	Press a key combo (e.g. `ctrl+s`, `return`)	no (`text`)
`hold_key`	Hold a key combo for `duration` ms then release	no (`text` + `duration`)
`wait`	Pause for `duration` ms (100–10000)	no
`left_mouse_down` / `left_mouse_up`	Hold/release the left button	yes
`right_mouse_down` / `right_mouse_up`	Hold/release the right button	yes

hold_key keeps a combo pressed across a following action — e.g. hold_key "shift" then left_click to extend a selection, or hold_key "ctrl" to multi-select. On macOS it supports only modifier combos (ctrl, shift, alt, super/cmd, fn); to hold a regular key on macOS, use repeated key calls instead.

Multi-display

If you have more than one monitor, computer use captures and operates on a single selected display — by default the primary. You can switch to whichever monitor you actually want the agent to drive (e.g. keep tfcode on one screen and let the agent work the other).

List detected displays:

tfcode computer display

This prints each display's id, name, and origin, marking the active one:

  1  Display 1 (main) — 1728x1117  origin (0,0)  (active)
  2  Display 2 — 2560x1440  origin (1728,0)

Choose one — the choice persists across restarts:

tfcode computer display --use 2

Revert to the primary display:

tfcode computer display --reset

tfcode computer doctor also lists displays when more than one is detected.

TUI — run /display (alias /monitor) to cycle to the next monitor.
Web & Desktop app — the computer-use dialog shows a display picker when more than one monitor is detected.

When a specific display is selected, the screenshot the agent receives is that monitor only, and its coordinate space is local to that monitor — top-left (0,0) is the top-left of the chosen display, not the global desktop. The backend translates coordinates to the monitor's global origin transparently, so the agent reads coordinates straight off the returned screenshot as usual.

Safety model

Computer use drives the real OS, so it is gated by several guardrails:

Permission-gated — the computer permission key defaults to ask for every agent, including workspace agents loaded with *: allow. Destructive actions (clicks, typing, keys, drags) prompt for confirmation every time. Read-only actions (screenshot, cursor_position, mouse_move) are allowed without prompting. See Permissions.
Auto-accept when opted in — when you enable computer use via a toggle, --computer, or the env flag, the per-step confirmation is skipped because you have already opted in.
Input mutex — only one computer-use operation owns input at a time, so parallel agents/subagents can't fight over the real mouse and keyboard.
Per-session action budget — a runaway loop is stopped after 100 actions per session (OPENCODE_COMPUTER_USE_BUDGET).
Audit log — every action is appended as JSONL to ~/.local/share/tfcode/computer/audit.jsonl.
Abort/panic — interrupting a session releases any held keys/buttons immediately.

Configuration

Environment variables

Variable	Default	Description
`OPENCODE_ENABLE_COMPUTER_USE`	unset (off)	Enable computer use process-wide for all frontends and sessions
`OPENCODE_COMPUTER_USE_IMAGE_MAX`	`1280`	Longest edge (px) screenshots are resampled to before sending
`OPENCODE_COMPUTER_USE_BUDGET`	`100`	Max computer actions per session before the loop is stopped

Environment variable

OPENCODE_ENABLE_COMPUTER_USE=true is the process-wide switch. It turns computer use on for every session regardless of frontend, auto-accepts all actions, and is the only path that works for:

tfcode run --command (the generated SDK can't carry computerUse on that path yet)
non-interactive and scheduled runs (where there is no UI to toggle and no one to confirm prompts)

For interactive use you usually do not need it — the per-session toggles and --computer flag are enough.

Permission key

Control the computer tool with the computer permission key in config:

{
  "permission": {
    "computer": "ask"
  }
}

Set it to "deny" to disable computer use for an agent entirely, or per-agent:

{
  "agent": {
    "build": {
      "permission": {
        "computer": "ask"
      }
    }
  }
}

See Permissions for the full permission model.

Troubleshooting

Run the doctor first — it pinpoints what's missing:

tfcode computer doctor

macOS — cliclick not found: install it.

brew install cliclick

macOS — Accessibility / Screen Recording denied: grant the permission to the terminal or app running tfcode in System Settings → Privacy & Security, then restart tfcode. The doctor probes these and reports them as ✗ until granted.

Linux — tools missing: install the stack for your session type.

# X11
sudo apt install xdotool maim scrot

# Wayland
sudo apt install ydotool wtype grim

Coordinates are off / clicks land in the wrong place: this is almost always a coordinate-space mismatch. Ensure you are on a current build — screenshots are resampled to ≤1280px so the reported size matches the image the model sees. The agent should read coordinates straight off the returned screenshot and not apply any scaling. To send larger screenshots, raise OPENCODE_COMPUTER_USE_IMAGE_MAX. With a multi-display setup, also confirm you're operating on the intended monitor (tfcode computer display).

Can't list or choose displays (tfcode computer display shows none): display enumeration needs a helper. On macOS that's swiftc (install Xcode Command Line Tools with xcode-select --install); on Linux X11 use xrandr, on Wayland use wlr-randr. Without it, computer use falls back to capturing the whole composited desktop.

Agent keeps looping / hit the action budget: the per-session budget (default 100) stops runaway loops. Raise it with OPENCODE_COMPUTER_USE_BUDGET, or interrupt the session (Escape in the TUI) and refine the task.

Audit trail: inspect what the agent did.

tail -n 50 ~/.local/share/tfcode/computer/audit.jsonl

Requirements​

First-time setup​

Enabling​

Web & Desktop app​

TUI​

CLI​

Server mode & non-interactive runs​

How it works​

Actions​

Multi-display​

Safety model​

Configuration​

Environment variables​

Environment variable​

Permission key​

Troubleshooting​