aq.dev / docs/background-workers

Running background workers on AQ

A background worker is a long-running process that belongs to a work item: a training job, a queue consumer, a recovery loop, a watchdog. AQ launches it, keeps it running, restarts it when it fails and starts it again when its machine restarts. It does not depend on an agent's turn, a terminal tab or a browser being open.

Start one

  • Ask the agent. Every work item thread has the tools worker_start, worker_status, worker_logs, worker_stop and worker_delete. Say "run the snapshot worker in the background" and the agent starts it as a worker instead of a command in its shell.
  • In the app. Changes panel, Run dock, Workers tab: + Worker, a name, the command and a restart policy. Each row shows its state, restarts, who started it, and its output.
  • From the API. POST /v1/workers with workspace_id, name and command; then GET /v1/workers/{id}, GET /v1/workers/{id}/logs, POST /v1/workers/{id}/stop|start|restart and DELETE /v1/workers/{id}. See the API docs.

What a worker gets

It runs in the work item's checkout (or a folder inside it), as the person who started it, with the same environment a terminal there has: your secrets as environment variables, the preview addresses and the work item's ports, plus AQ_WORKER_ID, AQ_WORKER_NAME and AQ_WORK_ITEM_ID. Each worker is one process tree with its own memory limit (75% of the machine by default); stopping it stops everything it started. Its output (stdout and stderr, with timestamps) is kept on the machine across restarts, and saved secret values are masked when AQ shows it.

Restart policy and health checks

  • on-failure (default): restarted 5 seconds after a crash or a non-zero exit, with no limit on retries. A clean exit (code 0) ends it.
  • always: restarted after any exit.
  • never: runs once.

An optional health check runs every minute or more: an HTTP request to a port the worker listens on, or a command whose exit code says healthy. After three failures in a row (configurable, or 0 to only report) AQ restarts the worker.

Attribution

Every worker records who created it and who last started it, from where (an agent, the API or the app) and from which thread. Starts, stops and deletes are in the organization's audit log.

What a worker survives

EventWorker
The agent's turn ends, the thread stops or restsKeeps running
Nobody has the work item openKeeps running
An AQ deploy or a runner updateKeeps running (it is not a child of AQ's processes)
The process crashesRestarted by its policy
The machine restarts (resize, maintenance, recovery)Started again within about a minute of the machine reconnecting
The work item moves to another machineStopped on the old machine and started on the new one
The work item is closedStopped

By plan and machine

  • Pro, AQ-managed machines. The machine stays on; workers run around the clock.
  • Your own Linux VM. The same supervision, for as long as you keep the VM running. It needs systemd, which every supported Linux has.
  • Free. A running worker counts as activity, so the machine is not put to sleep while one runs, for up to 8 hours with nobody using AQ. After that the machine sleeps: the worker is paused in memory and continues where it was when the machine wakes. A machine asleep for 45 days is archived; when it comes back, AQ starts its workers again.
  • Your Mac. Workers are not available on a Mac (it sleeps when its lid closes and has no service manager AQ can use). Move the work item to a cloud machine to run them.

Processes that are not workers

  • A command an agent runs in its shell, including one sent to the background with & or nohup: may be stopped when the agent is stopped or its run ends, is never restarted, and does not survive a machine restart.
  • A process in a terminal tab: ends with the terminal, which closes 15 minutes after nobody is looking at it.
  • The preview's dev server: runs while you use the preview and is started again when you open it. Use it for what you look at, and a worker for what must keep running.

Triggering agent runs from outside

A worker is a process. To have an outside system start an agent with a prompt and settings, use one of these:

  • A routine with a webhook trigger. A bot routine gets its own URL and secret; each delivery runs the routine's instruction with the delivery's body passed along as data, on the routine's model, budget and permissions. Schedules, Linear, GitHub, Gmail and Calendar triggers work the same way.
  • The API. POST /v1/workspaces creates a work item and POST /v1/workspaces/{id}/agent starts an agent on it with your message and settings (model_selection_id or a connected subscription, effort, speed, crew_id). Follow it with GET /v1/agent-sessions/{id}/events, send more with POST /v1/agent-sessions/{id}/messages, and register a signed webhook for its events, such as a pull request opening.

Coming from Conductor

  • Code that asks Conductor to launch and check workers maps to /v1/workers: create and start, status, logs, stop and restart, with the platform keeping the process alive.
  • A scheduled or webhook routine maps to a bot routine with the same trigger and instruction.
  • An API call that starts an agent with a prompt maps to POST /v1/workspaces/{id}/agent.
  • Setup and run scripts from .conductor/settings.toml or conductor.json are offered, when you connect the repository, as its setup command and services for an admin to apply.