Skip to content

System visibility — what each run costs and does

Itervox already knows every agent run’s process group: it starts each agent turn and each hook in its own group (and, on Windows, its own Job Object). System visibility watches those groups and answers three questions without opening a terminal:

  • What is this run costing the machine? CPU, memory, process count, disk I/O and open network connections, with the busiest child commands.
  • What did this run do? The commands it started, the files it wrote or had open, and the network connections it made.
  • Is the machine all right? Load, memory pressure and free disk on the volume that holds the workspaces, with a warning (and optionally a dispatch pause) when the disk runs low.

Everything is on by default. Nothing here leaves the machine except through the daemon’s own API.

An issue in full view: the Now panel with CPU, memory, processes and disk written, the run's commands, and the Activity audit with commands, files and connections An issue in full view: the Now panel with CPU, memory, processes and disk written, the run's commands, and the Activity audit with commands, files and connections
A run's resource use and activity audit in the issue view.

Each local run is sampled at most every 2 seconds by one sampler goroutine in the daemon (never on the orchestrator’s event loop). A sample covers the run’s whole process tree: the members of its process groups, their leaders, and every descendant — including a child that started its own session.

FigureMeaning
CPUPercent of one core over the last interval, summed over the tree (200% = two busy cores). The total CPU time is kept too.
MemoryResident memory (RSS) summed over the tree, with the run’s peak. Pages shared between processes count once per process.
ProcessesLive processes in the tree, with the peak.
Disk written / readStorage I/O of the tree since the run started (Linux and Windows; macOS does not report it per process).
ConnectionsOpen TCP/UDP connections of the tree (listening sockets are not counted).
Top child commandsThe five busiest processes: name, arguments, CPU and memory.

Where you see it:

  • the issue’s Now panel (issue detail and the issue view) while the run is live, and its last sample after it finishes;
  • resources on GET /api/v1/issues/{identifier}/context and on each running[] row of GET /api/v1/state;
  • resources on each run-history row: CPU time, peak CPU, last and peak memory, peak process count, disk I/O and activity counts, kept with the history file;
  • Prometheus gauges itervox_run_cpu_percent, itervox_run_memory_rss_bytes and itervox_run_processes (labelled by issue identifier, live runs only);
  • the desktop tray: each project’s line reads, for example, running · 3 agents · 1.2 GB.

Totals are monotonic: a process that exits keeps the CPU time and I/O it was last seen with. Work a process does between its last sample and its exit is not counted.

An SSH worker runs its agent on another host: only the local ssh client is visible, and the panel says so.

Command lines are redacted before they are kept or served: the log redaction (known key shapes, bearer tokens, high-entropy strings), plus the values of NAME=value assignments, of flags named like credentials (--token=…, --password …, --api-key …) and URL credentials. A viewer token sees each child command’s name only; arguments need an operator token.

Turn sampling off with agent.resource_sampling: false.

The Activity section of the Now panel (collapsed until you open it) lists, for the issue’s latest run or a run you pick:

  • Commands started by the run, each distinct command line once with a count;
  • Files it wrote (and, in sampled mode, other files it had open), with outside marked on a write outside the run’s workspace and a warning chip counting them;
  • Connections: the remote address, protocol, state and the command that made it.

A filter narrows the lists. The audit is served by GET /api/v1/issues/{identifier}/activity?run=<runId>, an operator route: command lines, paths and remote addresses say a lot about your code and machine even after redaction. A viewer sees a note instead.

agent.activity_audit chooses how activity is recorded.

ModeLinuxmacOSWindows
sampled (default)Commands, files (open files from /proc/<pid>/fd, write or read from fdinfo), connections (sockets matched to /proc/net/tcp, tcp6, udp, udp6 of the run’s network namespace)Commands, connections. No files: reading another process’s file activity needs Apple’s Endpoint Security entitlement.Commands, connections. No files.
preciseEvery exec, every openat for writing (O_WRONLY, O_RDWR, O_CREAT, O_TRUNC) and every connect of the run’s process tree, from eBPF tracepointsfalls back to sampledfalls back to sampled
offnothing is recorded

The badge next to Activity names the mode. sampled means what it says: anything that starts and ends between two samples, 2 seconds apart, can be missed — a quick curl, a file written and closed at once. Use precise on Linux when you need every event.

Precise mode loads a small eBPF program (assembled in the daemon; no compiler or kernel headers are needed) on the sched_process_exec, sched_process_fork, sched_process_exit, sys_enter_openat and sys_enter_connect tracepoints. A run’s group leaders are added to a kernel map as they start and the kernel adds every process they fork, so only the runs’ own process trees produce events. Field offsets are read from tracefs at start, so the program follows the running kernel’s layout.

It needs:

  • CAP_BPF and CAP_PERFMON (kernel 5.8+), or root — for example sudo setcap cap_bpf,cap_perfmon+ep "$(command -v itervox)";
  • tracefs mounted at /sys/kernel/tracing (or /sys/kernel/debug/tracing);
  • a kernel with BPF ring buffers (5.8+) and syscall tracepoints (CONFIG_FTRACE_SYSCALLS).

When any of that is missing the daemon logs the reason and samples instead; the Activity notes and system.activityModeReason on /api/v1/state say why, for example precise mode unavailable: tracefs is not mounted at /sys/kernel/tracing. Precise mode lists files the run wrote, not files it only read. A relative path is resolved against the process’s working directory when it is still alive.

Runs under process isolation use sandbox-exec on macOS. Its deny events go to the unified system log, which the daemon does not read yet, so they are not part of the activity audit.

Activity is runtime state under .itervox/runtime/activity/ (gitignored): one JSON file per run and an index. Per run at most 500 distinct commands, 1000 files and 300 connections are kept (further distinct entries are counted as dropped and the panel says how many); the directory keeps the 200 newest runs and at most 64 MiB, dropping the oldest first. A live run’s file is rewritten at most every 15 seconds while it changes, and once when the run ends.

The daemon also reads, every 10 seconds:

  • the load averages (none on Windows) and CPU count;
  • memory: total and available, and a pressure level — ok, moderate or high — from Linux PSI (/proc/pressure/memory, share of time tasks stalled on memory) where available, else from the share still available;
  • the free space on the volume that holds workspace.root.

They are the system block of GET /api/v1/state (with the live runs’ totals and the activity mode), Prometheus gauges (itervox_host_load1, itervox_host_memory_total_bytes, itervox_host_memory_available_bytes, itervox_workspace_disk_free_bytes, itervox_workspace_disk_low) and a rounded system block on GET /api/v1/summary for the desktop tray.

workspace:
disk_warn_free_gb: 5 # default; 0 = no warning
disk_pause_free_gb: 2 # default 0 = never pause

Below disk_warn_free_gb the dashboard shows a warning banner and the desktop tray a line under its header (⚠ web-app: low disk — 3.2 GB free). Below disk_pause_free_gb the daemon also pauses all dispatch with the regular dispatch pause (reason Low disk space: …), so running agents finish and nothing new starts; it resumes on its own once free space is back above the threshold plus 0.5 GB. It only ever resumes its own pause: a pause you set by hand stays.

KeyDefault
agent.resource_samplingtruePer-run resource sampling.
agent.activity_auditsampledoff, sampled or precise.
workspace.disk_warn_free_gb5Low-disk warning threshold.
workspace.disk_pause_free_gb0Pause dispatch below this.

All four are read at startup and can live in a settings file; see the configuration reference.