Skip to content

Local models (colibri)

Itervox runs agent CLIs, and most of them can talk to a model server other than their vendor’s. Point one at a server on your own machine and that profile’s runs cost nothing per token, and your code and prompts never leave the machine.

This guide uses colibri, an open-source engine for very large open models. It serves an OpenAI-compatible API and an Anthropic-compatible /v1/messages. The steps are the same for any server with those APIs.

Follow colibri’s own setup. It checks your hardware and recommends a model. Then start its API server:

Terminal window
coli serve # OpenAI-compatible /v1 and Anthropic-compatible /v1/messages on 127.0.0.1:8000

Set the agent CLI’s base-URL variable at the start of the profile’s command. Itervox skips NAME=value assignments when it works out the backend, so this profile is still the claude backend, with the same flags, resume and parsing.

agent:
profiles:
deps-analyzer-local:
command: ANTHROPIC_BASE_URL=http://127.0.0.1:8000 claude --model <model name colibri serves>
soul_file: .itervox/agents/deps-analyzer/SOUL.md
instructions_file: .itervox/agents/deps-analyzer/INSTRUCTIONS.md
  • Claude Code reads ANTHROPIC_BASE_URL, without /v1. If the server wants a key, put ANTHROPIC_AUTH_TOKEN in .itervox/.env, not in the command.
  • OpenCode talks to any OpenAI-compatible provider you define in opencode.json (base URL http://127.0.0.1:8000/v1). Choose it with --model <provider>/<model>. See Agent backends.
  • Codex reads custom providers from ~/.codex/config.toml, not from the command.

To keep a local profile on this machine, don’t combine it with agent.ssh_hosts. A run on a worker host would look for the server on that host.

A local model is slower than a vendor’s, so raise the timeouts that would otherwise kill a healthy run:

SettingDefaultWhat to set
agent.turn_timeout_ms3600000 (1 h)the longest a whole run may take
agent.stall_timeout_ms300000 (5 min)longer than the slowest gap between two events
agent.read_timeout_ms30000longer than the slowest single response chunk

itervox doctor probes every model server that a profile’s command points at. It lists which profiles use each one, and warns when one does not answer:

Model servers
ANTHROPIC_BASE_URL=http://127.0.0.1:8000, used by deps-analyzer-local: not answering
WARNING: deps-analyzer-local point their agent at http://127.0.0.1:8000, which did not answer (…); their runs will fail until it does — start the model server (for colibri: `coli serve`) or fix ANTHROPIC_BASE_URL

The warning does not change doctor’s exit code, because the server may just not be started yet. A URL taken from a variable ($LOCAL_MODEL_URL) is listed but not checked. Doctor sees only assignments in the command itself, not a CLI’s own config file.

5. Fall back to it when a vendor’s limit is hit

Section titled “5. Fall back to it when a vendor’s limit is hit”

Itervox pauses dispatch to a backend whose usage limit was hit, and can reroute work through agent.backend_fallback. Give the local profile its own pool, so a limit on Anthropic doesn’t hold it, and add the pool as a step in the chain:

agent:
profiles:
coder:
command: claude
coder-local:
command: ANTHROPIC_BASE_URL=http://127.0.0.1:8000 claude --model <model>
pool: local
backend_fallback:
chain: [claude, codex, claude/local]
profile_map:
coder: { "claude/local": coder-local }

When claude is limited and nothing else in the chain is healthy, coder issues run on coder-local. They switch back once the limit resets, using the usual switch_back policy.

The desktop app watches these servers for you. It reads each running daemon’s list of model servers (GET /api/v1/model-servers) and merges them across projects. The tray menu gets a Model servers section:

Model servers
local 127.0.0.1:8000 — answering · used by 2 projects
GPU: NVIDIA GeForce RTX 4090 · 37% · 10.2/24.0 GB

A server is named after its profiles’ pool, or “model server” when they set none. The end of the project tabs shows the same list (hover it). The app probes each server at most every 30 seconds, with a 3-second timeout, and any HTTP answer counts, as in doctor. It notifies you when a server that was answering misses two probes in a row, and again when it answers.

The app only probes 127.0.0.1, localhost and ::1. It shows these servers as not checked, with the reason:

  • a URL from a variable;
  • any other host;
  • a project with agent.ssh_hosts, whose runs go to the worker hosts.

The GPU line comes from nvidia-smi when it is on your PATH. On Apple silicon the app shows the integrated GPU, without its load. Otherwise it shows “not reported”. CPU, memory and disk are on the dashboard’s machine status.

A local model does not make a run safer. The agent CLI runs with the same permissions whichever model it talks to. A run with isolation: process keeps network access, so it can still reach the server on 127.0.0.1. See Safety.