Health and auto-restart

How the orchestrator probes engines, detects failures, and restarts with backoff.

Engines fail. Containers crash. A network blip drops a probe. The orchestrator runs a background health loop that detects failures quickly and restarts with exponential backoff. This page covers the loop and how to tune it.

The loop

Configuration

The loop is tuned via env vars. Defaults are sensible for most deployments; tune for special cases.

Variable	Default	Effect
ORCH_HEALTH_CHECK_INTERVAL_S	30	How often the loop sweeps every engine. Lower = faster failure detection, more probe traffic.
ORCH_HEALTH_CHECK_TIMEOUT_S	10	Per-probe timeout. Should be < interval.
ORCH_HEALTH_MAX_FAILURES	3	Consecutive failed probes before marking failed. Higher = tolerates more flakes.
ORCH_RESTART_BACKOFF_BASE_S	5	First retry waits this long. Doubles each attempt.
ORCH_RESTART_BACKOFF_MAX_S	300	Cap on backoff (5 min).
ORCH_RESTART_MAX_ATTEMPTS	8	Give up auto-restart after this many failures.
ORCH_BOOT_TIMEOUT_S	60	Provision-time boot deadline (separate from steady-state).

What “healthy” means

The orchestrator probes GET /health on the engine. The engine must return:

{ "status": "ok", ... }

Any of these counts as a failure:

• HTTP timeout (connection or read).
• Non-2xx status code.
• 200 status but status != "ok" in the body.

The engine’s /health is implemented in engine/src/server.py. It checks: database reachable, LLM provider reachable (last call succeeded), Asset Directory reachable. If any subsystem is down, the body’s status is non-"ok" and the orchestrator counts it as a failure.

What “failed” means

After ORCH_HEALTH_MAX_FAILURES consecutive failures, the orchestrator:

1. Sets status='failed' in the registry.
2. Writes an audit row with action health_failed.
3. Schedules an auto-restart for BACKOFF_BASE * 2^0 seconds later (5s by default).

A failed engine is invisible to the product unless it explicitly queries — GET /admit returns admitted: false with reason engine_unhealthy. If auto_provision=true is passed and policy allows, a new engine takes its place; otherwise the product sees the failure.

Auto-restart

The orchestrator restarts a failed engine by calling the backend’s start(). For Docker, this is docker.containers.start().

After each restart attempt:

• If GET /health returns ok within ORCH_HEALTH_CHECK_TIMEOUT_S, reset to status='running', health_failures=0. Audit auto_restart_success.
• If the restart itself fails, or /health keeps failing, increment the attempt counter and schedule the next retry with doubled backoff.

After ORCH_RESTART_MAX_ATTEMPTS (default 8) consecutive failed restarts, the orchestrator stops trying. The engine stays failed. Audit auto_restart_gave_up. Ops alerts should fire here.

The total elapsed time before giving up:

sum(min(5*2^i, 300) for i in range(8))
= 5 + 10 + 20 + 40 + 80 + 160 + 300 + 300
= 915 seconds (~15 min)

So a truly broken engine spends about 15 minutes thrashing before the orchestrator quits. Tune down for smaller fleets where you’d rather page sooner; tune up for large fleets where transient provider outages would otherwise destroy half the fleet.

What auto-restart fixes

• Container OOM kills.
• LLM provider transient outage (the engine’s /health will go ok again once the provider recovers).
• Brief network partitions.
• Engines that crashed during a deploy and haven’t been bumped.

What it doesn’t fix

• Misconfiguration. If LLM_API_KEY is wrong, restarting won’t help.
• Disk full. Restart hits the same disk error.
• Deeper bugs that surface every time. Restart loops them.

For these, the auto-restart eventually gives up. The audit table tells you the engine is hopelessly stuck; you investigate.

Concurrency

The health loop runs all probes for one tick in parallel up to a semaphore of 50. For larger fleets, the per-tick wall time is (num_engines / 50) × ORCH_HEALTH_CHECK_TIMEOUT_S. With 1000 engines and a 10s timeout, that’s 200 seconds per sweep — uncomfortable when the interval is 30s.

If you run >500 engines, raise the semaphore (a code change today; config in a future version) or increase the interval to give the loop time to finish.

Tuning by use case

Demo / small fleet

ORCH_HEALTH_CHECK_INTERVAL_S=10
ORCH_HEALTH_MAX_FAILURES=2
ORCH_RESTART_MAX_ATTEMPTS=5

Fast detection, fast give-up. Engineers see failures within seconds.

Production / large fleet

ORCH_HEALTH_CHECK_INTERVAL_S=30
ORCH_HEALTH_MAX_FAILURES=3
ORCH_RESTART_BACKOFF_BASE_S=10
ORCH_RESTART_MAX_ATTEMPTS=8

Tolerant of transient noise. Doesn’t thrash on provider blips.

Provider outage tolerance

ORCH_HEALTH_MAX_FAILURES=5
ORCH_RESTART_BACKOFF_MAX_S=600
ORCH_RESTART_MAX_ATTEMPTS=12

Survives a 30-minute provider outage without destroying engines. Slower to restart genuinely-stuck engines.

Observability

Watch:

• engines.health_failures — sustained > 0 across many engines means a systemic issue (provider, network).
• Audit auto_restart_* actions — high rate = thrashing.
• engines.status='failed' count over time — should approach 0 in steady state.

Wire alerts to:

• Restart rate > 5/min (Sev2).
• Failed engines > 1% of fleet sustained 5 min (Sev2).
• Single engine in failed for > 1 hour (Sev3).

See also

• Lifecycle — where the failed state fits in the state machine.
• Architecture — the implementation.
• Audit — the audit record of every restart.
• Config: env vars — every knob.