Deploy multi-replica high availability¶

This guide walks through running Repod as multiple backend-api replicas — first the minimal active-passive setup (survives a replica dying, no extra config), then active-active job state (Redis) so scan/install/mirror/sync jobs and the live event/log streams work regardless of which replica a request lands on. For the concepts and mechanisms behind both modes, see Architecture — High availability.

1. Prerequisites¶

Both modes below need the same two things in place first — they are not optional extras, they are what makes running more than one backend-api replica safe at all:

An external, HA PostgreSQL endpoint. DATABASE_URL on every replica must point at the same database — a VIP/endpoint in front of a Patroni/pgpool cluster, RDS Multi-AZ, Cloud SQL HA, or equivalent. Do not point replicas at the bundled db service (a single container, itself a single point of failure). The SQLAlchemy pool (pool_pre_ping=True, pool_recycle=1800) already handles reconnecting after a Postgres-side failover. See External PostgreSQL (install, harden, HA, backup) for how to actually build that endpoint — replication, automated-failover tooling (Patroni/repmgr/pg_auto_failover), and how to present it to backend-api as one stable connection string.
/repos shared read-write across every replica. All package artifacts, manifests, the GPG keyring, staging, audit logs, and settings.json live under /repos/ — every replica needs the identical tree, not a per-host copy. Use NFS, EFS (AWS), Filestore (GCP), or an equivalent shared filesystem mounted on every Docker host running a replica.

Both prerequisites are required for active-passive too

Even the simplest multi-replica deployment — no Redis, nothing else configured — needs both of these. Leader election alone does not remove the need for a shared database and a shared /repos; it only decides which replica runs the scheduler.

2. Minimal setup: active-passive¶

Active-passive requires no new environment variable and no extra service. Point every replica's DATABASE_URL at your HA PostgreSQL endpoint, mount the shared /repos on every host, and start as many backend-api containers as you want. Each replica elects leadership automatically on startup via a PostgreSQL advisory lock (backend/services/leader_election.py) — no coordination step is required from you.

Step 1 — point every replica at the shared database and /repos

export DATABASE_URL_HA=postgresql://repod:CHANGE_ME@pg-ha-endpoint:5432/repod
export REPOS_NFS_MOUNT=/mnt/repod-nfs   # already mounted on every Docker host

Step 2 — start replicas using the documented overlay

docker-compose.ha.yml at the repository root is a reference overlay that disables the bundled db service, points backend at DATABASE_URL_HA, and swaps the local ./repos bind mount for the shared NFS/EFS mount:

docker compose -f docker-compose.yaml -f docker-compose.ha.yml \
  up -d --scale backend=3

Read docker-compose.ha.yml before using it — it is documented as an illustrative starting point, not a drop-in for every environment (it assumes a single Docker host running multiple replicas via --scale; a real multi-host deployment typically runs one backend-api container per host instead, behind a load balancer, using the same environment variables).

Step 3 — put a load balancer in front

Route ordinary /api/v1/* traffic round-robin across every replica — reads, uploads, and most endpoints work identically on any replica because they only touch the shared database and shared /repos. No sticky routing is required for active-passive by itself.

Only the leader runs the six APScheduler cron jobs (security_sync_daily, sla_check_daily, retention_daily, inventory_scan, backup_daily, mirror_daily, plus the additional jobs listed in main.py) — scheduler_state.scheduler stays unset on passive replicas. A handful of endpoints that start an in-memory-tracked background job are gated behind Depends(require_leader) and return 503 on a passive replica: POST /import/sync/start, POST /import/mirror/start/{source_id}, POST /install/jobs, plus (via require_leader() directly, no distributed-flow exception) POST /app-deps/clients/{id}/scan, POST /drift/clients/{id}/scan, POST /drift/scan-by-tag, POST /inventory/clients/{id}/compliance/scan, and POST /inventory/compliance/scan-by-tag. Inventory scan endpoints (POST /inventory/clients/{id}/scan, /scan-all, /scan-by-tag) use require_leader_for("scan") instead, which becomes a no-op once active-active job state (below) is enabled for that flow.

If you never need Redis or job flows to be replica-agnostic, this is a complete, working HA deployment: a leader dies, its session-scoped PostgreSQL advisory lock is released automatically, and another replica picks up leadership — see Failover behavior below for exactly what "picks up" means in practice.

3. Enabling active-active job state (Redis)¶

Active-passive alone means a job's progress, cancellation state, and logs live only in the memory of whichever replica created it — a client polling a different replica sees nothing, which is why job creation is leader-gated in the first place. Setting JOB_STATE_BACKEND=redis moves that state into Redis so any replica can read or act on a job regardless of which replica started it, and lifts the leader gate for the flows that are verified as actually running on the distributed backend.

This is opt-in per deployment (not per component — see the note below) and off by default (JOB_STATE_BACKEND=local, unchanged behavior).

Step 1 — start the redis service

docker-compose.yaml already defines a redis service (redis:7-alpine, no persistence — job state is transient by design, a Redis restart just loses in-flight job history, never causes corruption), gated behind the ha-active-active Compose profile so it is never started by a bare docker compose up:

docker compose -f docker-compose.yaml -f docker-compose.ha.yml \
  --profile ha-active-active up -d --scale backend=3

Step 2 — point every backend-api replica at Redis

JOB_STATE_BACKEND=redis
JOB_STATE_REDIS_URL=redis://repod-redis:6379/0

JOB_STATE_REDIS_URL is optional if REDIS_URL (the same variable used by services/cache.py's response cache) is already set — JOB_STATE_REDIS_URL only exists so job-coordination traffic can be pointed at a separate Redis instance/DB from the response cache if you want that split. If neither is set while JOB_STATE_BACKEND=redis, every component falls back to local behavior (see Troubleshooting below).

What "opt-in per component" means operationally

One JOB_STATE_BACKEND=redis toggle activates every component at once — there is no per-flow environment variable. But each component reports its actual running backend independently in GET /health (below), because each one falls back to local independently if Redis is unreachable at the moment it needs it. Once genuinely active, it covers:

The four job trackers: inventory scan, install, mirror, sync — job creation, polling, cancellation, and confirmation all become safe on any replica, and the require_leader/require_leader_for("scan") gates for these flows stop applying.
The dashboard SSE event bus (GET /dashboard/events) — an event published on one replica reaches subscribers connected to every replica.
The live backend-log stream (GET /logs/stream, GET /logs, GET /logs/services) — same cross-replica delivery, plus a shared bounded history (last 2000 entries) so a replica that never recorded an entry locally can still serve it from Redis.

Neither the SSE bus nor the log stream ever had a require_leader gate to begin with (any replica already accepted subscribers) — enabling Redis for them changes delivery scope (single-replica → fleet-wide), not access.

drift / app-deps / compliance scans stay leader-only regardless

POST /app-deps/clients/{id}/scan, POST /drift/clients/{id}/scan, POST /drift/scan-by-tag, POST /inventory/clients/{id}/compliance/scan, and POST /inventory/compliance/scan-by-tag have no distributed-state backend of their own — they stay gated by the plain require_leader() whether or not JOB_STATE_BACKEND=redis is set. A passive replica always returns 503 on these five endpoints.

4. Verify it worked¶

Query GET /health on each replica and inspect checks.info.ha:

curl -s http://localhost:8000/health | jq '.checks.info.ha'

{
  "ok": true,
  "is_leader": true,
  "instance_id": "backend-a1b2c3d4",
  "scheduler_active": true,
  "job_state_backend": {
    "scan": "redis",
    "install": "redis",
    "mirror": "redis",
    "sync": "redis",
    "sse": "redis",
    "logs": "redis"
  }
}

is_leader / instance_id — confirm exactly one replica reports is_leader: true at a time, and that instance_id differs across replicas (it's derived from the hostname plus a random suffix, so two containers never collide even with identical hostnames prefixes).
scheduler_active — should be true only on the leader; false on every passive replica.
job_state_backend.{scan,install,mirror,sync,sse,logs} — each reports "redis" only when that component is genuinely running against a reachable Redis right now, never just the configured intent. If you set JOB_STATE_BACKEND=redis and still see "local" for any of these, see the troubleshooting section below before assuming active-active is live.

A quick end-to-end check: start an inventory scan job through one replica (behind the load balancer, so you don't control which one), then poll GET /inventory/clients/{id}/scan-status through a different replica (bypass the load balancer temporarily to target it directly) — with Redis active, the second replica should see live progress; with local, it would 404/report nothing.

5. Failover behavior¶

The advisory lock backing leadership (pg_try_advisory_lock) is session-scoped in PostgreSQL: it is tied to the specific database connection the leader opened, not to any timeout or heartbeat.

If the leader's process dies (container crash, OOM kill, host failure), its PostgreSQL connection closes and PostgreSQL releases the advisory lock automatically — no manual intervention needed on the database side.
Acquisition only happens at startup. acquire_leadership() runs once during a replica's lifespan startup and is never retried on a running process. A surviving passive replica does not poll for or pick up leadership while it keeps running — the lock only becomes acquirable again after it is released, and a replica only attempts to acquire it when it (re)starts. In practice this means failover requires restarting the dead leader's container (or starting a new one) — whichever replica's process next executes acquire_leadership() after the lock is released becomes the new leader. If your orchestrator (Docker Compose restart: unless-stopped, Kubernetes, systemd) already restarts a crashed container automatically, failover happens automatically too, on that restart — but it is the restart that triggers re-election, not a background watch.
Until a new leader is elected, the six cron jobs and the require_leader-gated endpoints listed above are unavailable (503) — everything else (reads, uploads, most of the API) continues working normally on the surviving replicas throughout.
Fail-open on non-PostgreSQL dialects or election errors: if DATABASE_URL points at a non-PostgreSQL dialect (e.g. SQLite, used in tests) or acquire_leadership() hits an unexpected error, that replica simply becomes leader (is_leader() == True) — this is what keeps a single-instance or test deployment fully functional without any of this machinery engaging.

6. Troubleshooting: Redis configured but unreachable¶

If JOB_STATE_BACKEND=redis is set but Redis cannot be reached (wrong URL, network partition, Redis itself down), each component falls back to local behavior independently — this is fail-soft by design, never a hard startup failure. What to look for:

Logs: each fallback is logged at ERROR level (not WARNING, unlike the same fallback pattern in services/cache.py) — a Redis outage affecting HA correctness is treated as more severe than a cache-layer degradation. Search backend-api logs for the component name (scan, install, mirror, sync, sse, logs) alongside the fallback message.
GET /health: checks.info.ha.job_state_backend.{flow} reports "local" for the affected component(s) — this is the authoritative, always-current signal, since it reflects the backend actually in use after any fallback, not the configured intent.
What still works: nothing breaks outright. A component that fell back to "local" behaves exactly as if JOB_STATE_BACKEND were never set for that component — scan/install/mirror/sync creation is rejected with 503 on passive replicas (the leader gate is still enforced, precisely to avoid a false sense of distributed safety), and the SSE/log streams simply stop delivering across replicas (each replica only serves its own local subscribers again) without rejecting anything.
Resolution: fix Redis reachability (check JOB_STATE_REDIS_URL/ REDIS_URL, network policy, the redis container's health) and restart the affected backend-api replicas — there is no live reconnect probe that flips a component back to "redis" without a restart.