Install & deploy · Operations

Part of the Install & deploy guide. This page is for running the platform after it is up: scaling from one machine to many, keeping every service available, data residency, and the checks before exposing an instance.

Scaling#

One machine, all of it#

A single instance already uses the whole machine. The container entrypoint forks one worker per available CPU and they share the port, so a 16- or 64-core host is used without configuration. Rendering a page costs roughly 30 ms of CPU, and that is what the extra workers buy — static assets were never the bottleneck.

"Available" means the CPU quota, not the host: in a container the count is capped by the cgroup limit, so --cpus=2 forks two however large the machine underneath is. Override with WEB_CONCURRENCY only on purpose — 1 is the right value when you run many small one-core containers instead, so workers don't fight over a fraction of a CPU. The number chosen is logged at startup.

Two things behave differently once workers multiply. The scheduler does not — it holds a fleet-wide lease, so exactly one sweep runs no matter how many workers or replicas exist. The lakehouse query engine does: it lives in each process, so its memory limit applies per worker. A 16-core host at LAKEHOUSE_MEMORY_LIMIT=16GB is 256 GB of intent, not 16 — size that limit against host RAM divided by workers. It is a ceiling rather than a reservation, so idle workers hold nothing, but the worst case is what an out-of-memory kill needs.

Analytics-only nodes#

A heavy lakehouse query and a page render share one Node process, so a thirty-second GROUP BY can stall interactive traffic on the node running it. Set APP_ROLE=analytics on the nodes you want kept out of the request path. Such a node reports not ready at /api/health/ready while staying alive at /api/health — readiness decides routing and liveness decides restarts, so it drains itself out of the interactive pool with no load-balancer feature required, and nothing restarts it. Point your readiness check at /api/health/ready for this to work.

It also defaults to a single worker, because each worker carries its own engine: forking would split the large memory limit you set into independent copies that could each claim the whole figure. WEB_CONCURRENCY still overrides. Scheduled work — BI refreshes, materialized-view rebuilds — still runs there, which is the point; pair it with DISABLE_INPROCESS_SCHEDULER on the interactive tier. The role is a routing declaration, not access control, so pointing a browser straight at one still works.

Then more machines#

The app tier is stateless — no sticky sessions needed, so put as many instances behind a load balancer as you like. Two things need attention when you do:

• The scheduler. Alerts, refreshes and purges run in-process. A cross-instance lease prevents double-firing, but the tidier arrangement is DISABLE_INPROCESS_SCHEDULER on the web tier and one external cron hitting the cron endpoint.
• Limits hold across the fleet. Rate limits and concurrency slots are counted in Postgres, so the number you configure is the number you get however many instances are running. If the database is briefly unreachable an instance falls back to counting locally and logs that it has — the limit weakens rather than vanishing. Budget caps are the other ceiling, and they are counted the same way; see Budgets.
• Set the proxy depth. TRUSTED_PROXY_HOPS must match how many proxies of yours sit in front — 1 for a load balancer alone, 2 with a CDN in front of it. MCP key IP allow-lists are checked against the address it selects.

The notebook Docker runtime is single-host by design — it launches containers on the host it runs on. Use the Kubernetes orchestrator to spread it.

Operations#

Health

A health endpoint reports process liveness — point your load balancer at it.

Backups

A self-hosted install has four things that cannot be regenerated: the application database, the lakehouse catalog (a separate Postgres that knows which Parquet files make up each table and every snapshot), the lakehouse data (Parquet in your bucket) and the secrets in .env. `npm run backup` captures the first three into backups/<timestamp>/ and lists the fourth by name — PROVIDER_CREDS_SECRET decrypts every stored credential and PROVENANCE_SIGNING_SECRET verifies every Answer Passport, so store both in your secret manager; values are never written to a backup. Give it a database credential (--db-url for self-hosted Supabase, SUPABASE_DB_PASSWORD for a linked hosted project) or that step is skipped and recorded in manifest.json — it never prompts, so it is safe to schedule. Rehearse before you need it: `npm run restore -- backups/<timestamp> --drill` restores into scratch targets, compares, cleans up and prints DRILL PASSED. Real restores opt into --catalog, --lake and --supabase and require --yes. Full runbook, including the order for a host migration, in docs/DEPLOYMENT.md.

Upgrades

Take a backup first (npm run backup), then pull, rebuild (docker compose up -d --build) and run npx supabase db push --include-all. Migrations only add and are never reverted, so the rollback is the backup plus a git checkout of the previous tag. Check the release notes before skipping several versions.

Logs

Container logs for the platform; in-app Traces for what agents did. They answer different questions — reach for Traces first when an agent misbehaves.

Keeping every service available#

Two promises worth separating: availability is the service still answering when one instance is lost, and durability is the data still being there afterwards. Replicas buy the first and never the second. A single host has neither — what it does have is a restart policy and a health check on every service that can answer for itself, so a crashed or wedged container comes back on its own. Losing the host is a restore, not a failover.

On Kubernetes every stateless tier already ships with two replicas, a topology spread so they do not share a node, and a PodDisruptionBudget so a drain or cluster upgrade cannot evict them all at once: the web tier, the analytics tier that carries the scheduler, the Office renderer, the JS sandbox, the notebook gateway and the sandbox egress proxy. Running the scheduler on more than one is safe because each pass is claimed through a lease with an atomic conditional update.

Verify with kubectl -n agentswarms get deploy,statefulset,pdb: every Deployment should report at least two ready, each with a budget beside it. The full per-service table, including what breaks when each one is lost, is in docs/DEPLOYMENT.md under High availability.

Data residency: one deployment per region#

"Multi-region" gets asked for in two quite different senses. Failover — a second region already holding the same data, serving it the moment the first one goes — needs a database that is multi-master across regions, and this platform does not do it; the answer there is the restore runbook. Residency — this customer's data must stay inside this jurisdiction — is the one enterprises usually mean, has nothing to do with failover, and is supported today by the plainest mechanism available: run a separate deployment in each region.

Nothing ties one install to another. A deployment is pinned to its data entirely by environment — SUPABASE_URL, the lake bucket and catalog, the key ring — and there is no notion of a region, an instance id or a registry of peers anywhere in the code. Two deployments are two installs that happen to run the same image, each deployed, upgraded, backed up and restored exactly as a single one is. Route people to theirs with DNS, or from your identity provider: one SAML/OIDC application per deployment, which is also what gives each region its own SCIM sync.

What you get is that data written in a region stays there — rows, files, traces, audit, embeddings — with no replication link to switch off and no setting to get wrong, because there is no connection between them to begin with. What you give up is worth deciding on before you choose the shape:

• No cross-region anything. A query, dashboard, agent or knowledge base in one deployment cannot see another's data. A report spanning both is assembled outside the platform.
• No single pane of glass. Users, agents, IAM groups, budgets and audit are per deployment, and an administrator manages each one.
• A region's outage is that region's outage. Residency is not availability.
• Upgrades are per deployment, so versions drift unless you drive them together.

Before you expose it#

• Turn off public signup, or enforce SSO — Access control.
• Set ENFORCE_BUDGET_CAP and give every embed and API key a cap.
• Serve over TLS; the service-role key must never reach a browser.
• Restrict embed keys to your own domains.
• Review retention windows for chats, transcripts and audit.
• Back up PROVIDER_CREDS_SECRET somewhere you can actually retrieve it.