Install & deploy · Kubernetes

Part of the Install & deploy guide. This page is the Kubernetes path: what each manifest is for, then Amazon EKS, Google GKE, Azure AKS and Oracle OKE step by step, and how to verify any of them.

Kubernetes#

The manifests, what each one is for, and a walk-through per cloud. The other targets — Docker Compose, a plain VM, a PaaS — are on the deployment targets section of the install guide.

Kubernetes, in detail#

Fully self-hosted, one command. Supabase runs in your cluster too — nothing leaves it, and nothing is optional:

[email protected] ADMIN_PASSWORD='...' bash scripts/setup-k8s.sh
kubectl -n agentswarms port-forward svc/agentswarms 8080:80

Needs kubectl, helm and a cluster. It generates every secret — including the anon and service-role keys signed from the JWT secret, since they are JWTs and a random string there gives you a stack that starts and then rejects every request — installs Supabase, applies the schema, creates your admin user, then starts the app with the Office renderer, the JS sandbox and the lakehouse catalog. Re-running is safe.

Deploying to a cloud cluster? Push the images first. The command above defaults to the three images this repo builds locally. A local cluster — Docker Desktop, kind, minikube, k3d — shares the machine’s image store and runs them as they are. No other cluster can: its nodes pull from a registry, and an image that exists only on your laptop ends in ImagePullBackOff. Push all three and name them:

AGENTSWARMS_IMAGE=ghcr.io/you/agentswarms:1.2.3 \
DOCGEN_IMAGE=ghcr.io/you/docgen:1.2.3 \
JS_SANDBOX_IMAGE=ghcr.io/you/js-sandbox:1.2.3 \
[email protected] ADMIN_PASSWORD='...' bash scripts/setup-k8s.sh

The installer substitutes all three as it applies the manifests, and warns before it starts if the current context does not look local while the images still do. Beyond that, the manifests use only core APIs and name no StorageClass, so every volume takes the cluster’s default. What still needs a decision per cloud: an Ingress or LoadBalancer in front of svc/agentswarms and Kong; a CNI that actually enforces NetworkPolicy (Calico, Cilium, GKE Dataplane V2, AKS or EKS with policy enabled) or the sandbox’s egress ban is inert; and roughly 3 CPU and 6 GiB of requests for our pods before the Supabase chart’s own.

Those decisions land differently on each managed cluster. docs/DEPLOYMENT.md has a step-by-step runbook per cloud — cluster creation, the registry, storage, policy enforcement, ingress and TLS, the managed Postgres and object storage the lakehouse prefers, and a GPU pool if you train — under “Managed clusters: AWS, GCP, Azure, OCI”. The short version:

Push five images, not three, if you use the developer workspace, ETL or the ML platform: the notebook gateway (./services/notebook-gateway) is named in deploy/k8s/notebooks/notebook-runtime.yaml, and the runtime image (./docker/notebook-runtime) in NOTEBOOK_RUNTIME_IMAGE. Training, prediction and pipeline runs are all batch pods in that namespace. The build loop is the same everywhere; only the registry host changes.

export REGISTRY=<your registry host and path>
export TAG=1.5.0
docker build -t "$REGISTRY/agentswarms:$TAG" .
docker build -t "$REGISTRY/docgen:$TAG" ./docgen-service
docker build -t "$REGISTRY/js-sandbox:$TAG" ./services/js-sandbox
docker build -t "$REGISTRY/notebook-gateway:$TAG" ./services/notebook-gateway
docker build -t "$REGISTRY/notebook-runtime:$TAG" ./docker/notebook-runtime
for i in agentswarms docgen js-sandbox notebook-gateway notebook-runtime; do docker push "$REGISTRY/$i:$TAG"; done

Amazon EKS, step by step#

Needs aws v2, eksctl, kubectl and helm, and an IAM principal that can create EKS clusters, IAM roles, ECR repositories, RDS instances and S3 buckets.

export AWS_REGION=us-east-1
export CLUSTER=agentswarms
export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
export REGISTRY="$ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com"

# 1. The cluster. --with-oidc is required: every add-on below uses IRSA.
eksctl create cluster --name "$CLUSTER" --region "$AWS_REGION" --version 1.31   --nodegroup-name app --node-type m6i.xlarge --nodes 3 --managed --with-oidc

# 2. A working default StorageClass. EKS ships gp2 with no driver behind it,
#    so every PersistentVolumeClaim sits in Pending for ever without this.
eksctl create iamserviceaccount --cluster "$CLUSTER" --region "$AWS_REGION"   --namespace kube-system --name ebs-csi-controller-sa --role-name AgentSwarmsEBSCSIRole   --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy   --approve --role-only
eksctl create addon --cluster "$CLUSTER" --region "$AWS_REGION" --name aws-ebs-csi-driver   --service-account-role-arn "arn:aws:iam::$ACCOUNT:role/AgentSwarmsEBSCSIRole" --force
kubectl annotate storageclass gp2 storageclass.kubernetes.io/is-default-class- --overwrite

# 3. Network policy, or the JS sandbox's egress ban is inert.
aws eks update-addon --cluster-name "$CLUSTER" --region "$AWS_REGION" --addon-name vpc-cni   --resolve-conflicts PRESERVE --configuration-values '{"enableNetworkPolicy":"true"}'

# 4. Registry.
for r in agentswarms docgen js-sandbox notebook-gateway notebook-runtime; do
  aws ecr create-repository --repository-name "$r" --region "$AWS_REGION" >/dev/null || true
done
aws ecr get-login-password --region "$AWS_REGION" | docker login --username AWS --password-stdin "$REGISTRY"

# 5. Install, then watch.
AGENTSWARMS_IMAGE="$REGISTRY/agentswarms:$TAG" DOCGEN_IMAGE="$REGISTRY/docgen:$TAG"   JS_SANDBOX_IMAGE="$REGISTRY/js-sandbox:$TAG"   [email protected] ADMIN_PASSWORD='...' bash scripts/setup-k8s.sh
kubectl -n agentswarms get pods -w

A gp3 StorageClass marked default has to exist as well — the full manifest, the AWS Load Balancer Controller install with its IAM policy, the ACM certificate and the Ingress are in docs/DEPLOYMENT.md. The two that catch people: health-check /api/health/ready rather than /api/health, and leave LAKEHOUSE_S3_ENDPOINT unset for real S3 — it exists for MinIO, and pointing it at an AWS host is the usual reason a lakehouse cannot read its own Parquet.

# 9-10. Managed Postgres for the catalog, S3 for the lake.
aws rds create-db-instance --db-instance-identifier agentswarms-catalog --engine postgres   --engine-version 16.4 --db-instance-class db.t4g.medium --allocated-storage 50   --storage-encrypted --master-username lakehouse --master-user-password '<password>'   --db-name lakehouse_catalog --no-publicly-accessible --region "$AWS_REGION"
aws s3api create-bucket --bucket agentswarms-lake --region "$AWS_REGION"
kubectl -n agentswarms patch secret agentswarms-env --type merge -p '{"stringData":{
  "LAKEHOUSE_CATALOG_URL":"postgres://lakehouse:<password>@<endpoint>:5432/lakehouse_catalog",
  "LAKEHOUSE_DATA_URL":"s3://agentswarms-lake/main",
  "LAKEHOUSE_S3_URL_STYLE":"vhost","LAKEHOUSE_S3_USE_SSL":"true"}}'

Google GKE, step by step#

Dataplane V2 is chosen at creation and cannot be added later. It is what enforces NetworkPolicy; a cluster without it needs rebuilding, not patching.

export PROJECT=$(gcloud config get-value project)
export REGION=us-central1
export REGISTRY="$REGION-docker.pkg.dev/$PROJECT/agentswarms"
gcloud services enable container.googleapis.com artifactregistry.googleapis.com   sqladmin.googleapis.com storage.googleapis.com

# 1. Cluster. --num-nodes is per zone, so a regional cluster gives three.
gcloud container clusters create agentswarms --region "$REGION" --num-nodes 1   --machine-type e2-standard-4 --enable-dataplane-v2 --enable-ip-alias   --workload-pool="$PROJECT.svc.id.goog" --release-channel regular
gcloud container clusters get-credentials agentswarms --region "$REGION"

# 2. Storage needs nothing: standard-rwo is already the default.
# 3. Registry.
gcloud artifacts repositories create agentswarms --repository-format=docker --location="$REGION"
gcloud auth configure-docker "$REGION-docker.pkg.dev"

# 5. A static address first, so the DNS record outlives the Ingress.
gcloud compute addresses create agentswarms-ip --global
gcloud compute addresses describe agentswarms-ip --global --format='value(address)'

# 6-7. Cloud SQL for the catalog; GCS speaks S3 with an HMAC key.
gcloud sql instances create agentswarms-catalog --database-version=POSTGRES_16   --tier=db-custom-2-7680 --region="$REGION"
gcloud storage buckets create gs://agentswarms-lake --location="$REGION"
gcloud storage hmac create <service-account-email>

The Ingress wants three objects: a ManagedCertificate, a BackendConfig whose health check is /api/health/ready, and the Ingress itself annotated with the static IP and the certificate. The certificate stays Provisioning until the A record resolves, which takes tens of minutes on first issue. For GCS set LAKEHOUSE_S3_ENDPOINT to storage.googleapis.com with LAKEHOUSE_S3_URL_STYLE=path.

Azure AKS, step by step#

export RG=agentswarms-rg LOCATION=eastus ACR=agentswarmsacr
az group create --name "$RG" --location "$LOCATION"

# 1. The dataplane is chosen at creation, like GKE's.
az aks create --resource-group "$RG" --name agentswarms --node-count 3   --node-vm-size Standard_D4s_v5 --network-plugin azure --network-dataplane cilium   --network-policy cilium --enable-managed-identity --generate-ssh-keys
az aks get-credentials --resource-group "$RG" --name agentswarms

# 3. Registry, attached so no imagePullSecret is needed anywhere.
az acr create --resource-group "$RG" --name "$ACR" --sku Standard
az aks update --resource-group "$RG" --name agentswarms --attach-acr "$ACR"
az acr login --name "$ACR"

# 5. Managed NGINX with its own public IP.
az aks approuting enable --resource-group "$RG" --name agentswarms
kubectl -n app-routing-system get service nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

# 6. Managed Postgres for the catalog.
az postgres flexible-server create --resource-group "$RG" --name agentswarms-catalog   --location "$LOCATION" --tier Burstable --sku-name Standard_B2s --version 16   --admin-user lakehouse --admin-password '<password>' --public-access None --yes

Oracle OKE, step by step#

OKE’s Quick create in the console builds the VCN, subnets and node pool in one pass; the CLI needs those OCIDs to exist already. Two OCI-specific credentials catch people out: the registry password is an auth token, and the lake needs a Customer Secret Key — they are different things, and neither is your console password.

export OCI_REGION=us-ashburn-1 REGION_KEY=iad
export NAMESPACE=$(oci os ns get --query data --raw-output)
export REGISTRY="$REGION_KEY.ocir.io/$NAMESPACE/agentswarms"

oci ce cluster create-kubeconfig --cluster-id <cluster OCID> --file "$HOME/.kube/config"   --region "$OCI_REGION" --token-version 2.0.0

# 4. OCIR. The password is an auth token from the console.
docker login "$REGION_KEY.ocir.io" --username "$NAMESPACE/oracleidentitycloudservice/[email protected]"
kubectl create secret docker-registry ocir --namespace agentswarms   --docker-server="$REGION_KEY.ocir.io"   --docker-username="$NAMESPACE/oracleidentitycloudservice/[email protected]"   --docker-password='<auth-token>'
kubectl -n agentswarms patch serviceaccount default   -p '{"imagePullSecrets":[{"name":"ocir"}]}'

# 8. Object Storage through its S3 compatibility endpoint.
oci os bucket create --compartment-id "$COMPARTMENT" --name agentswarms-lake

OKE ships oci-bv as the default StorageClass, so storage needs nothing — but block volumes are zonal, so keep the lakehouse catalog in one availability domain or move it to managed Postgres. Install Calico if you want the sandbox’s egress ban enforced: OKE’s flannel and VCN-native pod networking do not enforce NetworkPolicy, and the policy applies cleanly and does nothing without it.

After any of them#

Seven checks catch every decision above. The one worth running first is the sandbox, because its failure is silent:

kubectl -n agentswarms exec deploy/agentswarms-js-sandbox --   sh -c 'wget -qO- --timeout=5 https://example.com || echo DENIED'

DENIED is the pass. Anything else means the CNI is not enforcing policy and user-supplied code can reach the internet. Then: every pod Running and Ready; the Office renderer actually has a pod (a restricted namespace refuses it quietly); resources.limits.cpu set on the web Deployment, because the worker count follows it; the load balancer health-checking /api/health/ready; PUBLIC_APP_URL matching the hostname people type; and a CREATE TABLE smoke.t AS SELECT 1 in the Lakehouse SQL editor, which exercises the catalog Postgres and the object store together.

Bringing your own Supabase (Cloud, or one you already run)? deploy/k8s/app/agentswarms.yaml is the app on its own — namespace, web Deployment, optional analytics Deployment, Service, an HPA and the cron CronJob. It has been applied to a real cluster; the notes below are what failed there.

# kubectl keeps the quotes docker compose strips
sed -E 's/^([A-Za-z_][A-Za-z0-9_]*)="(.*)"$/\1=\2/' .env > .env.k8s
kubectl create secret generic agentswarms-env -n agentswarms --from-env-file=.env.k8s && rm .env.k8s
kubectl apply -f deploy/k8s/app/agentswarms.yaml

Strip quotes from .env first

Docker Compose removes the quotes around a value; kubectl create secret keeps them, so SUPABASE_URL arrives as a literal "https://…". Every pod then fails readiness with Invalid supabaseUrl and the Service ends up with no endpoints at all.

Always set resources.limits.cpu

The worker count follows the pod’s CPU limit. With no limit, a pod on a 64-core node forks 64 workers at ~0.5–1 GB each and is OOMKilled — a crash loop with no obvious cause. Measured on an 8-core node: cpu: "2" gives two workers, cpu: 500m gives one.

Probe liveness and readiness separately

Liveness on /api/health decides restarts; readiness on /api/health/ready decides routing. A pool that probes only liveness keeps sending traffic to pods that cannot reach the database.

No readiness probe on analytics pods

APP_ROLE=analytics answers readiness 503 for ever, and Kubernetes gates rollout progress on readiness — so a new pod never becomes Ready, the old one is never retired, and kubectl rollout status hangs on “1 old replicas are pending termination”. Exclude those pods from the Service by label instead, as the shipped manifest does.

Add what Compose was defaulting for you

docker-compose.yml fills a dozen values with ${VAR:-default} and Kubernetes has no equivalent, so a Secret built from that .env is missing them and the pod stops with couldn't find key … in Secret. Three matter: BI_CRON_TOKEN (the CronJob), INTERNAL_RUN_SECRET (the JS sandbox, which Compose defaulted to the service-role key) and LAKEHOUSE_CATALOG_PASSWORD (the catalog, which Compose defaulted to change-me).

Optional services get their own pods

deploy/k8s/app/services.yaml covers the Office renderer, the JS sandbox and the lakehouse catalog — each its own Deployment and Service, found by the same name the app uses under Compose. The catalog is a StatefulSet with a volume rather than a Deployment, because it holds the one part of the lakehouse that cannot be rebuilt from object storage; in production prefer a managed Postgres. The notebook runtime is separate again, in deploy/k8s/notebooks/.

Hardened by default

The image drops to a non-root user, and both app Deployments run runAsNonRoot with a read-only root filesystem, all capabilities dropped and no Kubernetes API token mounted — /tmp is an emptyDir because the lakehouse engine spills there. The JS sandbox, which runs user-supplied code, adds a NetworkPolicy denying it egress outright (your CNI has to enforce policy — Docker Desktop’s default does not).

Monitoring shows one pod, not the fleet

Observability → Monitoring reports the replica that answered — the page names it, and on Kubernetes that name is the pod. For fleet-wide numbers use your cluster metrics; this page is for looking at one instance and at what is down.