> For the complete documentation index, see [llms.txt](https://docs.warp.dev/llms.txt). > Markdown versions of each page are available by appending .md to any URL. # Managed: Kubernetes backend Deploy the Automation Platform managed worker into a Kubernetes cluster with the included Helm chart. Each agent task runs as a Kubernetes Job in your cluster. Deploy the `oz-agent-worker` daemon into a Kubernetes cluster using the included Helm chart. Each agent task runs as a Kubernetes Job in your cluster. The Automation Platform orchestrates runs end to end (Slack, Linear, schedules, API, `oz agent run-cloud`); your cluster provides the compute, scheduling, and policy enforcement. ## When to use the Kubernetes backend - You already operate a Kubernetes cluster and want agents to run there. - You need Kubernetes-native scheduling, resource management, or policy enforcement. - You want to use Kubernetes Secrets, ServiceAccounts, and admission policies to control task behavior. * * * ## How it works 1. The worker connects to the Kubernetes API server (using in-cluster auth by default, or an explicit kubeconfig). 2. On startup, the worker creates a short-lived **preflight Job** with the configured task pod shape. This catches Job creation, admission policy, Pod Security, and some image-volume failures before the worker accepts tasks. 3. For each assigned task, the worker creates a Kubernetes Job in the configured namespace. 4. The worker monitors the Job and Pod status via Kubernetes Watch (with a 30-second safety-net poll for watch disconnects). 5. After the task completes, the worker removes successful Jobs. Failed Jobs remain available for diagnosis for 24 hours by default. * * * ## Prerequisites - **Enterprise plan with self-hosting enabled** — [Contact sales](https://www.warp.dev/contact-sales) if self-hosting is not yet enabled for your team. - **A Kubernetes cluster** with the worker process able to reach the API server. The cluster must: - Allow the task namespace to create Jobs with a **root init container**, unless you enable native image volumes with `kubernetesBackend.useImageVolumes=true`. - Grant the worker these namespace-scoped permissions: `create`, `get`, `list`, `watch`, `delete` on `jobs`; `get`, `list`, `watch` on `pods`; `get` on `pods/log`; `list` on `events`. - **[Helm](https://helm.sh/docs/intro/install/)** installed locally, plus `kubectl` authenticated against the target cluster. - **An agent API key** — Create one in the [Oz web app](https://oz.warp.dev/settings) so the worker can authenticate to the Automation Platform. You can bind the key to any cloud agent — that choice doesn’t restrict which agents can run on the worker. See [API Keys](https://docs.warp.dev/reference/cli/api-keys/) for the full creation flow. * * * ## Install with the Helm chart The `oz-agent-worker` repository includes a namespace-scoped Helm chart at `charts/oz-agent-worker`. This is the recommended way to deploy the worker into a cluster. ### What the chart deploys - A long-lived `Deployment` running `oz-agent-worker` with the Kubernetes backend. - A namespaced `ServiceAccount` for the worker. - A namespaced `Role` / `RoleBinding` with the minimum permissions needed to manage task Jobs and Pods. - A `ConfigMap` containing the worker config YAML. - An optional `Secret` for `WARP_API_KEY` (or a reference to an existing Secret). The chart does not create CRDs or cluster-scoped RBAC resources. ### 1\. Set your API key and namespace ```bash export WARP_API_KEY="your_agent_api_key" ``` Create the namespace if it doesn’t exist: ```bash kubectl create namespace warp-oz ``` ### 2\. Create the API key Secret If you’re not using an existing Secret, create one with the API key: ```bash kubectl create secret generic oz-agent-worker \ --from-literal=WARP_API_KEY="$WARP_API_KEY" \ --namespace warp-oz ``` **Expected outcome:** `kubectl get secret -n warp-oz oz-agent-worker` shows the Secret. ### 3\. Install the chart Clone the worker repo and install the chart: ```bash git clone https://github.com/warpdotdev/oz-agent-worker.git helm install oz-agent-worker ./oz-agent-worker/charts/oz-agent-worker \ --namespace warp-oz \ --set worker.workerId=oz-k8s-worker \ --set image.tag= ``` Caution Set `image.tag` explicitly to pin the worker image. Check the [oz-agent-worker releases](https://github.com/warpdotdev/oz-agent-worker/releases) for the latest version. Do not rely on `latest`. **Expected outcome:** `kubectl get pods -n warp-oz` shows the worker Deployment pod as `Running`, and the worker logs include `Successfully connected to server`. To scale horizontally, deploy multiple Helm releases with distinct worker IDs rather than increasing replicas on a single release. * * * ## Key chart values **Required:** - `worker.workerId` — The worker ID (same as `--worker-id`). - `image.tag` — The worker image tag to deploy. **Worker configuration:** - `worker.logLevel` — Log verbosity (`debug`, `info`, `warn`, `error`). Defaults to `info`. - `worker.cleanup` — Whether to clean up task Jobs after execution. Defaults to `true`. - `worker.maxConcurrentTasks` — Maximum concurrent tasks. Defaults to `0` (unlimited). - `worker.idleOnComplete` — Duration to keep the oz process alive after task completion. - `worker.resources` — Resource requests/limits for the worker Deployment. Defaults to `100m` CPU and `128Mi` memory. - `worker.livenessProbe` — Liveness probe for the worker Deployment. Defaults to an `exec` probe (`kill -0 1`). Override with a custom probe or set to `null` to disable. - `worker.terminationGracePeriodSeconds` — Grace period for worker Deployment shutdown. Defaults to `30`. - `worker.nodeSelector`, `worker.tolerations`, `worker.affinity` — Scheduling constraints for the worker Deployment pod. **Kubernetes backend:** - `kubernetesBackend.namespace` — Namespace for task Jobs. Defaults to the release namespace. - `kubernetesBackend.defaultImage` — Default Docker image for task pods when no [Warp environment](https://docs.warp.dev/platform/environments/) has been supplied. Leave empty (default) to fall back to `ubuntu:22.04`. - `kubernetesBackend.imagePullPolicy` — Image pull policy for task pods. Defaults to `IfNotPresent`. - `kubernetesBackend.useImageVolumes` — Use native Kubernetes image volumes instead of root init containers to materialize sidecars. Defaults to `false`. - `kubernetesBackend.preflightImage` — Image for the startup preflight Job. Set this if your cluster restricts allowed registries. - `kubernetesBackend.preflightResources` — CPU and memory requests and limits for preflight containers. - `kubernetesBackend.sidecarImage` — Internal-registry override for the Warp agent sidecar image. - `kubernetesBackend.unschedulableTimeout` — How long a pod may remain unschedulable before failing. Defaults to `30s`. - `kubernetesBackend.setupCommand` — Shell command to run before each task. - `kubernetesBackend.teardownCommand` — Shell command to run after each task. - `kubernetesBackend.extraLabels` — Additional labels for task Jobs and Pods. - `kubernetesBackend.extraAnnotations` — Additional annotations for task Jobs and Pods. - `kubernetesBackend.activeDeadlineSeconds` — Maximum task Job lifetime. Defaults to eight hours. - `kubernetesBackend.ttlSecondsAfterFinished` — Retention period for failed Jobs and Jobs orphaned by worker disruption. Defaults to 24 hours when cleanup is enabled. - `kubernetesBackend.workspaceSizeLimit` — Size limit for workspace `emptyDir` volume. - `kubernetesBackend.podTemplate` — Raw PodSpec YAML for task Jobs (same as `backend.kubernetes.pod_template` in the [config file](https://docs.warp.dev/platform/self-hosting/reference/#config-file)). **API key Secret:** - `warp.apiKeySecret.create` — Set to `true` to have the chart create a Secret from `warp.apiKeySecret.value`. Defaults to `false` (expects a pre-existing Secret). - `warp.apiKeySecret.value` — The API key value to store in the chart-managed Secret. Only used when `warp.apiKeySecret.create` is `true`. - `warp.apiKeySecret.name` — Name of the Secret containing `WARP_API_KEY`. Defaults to `oz-agent-worker`. - `warp.apiKeySecret.key` — Key within the Secret. Defaults to `WARP_API_KEY`. See the [self-hosted worker reference](https://docs.warp.dev/platform/self-hosting/reference/#kubernetes-backend-config) for the full config file schema. * * * ## Cluster selection Cluster selection follows Kubernetes client config conventions: - Set `backend.kubernetes.kubeconfig` to use an explicit kubeconfig file. - If `kubeconfig` is omitted and the worker runs inside a Kubernetes pod, the worker uses in-cluster config automatically. - Otherwise, the worker falls back to the default kubeconfig loading rules and uses the current context. `namespace` selects the namespace inside the chosen cluster. It defaults to `default` when omitted. * * * ## Pod template The `pod_template` field accepts standard Kubernetes PodSpec YAML and is the declarative way to configure task pod scheduling, service accounts, image pull secrets, resources, and environment variables. When using `pod_template`, define a container named `task` to customize the main task container directly. Otherwise, the worker appends its own `task` container to the PodSpec. Use `valueFrom.secretKeyRef` to inject Kubernetes Secret values into task container environment variables: ```yaml pod_template: serviceAccountName: agent-task-sa imagePullSecrets: - name: my-registry-creds containers: - name: task resources: requests: cpu: "2" memory: 4Gi limits: memory: 8Gi env: - name: GITHUB_TOKEN valueFrom: secretKeyRef: name: my-k8s-secret key: github-token tolerations: - key: "dedicated" operator: "Equal" value: "agents" effect: "NoSchedule" ``` The worker Deployment’s ServiceAccount is separate from the task Job `serviceAccountName` you configure in `pod_template`. The Deployment ServiceAccount needs RBAC to manage Jobs and Pods. The task ServiceAccount (if any) controls what the agent process can access at runtime. When a run specifies an instance shape, the worker sets the `task` container’s CPU and memory requests and limits from that shape. Those values override CPU and memory configured on the `task` container in `pod_template`. * * * ## Preflight check On startup, the worker creates a short-lived preflight Job that uses the configured task PodSpec. The preflight confirms that the worker can create the Job and that the cluster can create the expected pod shape. It surfaces early failures from: - Insufficient worker RBAC permissions in the target namespace. - Cluster admission policies such as Pod Security Standards, OPA Gatekeeper, and Kyverno. - Sidecar-loading requirements, including native image-volume support when enabled. If the preflight fails, the worker logs a diagnostic error and exits before accepting tasks. A successful preflight does not validate every task image, task Secret, setup command, or network dependency. The preflight image defaults to `busybox:1.36`. If your cluster restricts allowed registries or images, set `kubernetesBackend.preflightImage` to an allowlisted image. When `imagePullSecrets` is configured in `kubernetesBackend.podTemplate`, those secrets apply to the preflight Job as well, so you can point `kubernetesBackend.preflightImage` at an image in your private registry. * * * ## Environment variables for Kubernetes tasks There are two ways to pass environment variables to Kubernetes task containers: 1. **`pod_template`** (recommended for Kubernetes-native config) — Use standard Kubernetes `env` syntax in the `task` container, including `valueFrom.secretKeyRef` for Kubernetes Secrets. 2. **`-e` / `--env` flags** — Backend-agnostic runtime overrides that work across all managed backends. When configuring the Kubernetes backend via YAML or Helm, declarative task-container env belongs in `pod_template` rather than a separate top-level list. If your organization uses an external secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, etc.), you can inject secrets into task pods via the CSI Secrets Store Driver or a similar operator. Configure the required `volumes`, `volumeMounts`, and annotations in `pod_template` just as you would for any other Kubernetes workload. See your secrets provider’s documentation for details. * * * ## Setup and teardown commands Use `kubernetesBackend.setupCommand` (Helm value) or `backend.kubernetes.setup_command` ([config file](https://docs.warp.dev/platform/self-hosting/reference/#kubernetes-backend-config)) to run a shell command before each task. Use `teardownCommand` / `teardown_command` for cleanup after the task finishes. These run inside the task Pod and are useful for workspace bootstrapping or post-run reporting. * * * ## Protect active task pods from disruption The worker Deployment and task Jobs have different disruption requirements. When Kubernetes terminates the worker pod normally, the worker leaves active task Jobs running. Replacing the worker pod does not intentionally stop their task pods. The task pod holds the live agent process and an `emptyDir` workspace. Evicting that pod interrupts the run and loses pod-local state. Configure node lifecycle tooling to avoid voluntary disruption of active task pods. For Karpenter, add its pod-level disruption annotation to every task Job through the Helm values: ```yaml title="values.yaml" kubernetesBackend: extraAnnotations: karpenter.sh/do-not-disrupt: "true" ``` Task pods exist only for the life of their Jobs. The annotation blocks Karpenter consolidation and, depending on your NodePool configuration, drift while a run is active. It does not prevent forceful expiration, interruption, node repair, or manual deletion. A configured NodePool `terminationGracePeriod` can also make a node eligible for drift and eventually terminate blocking pods. Review [Karpenter’s disruption controls](https://karpenter.sh/docs/concepts/disruption/) against your NodePool policy. A PodDisruptionBudget only constrains tools that use the Kubernetes Eviction API. It does not preserve the task process or workspace after deletion or protect against involuntary node loss. If your platform requires a PDB, validate its selector and availability rule against concurrent task Jobs. Keep the disruption protection required by your node lifecycle tool. For other autoscalers or node lifecycle controllers, use their equivalent protection for active task pods. Do not assume that a replacement pod can resume an interrupted run. * * * ## Plan capacity and scheduling Task pods need capacity for their configured requests before the worker’s unschedulable timeout expires. - Set `worker.maxConcurrentTasks` to a finite value based on the task capacity your cluster can provide. The default, `0`, does not cap concurrency. - Set task CPU and memory requests through a runner instance shape or the `task` container in `kubernetesBackend.podTemplate`. Leave headroom for init containers, node DaemonSets, and workload spikes. - Set `kubernetesBackend.unschedulableTimeout` longer than the slowest expected node provisioning time. The default is `30s`; set it to `0s` only when you want to disable the fail-fast check. - Use `worker.nodeSelector`, `worker.tolerations`, and `worker.affinity` for the worker Deployment. Use `kubernetesBackend.podTemplate` for task-pod selectors, tolerations, affinity, and topology constraints. - A toleration makes a pod eligible for a tainted node; it does not reserve capacity. Pair dedicated-node tolerations with matching selectors or affinity and autoscaler capacity. * * * ## Metrics The Helm chart includes built-in support for exporting OpenTelemetry metrics from the worker. Enable metrics by setting `metrics.enabled=true`: ```bash helm install oz-agent-worker ./charts/oz-agent-worker \ --namespace warp-oz \ --set worker.workerId=oz-k8s-worker \ --set image.tag=VERSION \ --set metrics.enabled=true ``` With the default `metrics.exporter=prometheus`, the chart creates a `Service` with Prometheus scrape annotations and exposes port `9464`. For clusters using the Prometheus Operator, set `metrics.podMonitor.create=true` to create a `PodMonitor`. To push metrics to an OTLP collector instead, set `metrics.exporter=otlp` and configure the endpoint via `metrics.extraEnv`. See [Monitoring](https://docs.warp.dev/platform/self-hosting/monitoring/) for the full list of Helm values, the metric catalog, and sample PromQL queries. * * * ## Operational notes - **Scaling** — The chart always deploys a single replica for a given `worker.workerId`. To run multiple workers, deploy multiple Helm releases with distinct worker IDs rather than scaling a single release horizontally. - **Security context** — The Deployment defaults to a non-root security context (`runAsUser: 10001`) with `allowPrivilegeEscalation: false` and all capabilities dropped. - **Liveness probe** — The Deployment includes a default `exec` liveness probe (`kill -0 1`). Override `worker.livenessProbe` for a custom probe, or set it to `null` to disable. - **In-cluster auth** — The chart assumes the worker runs inside the target cluster and uses in-cluster Kubernetes auth by default. * * * ## Related pages - [Self-hosted worker reference](https://docs.warp.dev/platform/self-hosting/reference/) — Full CLI flag and config file schema, including every Kubernetes backend field. - [Self-hosting overview](https://docs.warp.dev/platform/self-hosting/) — Managed vs unmanaged and the backend decision guide. - [Routing runs to this worker](https://docs.warp.dev/platform/self-hosting/#routing-runs-to-self-hosted-workers) — How to send tasks to your connected worker from the CLI, schedules, integrations, the API, and the web UI. - [Environments](https://docs.warp.dev/platform/environments/) — Define the task image, repos, and setup commands. - [Monitoring](https://docs.warp.dev/platform/self-hosting/monitoring/) — OpenTelemetry metrics, including Helm chart metrics values. - [Security and networking](https://docs.warp.dev/platform/self-hosting/security-and-networking/) — RBAC, admission policies, and data boundaries. - [Troubleshooting](https://docs.warp.dev/platform/self-hosting/troubleshooting/#kubernetes-backend) — Common Kubernetes-backend issues.