> For the complete documentation index, see [llms.txt](https://docs.warp.dev/llms.txt).
> Markdown versions of each page are available by appending .md to any URL.

# Self-hosting troubleshooting

Diagnose and fix common problems with self-hosted Automation Platform worker daemons across Docker, Kubernetes, and Direct backends.

Diagnostic guides for the `oz-agent-worker` daemon and its task execution. Use this page when a worker won’t start, won’t connect, tasks stay queued, or tasks fail.

Note

The steps below apply to the [managed architecture](https://docs.warp.dev/platform/self-hosting/#managed-architecture) (`oz-agent-worker` daemon). For [unmanaged](https://docs.warp.dev/platform/self-hosting/unmanaged/) deployments, refer to the documentation for the environment running `oz agent run` (e.g., GitHub Actions, Kubernetes).

* * *

## Worker won’t start

### Docker backend

**Cause:** Docker isn’t running, or the daemon platform isn’t supported.

**Fix:**

1.  Verify Docker is running: `docker info`.
2.  Confirm the daemon platform is `linux/amd64` or `linux/arm64`. Windows containers are not supported.
3.  If the worker runs inside Docker, confirm the `/var/run/docker.sock` mount is correct and the mounting user has permission to the socket.

### Kubernetes backend

**Cause:** The worker Deployment could not start, reach the Kubernetes API, or create its startup preflight Job. Missing Secrets, insufficient RBAC, and admission policies are common causes.

**Fix:**

1.  Run `kubectl describe pod -n NAMESPACE WORKER_POD`. For Helm installs, if the pod reports `CreateContainerConfigError`, confirm the Secret configured by `warp.apiKeySecret.name` exists and contains the key configured by `warp.apiKeySecret.key`.
2.  Check the worker logs for Kubernetes API or preflight diagnostics: `kubectl logs -n NAMESPACE WORKER_POD`.
3.  Confirm the worker’s namespace has these permissions: `create`, `get`, `list`, `watch`, `delete` on `jobs`; `get`, `list`, `watch` on `pods`; `get` on `pods/log`; `list` on `events`.
4.  Confirm the task namespace allows pods with a root init container, unless you enabled native image volumes with `kubernetesBackend.useImageVolumes=true`.
5.  If your cluster restricts image sources, set `kubernetesBackend.preflightImage` to an allowlisted image. The default is `busybox:1.36`.
6.  To pull the preflight image from a private registry, configure `imagePullSecrets` in `kubernetesBackend.podTemplate`.

A successful preflight confirms that the cluster can create the configured pod shape. It does not validate every task image, task Secret, setup command, or runtime network dependency.

### Direct backend

**Cause:** The `oz` CLI isn’t installed or isn’t on the worker’s `PATH`.

**Fix:**

1.  Install the Oz CLI on the worker host. See [Installing the CLI](https://docs.warp.dev/reference/cli/#installing-the-cli).
2.  If the CLI isn’t on `PATH`, set `oz_path` in the config file to the absolute path of the `oz` binary.

* * *

## Worker won’t connect

**Cause:** The API key is invalid, expired, or the host cannot reach the Automation Platform‘s backend.

**Fix:**

1.  Confirm your API key is correct, not expired, and has team scope.
2.  Regenerate the API key in **Settings** > **Cloud platform** > **API keys** if you suspect it’s invalid.
3.  Ensure the host has outbound internet access to `oz.warp.dev:443`.
4.  Check that no firewall rules are blocking WebSocket connections to `wss://oz.warp.dev`.
5.  Increase log verbosity with `--log-level debug` to see connection details.

See [Security and networking](https://docs.warp.dev/platform/self-hosting/security-and-networking/#network-requirements) for the full list of outbound endpoints the worker needs.

* * *

## Tasks not being picked up

**Cause:** The worker isn’t running, the `--host` value doesn’t match the worker’s `--worker-id`, or the worker and task belong to different teams.

**Fix:**

1.  Confirm the worker is running and connected. Check the worker logs for `Successfully connected to server`.
2.  Verify the `--host` (or `worker_host`) value you passed matches your `--worker-id` exactly. Case-sensitive.
3.  Ensure the worker’s team matches the team creating the task.

* * *

## Metrics not appearing

**Cause:** The worker is running but metrics aren’t showing up in Prometheus or your collector.

**Fix:**

1.  Verify `OTEL_METRICS_EXPORTER` is set correctly on the worker process. Run `curl -s localhost:9464/metrics` from the worker host (for `prometheus` mode) to confirm the endpoint is serving.
2.  For Prometheus scrape mode, confirm the bind address is `0.0.0.0` (not `localhost`) when running in Docker or Kubernetes. `localhost` is only reachable from inside the container.
3.  Confirm no firewall or network policy blocks the metrics port (default `9464`).
4.  For OTLP push mode, verify `OTEL_EXPORTER_OTLP_ENDPOINT` points to a reachable collector and that the protocol matches (`http/protobuf` vs `grpc`).
5.  When using the Helm chart, confirm `metrics.enabled=true` is set. Check that the `Service` and (optionally) `PodMonitor` were created: `kubectl get svc,podmonitor -n NAMESPACE`.
6.  If using `metrics.podMonitor.create=true`, verify the `monitoring.coreos.com` CRDs are installed in the cluster. The `PodMonitor` resource requires the Prometheus Operator.
7.  Restart the worker with `--log-level debug` and look for metrics-related error messages at startup.

See [Monitoring](https://docs.warp.dev/platform/self-hosting/monitoring/) for the full setup guide.

* * *

## Task failures

**Cause:** A variety of reasons depending on backend. Start with the diagnostic steps common to all backends, then follow the backend-specific checks.

**Fix (all backends):**

1.  Review task logs in the [cloud agent dashboard](https://oz.warp.dev) or via [session sharing](https://docs.warp.dev/agents/local-agents/session-sharing/).
2.  Use `--no-cleanup` to keep the container, Job, or workspace indefinitely for inspection. With Kubernetes cleanup enabled, failed Jobs remain for 24 hours by default.
3.  Use `--log-level debug` to see detailed execution logs.
4.  Ensure the worker machine or cluster has sufficient resources (CPU, memory, disk).

### Docker backend (task failures)

1.  Verify Docker is running (`docker info`).
2.  If using a custom image, confirm it is **glibc-based** (not Alpine/musl) and that its architecture matches the worker’s Docker daemon platform.

### Kubernetes backend (task failures)

Start with the pod phase and recent events:

```bash
kubectl get jobs,pods -n NAMESPACE
kubectl describe pod -n NAMESPACE TASK_POD
kubectl logs -n NAMESPACE TASK_POD -c CONTAINER_NAME
kubectl logs -n NAMESPACE TASK_POD -c CONTAINER_NAME --previous
```

Then match the reported reason to these checks:

-   **`Pending` or `Unschedulable`:** Read the pod’s scheduling events. Confirm that at least one eligible node has enough unallocated CPU and memory for the complete pod request. Also check task-pod selectors, affinity, taints and tolerations, topology constraints, resource quotas, and persistent volume requirements. If you depend on a node autoscaler, allow for provisioning latency and keep enough headroom to start task pods before the configured unschedulable timeout.
-   **`OOMKilled`:** Increase the memory in the agent profile’s instance shape. The selected instance shape sets the task container’s CPU and memory requests and limits, overriding those fields for the task container in the pod template. Also confirm that sidecars and init containers have appropriate resources.
-   **`ErrImagePull`, `ImagePullBackOff`, or `InvalidImageName`:** Confirm the image exists for the node architecture, the registry is reachable, and the task pod or service account references the required `imagePullSecrets`. A successful startup preflight does not validate a later task image.
-   **`CreateContainerConfigError` or `FailedMount`:** Inspect the pod events for the named Secret, ConfigMap, service account, or volume. Confirm the object exists in the task namespace and that every referenced key is present.
-   **Init container failure:** Inspect each init container’s status and logs. Admission policies must allow the worker’s sidecar-materialization init container to run as root unless native image volumes are enabled. Custom init containers must finish successfully before the task containers start.
-   **`Evicted`, `DeadlineExceeded`, or exit code `143`:** Check node pressure, lifecycle-tool events, the Job’s active deadline, and termination events. A task pod stores its workspace in pod-local ephemeral storage. Replacing an interrupted pod does not resume that run, so protect active task pods from voluntary node disruption. See [Protect active task pods from disruption](https://docs.warp.dev/platform/self-hosting/managed-kubernetes/#protect-active-task-pods-from-disruption).
-   **The task starts but cannot reach a dependency:** Test DNS resolution, TLS, and the destination from a task pod, not only from the worker pod. Worker connectivity to Warp does not prove that task-pod network policies, service mesh rules, proxies, or egress controls allow the task’s own dependencies.

The worker API key authenticates the worker to Warp. It is not injected as a general-purpose task credential. Provide repository, registry, and application credentials to task pods through your approved Secret integration and pod template.

### Direct backend (task failures)

1.  Verify the Oz CLI is accessible.
2.  Verify the workspace root directory has write permissions for the user running the worker.

* * *

## Image pull failures

### Docker backend (image pull)

1.  If using a private registry, ensure Docker credentials are available to the worker. See [Private Docker registries](https://docs.warp.dev/platform/self-hosting/managed-docker/#private-docker-registries).
2.  Try pulling the image manually on the worker host: `docker pull <image>`.

### Kubernetes backend (image pull)

1.  Configure `imagePullSecrets` in the `pod_template` section of your worker config.
2.  Verify the Secret exists in the task namespace and contains valid credentials.

### Both backends (image pull)

-   Verify the image exists and the tag is correct.
-   Check network connectivity from the worker/cluster to the registry.

* * *

## Related pages

-   [Self-hosting overview](https://docs.warp.dev/platform/self-hosting/) — Architecture and decision guide.
-   [Self-hosted worker reference](https://docs.warp.dev/platform/self-hosting/reference/) — CLI flags and config schema, including every flag mentioned here.
-   [Security and networking](https://docs.warp.dev/platform/self-hosting/security-and-networking/) — Outbound endpoints the worker needs.
-   [Agent Session Sharing](https://docs.warp.dev/agents/local-agents/session-sharing/) — Attach to running tasks to debug interactively.