# Troubleshooting failed deploys: errors and fixes

Fix a failed deploy: health check failed, build failed, domain not working, disk full, out of memory, server offline. The exact message, cause and fix.

When a deploy fails, the previous release keeps serving. The run's log names the failed step, the cause and the fix, then shows the last lines of output. Every failure has this shape:

Four common failures, each with what the log or the domain row says, what to look at, and the fix. While you fix a failed deploy, the previous release is still serving.

How a failure reads:

```text
<step>: <cause>
fix: <fix>
--- last output ---
<the last lines the command or the app printed>
```

To read a run's log, open the run in the dashboard, or run `ox runs <project>` and then `ox logs <project> --run <id>`. A failed `ox deploy --wait` exits with status 1. Then find the message you see below.

## Health check failed: the app crashed or did not answer within 120s

### Symptom

The log shows `waiting for … (up to 120s)`, then a step named `health (<unit>)` with one of these causes.

The health causes:

```text
the app exited while starting (systemd restarted it 3 times)
the app exited while starting (failed)
… did not answer within 120s: …
```

### Cause

The new release started but did not answer in time. ox checks it every 2 seconds for up to 120 seconds:

- When `[app] health` is set, ox sends a GET to that path and needs a 2xx or 3xx status.
- When it is not set, ox sends `GET /` and accepts any answer within 5 seconds except 502, 503 or 504.
- On a server whose agent is out of date, a crash can read `did not answer within 120s`. Run `ox servers update <server>`.

The cause line says what went wrong:

- `the app exited while starting` means the app crashed. systemd restarts it after each crash, and ox stops waiting at the third restart, usually within about 15 seconds. The output above the message ends with the error that stopped it. A worker says `the worker exited`, and a service says `the service exited`.
- `did not answer within 120s` means the app stayed up but never answered.
- `did not answer GET / within 5 seconds` means the home page is too slow to use as a health check.

The usual reasons are a start command that does not listen on `$PORT`, a missing variable that makes the app crash, or a health path that returns 404.

Some causes have a message of their own:

- The app refused its host name. The cause names the framework and the name, for example `Django refused the host name shop.k7mq2xw4ab.preview.getox.in (status 400): ALLOWED_HOSTS does not list it`. Rails' `config.hosts` and Vite's `allowedHosts` read the same way. A staging copy or a preview fails like this when it copied production's host list as plain text.
- A PHP app wrote to its home folder. A PHP app's home folder is read-only while it runs, so the cause says `the app wrote to <path> in its home folder, which is read-only while it runs`. Other apps may write to their home folder.
- The app wrote a database file into its release. A file with a `.db`, `.sqlite`, `.sqlite3` or `.db3` name stops the next deploy before anything changes, and the live version keeps serving. The message names the file and gives the `[storage] keep` line to set in `ox.toml`.

### Fix

Nothing changed for your visitors: the fix line ends with `the previous release is still serving`. Read the app's own log above the failure, fix the cause, and push.

- Make the app listen on `127.0.0.1:$PORT`, as in [[app] start](https://console.t12.sorv.dev/docs/config#app).
- Point `[app] health` at a path that answers quickly and without a login.
- For `the app exited while starting`, read the output above it for the error, then run the start command on your own computer to see it fail. A missing variable, a wrong path to the built file and a missing module are the usual causes.
- For `a port it tried to listen on is already in use`, listen for HTTP on `$PORT`, not a fixed number. If the app already does, another fixed port it opens, such as a gRPC or admin port, is probably held by another environment's copy of the app. Derive that port from `$PORT` too.
- For `killed by the out-of-memory killer as it started`, see [Not enough memory](https://console.t12.sorv.dev/docs/troubleshooting#out-of-memory).
- For a refused host name, read the name from `PUBLIC_HOST`, which ox sets to each environment's own host name. For Django, set `ALLOWED_HOSTS=${PUBLIC_HOST}` on the [Variables](https://console.t12.sorv.dev/docs/operate/variables#reference) screen. Set it in production too, so its staging copies and previews inherit the reference and not production's name. For Rails and Vite, the fix line gives the line to add to your config.
- For a write in a PHP app's home folder, point that path at a folder under `$OX_DATA_DIR` with the app's own setting, or set `[app] sandbox = "relaxed"`.
- For a database file in the release, set the `[storage] keep` line the message gives and deploy again. That deploy stops the app for a moment, moves the database into the project's data folder, and starts the new version on the same rows.

Apps other than PHP apps may write other files into their release, but they are gone after the next deploy. The deploy's log lists what the app wrote there. If it is data, add its folder to [`[storage] keep`](https://console.t12.sorv.dev/docs/config#tools), which links it to a writable folder that survives deploys.

## Build failed: exit status 1, or set these before deploying

### Symptom

The run stops at an install, build or migrate step, or before anything starts.

A failed build step:

```text
build 1/2: npm run build: exit status 1
fix: the command's own output is above; fix it in the repo and push
```

A deploy that is missing variables:

```text
preflight: variables: set these before deploying: SECRET_KEY; nothing on the host changed
fix: set them with `ox vars set shop SECRET_KEY`, or run `ox review shop --skip KEY` for each one the app does without
```

### Cause

A command from [[build]](https://console.t12.sorv.dev/docs/config#build) exited with an error, or the checks before the deploy (preflight) found a problem. `set these before deploying: SECRET_KEY` means a variable has no value. A failed build or migrate never touches the live release.

### Fix

Run `ox check` in the repo. It prints `Ready to deploy.`, or a list of problems with the field each one is about. For a failed command, run it on your own computer: the log shows each command as it ran.

For a missing variable, set it in the dashboard or with the `ox vars set` line the fix gives. If the app works without it, run `ox review <project> --skip KEY`. [Variables](https://console.t12.sorv.dev/docs/operate/variables#set) shows how.

## Domain not working: waiting for DNS, points elsewhere, no certificate

### Symptom

The domain's row on the project's Settings tab says **waiting for DNS**, **points elsewhere** or **No certificate** instead of **HTTPS on**.

### Cause

The name does not point at the server yet, it points somewhere else, or a Cloudflare proxy in front of it has no certificate for it. The row says which, for example `It resolves to 203.0.113.9, not this server. Change the record to:` followed by the A record to create.

### Fix

At your DNS provider, create or change the record the row shows ([how to point a domain](https://console.t12.sorv.dev/docs/operate/domains#a-record)). The row turns to **HTTPS on** by itself once the record resolves. From a terminal, `ox domains <project>` lists each domain with its DNS state and prints `Point each domain's A record at <ip>`.

For **No certificate** behind Cloudflare, check the name's certificate settings in Cloudflare. Or set its record to DNS only, and the server gets its own certificate.

## another operation is already running

### Symptom

A refused deploy:

```text
another operation is already running for shop: the daily backup, run Qd8xT2LmA0c (ox logs shop --run Qd8xT2LmA0c --follow); wait for it to finish, then retry
```

### Cause

ox runs one deploy, rollback, backup, restore or delete at a time per project. A push, the daily backup or another person started one just before you.

### Fix

Wait for it to end, then run your command again. The message names the run that holds the project. The `ox logs` command in the message follows that run until it ends. `ox runs shop` lists the project's other runs.

## interrupted by ox, not the app

### Symptom

An interrupted deploy:

```text
deploy interrupted: interrupted by ox, not the app: the ox agent stopped mid-run (an ox update or a reboot); ox is deploying it again as run Qd8xT2LmA0c (ox logs shop --run Qd8xT2LmA0c --follow)
```

### Cause

ox cut the run short, and your app did nothing wrong. The ox agent on your server restarted for an update or a reboot, or the server was offline. The previous release keeps serving.

### Fix

Usually nothing. ox runs an interrupted deploy once more, right away or when the server is back within the hour, and the message names that run. Follow it with the `ox logs` command it gives.

When the message ends with `deploy again` or `run it again`, ox did not retry, so run your command again. That happens for a rollback, for a retry that was interrupted too, when a newer run replaced it, or when the server came back after more than an hour.

## Disk full: a deploy needs at least 2 GB

### Symptom

A deploy stopped by preflight:

```text
disk: only 1400 MB free on /srv/ox; a deploy needs at least 2 GB
fix: delete old projects or grow the disk
```

A deploy is also refused when less than 10% of the disk is free. That message ends with `ox keeps at least 10% free so PostgreSQL never meets a full disk` and names what uses the most space. Once less than 5% is free, `sudo ox doctor` on the server says `only 4% of the disk is free (…); deploys and backups fail soon`.

### Cause

The server's disk is close to full. Releases, backups, logs and your app's own files all share it. ox cleans up by itself: when less than 15% of the disk is free, it removes old preview releases, deleted projects' backups, backups your S3 bucket already holds, build caches and older releases, until 20% is free. It also keeps each project's backups under 20% of the disk. It always keeps the newest release you can roll back to and the newest backup of each service.

### Fix

Delete projects you no longer need, or resize the disk at your provider. Adding an S3 bucket also helps, because ox can then remove local backups the bucket holds. See [how many backups ox keeps](https://console.t12.sorv.dev/docs/operate/backups#retention).

## Not enough memory: killed by the out-of-memory killer

### Symptom

Preflight refuses the deploy, a build step dies, or an app, worker or service is stopped as it starts:

Nine memory failures:

```text
capacity: only 1731 MB of the server's 1987 MB is free to reserve, 573 MB less than the 2304 MB shop needs. …
capacity: only 1920 MB of the 3968 MB previews share is free to reserve, 128 MB less than the 2048 MB shop-feat needs. …
killed by the out-of-memory killer (the build needs more memory than the project or host allows)
killed by the out-of-memory killer: the whole server ran out of memory while it ran (the apps and services running on it held the rest)
killed by the out-of-memory killer: the build used all 5952 MB a build may use on this server (three quarters of it)
killed by the out-of-memory killer: the step used all 3968 MB previews on this server share (half of it, so production keeps the rest)
killed by the out-of-memory killer as it started: it needs more memory than its limit allows
killed by the out-of-memory killer as it started: the previews on this server ran out of the memory they share
killed by the out-of-memory killer as it started: the whole server ran out of memory (what else runs on it held the rest)
```

### Cause

Each message points at one limit:

- `capacity:` lines come from the check before a deploy. ox reserves each project's `[limits] memory` and its services' memory, even while the app uses less. The apps, their services and their staging copies must fit in the server's memory. A stopped project gives back its `[limits] memory`, but its services stay reserved. Previews share half of the server's memory, and production keeps the other half.
- A build step may use at most three quarters of the server's memory, or the project's `[limits] memory` when that is lower. This holds for a preview's build too. A preview's migration stays inside the previews' half.
- An app, worker or service has no memory limit unless `ox.toml` sets one, with its own `memory` or the project's `[limits] memory`. One stopped `as it started` went past that limit, or, for a preview, past the previews' shared half.
- `the whole server ran out` means the step stayed under its own limit, but the apps and services running on the server held the rest.

Build and migrate steps run one at a time on a server, so another build is never the cause. A step that has to wait names the one it waits for: `waiting for the build of shop (run ...) to finish`. A `[models]` download is not a build step and may run beside one. ox stops a step whose run was cancelled or whose agent restarted.

### Fix

For a `capacity:` refusal, the first line says how much of the server is still free to reserve and how far short this project is. The lines under it say which projects reserve the rest and what this project needs the memory for. Each fix it lists makes enough room on its own, with its numbers:

- Lower this project's `[limits] memory` (`set memory under [limits] in ox.toml to at most 3040 MB`).
- Remove a service the app does not need (`remove postgres (frees 256 MB) from ox.toml if the app does not need it`).
- Stop projects that free enough (`stop discord-ai-mod-staging (frees 1536 MB)`, or `stop a (frees 600 MB) and b (frees 500 MB)` when it takes both).
- Use a bigger server.

When none of those is enough, delete a project you do not need. `ox stop` frees a project's `[limits] memory` at once, but not its services' memory. To free that too, delete the project or make the service shared.

The same check runs for `ox start` and `ox rollback`. A refused `ox start` leaves the project stopped: stop a project the fix names first, or set `[limits] memory` to the size it gives and deploy before you start it. A refused `ox rollback` to a release with a larger `[limits] memory` changes nothing.

For the other messages:

- A build step: raise `memory` in `ox.toml` if it sets one, or use a bigger server. For a preview's build, use a server with more memory, or make the build use less, for example by splitting it into smaller commands.
- An app, worker or service stopped as it starts: raise its `memory` in `ox.toml` (or `[limits] memory` if the project sets one), or use a bigger server. For a preview, remove another preview.
- A preview's migration that used the previews' share: use a server with more memory, or make the step use less. The message says to remove another preview only when other previews held part of the share at the time.
- The whole server ran out: use a bigger server, or stop a project, staging copy or preview you do not need. Running [fewer previews at once](https://console.t12.sorv.dev/docs/operate/previews#limits) also frees memory.

## A Node build stops: ran out of memory at its heap limit

### Symptom

A build step ends with exit code 134 and the build output shows `JavaScript heap out of memory`. The failure names the heap Node reached, read from its last trace:

The heap failure:

```text
Node ran out of memory at its heap limit, about 2082 MB
```

### Cause

Node limits its heap to about half the memory it may use, and never more than about 2.2 GB. A large front-end build, such as a big Vite or webpack bundle in a monorepo, can need more. When a build step may use more than 5 GB, ox raises the heap to half of the step's memory. A running app keeps Node's own limit.

### Fix

On the Variables screen, set `NODE_OPTIONS` to the size the failure suggests. That is half again above the heap Node reached, and at least `--max-old-space-size=4096` (megabytes). If the variable already holds other flags, add this one beside them. Your variable replaces the one ox sets.

If the build command sets `--max-old-space-size` itself, that flag wins over the variable, so the failure tells you to raise it in the command instead. The server needs that much free memory while the build runs, and any `[limits] memory` must be higher than the number you set.

## A compiled module needs a system library this server does not have

### Symptom

A build, a migration or the app's start fails, and the output shows `cannot open shared object file`. The failure names the library:

The library failure:

```text
a compiled module needs the system library libcairo.so.2, which this server does not have
```

### Cause

A package with compiled code, such as a Node addon or a Python extension, loads a library from the operating system when it starts. A Docker image often has that library already. A fresh Ubuntu server has only what ox and your `packages` install.

### Fix

Add the Ubuntu package that ships the file to `packages` in `ox.toml`. For `libcairo.so.2`, that is `packages = ["libcairo2"]`. [packages.ubuntu.com](https://packages.ubuntu.com) finds the package by the file's name.

Sometimes this Ubuntu ships only another version of the library. Ubuntu 26.04 has `libxml2.so.16`, not `libxml2.so.2`, for example. Then you need a build of the module made for that version, or a version of the module that does not use the library.

## The app keeps crashing and restarting

### Symptom

The deploy passed, but later an alert says:

The crash alert:

```text
web of shop keeps crashing

It restarted 5 times in the last 10 minutes. The logs show why.
```

When the server stopped it for running out of memory, the alert says that instead, with the fix:

The crash alert for memory:

```text
web of shop keeps crashing

It restarted 5 times in the last 10 minutes, because it ran out of memory. It needs more memory than its limit allows.

What to do:
- Raise its memory in ox.toml (the memory of its own entry, or [limits] memory if the project sets one).
- Or use a bigger server.
```

### Cause

The process starts, then exits, and the server restarts it. The first restart comes after 2 seconds, and each one after that waits longer, up to once a minute, so a broken process never floods the server. The server never gives up, so the process comes back by itself once what it needs, such as a database or the network, is back. A missing variable, a lost database connection, a full disk or too little memory are common reasons.

A deploy never leaves a process in this loop. When a new service or HTTP worker does not come up, the deploy puts back the version that ran before, or stops it.

### Fix

Open the **Logs** tab, pick **Errors**, and read the lines just before each restart. From a terminal, run `ox logs shop`. See [Logs](https://console.t12.sorv.dev/docs/operate/logs). If the last deploy caused it, [roll back](https://console.t12.sorv.dev/docs/operate/rollback) while you fix it. An alert about memory carries its own fix, and [Not enough memory](https://console.t12.sorv.dev/docs/troubleshooting#out-of-memory) explains more.

## Server offline: the server has not connected yet

### Symptom

A deploy or a command is refused with one of these:

An unreachable server:

```text
server my-server has not connected yet
server my-server went offline 12 minutes ago
```

### Cause

The ox agent on the server is not talking to ox. The server may be off or off the network, or the agent may have stopped. Your sites keep running unless the server itself is down, but ox cannot deploy to it until it reconnects.

A deploy that starts while the server is offline waits 30 seconds for it, then ends [interrupted](https://console.t12.sorv.dev/docs/troubleshooting#interrupted). ox runs it again if the server is back within the hour. A push you made meanwhile deploys within a minute of the server reconnecting.

`has not connected yet` means the agent never reached ox after the install, so the install did not finish. The dashboard shows the server as Not connected yet, and ox sends no offline alert for it. A firewall or proxy that blocks the agent's connection out to ox is a common reason.

### Fix

Check that the server is on at your provider. Then, on the server:

Check the agent:

```sh
sudo systemctl status ox-agent
sudo journalctl -u ox-agent -n 50
```

`sudo systemctl restart ox-agent` starts the agent again. If the server was rebuilt, add it again from the dashboard.

## Install refused: this server can't take ox yet

### Symptom

The Add server command stops before it changes anything:

A server that cannot take ox:

```text
this server can't take ox yet; nothing was changed:
  - port 80 is in use by nginx; ox's proxy (Caddy) needs it: stop that program, or use a fresh server
  - the server has 512 MB of RAM and ox needs at least 1 GB: resize it
  - the disk has 6.2 GB free on / and ox needs at least 10 GB: free space or resize the disk
```

### Cause

ox needs ports 80 and 443 free for its own Caddy, and no Caddy you installed yourself, even a stopped one: ox would take it over. It also needs IPv4 internet access to reach GitHub, at least 1 GB of memory, and at least 10 GB of free disk.

### Fix

Do what each line says, then run the same Add server command again on the same server. A failed run hands the command back, so it works again on that server until it expires. `sudo ox install --check` runs only these checks, without installing anything. A fresh Ubuntu LTS server avoids most of these problems.

Press Add server for a new command when the old one has expired, or when you add a different server. Also get a new one if the rerun says the command was already taken. That means the server's address changed since the first run, for example because it reached ox over IPv6 once and IPv4 the next time, or its NAT or DHCP address changed.

## Cloudflare answered this server with a bot challenge

### Symptom

The Add server command, `ox install`, or the agent's log (`sudo journalctl -u ox-agent`) says:

A Cloudflare challenge:

```text
Cloudflare, which sits in front of ox Cloud, answered this server with a bot challenge
instead of passing it on to ox (Cloudflare Ray ID 8d1f2e3a4b5c6d7e-FRA).
This is on ox's side, not this server's
```

### Cause

ox sits behind Cloudflare, and one of Cloudflare's bot settings stopped your server's request before it reached ox. A server can't answer a challenge meant for a browser. Nothing on your server is wrong.

### Fix

Send ox support the Ray ID from the message. Once ox has fixed it, run the same Add server command again. If the run says so, or the command has expired, press Add server for a new one. An agent that was already installed reconnects by itself.

## repository access removed: GitHub gives ox no access

### Symptom

A deploy that cannot read the repo:

```text
repository access removed: GitHub gives ox no access to acme/shop (Not Found); connect GitHub again and pick the repository
the repository moved to another account on GitHub: ox deploys acme/shop, and GitHub now names it bob/shop (ID 123456), outside the account ox's GitHub App is installed on, so ox stopped fetching it; install the App there, connect GitHub again and pick the repository
```

### Cause

The ox GitHub App was removed from the repository or the account on GitHub, or the repository was transferred to another account. ox tracks each repository by its GitHub ID, which never changes. So when you rename the repository, its organization or your user, ox follows the new name by itself. A transfer to an account the App is not installed on is different: ox stops fetching from it.

### Fix

Connect GitHub again from the dashboard, pick the repository, and deploy. After a transfer, install the App on the new account, add the repository there as a new project, then delete the old project.

## organization repository: your GitHub access removed

### Symptom

A deploy from an organization's repo you can no longer read:

```text
your GitHub access removed: your GitHub account can no longer read acme/shop, so ox stopped fetching it; once GitHub gives you access again, sign in to ox again
```

### Cause

The project comes from an organization's repository, and your own GitHub account can no longer read it. The organization removed you, or took you off the repository. When the GitHub App was installed by someone else, ox asks GitHub whether your account can still read the repository before it fetches. It trusts a yes for up to an hour. Checks, pull request states, check runs and GitHub Deployments also stop once that hour runs out without a new yes.

### Fix

Ask the organization for access to the repository. Once GitHub shows it to you, sign in to ox again to clear the mark, then deploy.

## organization repository: ox could not check with GitHub

### Symptom

A deploy ox could not check:

```text
ox could not check with GitHub that your GitHub account can still read acme/shop, so it did not fetch it; sign in to ox again (...)
ox could not check with GitHub that your GitHub account can still read acme/shop, so it did not fetch it; start an SSO session for acme on GitHub, then sign in to ox again (...)
ox could not check with GitHub that your GitHub account can still read acme/shop, so it did not fetch it; try again in a few minutes (...)
ox could not check with GitHub which account the installation of acme/shop is on, so it did not fetch it; try again in a few minutes (...)
ox has not checked in the last hour that your GitHub account can still read acme/shop, so it did not call GitHub for it; the next fetch checks
```

### Cause

Before it fetches an organization's repository for you, ox asks GitHub whether your account can still read it, and GitHub did not answer. Your sign-in expired or was revoked, the organization requires SAML single sign-on and your session for it ended, or GitHub was down or rate limited. The fourth message means ox could not ask GitHub which account the App is installed on. ox does not fetch a repository it could not check, and it does not mark the project. Checks, pull request states, check runs and GitHub Deployments wait for the next fetch to check.

### Fix

Do what the message says:

- `sign in to ox again`: sign in again.
- `start an SSO session`: open the organization's page on GitHub and follow its single sign-on prompt, then sign in to ox again.
- `try again in a few minutes`: wait, then deploy again. ox asks GitHub again after 5 minutes, or after a minute for the installation's account.
- `the next fetch checks`: deploy, or wait for the next push.

## not signed in; run ox login

### Symptom

CLI sign-in errors:

```text
not signed in; run ox login
the plane refused the token (expired or revoked); sign in again with ox login
```

### Cause

This computer has no token yet, or its token expired or was revoked in the dashboard. The command exits with status 3.

### Fix

Run `ox login` and approve the code in the browser. A `could not reach …` message with status 4 is a network problem, and signing in again does not fix it. See [exit codes](https://console.t12.sorv.dev/docs/cli#exit-codes).

Last updated 2026-10-08. The page as HTML: https://console.t12.sorv.dev/docs/troubleshooting