Troubleshooting failed deploys: errors and fixes

When a deploy fails, the previous release keeps serving. The run's log names the failed step, the cause and the fix, then shows the last lines of output. Every failure has this shape:

<step>: <cause>
fix: <fix>
--- last output ---
<the last lines the command or the app printed>

To read a run's log, open the run in the dashboard, or run ox runs <project> and then ox logs <project> --run <id>. A failed ox deploy --wait exits with status 1. Then find the message you see below.

Health check failed: the app crashed or did not answer within 120s

Symptom

The log shows waiting for … (up to 120s), then a step named health (<unit>) with one of these causes.

the app exited while starting (systemd restarted it 3 times)
the app exited while starting (failed)
… did not answer within 120s: …

Cause

The new release started but did not answer in time. ox checks it every 2 seconds for up to 120 seconds:

• When [app] health is set, ox sends a GET to that path and needs a 2xx or 3xx status.
• When it is not set, ox sends GET / and accepts any answer within 5 seconds except 502, 503 or 504.
• On a server whose agent is out of date, a crash can read did not answer within 120s. Run ox servers update <server>.

The cause line says what went wrong:

• the app exited while starting means the app crashed. systemd restarts it after each crash, and ox stops waiting at the third restart, usually within about 15 seconds. The output above the message ends with the error that stopped it. A worker says the worker exited, and a service says the service exited.
• did not answer within 120s means the app stayed up but never answered.
• did not answer GET / within 5 seconds means the home page is too slow to use as a health check.

The usual reasons are a start command that does not listen on $PORT, a missing variable that makes the app crash, or a health path that returns 404.

Some causes have a message of their own:

• The app refused its host name. The cause names the framework and the name, for example Django refused the host name shop.k7mq2xw4ab.preview.getox.in (status 400): ALLOWED_HOSTS does not list it. Rails' config.hosts and Vite's allowedHosts read the same way. A staging copy or a preview fails like this when it copied production's host list as plain text.
• A PHP app wrote to its home folder. A PHP app's home folder is read-only while it runs, so the cause says the app wrote to <path> in its home folder, which is read-only while it runs. Other apps may write to their home folder.
• The app wrote a database file into its release. A file with a .db, .sqlite, .sqlite3 or .db3 name stops the next deploy before anything changes, and the live version keeps serving. The message names the file and gives the [storage] keep line to set in ox.toml.

Fix

Nothing changed for your visitors: the fix line ends with the previous release is still serving. Read the app's own log above the failure, fix the cause, and push.

• Make the app listen on 127.0.0.1:$PORT, as in [app] start.
• Point [app] health at a path that answers quickly and without a login.
• For the app exited while starting, read the output above it for the error, then run the start command on your own computer to see it fail. A missing variable, a wrong path to the built file and a missing module are the usual causes.
• For a port it tried to listen on is already in use, listen for HTTP on $PORT, not a fixed number. If the app already does, another fixed port it opens, such as a gRPC or admin port, is probably held by another environment's copy of the app. Derive that port from $PORT too.
• For killed by the out-of-memory killer as it started, see Not enough memory.
• For a refused host name, read the name from PUBLIC_HOST, which ox sets to each environment's own host name. For Django, set ALLOWED_HOSTS=${PUBLIC_HOST} on the Variables screen. Set it in production too, so its staging copies and previews inherit the reference and not production's name. For Rails and Vite, the fix line gives the line to add to your config.
• For a write in a PHP app's home folder, point that path at a folder under $OX_DATA_DIR with the app's own setting, or set [app] sandbox = "relaxed".
• For a database file in the release, set the [storage] keep line the message gives and deploy again. That deploy stops the app for a moment, moves the database into the project's data folder, and starts the new version on the same rows.

Apps other than PHP apps may write other files into their release, but they are gone after the next deploy. The deploy's log lists what the app wrote there. If it is data, add its folder to [storage] keep, which links it to a writable folder that survives deploys.

Build failed: exit status 1, or set these before deploying

Symptom

The run stops at an install, build or migrate step, or before anything starts.

build 1/2: npm run build: exit status 1
fix: the command's own output is above; fix it in the repo and push

preflight: variables: set these before deploying: SECRET_KEY; nothing on the host changed
fix: set them with `ox vars set shop SECRET_KEY`, or run `ox review shop --skip KEY` for each one the app does without

Cause

A command from [build] exited with an error, or the checks before the deploy (preflight) found a problem. set these before deploying: SECRET_KEY means a variable has no value. A failed build or migrate never touches the live release.

Fix

Run ox check in the repo. It prints Ready to deploy., or a list of problems with the field each one is about. For a failed command, run it on your own computer: the log shows each command as it ran.

For a missing variable, set it in the dashboard or with the ox vars set line the fix gives. If the app works without it, run ox review <project> --skip KEY. Variables shows how.

Domain not working: waiting for DNS, points elsewhere, no certificate

Symptom

The domain's row on the project's Settings tab says waiting for DNS, points elsewhere or No certificate instead of HTTPS on.

Cause

The name does not point at the server yet, it points somewhere else, or a Cloudflare proxy in front of it has no certificate for it. The row says which, for example It resolves to 203.0.113.9, not this server. Change the record to: followed by the A record to create.

Fix

At your DNS provider, create or change the record the row shows (how to point a domain). The row turns to HTTPS on by itself once the record resolves. From a terminal, ox domains <project> lists each domain with its DNS state and prints Point each domain's A record at <ip>.

For No certificate behind Cloudflare, check the name's certificate settings in Cloudflare. Or set its record to DNS only, and the server gets its own certificate.

another operation is already running

Symptom

another operation is already running for shop: the daily backup, run Qd8xT2LmA0c (ox logs shop --run Qd8xT2LmA0c --follow); wait for it to finish, then retry

Cause

ox runs one deploy, rollback, backup, restore or delete at a time per project. A push, the daily backup or another person started one just before you.

Fix

Wait for it to end, then run your command again. The message names the run that holds the project. The ox logs command in the message follows that run until it ends. ox runs shop lists the project's other runs.

interrupted by ox, not the app

Symptom

deploy interrupted: interrupted by ox, not the app: the ox agent stopped mid-run (an ox update or a reboot); ox is deploying it again as run Qd8xT2LmA0c (ox logs shop --run Qd8xT2LmA0c --follow)

Cause

ox cut the run short, and your app did nothing wrong. The ox agent on your server restarted for an update or a reboot, or the server was offline. The previous release keeps serving.

Fix

Usually nothing. ox runs an interrupted deploy once more, right away or when the server is back within the hour, and the message names that run. Follow it with the ox logs command it gives.

When the message ends with deploy again or run it again, ox did not retry, so run your command again. That happens for a rollback, for a retry that was interrupted too, when a newer run replaced it, or when the server came back after more than an hour.

Disk full: a deploy needs at least 2 GB

Symptom

disk: only 1400 MB free on /srv/ox; a deploy needs at least 2 GB
fix: delete old projects or grow the disk

A deploy is also refused when less than 10% of the disk is free. That message ends with ox keeps at least 10% free so PostgreSQL never meets a full disk and names what uses the most space. Once less than 5% is free, sudo ox doctor on the server says only 4% of the disk is free (…); deploys and backups fail soon.

Cause

The server's disk is close to full. Releases, backups, logs and your app's own files all share it. ox cleans up by itself: when less than 15% of the disk is free, it removes old preview releases, deleted projects' backups, backups your S3 bucket already holds, build caches and older releases, until 20% is free. It also keeps each project's backups under 20% of the disk. It always keeps the newest release you can roll back to and the newest backup of each service.

Fix

Delete projects you no longer need, or resize the disk at your provider. Adding an S3 bucket also helps, because ox can then remove local backups the bucket holds. See how many backups ox keeps.

Not enough memory: killed by the out-of-memory killer

Symptom

Preflight refuses the deploy, a build step dies, or an app, worker or service is stopped as it starts:

capacity: only 1731 MB of the server's 1987 MB is free to reserve, 573 MB less than the 2304 MB shop needs. …
capacity: only 1920 MB of the 3968 MB previews share is free to reserve, 128 MB less than the 2048 MB shop-feat needs. …
killed by the out-of-memory killer (the build needs more memory than the project or host allows)
killed by the out-of-memory killer: the whole server ran out of memory while it ran (the apps and services running on it held the rest)
killed by the out-of-memory killer: the build used all 5952 MB a build may use on this server (three quarters of it)
killed by the out-of-memory killer: the step used all 3968 MB previews on this server share (half of it, so production keeps the rest)
killed by the out-of-memory killer as it started: it needs more memory than its limit allows
killed by the out-of-memory killer as it started: the previews on this server ran out of the memory they share
killed by the out-of-memory killer as it started: the whole server ran out of memory (what else runs on it held the rest)

Cause

Each message points at one limit:

• capacity: lines come from the check before a deploy. ox reserves each project's [limits] memory and its services' memory, even while the app uses less. The apps, their services and their staging copies must fit in the server's memory. A stopped project gives back its [limits] memory, but its services stay reserved. Previews share half of the server's memory, and production keeps the other half.
• A build step may use at most three quarters of the server's memory, or the project's [limits] memory when that is lower. This holds for a preview's build too. A preview's migration stays inside the previews' half.
• An app, worker or service has no memory limit unless ox.toml sets one, with its own memory or the project's [limits] memory. One stopped as it started went past that limit, or, for a preview, past the previews' shared half.
• the whole server ran out means the step stayed under its own limit, but the apps and services running on the server held the rest.

Build and migrate steps run one at a time on a server, so another build is never the cause. A step that has to wait names the one it waits for: waiting for the build of shop (run ...) to finish. A [models] download is not a build step and may run beside one. ox stops a step whose run was cancelled or whose agent restarted.

Fix

For a capacity: refusal, the first line says how much of the server is still free to reserve and how far short this project is. The lines under it say which projects reserve the rest and what this project needs the memory for. Each fix it lists makes enough room on its own, with its numbers:

• Lower this project's [limits] memory (set memory under [limits] in ox.toml to at most 3040 MB).
• Remove a service the app does not need (remove postgres (frees 256 MB) from ox.toml if the app does not need it).
• Stop projects that free enough (stop discord-ai-mod-staging (frees 1536 MB), or stop a (frees 600 MB) and b (frees 500 MB) when it takes both).
• Use a bigger server.

When none of those is enough, delete a project you do not need. ox stop frees a project's [limits] memory at once, but not its services' memory. To free that too, delete the project or make the service shared.

The same check runs for ox start and ox rollback. A refused ox start leaves the project stopped: stop a project the fix names first, or set [limits] memory to the size it gives and deploy before you start it. A refused ox rollback to a release with a larger [limits] memory changes nothing.

For the other messages:

• A build step: raise memory in ox.toml if it sets one, or use a bigger server. For a preview's build, use a server with more memory, or make the build use less, for example by splitting it into smaller commands.
• An app, worker or service stopped as it starts: raise its memory in ox.toml (or [limits] memory if the project sets one), or use a bigger server. For a preview, remove another preview.
• A preview's migration that used the previews' share: use a server with more memory, or make the step use less. The message says to remove another preview only when other previews held part of the share at the time.
• The whole server ran out: use a bigger server, or stop a project, staging copy or preview you do not need. Running fewer previews at once also frees memory.

A Node build stops: ran out of memory at its heap limit

Symptom

A build step ends with exit code 134 and the build output shows JavaScript heap out of memory. The failure names the heap Node reached, read from its last trace:

Node ran out of memory at its heap limit, about 2082 MB

Cause

Node limits its heap to about half the memory it may use, and never more than about 2.2 GB. A large front-end build, such as a big Vite or webpack bundle in a monorepo, can need more. When a build step may use more than 5 GB, ox raises the heap to half of the step's memory. A running app keeps Node's own limit.

Fix

On the Variables screen, set NODE_OPTIONS to the size the failure suggests. That is half again above the heap Node reached, and at least --max-old-space-size=4096 (megabytes). If the variable already holds other flags, add this one beside them. Your variable replaces the one ox sets.

If the build command sets --max-old-space-size itself, that flag wins over the variable, so the failure tells you to raise it in the command instead. The server needs that much free memory while the build runs, and any [limits] memory must be higher than the number you set.

A compiled module needs a system library this server does not have

Symptom

A build, a migration or the app's start fails, and the output shows cannot open shared object file. The failure names the library:

a compiled module needs the system library libcairo.so.2, which this server does not have

Cause

A package with compiled code, such as a Node addon or a Python extension, loads a library from the operating system when it starts. A Docker image often has that library already. A fresh Ubuntu server has only what ox and your packages install.

Fix

Add the Ubuntu package that ships the file to packages in ox.toml. For libcairo.so.2, that is packages = ["libcairo2"]. packages.ubuntu.com finds the package by the file's name.

Sometimes this Ubuntu ships only another version of the library. Ubuntu 26.04 has libxml2.so.16, not libxml2.so.2, for example. Then you need a build of the module made for that version, or a version of the module that does not use the library.

The app keeps crashing and restarting

Symptom

The deploy passed, but later an alert says:

web of shop keeps crashing

It restarted 5 times in the last 10 minutes. The logs show why.

When the server stopped it for running out of memory, the alert says that instead, with the fix:

web of shop keeps crashing

It restarted 5 times in the last 10 minutes, because it ran out of memory. It needs more memory than its limit allows.

What to do:
- Raise its memory in ox.toml (the memory of its own entry, or [limits] memory if the project sets one).
- Or use a bigger server.

Cause

The process starts, then exits, and the server restarts it. The first restart comes after 2 seconds, and each one after that waits longer, up to once a minute, so a broken process never floods the server. The server never gives up, so the process comes back by itself once what it needs, such as a database or the network, is back. A missing variable, a lost database connection, a full disk or too little memory are common reasons.

A deploy never leaves a process in this loop. When a new service or HTTP worker does not come up, the deploy puts back the version that ran before, or stops it.

Fix

Open the Logs tab, pick Errors, and read the lines just before each restart. From a terminal, run ox logs shop. See Logs. If the last deploy caused it, roll back while you fix it. An alert about memory carries its own fix, and Not enough memory explains more.

Server offline: the server has not connected yet

Symptom

A deploy or a command is refused with one of these:

server my-server has not connected yet
server my-server went offline 12 minutes ago

Cause

The ox agent on the server is not talking to ox. The server may be off or off the network, or the agent may have stopped. Your sites keep running unless the server itself is down, but ox cannot deploy to it until it reconnects.

A deploy that starts while the server is offline waits 30 seconds for it, then ends interrupted. ox runs it again if the server is back within the hour. A push you made meanwhile deploys within a minute of the server reconnecting.

has not connected yet means the agent never reached ox after the install, so the install did not finish. The dashboard shows the server as Not connected yet, and ox sends no offline alert for it. A firewall or proxy that blocks the agent's connection out to ox is a common reason.

Fix

Check that the server is on at your provider. Then, on the server:

sudo systemctl status ox-agent
sudo journalctl -u ox-agent -n 50

sudo systemctl restart ox-agent starts the agent again. If the server was rebuilt, add it again from the dashboard.

Install refused: this server can't take ox yet

Symptom

The Add server command stops before it changes anything:

this server can't take ox yet; nothing was changed:
  - port 80 is in use by nginx; ox's proxy (Caddy) needs it: stop that program, or use a fresh server
  - the server has 512 MB of RAM and ox needs at least 1 GB: resize it
  - the disk has 6.2 GB free on / and ox needs at least 10 GB: free space or resize the disk

Cause

ox needs ports 80 and 443 free for its own Caddy, and no Caddy you installed yourself, even a stopped one: ox would take it over. It also needs IPv4 internet access to reach GitHub, at least 1 GB of memory, and at least 10 GB of free disk.

Fix

Do what each line says, then run the same Add server command again on the same server. A failed run hands the command back, so it works again on that server until it expires. sudo ox install --check runs only these checks, without installing anything. A fresh Ubuntu LTS server avoids most of these problems.

Press Add server for a new command when the old one has expired, or when you add a different server. Also get a new one if the rerun says the command was already taken. That means the server's address changed since the first run, for example because it reached ox over IPv6 once and IPv4 the next time, or its NAT or DHCP address changed.

Cloudflare answered this server with a bot challenge

Symptom

The Add server command, ox install, or the agent's log (sudo journalctl -u ox-agent) says:

Cloudflare, which sits in front of ox Cloud, answered this server with a bot challenge
instead of passing it on to ox (Cloudflare Ray ID 8d1f2e3a4b5c6d7e-FRA).
This is on ox's side, not this server's

Cause

ox sits behind Cloudflare, and one of Cloudflare's bot settings stopped your server's request before it reached ox. A server can't answer a challenge meant for a browser. Nothing on your server is wrong.

Fix

Send ox support the Ray ID from the message. Once ox has fixed it, run the same Add server command again. If the run says so, or the command has expired, press Add server for a new one. An agent that was already installed reconnects by itself.

repository access removed: GitHub gives ox no access

Symptom

repository access removed: GitHub gives ox no access to acme/shop (Not Found); connect GitHub again and pick the repository
the repository moved to another account on GitHub: ox deploys acme/shop, and GitHub now names it bob/shop (ID 123456), outside the account ox's GitHub App is installed on, so ox stopped fetching it; install the App there, connect GitHub again and pick the repository

Cause

The ox GitHub App was removed from the repository or the account on GitHub, or the repository was transferred to another account. ox tracks each repository by its GitHub ID, which never changes. So when you rename the repository, its organization or your user, ox follows the new name by itself. A transfer to an account the App is not installed on is different: ox stops fetching from it.

Fix

Connect GitHub again from the dashboard, pick the repository, and deploy. After a transfer, install the App on the new account, add the repository there as a new project, then delete the old project.

organization repository: your GitHub access removed

Symptom

your GitHub access removed: your GitHub account can no longer read acme/shop, so ox stopped fetching it; once GitHub gives you access again, sign in to ox again

Cause

The project comes from an organization's repository, and your own GitHub account can no longer read it. The organization removed you, or took you off the repository. When the GitHub App was installed by someone else, ox asks GitHub whether your account can still read the repository before it fetches. It trusts a yes for up to an hour. Checks, pull request states, check runs and GitHub Deployments also stop once that hour runs out without a new yes.

Fix

Ask the organization for access to the repository. Once GitHub shows it to you, sign in to ox again to clear the mark, then deploy.

organization repository: ox could not check with GitHub

Symptom

ox could not check with GitHub that your GitHub account can still read acme/shop, so it did not fetch it; sign in to ox again (...)
ox could not check with GitHub that your GitHub account can still read acme/shop, so it did not fetch it; start an SSO session for acme on GitHub, then sign in to ox again (...)
ox could not check with GitHub that your GitHub account can still read acme/shop, so it did not fetch it; try again in a few minutes (...)
ox could not check with GitHub which account the installation of acme/shop is on, so it did not fetch it; try again in a few minutes (...)
ox has not checked in the last hour that your GitHub account can still read acme/shop, so it did not call GitHub for it; the next fetch checks

Cause

Before it fetches an organization's repository for you, ox asks GitHub whether your account can still read it, and GitHub did not answer. Your sign-in expired or was revoked, the organization requires SAML single sign-on and your session for it ended, or GitHub was down or rate limited. The fourth message means ox could not ask GitHub which account the App is installed on. ox does not fetch a repository it could not check, and it does not mark the project. Checks, pull request states, check runs and GitHub Deployments wait for the next fetch to check.

Fix

Do what the message says:

• sign in to ox again: sign in again.
• start an SSO session: open the organization's page on GitHub and follow its single sign-on prompt, then sign in to ox again.
• try again in a few minutes: wait, then deploy again. ox asks GitHub again after 5 minutes, or after a minute for the installation's account.
• the next fetch checks: deploy, or wait for the next push.

not signed in; run ox login

Symptom

not signed in; run ox login
the plane refused the token (expired or revoked); sign in again with ox login

Cause

This computer has no token yet, or its token expired or was revoked in the dashboard. The command exits with status 3.

Fix

Run ox login and approve the code in the browser. A could not reach … message with status 4 is a network problem, and signing in again does not fix it. See exit codes.