Initialise embedded PostgreSQL where a platform volume lets it - #281
Merged
davidmckayv merged 2 commits intoAug 28, 2026
Merged
Conversation
`EMBEDDED_POSTGRES=on` could not create its cluster on a platform whose volume is an ext4 mount, which is most of them, and it failed differently depending on which of two paths the volume was mounted at. Neither worked, and the two suggestions in this repo disagreed: docs/deployment.md said /var/lib/postgresql/data, the Dockerfile comment said /var/lib/postgresql. Mounted at the parent, the mount arrives owned by root, `data` is not in it, and the image's build-time chown is underneath the mount. `initdb` runs as `postgres` by then and cannot create the directory. `postgres-init` is a oneshot whose `up` runs as root, so it creates and chowns it first. That also fixes plain `docker run -v openbot-data:/var/lib/postgresql`, which relied on the hidden chown. Mounted on the data directory itself, the mount holds a `lost+found` and `initdb` refuses a directory with anything in it. The documented mount is now the parent, leaving `data` a subdirectory — what initdb's own hint asks for and what the Dockerfile already said. A Docker named volume at the old path arrives empty rather than with a lost+found, so a deployment already working that way keeps working and needs no change; this is why the data directory is not moved into a subdirectory of itself, which would have looked for a cluster somewhere that deployment has none and initialised a fresh one over the top. The failure also said nothing useful. `api` waits on `postgres` and `migrate`, so neither started, the container came up regardless, the platform reported the deploy a success, and the URL served a persistent 502 with the reason only in the container log. A data directory holding no cluster and not empty is now refused with a sentence naming the mount to use instead. Reported by Jerel Velarde in CopilotKit#269, with the logs for both mount paths, which is what made the two failure modes separable. Closes CopilotKit#269 Verification is honest about its limits: no Docker on this machine, so the image was not built or booted. `sh -n` is clean, and the decision logic was exercised against fixtures for all four states — no volume, volume at the parent, volume on the data directory, and an existing cluster — with the last confirming an initialised cluster is still never touched. CI's image job boots the no-volume path; neither it nor I cover a real platform volume.
Hotragn
requested review from
MikeRyanDev,
davidmckayv,
guidovizoso and
tylerslaton
as code owners
August 28, 2026 00:15
# Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Closes #269. @jerelvelarde reported it with the container logs for both mount paths, which is what made
the two failure modes separable — the diagnosis below is theirs.
EMBEDDED_POSTGRES=oncould not create its cluster on a platform whose persistent volume is an ext4mount, and it failed differently depending on where the volume was mounted. Neither of the two paths
this repo suggested worked, and the two suggestions disagreed with each other:
docs/deployment.mdsaid
/var/lib/postgresql/data, theDockerfilecomment at line 126 said/var/lib/postgresql.Mounted at the parent, the mount arrives owned by root,
datais not in it, and the image'sbuild-time
chownis underneath the mount.initdbhas already dropped topostgresby the time itruns, so it cannot create the directory.
postgres-initis a oneshot whoseupruns as root, so itnow creates and chowns it first — which is also the fix for plain
docker run -v openbot-data:/var/lib/postgresql, a case that relied entirely on a chown a mount hides.Mounted on the data directory itself, the mount holds a
lost+foundandinitdbrefuses adirectory with anything in it. The documented mount is now the parent, leaving
dataan ordinarysubdirectory — which is what
initdb's own hint asks for and what theDockerfilealready said.The failure said nothing useful, which is the half worth fixing on its own.
apiwaits onpostgresandmigrate, so neither started, the container came up regardless, the platform reportedthe deploy a success, and the public URL served a persistent 502 with the real reason only in the
container log. A data directory that holds no cluster and is not empty is now refused with a sentence
naming the mount to use instead.
Why the data directory is not moved into a subdirectory of itself
That would fix both mount paths in code with no documentation change, and it is the wrong trade. A
Docker named volume at
/var/lib/postgresql/dataarrives empty rather than with alost+found,so that configuration works today —
docker-compose.ymland the olddocker runline both use it.Looking for the cluster one level deeper would find nothing there, initialise a fresh one, and leave
the existing cluster sitting in the same volume unreferenced. That is losing somebody's audit trail to
fix a first-boot error, and the audit trail is the product.
So: existing working deployments are untouched and need no migration, and the change for new ones is a
mount path in the docs.
docker-compose.ymlis deliberately not touched. Itspostgres:service is the official PostgreSQLimage, where
/var/lib/postgresql/datais that image's own documented mount and none of this applies.Where it runs
definition —
EMBEDDED_POSTGRESexists for the deployment that runs one thing. A deployment withreplicas points
DATABASE_URLat a managed database and never runs this script.postgres-initis a oneshot thatpostgresandmigratedepend on, so itcompletes before either starts. That ordering is unchanged; what changed is that it now fails
loudly instead of leaving the two never starting for an unexplained reason.
Boundary and audit
touches nothing about who may do what.
A Bot's shell has passwordless owner access to the database in the all-in-one image #226 is about that premise and this PR deliberately does not touch it — the two would collide in
this file, and this one is a first-boot failure rather than a boundary.
Changelog
Added under
Unreleased, naming both failure modes, the mount change, and that an existing volume atthe old path keeps working.
Proof
Stated plainly: I did not build or boot the image. There is no working Docker on this machine, so
the container itself is unverified by me. What I did run:
sh -n docker/s6/scripts/postgres-init.sh— clean.The decision logic exercised against real directory fixtures, for all four states the script can meet:
datainitdblost+foundbesidedatainitdb— this is the mode 2 fixdata,lost+foundinside itPG_VERSIONpresent)The last row is the one I most wanted to see: an initialised cluster is still never re-initialised, so
nothing here can take a working deployment's data.
Also checked, because a CRLF shebang would break the image and this machine is Windows: the committed
blob has zero carriage returns, matching upstream's.
CI's
imagejob covers the no-volume path. Neither it nor I cover a real platform volume, which isthe case the issue is about — a Railway deploy at the new mount path is the check that actually
closes this, and @jerelvelarde is the one who can do it.
What is not covered
healthy deploy is a real gap, and the issue raises it as worth considering separately. This makes
the reason legible in the log; it does not make the container report itself unhealthy. That wants a
healthcheck decision rather than a line in this script.
/var/lib/postgresql/datastill has tochange its mount. Nothing is lost by doing so — the cluster never initialised, so the volume holds
no data — and the new message says which path to use.