Skip to content

Initialise embedded PostgreSQL where a platform volume lets it - #281

Merged
davidmckayv merged 2 commits into
CopilotKit:mainfrom
Hotragn:initialise-postgres-where-a-volume-lets-it
Aug 28, 2026
Merged

Initialise embedded PostgreSQL where a platform volume lets it#281
davidmckayv merged 2 commits into
CopilotKit:mainfrom
Hotragn:initialise-postgres-where-a-volume-lets-it

Conversation

@Hotragn

@Hotragn Hotragn commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

What this changes

Closes #269. @jerelvelarde reported it with the container logs for both mount paths, which is what made
the two failure modes separable — the diagnosis below is theirs.

EMBEDDED_POSTGRES=on could not create its cluster on a platform whose persistent volume is an ext4
mount, and it failed differently depending on where the volume was mounted. Neither of the two paths
this repo suggested worked, and the two suggestions disagreed with each other: docs/deployment.md
said /var/lib/postgresql/data, the Dockerfile comment at line 126 said /var/lib/postgresql.

Mounted at the parent, the mount arrives owned by root, data is not in it, and the image's
build-time chown is underneath the mount. initdb has already dropped to postgres by the time it
runs, so it cannot create the directory. postgres-init is a oneshot whose up runs as root, so it
now creates and chowns it first — which is also the fix for plain
docker run -v openbot-data:/var/lib/postgresql, a case that relied entirely on a chown a mount hides.

Mounted on the data directory itself, the mount holds a lost+found and initdb refuses a
directory with anything in it. The documented mount is now the parent, leaving data an ordinary
subdirectory — which is what initdb's own hint asks for and what the Dockerfile already said.

The failure said nothing useful, which is the half worth fixing on its own. api waits on
postgres and migrate, so neither started, the container came up regardless, the platform reported
the deploy a success, and the public URL served a persistent 502 with the real reason only in the
container log. A data directory that holds no cluster and is not empty is now refused with a sentence
naming the mount to use instead.

Why the data directory is not moved into a subdirectory of itself

That would fix both mount paths in code with no documentation change, and it is the wrong trade. A
Docker named volume at /var/lib/postgresql/data arrives empty rather than with a lost+found,
so that configuration works today — docker-compose.yml and the old docker run line both use it.
Looking for the cluster one level deeper would find nothing there, initialise a fresh one, and leave
the existing cluster sitting in the same volume unreferenced. That is losing somebody's audit trail to
fix a first-boot error, and the audit trail is the product.

So: existing working deployments are untouched and need no migration, and the change for new ones is a
mount path in the docs.

docker-compose.yml is deliberately not touched. Its postgres: service is the official PostgreSQL
image, where /var/lib/postgresql/data is that image's own documented mount and none of this applies.

Where it runs

  • New state that outlives a request? None. This is container first-boot, before anything serves.
  • What happens on the second replica? Unchanged, and this is the single-container shape by
    definition — EMBEDDED_POSTGRES exists for the deployment that runs one thing. A deployment with
    replicas points DATABASE_URL at a managed database and never runs this script.
  • Anything serialised? postgres-init is a oneshot that postgres and migrate depend on, so it
    completes before either starts. That ordering is unchanged; what changed is that it now fails
    loudly instead of leaving the two never starting for an unexplained reason.
  • Anything fanned out to a browser? No.
  • New listener, port, or schedule? No.

Boundary and audit

  • No boundary change. This runs before the API exists, creates the cluster the API then migrates, and
    touches nothing about who may do what.
  • Trust-auth and loopback-only are unchanged, including the comment explaining them. Worth noting that
    A Bot's shell has passwordless owner access to the database in the all-in-one image #226 is about that premise and this PR deliberately does not touch it — the two would collide in
    this file, and this one is a first-boot failure rather than a boundary.
  • No audit rows: there is no database to write one to at this point in the boot.

Changelog

Added under Unreleased, naming both failure modes, the mount change, and that an existing volume at
the old path keeps working.

Proof

Stated plainly: I did not build or boot the image. There is no working Docker on this machine, so
the container itself is unverified by me. What I did run:

sh -n docker/s6/scripts/postgres-init.sh — clean.

The decision logic exercised against real directory fixtures, for all four states the script can meet:

Fixture Result
no volume — image-built empty data proceeds to initdb
platform volume at the parent, lost+found beside data proceeds to initdb — this is the mode 2 fix
platform volume on data, lost+found inside it refused, with the message
existing cluster (PG_VERSION present) skipped entirely

The last row is the one I most wanted to see: an initialised cluster is still never re-initialised, so
nothing here can take a working deployment's data.

Also checked, because a CRLF shebang would break the image and this machine is Windows: the committed
blob has zero carriage returns, matching upstream's.

CI's image job covers the no-volume path. Neither it nor I cover a real platform volume, which is
the case the issue is about — a Railway deploy at the new mount path is the check that actually
closes this, and @jerelvelarde is the one who can do it.

What is not covered

  • The 502-on-success problem itself. A first-start failure that leaves the platform reporting a
    healthy deploy is a real gap, and the issue raises it as worth considering separately. This makes
    the reason legible in the log; it does not make the container report itself unhealthy. That wants a
    healthcheck decision rather than a line in this script.
  • An existing broken deployment with a platform volume on /var/lib/postgresql/data still has to
    change its mount. Nothing is lost by doing so — the cluster never initialised, so the volume holds
    no data — and the new message says which path to use.

`EMBEDDED_POSTGRES=on` could not create its cluster on a platform whose volume
is an ext4 mount, which is most of them, and it failed differently depending on
which of two paths the volume was mounted at. Neither worked, and the two
suggestions in this repo disagreed: docs/deployment.md said
/var/lib/postgresql/data, the Dockerfile comment said /var/lib/postgresql.

Mounted at the parent, the mount arrives owned by root, `data` is not in it, and
the image's build-time chown is underneath the mount. `initdb` runs as `postgres`
by then and cannot create the directory. `postgres-init` is a oneshot whose `up`
runs as root, so it creates and chowns it first. That also fixes plain
`docker run -v openbot-data:/var/lib/postgresql`, which relied on the hidden
chown.

Mounted on the data directory itself, the mount holds a `lost+found` and `initdb`
refuses a directory with anything in it. The documented mount is now the parent,
leaving `data` a subdirectory — what initdb's own hint asks for and what the
Dockerfile already said. A Docker named volume at the old path arrives empty
rather than with a lost+found, so a deployment already working that way keeps
working and needs no change; this is why the data directory is not moved into a
subdirectory of itself, which would have looked for a cluster somewhere that
deployment has none and initialised a fresh one over the top.

The failure also said nothing useful. `api` waits on `postgres` and `migrate`, so
neither started, the container came up regardless, the platform reported the
deploy a success, and the URL served a persistent 502 with the reason only in the
container log. A data directory holding no cluster and not empty is now refused
with a sentence naming the mount to use instead.

Reported by Jerel Velarde in CopilotKit#269, with the logs for both mount paths, which is
what made the two failure modes separable.

Closes CopilotKit#269

Verification is honest about its limits: no Docker on this machine, so the image
was not built or booted. `sh -n` is clean, and the decision logic was exercised
against fixtures for all four states — no volume, volume at the parent, volume on
the data directory, and an existing cluster — with the last confirming an
initialised cluster is still never touched. CI's image job boots the no-volume
path; neither it nor I cover a real platform volume.
@davidmckayv
davidmckayv merged commit cc06097 into CopilotKit:main Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Embedded Postgres cannot initialise on Railway: initdb fails at both documented volume paths, and the deploy still reports success

2 participants