✨ Moved installations of the released main layout onto this one (S6b)

.env.example

@@ -129,5 +129,8 @@ LOG_MAX_FILE="3"
 # TINYBIRD_TRACKER_TOKEN="p.eyJxxxxx"
 # TINYBIRD_ADMIN_TOKEN="p.eyJxxxxx"
 # TINYBIRD_WORKSPACE_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
+# Shared secret Ghost uses to authenticate Tinybird sync requests to Traffic
+# Analytics, for automation analytics. Generate one with `openssl rand -hex 32`.
+# TINYBIRD_SYNC_AUTH="xxxxxxxx"
 # SALT_STORE_TYPE="file"
 # TRAFFIC_ANALYTICS_LOG_LEVEL="info"

.github/workflows/test.yml

@@ -23,6 +23,9 @@ jobs:
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
 
+      # The released main layout the migration tests read (GD_RELEASED_MAIN).
+      - run: git fetch --no-tags --depth=1 origin main:refs/remotes/origin/main
+
       - uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
         with:
           node-version: "26"
@@ -118,6 +121,19 @@ jobs:
       - name: Self-update end to end
         run: tests/e2e/self-update.sh
 
+  # An installation of the released main layout, migrated by the served
+  # launcher: refusals before any change, recovery after a failed start, and
+  # the migration keeping its project, data, routes and overrides.
+  migrate-main-linux:
+    name: Migration from the released main layout on Linux
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Migration end to end
+        run: tests/e2e/migrate-main.sh
+
   # Real backups and restores on Docker Engine: a local site with ActivityPub
   # backed up, restored over itself and into a new directory, the lock
   # refusing a second operation, and a dump that fails.
@@ -209,6 +225,7 @@ jobs:
       - launcher-linux
       - install-linux
       - self-update-linux
+      - migrate-main-linux
       - backup-linux
       - import-linux
       - launcher-macos

.gitignore

@@ -98,7 +98,7 @@ caddy/custom/*
 caddy/global/*
 !caddy/global/.gitignore
 !caddy/global/README.md
-# Operator's own Caddyfile from pre-1.0 installations (see the S6 migration)
+# The released main layout's own Caddyfile, kept by its migration (docs/install.md)
 caddy/Caddyfile.local
 # The operator's own Compose overrides (docs/configuration.md)
 compose.override.yml

README.md

@@ -5,9 +5,9 @@ Configuration to run Ghost and its services with Docker Compose.
 > **This is the `next-docker` development branch.** It is being rebuilt around
 > a manager image: a small CLI in a container, started by a bash launcher that
 > uses Docker. It installs local and production sites, imports local and production
-> Ghost-CLI sites, backs up and restores them, and updates image installations
-> between beta releases. Migration from the released `main` layout remains on
-> the [roadmap](docs/ghost-cli-replacement.md). For a supported setup
+> Ghost-CLI sites, backs up and restores them, updates image installations
+> between beta releases, and moves installations of the released `main` layout
+> onto it. Remaining work is on the [roadmap](docs/ghost-cli-replacement.md). For a supported setup
 > today, use the `main` branch.
 
 ## The launcher
@@ -91,6 +91,7 @@ From `scripts/`, run `pnpm install --frozen-lockfile`, `pnpm run typecheck` and
 tests/e2e/launcher.sh        # stand-in Docker, then the real image
 tests/e2e/install.sh         # real installations: pulls images, binds 80 and 443
 tests/e2e/self-update.sh     # release updates, failed-update write retention and recovery
+tests/e2e/migrate-main.sh    # an installation of the released main layout, migrated
 tests/e2e/backup.sh          # real backups and restores, including ActivityPub
 tests/e2e/import.sh          # real Ghost-CLI sites exported and imported
 GD_TEST_HOST_CHANGES=1 tests/e2e/production-import.sh  # a Ghost-CLI production site moved; changes the host

TINYBIRD.md

@@ -8,6 +8,7 @@ Note: Currently Traffic Analytics features are behind a feature flag. For now, y
 1. Run `docker compose run --rm tinybird-deploy` and wait for the service to exit successfully. This will create your Tinybird datasources, pipes and API endpoints. It may take a minute or two to complete the first time. You should see "Deployment #1 is live!" in your terminal before the service exits.
 1. Run `docker compose run --rm tinybird-login get-tokens`
 1. Copy and paste the values from the previous step into your `.env` file (Tinybird credentials are operator settings, not Ghost application settings)
+1. If using automations analytics, generate a shared sync secret with `openssl rand -hex 32` and add it to `.env`: `./ghost-docker config set .env TINYBIRD_SYNC_AUTH <generated value>`. Ghost and Traffic Analytics share it; after adding or changing it, run `docker compose up -d` to recreate their containers (a restart alone does not apply environment changes).
 1. Run `docker compose --profile=analytics up -d` to start all services in the background
 1. Add `analytics` to `COMPOSE_PROFILES` in your `.env` file, alongside the site mode, to include the `analytics` profile automatically when running `docker compose` commands. Profiles are additive: adding `analytics` does not change the site mode.
 1. In production, add the analytics route to `caddy/sites/site.caddy`, inside the site's block: `import /etc/caddy/snippets/TrafficAnalytics traffic-analytics-<COMPOSE_PROJECT_NAME>:3000`, then reload Caddy with `docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile`

compose.yml

@@ -65,6 +65,7 @@ services:
       tinybird__tracker__datasource: analytics_events
       tinybird__adminToken: ${TINYBIRD_ADMIN_TOKEN:-}
       tinybird__workspaceId: ${TINYBIRD_WORKSPACE_ID:-}
+      tinybird__sync_auth_key: ${TINYBIRD_SYNC_AUTH:-}
       tinybird__stats__endpoint: ${TINYBIRD_API_URL:-https://api.tinybird.co}
     volumes:
       - ${UPLOAD_LOCATION:-./data/ghost}:${GHOST_CONTENT_PATH:-/home/ghost/content}
@@ -194,6 +195,7 @@ services:
       SALT_STORE_TYPE: ${SALT_STORE_TYPE:-file}
       SALT_STORE_FILE_PATH: /data/salts.json
       TINYBIRD_TRACKER_TOKEN: ${TINYBIRD_TRACKER_TOKEN:-}
+      TINYBIRD_SYNC_AUTH: ${TINYBIRD_SYNC_AUTH:-}
       LOG_LEVEL: ${TRAFFIC_ANALYTICS_LOG_LEVEL:-info}
     networks:
       ghost_network:

docs/architecture.md

@@ -257,4 +257,22 @@ useful fresh/damaged-install diagnostics. Bundle v1 has its own
 S12 must record the exact schema/format/launcher versions supported by the stable
 release here. Subsequent changes must be compatible or explicitly versioned with
 a migration or old-format reader; pinned launchers must still start newer update
-managers. Migration from the released `main` layout remains required in S6b.
+managers.
+
+### Migration from the released main layout
+
+`src/legacy.ts` is migration `0001-compose-profiles`: the one way an installation
+of the released `main` layout (a clone with no metadata) reaches this layout,
+entered through the served launcher's `self-update`. `src/legacy/caddy.ts`
+carries the operator's Caddyfile as written, with main's environment variables
+filled in and main's snippets kept beside it rather than translated into this
+layout's shape; Caddy decides whether it loads. `src/legacy/config.ts` splits
+its `.env`.
+It decides everything before changing anything: Compose resolves the staged
+configuration with the operator's overrides and Caddy loads the staged routes.
+It then follows self-update's recovery boundary, reusing its snapshot, writer
+pause and backup; the backup skips drift refusal because the stopped containers
+deliberately ran main's images. The metadata is written last and is the only
+record of completion: a site with it is never migrated again. There is no
+general migration framework; a later migration from a released layout adds
+what it needs.

docs/caddy.md

@@ -25,6 +25,16 @@ import /etc/caddy/sites/*.caddy
 import /etc/caddy/custom/*.caddy
 ```
 
+## Routes moved from the released main layout
+
+A site moved from `main` keeps its own Caddyfile as `caddy/sites/site.caddy`
+([install.md](install.md#moving-from-the-released-main-layout)). It imports
+main's snippets, which take no arguments, from `caddy/sites/legacy-snippets/`,
+and proxies to bare service names (`ghost:2368`), which resolve on the site's
+own network. Both keep working. Moving to this layout's snippets and aliases,
+as `install` writes them, is optional, and is needed before the site joins a
+shared Caddy.
+
 ## Changing routes
 
 Edit the file, then reload Caddy explicitly:

docs/configuration.md

@@ -295,8 +295,9 @@ minimum and exact source-version requirement.
 ## Existing installations
 
 Do not use `git pull` to move a released `main` installation to this layout.
-That migration is still [roadmap work](ghost-cli-replacement.md#s6b--migration-from-the-released-main-layout).
-Moving a Ghost-CLI site is a separate, supported import described in
+The served launcher's `self-update` migrates it, splitting its `.env` into
+`.env` and `ghost.env` as this page describes; [install.md](install.md#moving-from-the-released-main-layout)
+lists what each setting becomes. Moving a Ghost-CLI site is a separate, supported import described in
 [install.md](install.md#importing-a-ghost-cli-site).
 
 ## Installed image pins

docs/ghost-cli-replacement.md

@@ -11,8 +11,8 @@ Linear](https://linear.app/ghost/issue/PLA-412).
 
 | Outcome | Remaining work | Dependencies |
 | --- | --- | --- |
-| Tagged single-site production | S6b legacy-layout migration, S5e production import, S12 qualification | Existing image self-update, local import, backup and restore |
-| Several sites behind shared Caddy | S13 | Existing backup/restore; S6b for converting existing sites |
+| Tagged single-site production | S5e production import, S12 qualification | Existing image self-update, local import, backup and restore |
+| Several sites behind shared Caddy | S13 | Existing backup/restore and released-main migration |
 | Admin-driven Ghost upgrades | S7 host upgrade, S8 supervisor, S9 core adapter/API, S10 Admin | S7 before supervisor execution; S8 protocol before S9; S9 before S10; S13 integration if shipped |
 | Later extensions | S11 file secrets, S14 service references, S15 nightlies, S16 Redis | See each step |
 
@@ -43,37 +43,6 @@ its recovery, and the admin domain in CI; the cross-host move is run by hand. Th
 `scripts/migrate.sh`; S12 must not merge `next-docker` into `main` before these
 scenarios pass.
 
-### S6b — Migration from the released main layout
-
-Repo: ghost-docker. Depends on image self-update and backup/restore. This migration
-remains required regardless of the development-format [compatibility
-policy](architecture.md#compatibility), and gates merging into `main`.
-
-Implement ordered release migration scripts, recording completion in metadata when the
-first migration needs that field. `0001-compose-profiles` must:
-
-- Handle both absent profiles and existing `analytics,activitypub`, adding
-  `production`; preserve credentials and project identity.
-- Split Ghost application configuration into `ghost.env`, keeping operator
-  settings in `.env`; add `SITE_MODE`, `URL`, `PROJECT_DIR` and an exact pin for
-  the currently running Ghost version.
-- Preserve the existing untracked Caddyfile before managed files are written.
-  Custom routes must still work with snippets that take explicit upstream and
-  domain arguments. Preserve supported customizations automatically, or stop
-  before changing the live site and explain what must be resolved.
-
-Existing installations have no launcher or metadata. The entry point is the served
-launcher from inside the installation (`curl -fsSL https://docker.ghost.org/install.sh |
-bash -s -- self-update`), not a raw `git pull` across the breaking change. This is a
-migration of the released layout, distinct from self-update of a current `source:
-checkout` site.
-
-Acceptance: absent metadata, an untagged starting commit, existing optional profiles,
-custom Caddy routes and Compose overrides, skipped releases, repeated invocation and
-failed hooks. Failures during migration, pulling or startup must follow the [recovery
-boundary](architecture.md#recovery) and report the observed state accurately, retaining
-any writes after startup.
-
 ### S7 — Host-driven Ghost upgrades
 
 Repo: ghost-docker. Depends on backup/restore and release compatibility. Implement
@@ -244,13 +213,15 @@ environment-based installs, older imports, restart, and restore still work.
 
 ### S12 — Single-site release qualification and documentation
 
-Repo: ghost-docker, with cross-repo fixtures. Deps: S5e and S6b, plus existing install,
-backup/restore and self-update; qualify S13, S7-S10 and S11 when they have shipped.
+Repo: ghost-docker, with cross-repo fixtures. Deps: S5e, plus existing install,
+backup/restore, self-update and released-main migration; qualify S13, S7-S10 and S11 when they have shipped.
 Gates the first stable tag and merging `next-docker` into `main`. Include the launcher
 on Linux, macOS and WSL2. Consolidate CI and qualify the actual minimum supported tools
 and image versions. Run fresh local/production install, optional-service variants, CLI
 migration, legacy stack update, Ghost upgrade/recovery, supervisor/Admin, and restore
-scenarios. Include Linux runtime tests and macOS-compatible shell/configuration checks.
+scenarios. Before merging, pin the migration tests' `GD_RELEASED_MAIN`
+(default `origin/main`, in `manager/test/site.ts`, `tests/e2e/migrate-main.sh` and
+the manager CI job) to `main`'s last commit, and rerun them. Include Linux runtime tests and macOS-compatible shell/configuration checks.
 Record the [compatibility baseline](architecture.md#compatibility): the exact metadata
 schema, backup format, launcher contract and bundle versions the stable release
 supports, and the obligations that hold for them from then on.
@@ -265,7 +236,8 @@ shipped.
 ### S13 — Optional shared Caddy
 
 Repo: ghost-docker. Depends on backup/restore for new member sites; converting an
-existing site into a member (PLA-504) also needs S6b. Several sites on one server, as
+existing site into a member (PLA-504) starts from a site on this layout, which the
+released-main migration provides. Several sites on one server, as
 Ghost-CLI ran several sites behind one nginx. 80 and 443 can belong to one Caddy only,
 so that Caddy is shared; everything else stays per site, MySQL included, so backup,
 restore, upgrade and import are unchanged.

docs/install.md

@@ -590,7 +590,8 @@ site's launcher starts that image, not the one it is pinned to.
 | `--channel CHANNEL` | The newest release on `stable` or `beta`, which the site then follows. |
 | `--to vX.Y.Z` | Exactly that release. |
 
-The update refuses downgrades, incompatible Ghost versions, absent metadata,
+The update refuses downgrades, incompatible Ghost versions, absent metadata
+(except for [the released main layout](#moving-from-the-released-main-layout)),
 checkouts, another operation's lock and an unfinished update snapshot. It pulls
 images before the outage where possible, keeps a file snapshot in
 `.ghost-docker-update/`, then pauses Ghost and ActivityPub and takes a consistent
@@ -655,6 +656,63 @@ before, and restore the backup (`./ghost-docker restore --yes backups/<backup>`)
 the backup records the commit checked out when it was taken, and restore
 refuses a clone at any other.
 
+### Moving from the released main layout
+
+An installation made from `main` before this layout (a git clone with one
+`.env`, a hand-edited `caddy/Caddyfile`, and no `ghost-docker` launcher) moves
+onto it once, with the launcher as docker.ghost.org serves it, run in the
+site directory:
+
+```bash
+cd /path/to/your/ghost-docker
+curl -fsSL https://docker.ghost.org/install.sh | bash -s -- self-update --check
+curl -fsSL https://docker.ghost.org/install.sh | bash -s -- self-update
+```
+
+**Do not `git pull` across this change.** The new layout's Compose file selects
+no services without the new settings, and a pull cannot write them.
+
+`self-update` finds no `.ghost-docker.json` and main's `.env`, and runs migration
+`0001-compose-profiles`. Ghost and its database must be running, so it can
+read the Ghost version the site runs and back the site up. Before changing
+anything it works out and checks all of this, and stops, naming what to
+resolve, if any of it cannot be carried over:
+
+| What | Becomes |
+| --- | --- |
+| `COMPOSE_PROFILES` | `production`, plus `analytics` and `activitypub` if they were on |
+| Compose project | The same name, set as `COMPOSE_PROJECT_NAME`, so Caddy's certificates and every volume stay the site's |
+| `DOMAIN`, `ADMIN_DOMAIN` | `URL`, `ADMIN_URL` |
+| Credentials, ports, data locations, Tinybird settings | Kept in `.env`, with `SITE_MODE`, `PROJECT_DIR` and the rest of this layout's settings added |
+| Ghost's configuration in `.env` (`mail__*`, `labs__*`, anything Ghost read) | Moved to `ghost.env`, with the values Ghost actually received. Keys the container sets itself (`url`, `database__*`) are listed and left out, as they were overridden on main too |
+| `ghost:6-alpine` | The `next` image of **exactly the running version**, pinned by digest (`GHOST_IMAGE_REF`). Ghost 6.61.0 is the first with one: upgrade an older Ghost on main first (`docker compose pull ghost && docker compose up -d`) |
+| `caddy/Caddyfile` | Kept as `caddy/Caddyfile.local`, and carried into `caddy/sites/site.caddy` as written: `{$DOMAIN}`, `{$ADMIN_DOMAIN}` and `{$ACTIVITYPUB_TARGET}` filled in, and `import snippets/...` pointed at main's snippets, kept in `caddy/sites/legacy-snippets/`. A leading global options block moves to `caddy/global/legacy.caddy`. Caddy loads the result before anything changes; if it does not load, nothing is changed |
+| `compose.override.yml`, `GD_COMPOSE_OVERRIDES` | Kept, and resolved with the new layout and every `.env` setting before anything changes. A mount into `/var/lib/ghost`, where main's Ghost image kept its files, is refused: this layout's image keeps them in `/home/ghost/content`, and a backup holds content only in `data/ghost` |
+| Stack files (`compose.yml`, snippets, `mysql-init/`) | The release's. Git must show them unedited: a change in the work tree or in a commit no remote has stops the migration. Move such changes into `compose.override.yml` or a `.caddy` file of your own afterwards |
+
+Then it keeps a copy of every file it writes in `.ghost-docker-update/`, stops
+Ghost and ActivityPub, writes the new layout and `.ghost-docker.json`, takes a
+checked backup into `backups/`, starts the site on the new layout and verifies
+it through Caddy. Ghost is unavailable from the backup until the new layout
+starts.
+
+Recovery follows [self-update](#self-update)'s: a failure before startup puts
+main's files back and starts Ghost again in the same containers. A failure
+after startup stops the services, puts main's files back, leaves the data as
+it is, and says **the site needs you**. Ghost's version never changes, so
+`docker compose up -d` starts main's layout on that data. The backup holds the
+databases and content as they were before the migration started anything;
+once the migration has run, `./ghost-docker restore --yes backups/<backup>`
+puts them back.
+
+Afterwards the directory is a site installed from the image: update it with
+`./ghost-docker self-update`, not git (its `.git` is left in place, and shows
+the stack files as changed). Running the served command again is an ordinary
+self-update. Compose files you used with `-f` by hand, such as
+`compose.ipv6.yml`, are not seen by the manager: name them with
+`GD_COMPOSE_OVERRIDES` when you migrate, as for every manager command
+([configuration](configuration.md#the-compose-invocation-contract)).
+
 ## backup and restore
 
 ```text