Skip to content

Make the epoch repair scripts behave on 13.7.2.1+ - #2184

Draft
ArturWieczorek wants to merge 9 commits into
masterfrom
fix-epoch-repair-scripts
Draft

ArturWieczorek wants to merge 9 commits into
masterfrom
fix-epoch-repair-scripts

Conversation

@ArturWieczorek

@ArturWieczorek ArturWieczorek commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Since 13.7.2.1 (migration 2-0049) epoch is a view: db-sync stores finished epochs in epoch_finalized and computes the current one from block and tx. The three epoch scripts in scripts/ still assumed a table. Found while testing 13.8.0.0.

What went wrong

  • fix-epoch-table.sql tries to INSERT ... ON CONFLICT into the view, fails with cannot insert into view "epoch", and still exits 0: the script didn't stop on errors, so the COMMIT just rolled the failed transaction back.
  • check-and-fix-epoch-table.sh asks "Apply fix?" and then runs that fix.
  • With epoch sync not enabled (disable_epoch, or db-sync not started since the upgrade) the view is empty, so validate-epoch-table.sql lists every epoch as MISSING.

Changes, one per commit:

  1. fix-epoch-table.sql stops at the first error, and refuses up front on a view, before BEGIN:
    ERROR: epoch is a view since db-sync 13.7.2.1, so this script cannot repair it
    HINT: Finished epochs are stored in epoch_finalized, which db-sync maintains; a mismatch reported by validate-epoch-table.sql points to a db-sync bug.
    On the old table it works as before, and a failed run now exits non-zero.
  2. validate-epoch-table.sql: on a view with epoch sync not enabled, it says so and exits 0 instead of listing every epoch as MISSING.
  3. check-and-fix-epoch-table.sh: on a view it doesn't offer the fix. If epoch sync is not enabled it says so and exits 0 without validating. If there are mismatches it prints them, says the script can't repair them, and exits 1.
  4. check-and-fix-epoch-table.sh: with no terminal (closed stdin) the prompt hit end of input, and set -e ended the script with exit 1 and no message. No input now counts as "no", and an answer without a final newline is still taken.
  5. check-and-fix-epoch-table.sh: the header claimed it requires PGPASSFILE, could take the database name "via psql_args", and read "field 5" of the pgpass line. None of that matched the code.
  6. scripts/README.md: the epoch section says what applies since 13.7.2.1.
  7. check-and-fix-epoch-table.sh took only the database name from the pgpass file, so with a pgpass for another host, port or user it checked (and after "y" fixed) a same-named database on the default server. It now takes host, port, database and user from the first pgpass line, as db-sync does; PGHOST/PGPORT/PGDATABASE/PGUSER still win, and a * field is left to psql.
  8. validate-epoch-table.sql now also reports start_time / end_time mismatches. The fix rewrites both, but validate never flagged them, so the wrapper said "No mismatches" for an epoch it would have changed.
  9. Run without -v epoch_no, fix-epoch-table.sql and validate-epoch-table.sql now stop with missing -v epoch_no: run psql -v epoch_no=<epoch, or -1 for all epochs> ... and exit 3, instead of a raw syntax error (validate even exited 0).

Why not repair epoch_finalized instead: db-sync already maintains it with the same recompute from block and tx. It upserts at each epoch boundary, deletes from the rollback epoch on, and backfills at startup. The old fix also writes the current epoch, which would hide it from epoch_current until the next boundary.

Testing

  • Throwaway databases built from the real 13.7.1.0 (table) and 13.8.0.0 (view) schemas: 16 scenarios covering both forms, epoch sync on, off or not set up, mismatches, the answers "y" and "n", and no input. The new scenarios fail on master and pass with these commits, and removing any of the changes makes its scenario fail again.
  • Commits 7-9: 5 more scenarios (21 in total), red before each commit and green after, plus a pgpass test with a second PostgreSQL server holding a same-named database (before: the wrapper fixed the wrong one; now the one the pgpass names). validate on real preview / preprod (13.7.1.0 table, 13.8.0.0 view) still reports 0 rows with the time columns included.
  • shellcheck is clean.
  • Real databases, read-only:
┌─────────────────────────────────────────────────────┬──────────────────────────────────┬───────────────────────┬───────────────────────┐
│                      Database                       │               fix                │       validate        │        wrapper        │
├─────────────────────────────────────────────────────┼──────────────────────────────────┼───────────────────────┼───────────────────────┤
│ preprod / preview 13.8.0.0, preprod 13.7.2.1 (view) │ refuses, exit 3, nothing written │ exit 0, no mismatches │ exit 0, no mismatches │
├─────────────────────────────────────────────────────┼──────────────────────────────────┼───────────────────────┼───────────────────────┤
│ preprod / preview 13.7.1.0 (table)                  │ not run (writes)                 │ exit 0, no mismatches │ exit 0, no mismatches │
└─────────────────────────────────────────────────────┴──────────────────────────────────┴───────────────────────┴───────────────────────┘

scripts/README.md is also changed by #2183, in a different section, and the two merge cleanly.

Since 13.7.2.1 (migration 2-0049) epoch is a view: db-sync stores finished
epochs in epoch_finalized and computes the current one from block and tx.
The script still tried to INSERT ... ON CONFLICT into the view, failed with
"cannot insert into view", and psql exited 0 because the script did not
stop on errors: the COMMIT just rolled the failed transaction back.

Stop at the first error, and refuse up front, before BEGIN, when epoch is a
view:

  ERROR:  epoch is a view since db-sync 13.7.2.1, so this script cannot
  repair it
  HINT:  Finished epochs are stored in epoch_finalized, which db-sync
  maintains; a mismatch reported by validate-epoch-table.sql points to a
  db-sync bug.

On a table (before 13.7.2.1) the fix works as before, and a failed run now
exits non-zero.
…sing

Since 13.7.2.1 epoch is a view that returns no rows while epoch sync is not
enabled: when disable_epoch is set, and on a migrated database before
db-sync has started on it (migration 2-0049 creates the flag as false).
The validation then printed every epoch as MISSING, which reads like a
broken epoch table.

On the view form, check the flag first and, when it is off, say so and stop
with exit 0, as there is nothing to validate:

  Epoch sync is not enabled (disable_epoch is set, or db-sync has not
  started since the upgrade), so the epoch view is empty: nothing to
  validate.

The table form and an enabled view are validated as before.
…apply

When validation finds mismatches the wrapper asks "Apply fix?" and runs
fix-epoch-table.sql. Since 13.7.2.1 epoch is a view, so on a newer
database the fix can only refuse, after the operator was told it would run.

Check the epoch relation first. On a view:

- if epoch sync is not enabled (disable_epoch is set, or db-sync has not
  started since the upgrade), the view is empty by design: say so and exit
  0 without validating, instead of reporting every epoch as a mismatch
- if validation finds mismatches, print them, say the script cannot repair
  them and that they point to a db-sync bug, and exit 1 without asking

The messages now say "epoch data" rather than "epoch table", and the
header lists the exit codes. The table form behaves as before.
Run without a terminal, e.g. from a job or with stdin closed, read hits end
of input and fails, so set -e ended the script with exit 1 and no message
after it had listed the mismatches. Let read fail without ending the script:
no input at all now prints "No changes applied." and exits 0, as for "n",
and an answer without a final newline is still taken.
The header said the script requires PGPASSFILE and can take the database
name "via psql_args", and a comment said the name is field 5 of the pgpass
line. The script needs only PGDATABASE (or the pgpass file as a fallback),
has no psql_args, and reads field 3, which is where pgpass keeps the
database name.
…-sync uses

The wrapper took only the database name from the pgpass file and left host, port
and user to psql's defaults. db-sync takes all four from the first pgpass line
(cardano-db PGConfig), so with a pgpass for a non-default server, port or user
the wrapper validated - and after "Apply fix?" rewrote - a same-named database on
the default server instead of db-sync's.

Take host, port, database and user from the first pgpass line, as db-sync does.
PGHOST, PGPORT, PGDATABASE and PGUSER still take precedence, and a "*" field is
left to psql's default. The "Validating" line now names the host and port.

Tested with a second PostgreSQL server holding a same-named database: before,
a pgpass pointing at it made the wrapper fix the database on the default server;
now it fixes the one the pgpass names. Explicit PGHOST/PGPORT and wildcard
fields keep connecting to the default server.
…_time mismatches

fix-epoch-table.sql recomputes and rewrites start_time and end_time, but
validate-epoch-table.sql only reported out_sum, fees, tx_count and blk_count
differences (it computed the times and showed them, but never filtered on them).
An epoch whose only error was its start or end time passed validation, and
check-and-fix-epoch-table.sh then said "No mismatches" without offering the fix
that would have changed it.

Report a row when either time differs too, so validate checks every column the
fix rewrites. On preview and preprod (13.7.1.0 table, 13.8.0.0 view) there are
no such epochs today: still 0 rows.
Run directly without `-v epoch_no`, fix-epoch-table.sql and
validate-epoch-table.sql stopped on a raw "syntax error at or near ':'" at
`CAST(:epoch_no ...)`, and validate even exited 0 because it does not set
ON_ERROR_STOP. Both now check the variable first and stop with
"missing -v epoch_no: run psql -v epoch_no=<epoch, or -1 for all epochs> -f ..."
and exit code 3, before any statement that reads or writes the epoch data.
check-and-fix-epoch-table.sh always passes the variable and is unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant