Repository navigation
Make the epoch repair scripts behave on 13.7.2.1+ - #2184
Draft
ArturWieczorek wants to merge 9 commits into
Draft
ArturWieczorek wants to merge 9 commits into
ArturWieczorek wants to merge 9 commits into
Conversation
Since 13.7.2.1 (migration 2-0049) epoch is a view: db-sync stores finished epochs in epoch_finalized and computes the current one from block and tx. The script still tried to INSERT ... ON CONFLICT into the view, failed with "cannot insert into view", and psql exited 0 because the script did not stop on errors: the COMMIT just rolled the failed transaction back. Stop at the first error, and refuse up front, before BEGIN, when epoch is a view: ERROR: epoch is a view since db-sync 13.7.2.1, so this script cannot repair it HINT: Finished epochs are stored in epoch_finalized, which db-sync maintains; a mismatch reported by validate-epoch-table.sql points to a db-sync bug. On a table (before 13.7.2.1) the fix works as before, and a failed run now exits non-zero.
…sing Since 13.7.2.1 epoch is a view that returns no rows while epoch sync is not enabled: when disable_epoch is set, and on a migrated database before db-sync has started on it (migration 2-0049 creates the flag as false). The validation then printed every epoch as MISSING, which reads like a broken epoch table. On the view form, check the flag first and, when it is off, say so and stop with exit 0, as there is nothing to validate: Epoch sync is not enabled (disable_epoch is set, or db-sync has not started since the upgrade), so the epoch view is empty: nothing to validate. The table form and an enabled view are validated as before.
…apply When validation finds mismatches the wrapper asks "Apply fix?" and runs fix-epoch-table.sql. Since 13.7.2.1 epoch is a view, so on a newer database the fix can only refuse, after the operator was told it would run. Check the epoch relation first. On a view: - if epoch sync is not enabled (disable_epoch is set, or db-sync has not started since the upgrade), the view is empty by design: say so and exit 0 without validating, instead of reporting every epoch as a mismatch - if validation finds mismatches, print them, say the script cannot repair them and that they point to a db-sync bug, and exit 1 without asking The messages now say "epoch data" rather than "epoch table", and the header lists the exit codes. The table form behaves as before.
Run without a terminal, e.g. from a job or with stdin closed, read hits end of input and fails, so set -e ended the script with exit 1 and no message after it had listed the mismatches. Let read fail without ending the script: no input at all now prints "No changes applied." and exits 0, as for "n", and an answer without a final newline is still taken.
The header said the script requires PGPASSFILE and can take the database name "via psql_args", and a comment said the name is field 5 of the pgpass line. The script needs only PGDATABASE (or the pgpass file as a fallback), has no psql_args, and reads field 3, which is where pgpass keeps the database name.
…-sync uses The wrapper took only the database name from the pgpass file and left host, port and user to psql's defaults. db-sync takes all four from the first pgpass line (cardano-db PGConfig), so with a pgpass for a non-default server, port or user the wrapper validated - and after "Apply fix?" rewrote - a same-named database on the default server instead of db-sync's. Take host, port, database and user from the first pgpass line, as db-sync does. PGHOST, PGPORT, PGDATABASE and PGUSER still take precedence, and a "*" field is left to psql's default. The "Validating" line now names the host and port. Tested with a second PostgreSQL server holding a same-named database: before, a pgpass pointing at it made the wrapper fix the database on the default server; now it fixes the one the pgpass names. Explicit PGHOST/PGPORT and wildcard fields keep connecting to the default server.
…_time mismatches fix-epoch-table.sql recomputes and rewrites start_time and end_time, but validate-epoch-table.sql only reported out_sum, fees, tx_count and blk_count differences (it computed the times and showed them, but never filtered on them). An epoch whose only error was its start or end time passed validation, and check-and-fix-epoch-table.sh then said "No mismatches" without offering the fix that would have changed it. Report a row when either time differs too, so validate checks every column the fix rewrites. On preview and preprod (13.7.1.0 table, 13.8.0.0 view) there are no such epochs today: still 0 rows.
Run directly without `-v epoch_no`, fix-epoch-table.sql and validate-epoch-table.sql stopped on a raw "syntax error at or near ':'" at `CAST(:epoch_no ...)`, and validate even exited 0 because it does not set ON_ERROR_STOP. Both now check the variable first and stop with "missing -v epoch_no: run psql -v epoch_no=<epoch, or -1 for all epochs> -f ..." and exit code 3, before any statement that reads or writes the epoch data. check-and-fix-epoch-table.sh always passes the variable and is unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since
13.7.2.1(migration 2-0049)epochis a view: db-sync stores finished epochs inepoch_finalizedand computes the current one fromblockandtx. The three epoch scripts in scripts/ still assumed a table. Found while testing13.8.0.0.What went wrong
fix-epoch-table.sqltries toINSERT ... ON CONFLICTinto the view, fails with cannot insert into view "epoch", and still exits 0: the script didn't stop on errors, so the COMMIT just rolled the failed transaction back.check-and-fix-epoch-table.shasks "Apply fix?" and then runs that fix.disable_epoch, or db-sync not started since the upgrade) the view is empty, sovalidate-epoch-table.sqllists every epoch as MISSING.Changes, one per commit:
fix-epoch-table.sqlstops at the first error, and refuses up front on a view, before BEGIN:ERROR: epoch is a view since db-sync
13.7.2.1, so this script cannot repair itHINT: Finished epochs are stored in
epoch_finalized, which db-sync maintains; a mismatch reported byvalidate-epoch-table.sqlpoints to a db-sync bug.On the old table it works as before, and a failed run now exits non-zero.
validate-epoch-table.sql: on a view with epoch sync not enabled, it says so and exits 0 instead of listing every epoch as MISSING.check-and-fix-epoch-table.sh: on a view it doesn't offer the fix. If epoch sync is not enabled it says so and exits 0 without validating. If there are mismatches it prints them, says the script can't repair them, and exits 1.check-and-fix-epoch-table.sh: with no terminal (closed stdin) the prompt hit end of input, and set -e ended the script with exit 1 and no message. No input now counts as "no", and an answer without a final newline is still taken.check-and-fix-epoch-table.sh: the header claimed it requiresPGPASSFILE, could take the database name "via psql_args", and read "field 5" of the pgpass line. None of that matched the code.scripts/README.md: the epoch section says what applies since13.7.2.1.check-and-fix-epoch-table.shtook only the database name from the pgpass file, so with a pgpass for another host, port or user it checked (and after "y" fixed) a same-named database on the default server. It now takes host, port, database and user from the first pgpass line, as db-sync does;PGHOST/PGPORT/PGDATABASE/PGUSERstill win, and a*field is left to psql.validate-epoch-table.sqlnow also reportsstart_time/end_timemismatches. The fix rewrites both, but validate never flagged them, so the wrapper said "No mismatches" for an epoch it would have changed.-v epoch_no,fix-epoch-table.sqlandvalidate-epoch-table.sqlnow stop withmissing -v epoch_no: run psql -v epoch_no=<epoch, or -1 for all epochs> ...and exit 3, instead of a raw syntax error (validate even exited 0).Why not repair
epoch_finalizedinstead: db-sync already maintains it with the same recompute fromblockandtx. It upserts at each epoch boundary, deletes from the rollback epoch on, and backfills at startup. The old fix also writes the current epoch, which would hide it fromepoch_currentuntil the next boundary.Testing
13.7.1.0(table) and13.8.0.0(view) schemas: 16 scenarios covering both forms, epoch sync on, off or not set up, mismatches, the answers "y" and "n", and no input. The new scenarios fail on master and pass with these commits, and removing any of the changes makes its scenario fail again.scripts/README.mdis also changed by #2183, in a different section, and the two merge cleanly.