Skip to content

fix(db): audit partition catalog before creation - #6515

Open
Areson wants to merge 2 commits into
block:mainfrom
Areson:Areson/partition-audit-observability
Open

fix(db): audit partition catalog before creation#6515
Areson wants to merge 2 commits into
block:mainfrom
Areson:Areson/partition-audit-observability

Conversation

@Areson

@Areson Areson commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Why

The partition manager checks partition existence by name only and treats Postgres 42P17 "would overlap partition" errors as success. On databases where a catch-all partition (events_p_future / delivery_log_p_future) covers future months, the monthly creation path silently does nothing while real coverage erodes toward a write outage. There is currently no visibility into partition catalog health and no signal before writes start failing.

What

  • Read-only catalog audit in buzz-db (partition.rs): classifies the live partition catalog for events and delivery_log via pg_inherits and bound parsing — coverage bounds, catch-all detection, anomalous children, and trigger parity checked recursively down to routable leaf partitions.
  • Creation gated on the audit verdict: the 42P17-to-success swallow is removed; monthly CREATE runs only for months the audit proves uncovered, a canonical-name mismatch is an error, and per-month outcome capture replaces first-error abort. New BUZZ_PARTITION_MANAGER_CREATE_ENABLED kill switch (default true).
  • Prometheus metrics per table: buzz_partition_serving_safe, buzz_partition_uncovered_months (audit window is now through +3 months), buzz_partition_catch_all_covered_months, buzz_partition_anomalous_children, buzz_partition_trigger_parity_missing/_extra, buzz_partition_audit_duration_seconds, buzz_partition_audit_runs_total{outcome}, buzz_partition_audit_failures_total, buzz_partition_create_attempts_total, and buzz_partition_audit_last_success_timestamp_seconds for staleness alerting.
  • Readiness term: /_readiness now includes partition serving-safety. It serves from the last successful audit and fails only when no partition covers the current time (or before the first successful audit completes at boot). An audit-path failure alone never fails readiness — that is surfaced through the failure counter and the last-success timestamp instead, so a monitoring-path outage cannot take serving pods out of rotation while coverage is provably fine.
  • Periodic audit task in the relay, plus config surface for the audit interval and the monthly-create kill switch. Catch-alls are detected from catalog bounds rather than configured names.
  • Doc correction: the mesh heartbeat comment no longer claims readiness gating (it never was gated; behavior unchanged).

Behavior changes & risk

  • No DETACH/DROP/ATTACH and no schema migrations. The only DDL is the pre-existing monthly CREATE path, now strictly narrower (audit-proven-uncovered months only) and behind a kill switch.
  • Readiness: pods report NotReady until the first successful audit at boot (short window by design), and whenever no partition covers now() — a state that was previously a silent write outage. The partition term reflects shared catalog state, so it changes for all pods together.
  • Partition-overlap errors now surface as errors instead of being swallowed as success.
  • Rollback: revert the commit, or set BUZZ_PARTITION_MANAGER_CREATE_ENABLED=false to disable the creation path while keeping the audit and metrics.

Validation

  • New Postgres-backed regression tests for catalog classification and trigger parity (disabled child trigger, ENABLE ALWAYS parent, disabled nested leaf clone in a multi-level layout, healthy-catalog zero-parity checks). Each new regression was verified to fail against the previous logic before the fix.
  • Full buzz-db and buzz-relay test suites green at this head; cargo clippy --all-targets -- -D warnings clean.

Follow-up work (separate PRs): bounded catch-all advance (empty-only golden path; non-empty reattach only behind an explicit operator flag, default off), fresh-install convergence, and a proactive coverage-runway readiness threshold.

Update — 2026-08-21

  • Fixed nested-leaf trigger parity auditing so child-only row triggers are checked on every routable descendant leaf and reported as {leaf}:{trigger} under the owning root child.
  • Added a Postgres regression test for a child-only trigger on a nested leaf; mutation-reverting to immediate-child-only extra-trigger scanning makes it fail.

Co-authored-by: Ian Oberst <ioberst@squareup.com>
Signed-off-by: Ian Oberst <ioberst@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
Ai-assisted: true
@Areson
Areson requested a review from a team as a code owner August 21, 2026 20:21
Audit child-only row triggers on every routable descendant leaf,
qualifying nested leaves the same way missing triggers are qualified.
Add a Postgres regression test that fails when extra-trigger parity only
looks at the immediate root child.

Co-authored-by: coder 0 <d97ebdbb198c7237c94f84ea8bb8a73583ea067407eebd0062abbb3962527fb1@buzz.block.builderlab.xyz>
Signed-off-by: coder 0 <d97ebdbb198c7237c94f84ea8bb8a73583ea067407eebd0062abbb3962527fb1@buzz.block.builderlab.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants