feat: ducklake_catalog_connstr config for remote DuckLake catalog#72
Merged
feat: ducklake_catalog_connstr config for remote DuckLake catalog#72
Conversation
Allow flush threads and snapshot consumers to point at a remote PG for DuckLake metadata instead of the local PG. When unset, behavior is unchanged (local PG). Enables Citus worker-side execution where each worker flushes to a shared DuckLake catalog on an analytics node. - New string config key in GroupConfig/ResolvedConfig (global + per-group) - FlushCoordinator resolves catalog connstr internally from ResolvedConfig - Redact credentials in set_config logs for connstr-valued keys - Remove dead ServiceConfig.duckdb_pg_connstr field - Fix get_group_config false "unknown key" error for Option<String> fields Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ist) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Verifies flush threads actually use the configured ducklake_catalog_connstr for DuckDB ATTACH by creating a dedicated sync group with a Unix socket loopback connstr, inserting data, and confirming it arrives in the target. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merge config CRUD and E2E flush tests into a single file. Remove the separate _e2e test. Move to late schedule position (needs bgworker). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Applies the same redaction logic from set_config to set_group_config, preventing connection string passwords from leaking to PostgreSQL logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ducklake_catalog_connstrstring config key (global + per-group) that lets flush threads and snapshot consumers point at a remote PG for DuckLake metadata instead of the local PGResolvedConfig— zero changes to bgworker/daemon call sitesset_config()logs when the key is a connection stringServiceConfig.duckdb_pg_connstrfield and stalehost=127.0.0.1variableget_group_config(key)returning false "unknown key" error forOption<String>config fieldsEnables Citus worker-side execution (CITUS_DESIGN.md Phase 1a) where each worker flushes to a shared DuckLake catalog on an analytics node.
Test plan
ducklake_catalog_connstrregression test (config CRUD, empty string rejection, per-group override, cleanup)make installcheck)cargo fmt --all --checkcleanducklake_catalog_connstrpointing at a remote PG, verify flush threads ATTACH to the remote catalog🤖 Generated with Claude Code