feat: inject pg_duckdb S3/GCS/R2 secrets into DuckDB connections#74
Merged
feat: inject pg_duckdb S3/GCS/R2 secrets into DuckDB connections#74
Conversation
pg_duckpipe's in-memory DuckDB connections had no cloud storage credentials, so DuckLake tables with storage_location='s3://...' would fail on flush. This reads secrets from pg_duckdb's existing FDW catalogs (pg_foreign_server + pg_user_mapping) at bgworker/daemon startup and injects them as CREATE SECRET statements into every DuckDB connection. - New duckdb_secrets module with shared SQL query (SECRET_QUERY) - Secrets threaded through FlushCoordinator → flush threads + snapshots - Both bgworker (SPI) and daemon (TCP) modes supported - Secret failures are non-fatal (logged as warnings) - Arc<Vec<String>> for zero-cost sharing across flush threads Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The raw pg_user_mapping catalog has column-level ACL in PG 17+ that restricts umoptions visibility. Switch SECRET_QUERY to use the pg_user_mappings VIEW which applies proper access control (shows options when the user is the mapped user with USAGE on the server, or is a superuser). Also fixes test assertion for lowercase option names (PG lowercases unquoted FDW option identifiers). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pg_foreign_server+pg_user_mapping) and injects them asCREATE SECRETstatements into pg_duckpipe's in-memory DuckDB connectionsduckdb_secretsmodule induckpipe-corewith a shared SQL query (SECRET_QUERY) that builds completeCREATE SECRETstatements server-sideFlushCoordinator→ flush threads + snapshot workersArc<Vec<String>>for zero-cost cloningTest plan
build_create_secret_sql(option parsing, quoting, edge cases)duckdb_secretsregression test: creates S3 secret viaduckdb.create_simple_secret(), verifies catalog discovery, validates generated SQL formatducklake_catalog_connstrregression test passes (exercises full bgworker + flush path with modifiedFlushCoordinator::newsignature)group_configregression test passescargo fmt --all --checkcleancargo check --workspaceclean (no new warnings)🤖 Generated with Claude Code