Support custom dlt metadata table prefixes - #21
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10d8d9d151
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| prefix = self.config.internal_table_prefix.rstrip("_") | ||
| if not prefix: | ||
| raise ValueError("internal_table_prefix must contain at least one character") | ||
| return f"{prefix}_{table_name.removeprefix(logical_prefix)}" |
There was a problem hiding this comment.
When internal_table_prefix="dlt" and the pipeline has an ordinary resource named dlt_loads (similarly dlt_version or dlt_pipeline_state), both that resource and the corresponding _dlt_* metadata table resolve to the same catalog identifier. The user batch can therefore create the table first, after which metadata writes load it with an incompatible schema and fail the pipeline, or intermingle data if schemas happen to be compatible. Detect these collisions against the logical schema before loading rather than routing two logical tables to one physical name.
Useful? React with 👍 / 👎.
1 participant
Adds an opt-in
internal_table_prefixsetting so catalogs that reject leading underscores can store dlt metadata tables asdlt_loads,dlt_version, anddlt_pipeline_state.The default remains
_dlt, preserving normal dlt physical names and behavior. Logical schema names are unchanged in both modes, and every catalog read/write/drop/location path applies the same mapping.Validated with the full suite: 169 passed, 15 skipped.