Foreign Table: Persistent Postgres table for managing external data queried by DuckDB readers

[YuweiXiao]

This PR introduces foreign table support, allowing users to persist an external file's view queried through DuckDB's readers (read_csv, read_parquet, and read_json).

Previously, users had to embed file locations and options directly in queries, and use r[xx] syntax for column reference. Foreign tables simplify this by defining file paths and reader options once at CREATE time, enabling clean SELECT statements without r[xx] syntax. This also opens room for access control on external files, such as fine-grained permissions like column-level visibility for different users.

CREATE TABLE Syntax

CREATE FOREIGN TABLE external_csv () 
  SERVER duckdb 
  OPTIONS (
    location = '../../data/iris.csv',
    format = 'csv',
    options = '{"header": true}'
);

-- Query like a regular table
SELECT * FROM external_csv;
SELECT "sepal.length" FROM external_csv;

-- Raw SQL way
SELECT r['sepal.length'] FROM read_parquet('../../data/iris.csv')

Features

• DDL Support: CREATE FOREIGN TABLE, DROP FOREIGN TABLE, ALTER TABLE NAME
• Auto Schema Inference: Column names and types inferred by DuckDB, persisted in Postgres catalog
• Lazy loading: External tables are dynamically loaded as DuckDB views only when referenced in queries

[visardida]

This is a much needed improvement @YuweiXiao , thank you!

Does this implementation of external table support handle partitioned Parquet datasets for example, when using wildcard paths or recursive directory patterns such as:

read_parquet('/path/to/data/**/*.parquet')

In other words, if I create an external table pointing to a directory of Parquet partitions, will it automatically discover and read all matching files, or does it only support a single file path per table definition?

[YuweiXiao]

Does this implementation of external table support handle partitioned Parquet datasets for example, when using wildcard paths or recursive directory patterns such as:

read_parquet('/path/to/data/**/*.parquet')

In other words, if I create an external table pointing to a directory of Parquet partitions, will it automatically discover and read all matching files, or does it only support a single file path per table definition?

YES. External table tracks path / read_options in pg catalog. And file list is triggered for each query. Theoretically, all functionality supported by read_xxxx should also be available in external table.

[JelteF]

Thanks for the work on this! I also had something like this in mind, but I was thinking about using FOREIGN TABLES instead of table access methods for this. So I'm wondering why you went this route instead. (not saying that one is really better than the other, but I'm wondering what tradeoffs you considered)

[YuweiXiao]

Thanks for the work on this! I also had something like this in mind, but I was thinking about using FOREIGN TABLES instead of table access methods for this. So I'm wondering why you went this route instead. (not saying that one is really better than the other, but I'm wondering what tradeoffs you considered)

Yes, FOREIGN TABLE would definitely work too. I didn’t have a strong tradeoff in mind — mainly wanted to reuse the existing codebase as much as possible, e.g., the DuckDB AM that’s already properly hooked and the registered triggers.

I’ll take another look at the FOREIGN TABLE approach — it has a better semantic fit (i.e., metadata only table).

[JelteF]

Thinking about it more, I do think FOREIGN TABLE is a better fit for this semantically. Because the CREATE TABLE command that you have now isn't actually creating the backing files. It's only registering some already existing external data in postgres.

[YuweiXiao]

Thinking about it more, I do think FOREIGN TABLE is a better fit for this semantically. Because the CREATE TABLE command that you have now isn't actually creating the backing files. It's only registering some already existing external data in postgres.

Yeah. I will initiate a discussion thread and let's define the SQL interface (usage) before impl.

[AndrewJackson2020]

The above change (or a similar change using FDW instead) would be great. One of the issues with the current syntax is that it does not play nice with ORM's which is a big annoyance for a lot of teams. Also I could see a usage pattern with pg_duckdb whereby you keep "live data" in postgres tables (or partitions) and "archive data" on s3/parquet. Would be great to be able to access both of these with a uniform interface.

[YuweiXiao]

Hi @JelteF , the PR is ready for review:

• Switched to the FOREIGN TABLE interface for table management.
• A pre-created server (ddb_foreign_server) is added in the extension SQL — we can adjust the naming if needed.
• Added support for table rename DDLs (both ALTER FOREIGN TABLE and ALTER TABLE).

[JelteF]

Let's just use duckdb for this. I think that will make the SQL look nicer:

CREATE FOREIGN TABLE external_parquet ()
  SERVER duckdb
  OPTIONS (
    location '../../data/iris.parquet'
  );

Suggested change

	#define DUCKDB_FOREIGN_SERVER_NAME "ddb_foreign_server"
	#define DUCKDB_FOREIGN_SERVER_NAME "duckdb"

[YuweiXiao]

hi @JelteF , can we target it for 1.1.0 release? will put in effect to address any feedback.

[JelteF]

I would love to include this in 1.1.0, but it's primarily my own time that's the bottleneck here (not you). I need to spend some quality time playing with this, and reading the code. Your two PRs (this one and the INSERT one) are definitely the number one features that I'd like to get released. But they're both non-trivial to review.

But the main branch has been accumulating small fixes that I just want to release somewhere in the next few days. So that's why I moved this to the 1.2.0.

[JelteF]

One thing I noticed now. Can you update the PR description to use the new syntax?

[jan-swiatek]

I know that's though question, but when is version 1.2.0 roughly scheduled for release?

[visardida]

I am also very eager to get this feature, I was so hoping to get it in v1.1.0

[JelteF]

Could you fix the merge conflicts, that will make it easier to review this (which I'm planning to do early january).