Statistics-augmented Catalog Service API and DataFusion connector #16

rayhhome · 2025-11-19T05:36:20Z

Statistics-augmented Catalog Service API and DataFusion connector

Augment the Statistics module with a Service wrapper to make ready for integration and benchmark. Implement connectors to DataFusion to complete pipeline. Connect to the datafusion CLI for ease of use.

Architecture

Optd Catalog (`optd/catalog/`)

The main catalog service module. This package provides a runnable service that serves as a thread-safe metadata-augmented catalog backed by DuckLake. Below is the code architecture.

CatalogService: Catalog service in background. Uses mpsc channels to provide serialized access to the encapsulated DuckLake catalog and process incoming requests;
CatalogServiceHandle: Handle for sending requests to CatalogService; instantiated together with the backend service when generating new CatalogService. This handle can be cloned to be adapted in multi-threading situations. It currently allows updates to and retrievals from the service for the following information:
- Snapshot management (current_snapshot, current_snapshot_info);
- Schema retrieval (current_schema, current_schema_info);
- Statistics (table_statistics, update_table_column_stats).
DuckLakeCatalog: Wrapper of the DuckLake Catalog connection and implements the Catalog trait that determines the functionalities exposed.

Statistics are versioned: each update_table_column_stats creates a new snapshot with begin-to-end ranges, while supporting time-travel and advanced statistics stored as JSON.

DataFusion Connector (`connectors/datafusion/src/catalog.rs`, `connectors/datafusion/src/table.rs`)

Integration of the OptD catalog to fit in DataFusion's planning procedure:

OptdCatalogProviderList: Wrapper of DataFusion's catalog list, can take in a CatalogServiceHandle to communicate with its corresponding service and pass it down to its subordinate OptdCatalogProviders;
OptdCatalogProvider: Wrapper of DataFusion's catalog, can take in a CatalogServiceHandle;
OptdSchemaProvider and OptdTableProvider: Simple wrapper structs for schemas and tables.

CLI (`cli/src/main.rs`)

Allow default usage of the DataFusion Connector to interact with a background Optd Catalog Service process.

Catalog service is started in the background (tokio);
DuckLake catalog can be instantiated from OPTD_METADATA_CATALOG_PATH or default location.

Testing

Optd Catalog

Unit and edge tests for every single utility function (including time-travel and multiple-schema cases) for service and handle.
Stress and concurrency tests for the handle (single-thread heavy update and multi-thread update/retrieval).

DataFusion Connector

Integration tests that check interaction between the wrapper providers and the running background catalog service.

CLI

Two integration tests, one for thoroughly interacting with the DuckLakeCatalog, another for full integration among the Catalog service, the DataFusion Connector, and the CLI.
A smoke test (cli/src/smoke_test_cli.sh) for checking CLI usage in terminal (also provides intuition for how to use the catalog service).

Next Steps

The are two branching options for next steps:

Supporting updates to and retrievals from the Optd_query table, which tracks the queries executed at snapshot;
Exploring the more generic functionalities for the catalog, such as persistence (need research on delete vectors).

Note 1: currently the catalog service can buffer 1000 requests with mpsc (fixed). Do we make it a parameter that can be passed into the constructor of CatalogService?
Note 2: Should we make DuckLakeCatalog only public within the crate?

v0 of the Cascades-style optimizer. - Exhaustive optimization: expression and group returns only when the subgraph is optimized. - Applying enforcer rules and adding generated expressions to the memo table. - Special termination logic is required when the child has the same group + physical requirement as the parent. - Exhaustive exploration when applying rules, generate all bindings before doing the transform, but only expand based on specified rule patterns. Signed-off-by: Yuchen Liang <[email protected]>

Signed-off-by: Yuchen Liang <[email protected]>

## Problem Some methods are added to the IR as experimental features. We also got feedback from the dev meeting that the rule seems hard to read (or long). We would like to clean up these rough edges. ## Summary of changes - eliminate`try_bind_ref_xxx` and use `try_borrow` - add `borrow_raw_parts` so we always refer to `$node_name` instead of `$ref_name`. - Plumb through property methods to use shorthand. **_TODO:_** Pattern builder can also be generated by macros. --------- Signed-off-by: Yuchen Liang <[email protected]>

…ementation

…_provider

Signed-off-by: Yuchen Liang <[email protected]>

…ion pool

Signed-off-by: Yuchen Liang <[email protected]>

…to catalog_provider

… catalog_provider

yliang412 · 2025-11-20T04:59:30Z

.DS_Store

Let's remove this and add .DS_Store to .gitignore

Hmm, I've already added that to .gitignore, not sure why it sneaked through. Will fix this later today.

yliang412 · 2025-11-20T05:01:05Z

connectors/datafusion/src/catalog.rs

+#[derive(Debug)]
+pub struct OptdSchemaProvider {
+    inner: Arc<dyn SchemaProvider>,
+    catalog_handle: Option<CatalogServiceHandle>,


Why is this an Option

Oops, I think this is an artifact I left when I was trying to add caching to the catalog service and forgot to remove it from SchemaProvider or TableProvider. Let me fix this

…_provider

Copilot

Pull request overview

This PR introduces a statistics-augmented catalog service API backed by DuckLake and provides DataFusion integration to complete the query optimization pipeline. The implementation adds a thread-safe catalog service using async message passing, wrapper providers for DataFusion integration, and CLI support with optional persistent metadata.

Key Changes:

Implements CatalogService with async request handling via mpsc channels, supporting snapshot-based time travel for statistics
Provides DataFusion connector wrappers (OptdCatalogProviderList, OptdCatalogProvider, OptdSchemaProvider, OptdTableProvider) to integrate the catalog service into DataFusion's planning
Adds CLI integration with optional DuckLake metadata persistence via OPTD_METADATA_CATALOG_PATH environment variable

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`optd/storage/src/lib.rs`	Removed unused DuckLake storage module scaffolding
`optd/storage/Cargo.toml`	Removed unused storage crate
`optd/catalog/src/lib.rs`	Core catalog implementation with DuckDB backend, snapshot management, and statistics versioning
`optd/catalog/src/service.rs`	Async catalog service using tokio channels for thread-safe access
`optd/catalog/tests/statistics_tests.rs`	Comprehensive unit tests for catalog functionality including edge cases and time-travel
`optd/catalog/tests/service_tests.rs`	Service-level tests covering concurrency, shutdown, and handle cloning
`optd/catalog/Cargo.toml`	Dependencies for DuckDB, tokio, serde, and testing
`connectors/datafusion/src/catalog.rs`	Wrapper providers for DataFusion catalog integration
`connectors/datafusion/src/table.rs`	Table provider wrapper with statistics support
`connectors/datafusion/src/lib.rs`	Public API exports for connector
`connectors/datafusion/tests/integration_test.rs`	Integration tests for DataFusion connector with catalog service
`connectors/datafusion/Cargo.toml`	Dependencies including optd-catalog and async-trait
`cli/src/main.rs`	CLI integration with optional catalog service initialization
`cli/src/lib.rs`	Session context enhancements for external table creation
`cli/tests/catalog_service_integration.rs`	End-to-end integration tests for CLI with catalog service
`cli/smoke_test_cli.sh`	Shell script for manual CLI smoke testing
`cli/Cargo.toml`	CLI dependencies including optd-catalog
`Cargo.toml`	Workspace member cleanup removing optd-storage

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-13T22:50:44Z

connectors/datafusion/src/table.rs

+#[allow(dead_code)]
+pub struct OptdTable {
+    inner: Box<ListingTable>,
+    name: String,
+    table_reference: TableReference,
+}
+
+impl OptdTable {
+    pub fn try_new(
+        inner: ListingTable,
+        name: String,
+        table_reference: TableReference,
+    ) -> Result<Self> {
+        Ok(Self {
+            inner: Box::new(inner),
+            name,
+            table_reference,
+        })
+    }
+
+    pub fn new_with_inner(
+        inner: Box<ListingTable>,
+        name: String,
+        table_reference: TableReference,
+    ) -> Self {
+        Self {
+            inner,
+            name,
+            table_reference,
+        }
+    }
+
+    pub fn name(&self) -> &str {
+        &self.name
+    }
+
+    pub fn table_reference(&self) -> &TableReference {
+        &self.table_reference
+    }
+}


Setting #[allow(dead_code)] on the entire OptdTable struct and its methods suppresses warnings that could indicate unused code. If this is intended for future use, consider documenting why it exists or remove it if not needed in the current implementation.

Copilot · 2025-12-13T22:50:44Z

connectors/datafusion/src/table.rs

+    fn supports_filters_pushdown(
+        &self,
+        filters: &[&Expr],
+    ) -> Result<Vec<TableProviderFilterPushDown>> {
+        Ok(vec![
+            TableProviderFilterPushDown::Unsupported;
+            filters.len()
+        ])
+    }


The supports_filters_pushdown implementation returns Unsupported for all filters. This means filter pushdown optimizations are disabled. If filter pushdown is not yet implemented, add a TODO comment explaining the plan to enable it in the future.

Copilot · 2025-12-13T22:50:44Z

cli/smoke_test_cli.sh

+#!/usr/bin/env bash
+# CLI smoke test - verifies catalog integration is active
+
+set -e  # Exit on error


The shell script uses set -e but does not use set -u (fail on undefined variables) or set -o pipefail (fail if any command in a pipeline fails). This could mask errors in test execution. Consider adding these flags for more robust error detection.

Suggested change

set -e # Exit on error

set -euo pipefail # Exit on error, undefined variable, or failed pipeline

Copilot · 2025-12-13T22:50:45Z

optd/catalog/src/service.rs

+/// Max pending requests
+const CHANNEL_BUFFER_SIZE: usize = 1000;


The channel buffer size is hardcoded to 1000. As noted in the PR description, this might need to be configurable. Consider making CHANNEL_BUFFER_SIZE a parameter in the constructor to allow tuning based on expected load, or document why 1000 is the appropriate fixed value.

Suggested change

/// Max pending requests

const CHANNEL_BUFFER_SIZE: usize = 1000;

// The channel buffer size should be passed as a parameter to the constructor or initialization function

// to allow tuning based on expected load.

Copilot · 2025-12-13T22:50:45Z

cli/src/lib.rs

+    async fn create_external_table(&self, cmd: &CreateExternalTable) -> Result<DataFrame> {
+        let exist = self.inner.table_exist(cmd.name.clone())?;
+
+        if cmd.temporary {
+            return not_impl_err!("Temporary tables not supported");
+        }
+
+        if exist {
+            match cmd.if_not_exists {
+                true => return self.return_empty_dataframe(),
+                false => {
+                    return exec_err!("Table '{}' already exists", cmd.name);
+                }
+            }
+        }
+
+        let table_provider: Arc<dyn TableProvider> = self.create_custom_table(cmd).await?;
+        self.register_table(cmd.name.clone(), table_provider)?;
+
+        self.return_empty_dataframe()
+    }


Missing documentation for the CreateExternalTable handling. The new create_external_table method adds significant functionality but lacks documentation explaining its purpose, parameters, and behavior. Add docstring comments to clarify the intent and usage.

Copilot · 2025-12-13T22:50:45Z

optd/catalog/src/service.rs

+                source: duckdb::Error::ExecuteReturnedResults,
+            })?;
+
+        rx.await.map_err(|_| Error::QueryExecution {
+            source: duckdb::Error::ExecuteReturnedResults,
+        })?


The error handling here uses ExecuteReturnedResults as a catch-all error for channel communication failures. This is misleading because ExecuteReturnedResults is a DuckDB error meant for SQL execution, not channel communication errors. Consider creating a more specific error variant for channel/communication errors.

Copilot · 2025-12-13T22:50:45Z

optd/catalog/src/service.rs

+    /// Get a reference to the underlying DuckLakeCatalog for test setup only.
+    /// Only available in test/debug builds and should
+    /// only be used for setting up test fixtures.
+    #[cfg(any(test, debug_assertions))]
+    pub fn catalog_for_setup(&self) -> &DuckLakeCatalog {
+        &self.backend
+    }
+}


The method catalog_for_setup is only available in debug/test builds but returns a reference to the backend which could enable modifications outside the intended async service pattern. Even though this is test-only, consider returning a read-only interface or documenting that this should only be used before spawning the service.

rayhhome and others added 30 commits August 5, 2025 18:33

first commit

36cfae2

setup CLI and connectors skeleton (#4)

8716478

Signed-off-by: Yuchen Liang <[email protected]>

setup ci (#5)

deeac21

Signed-off-by: Yuchen Liang <[email protected]>

cargo.lock update

ef1a181

lib.rs intercept create external table

e1bf28c

Merge branch 'main' into catalog_provider

fa8e974

optclicontext create external table impl attempt

762fae7

Merge remote-tracking branch 'origin' into catalog_provider

f92efb1

Tentative statistics retriver structs and fetch_table_statistics impl…

40274ab

…ementation

Merge branch 'main' of https://github.com/yliang412/optd into catalog…

f2a17dd

…_provider

fetch stats, update stats, and preliminary tests impl

bad8c8f

update dependencies

af832ba

add some notes to refactoring

a610854

Signed-off-by: Yuchen Liang <[email protected]>

fix non unique table name query, use fromIterator, and remove connect…

c114adf

…ion pool

make tests use tempdir

b0e5329

attempted update_stats impl

a85ad15

add new snapshot logic

ff7ce20

more rubust tests for stats

1952737

move tests to directory and add basic stats update

86dcfa1

Upgrade fetch and update functions to use adv stats

c2f31c2

get schema implementation update

bf988e3

documentation & format

23192a5

refactor

dd3f9eb

Signed-off-by: Yuchen Liang <[email protected]>

fix clippy

54731f6

Signed-off-by: Yuchen Liang <[email protected]>

refactor methods as associated

1753080

cleanup

7bf7e27

modularity refactor + general cleanup

d25b78e

move to connectors

83e4cb4

rayhhome added 5 commits November 11, 2025 13:03

Merge branch 'statistics_api' of https://github.com/yliang412/optd in…

c658be8

…to catalog_provider

service implementation and test suite

e6e9253

rename error

ba0bfa0

update lock and remove superfluous main

740e1f3

catalog implementation and test suite + datafusion version bump

4b579be

rayhhome added enhancement New feature or request rust Pull requests that update rust code dependencies Pull requests that update a dependency file labels Nov 19, 2025

rayhhome added 5 commits November 19, 2025 00:40

cargo fmt

85205c4

bump version of datafusion & cli to 51.0

4bebc8f

Merge branch 'version-bump' of https://github.com/yliang412/optd into…

2103a8a

… catalog_provider

add datafusion features & update gitignore

4dba0d4

Merge branch 'version-bump' of https://github.com/yliang412/optd into…

b5709e9

… catalog_provider

yliang412 reviewed Nov 20, 2025

View reviewed changes

rayhhome added 7 commits November 20, 2025 22:24

refactoring inconsistent design and remove ds_store

03ebbbb

update cli to use catalog service

11e8240

add cli smoke and integration tests

a487782

remove lib extraneous comment

5727ced

Change path naming

664e55a

Add more comprehensive Datafusion integration tests

851be1f

Merge branch 'main' of https://github.com/yliang412/optd into catalog…

f6aa2c7

…_provider

Copilot AI review requested due to automatic review settings December 13, 2025 22:47

Copilot started reviewing on behalf of rayhhome December 13, 2025 22:47 View session

Copilot AI reviewed Dec 13, 2025

View reviewed changes

rayhhome mentioned this pull request Dec 17, 2025

Feature/persistent table registration #22

Open

	set -e # Exit on error
	set -euo pipefail # Exit on error, undefined variable, or failed pipeline

		/// Max pending requests
		const CHANNEL_BUFFER_SIZE: usize = 1000;

Statistics-augmented Catalog Service API and DataFusion connector #16

Are you sure you want to change the base?

Statistics-augmented Catalog Service API and DataFusion connector #16

Uh oh!

Conversation

rayhhome commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Statistics-augmented Catalog Service API and DataFusion connector

Architecture

Optd Catalog (optd/catalog/)

DataFusion Connector (connectors/datafusion/src/catalog.rs, connectors/datafusion/src/table.rs)

CLI (cli/src/main.rs)

Testing

Optd Catalog

DataFusion Connector

CLI

Next Steps

Uh oh!

yliang412 Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

rayhhome Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

yliang412 Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

rayhhome Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 13, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rayhhome commented Nov 19, 2025 •

edited

Loading

Optd Catalog (`optd/catalog/`)

DataFusion Connector (`connectors/datafusion/src/catalog.rs`, `connectors/datafusion/src/table.rs`)

CLI (`cli/src/main.rs`)