Skip to content

Conversation

@shiyasmohd
Copy link
Contributor

Add a Dataset trait in datasets-common to provide a unified interface for accessing dataset properties across all dataset types (raw extractors and derived).

  • Add DatasetWithFunctions sub-trait for datasets supporting JavaScript UDFs
  • Move Table struct from common::catalog::logical to datasets-common::dataset
  • Move js_udf.rs from common to js-runtime crate
  • Add udf re-export module in datasets-common for UDF-related types
  • Implement Dataset trait in each extractor crate (evm-rpc, eth-beacon,
    firehose, solana) and datasets-derived

@shiyasmohd shiyasmohd self-assigned this Jan 19, 2026
@shiyasmohd shiyasmohd requested a review from LNSD January 19, 2026 17:56
Copy link
Contributor

@LNSD LNSD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, check my comments 🙂

Copy link
Contributor

@LNSD LNSD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, check my comments 🙂

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary for the PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

js_runtime::IsolatePool is needed in datasets-common for DatasetWithFunctions trait here and datasets-derived. common imports js_runtime & datasets-common, hence best option was to move js_udf to js_runtime which is now imported by datasets-common.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's very incovenient :/

Then, we need to find a better way to handle the special case of derived datasets.

};
use datasets_common::udf::IsolatePool;
use futures::{TryStreamExt, stream};
use js_runtime::isolate_pool::IsolatePool;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep the IsolatePool in js-runtime

Copy link
Contributor

@LNSD LNSD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ✅

@LNSD
Copy link
Contributor

LNSD commented Jan 21, 2026

Note that there are some merge conflicts

@shiyasmohd shiyasmohd force-pushed the shiyasmohd/dataset-trait branch from 345d540 to ea74bd7 Compare January 21, 2026 11:26
@shiyasmohd shiyasmohd merged commit 2b14be2 into main Jan 21, 2026
8 checks passed
@shiyasmohd shiyasmohd deleted the shiyasmohd/dataset-trait branch January 21, 2026 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants