Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Nov 24, 2025

Description

Documents the new periodic refresh table jobs feature introduced in risingwavelabs/risingwave#23737. This experimental feature enables scheduled full reloads of Iceberg tables with refresh_mode = 'FULL_RELOAD'.

Changes:

  • New command: REFRESH TABLE for manual refresh triggers
  • New system catalog: rw_refresh_table_state for monitoring refresh job status
  • Enhanced Iceberg ingestion: Added refresh_mode and refresh_interval_sec parameters for periodic reloads
  • Updated feature matrix: Periodic loading now supported for both append-only and mutable Iceberg tables
  • Clarified default behavior: Continuous ingestion marked as the default approach
  • Important usage notes: FULL_RELOAD requires CREATE TABLE (not source), and data only loads after triggering a refresh
  • Scheduler configuration: Added documentation for stream_refresh_scheduler_interval_sec config parameter

Example:

CREATE TABLE iceberg_batch_table (
    id int primary key,
    name varchar
) WITH (
    connector = 'iceberg',
    refresh_mode = 'FULL_RELOAD',
    refresh_interval_sec = '60',
    -- ... other Iceberg connector parameters
);

-- Manual refresh (required to load initial data)
REFRESH TABLE iceberg_batch_table;

-- Monitor status
SELECT table_id, current_status, last_trigger_time, trigger_interval_secs
FROM rw_catalog.rw_refresh_table_state;

All features marked as experimental per source PR requirements.

Related code PR

risingwavelabs/risingwave#23737

Related doc issue

Fix #815

Checklist

  • I have run the documentation build locally to verify the updates are applied correctly.
  • For new pages, I have updated mint.json to include the page in the table of contents.
  • All links and references have been checked and are not broken.
Original prompt

This section details on the original issue you should resolve

<issue_title>Document: feat: add periodic refresh table jobs & refactor ProgressTracker</issue_title>
<issue_description>This issue tracks the documentation update needed for the merged PR risingwavelabs/risingwave-docs#23737.

Source PR URL: risingwavelabs/risingwave#23737
Source PR Merged At: 2025-11-24T08:59:25Z

If it is a major improvement that deserves a new page or a new section in the documentation, please check if we should label it as an experiment feature.</issue_description>

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI changed the title [WIP] Update documentation for periodic refresh table jobs docs: Add periodic refresh table jobs documentation (experimental) Nov 24, 2025
Copilot finished work on behalf of tabVersion November 24, 2025 09:49
Copilot AI requested a review from tabVersion November 24, 2025 09:49
@tabVersion tabVersion marked this pull request as ready for review November 25, 2025 05:03
…ements, and scheduler interval

Co-authored-by: tabVersion <[email protected]>
Copilot AI requested a review from tabVersion November 25, 2025 05:07
Copilot finished work on behalf of tabVersion November 25, 2025 05:07
@WanYixian
Copy link
Collaborator

@chenzl25 Please check the accuracy of the doc, thanks!

Copy link

@chenzl25 chenzl25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@WanYixian WanYixian merged commit a6372e2 into main Nov 28, 2025
5 checks passed
@WanYixian WanYixian deleted the copilot/update-documentation-refresh-table-jobs branch November 28, 2025 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document: feat: add periodic refresh table jobs & refactor ProgressTracker

4 participants