enh(wal): [TS-7567]set wal keep version to avoid deleting

[wangmm0220]

Description

Please briefly describe the code changes in this pull request.

Checklist

Please check the items in the checklist if applicable.

• Is the user manual updated?
• Are the test cases passed and automated?
• Is there no significant decrease in test coverage?

[gemini-code-assist]

Summary of Changes

Hello @wangmm0220, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances WAL (Write-Ahead Log) management by introducing explicit control over log retention and trimming. Users can now define a 'keep version' for vgroups to prevent critical WAL files from being deleted prematurely, and manually trigger WAL trimming for a database. Concurrently, the scheduler's task retry logic has been improved to make task execution more resilient and efficient, especially in distributed environments, by optimizing how failures are handled and tasks are re-routed.

Highlights

• New SQL Commands for WAL Management: Introduced two new SQL commands: TRIM DATABASE <db_name> WAL to explicitly trigger WAL trimming for a database, and ALTER VGROUP <vgId> SET KEEP VERSION <version> to set a minimum WAL version to retain for a specific vgroup.
• WAL Keep Version Mechanism: Implemented a 'keep version' feature for the Write-Ahead Log (WAL) to prevent premature deletion of WAL files. This ensures that logs below a specified version are not removed, crucial for data consistency and recovery. This version is persisted in the WAL metadata.
• Enhanced Scheduler Retry Logic: Refined the scheduler's retry mechanism, particularly for data-bound tasks. This includes adjustments to retry counts, delay calculations, and how candidate addresses (endpoints) are managed during task failures and redirects, improving overall robustness and efficiency.
• TMQ Client WAL Marker: Added an enableWalMarker configuration option and field to the TMQ (TDengine Message Queue) client, allowing the client to send WAL mark messages to the mnode when committing offsets, which can influence WAL retention.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a valuable feature to prevent WAL deletion up to a specified version, which is particularly useful for WAL consumers. It adds new SQL commands: ALTER VGROUP ... SET KEEP VERSION ... and TRIM DATABASE ... WAL. The implementation is comprehensive, touching multiple layers of the system from the SQL parser down to the WAL storage, and the changes are logically sound. Additionally, the pull request includes a significant and beneficial refactoring of the scheduler's retry and redirect logic, enhancing its robustness. My review has identified a few issues, including a memory leak in an error handling path, a misleading error message, and a potential bug in an iteration loop's error handling. Addressing these points will further improve the quality of this contribution.