enh: enh sys scan in vtable query.

[Simon9997]

Description

Please briefly describe the code changes in this pull request.

Checklist

Please check the items in the checklist if applicable.

• Is the user manual updated?
• Are the test cases passed and automated?
• Is there no significant decrease in test coverage?

[gemini-code-assist]

Summary of Changes

Hello @Simon9997, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces significant enhancements to the system scan capabilities within virtual table queries. The core objective is to improve query performance and efficiency by implementing a versioning mechanism for virtual tables. This allows the system to intelligently determine if a re-scan is necessary, reducing redundant data processing. The changes span across data structures, metadata management, query planning, and execution, ensuring a cohesive integration of virtual table versioning throughout the database system.

Highlights

• New System Scan Operator Parameter: A new SSysScanOperatorParam struct has been introduced to include/common/tmsg.h, defining parameters for system scan operations, including flags for virtual supertables (isVstb), unique identifiers (uid), and version numbers (version).
• Virtual Table Versioning API: New metadata API functions, metaGetVirtualSupertableVersion and metaGetVirtualNormalChildtableVersion, have been added to include/libs/executor/storageapi.h and source/dnode/vnode/inc/vnode.h. These functions allow retrieval of version information for virtual tables, which is crucial for optimizing scan operations.
• Dynamic Query Control Enhancements: The SDynQueryCtrlVtbScan and SVtbScanDynCtrlBasic structs in include/libs/nodes/plannodes.h have been extended with pSysScanVgIds. This addition enables the dynamic query control mechanism to manage virtual group IDs specifically for system scan operations.
• Serialization and Deserialization Support: The source/common/src/msg/tmsg.c file has been updated to include serialization and deserialization logic for the new SSysScanOperatorParam struct, ensuring that system scan parameters can be correctly transmitted and reconstructed across the system.
• Vnode Virtual Table Version Management: The SVnode structure in source/dnode/vnode/src/inc/vnodeInt.h now incorporates versionLock and pVtableVersion to manage and track versions of virtual tables. This allows for efficient caching and invalidation of virtual table metadata.
• Refactored Virtual Table Scan Operator: The virtual table scan logic in source/libs/executor/src/dynqueryctrloperator.c has been refactored into distinct vtbScanOpen and vtbScanNext functions. This change, along with the introduction of buildBatchExchangeOperatorParamForVSysScan, integrates the new system scan versioning into the query execution flow, enabling version-aware data retrieval.
• System Table Scan Logic Update: The sysTableScanUserVcCols function in source/libs/executor/src/sysscanoperator.c now performs version checks for virtual tables. This prevents unnecessary data collection if the table version has not changed, and sysTableUserColsFillOneVirtualTableCols has been updated to include the vstb_version in the output.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request enhances system table scans for virtual tables by introducing a versioning mechanism to avoid re-fetching data if it has not changed. The changes are extensive, touching the data structures for messaging, execution plans, and operator parameters, as well as the logic in the executor and storage layers. While the overall approach seems sound, the review identified several issues. These include a critical bug due to the removal of necessary data copying logic, high-severity issues related to potential memory leaks and data corruption from type mismatches, and several medium-severity issues concerning code style and potential data representation problems. Addressing these points will significantly improve the robustness and correctness of the implementation.