[C++][Parquet] GH-47628: Implement basic parquet file rewriter

[HuaHuaY]

This is a draft PR now. I follow Java's implementation but I think it is not a good enough design for C++. Because we must copy lots of code from file_writer.cc or file_reader.cc and it will be troublesome to maintain in the future. I prefer to implement some classes inheriting XXXWriter or XXXReader. I'll think about how to refactor the code. If anyone has any good suggestions, please comment.

Now I have written two kinds of tests. Test the horizontal splicing and vertical splicing of parquet files separately. But only horizontal splicing is implemented now because I don't find an efficient way to merge two parquet files' schema.

Rationale for this change

Allow to rewrite parquet files in binary data formats instead of reading, decoding all values and writing them.

What changes are included in this PR?

1. Add class ParquetFileRewriter and RewriterProperties.
2. Add some to_thrift and SetXXX methods to help me copy the metadata.
3. Add CopyStream methods to call memcpy between ArrowInputStream and ArrowOutputStream.
4. Add RowGroupMetaDataBuilder::NextColumnChunk(std::unique_ptr<ColumnChunkMetaData> cc_metadata, int64_t shift) which allows to add column metadata without creating ColumnChunkMetaDataBuilder.

Are these changes tested?

Yes

Are there any user-facing changes?

1. Add some new classes and methods mentioned above.
2. ReaderProperties::GetStream is changed to a const method. Only the signature has been changed. Its original implementation allows it to be declared as a const method.

• GitHub Issue: [C++][Parquet] Provide a rewriter to rewrite parquet files without decoding all the row groups/pages #47628

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[HuaHuaY]

@pitrou @adamreeve @mapleFU Do you have any suggestions about this draft? Is there any efficient way to merge two parquet files' schema?

[mapleFU]

Emm I'm thinking that just reuse the current code a ok way, since these logic in current impl would be a bit hacking with current interface...

[mapleFU]

Can this separate to different method? This reuse is a bit hacking to me

[wgtmac]

I haven't reviewed all the changes yet and will progressively post my comments.

[wgtmac]

The general workflow of the rewriter looks good to me. However, I don't believe we should directly manipulate the thrift objects.

[wgtmac]

Perhaps introduce a RowGroupContext to hold all row group xxx readers?

[wgtmac]

This is blocked by #37400

[wgtmac]

Perhaps we need to split this into smaller but dedicated functions?