feat(Nimble): New Flush Policy Implementation With Chunking (#14846)

Summary:
X-link: https://github.com/facebookexternal/presto-facebook/pull/3412

Pull Request resolved: #14846

X-link: facebookincubator/nimble#240

This is an implementation of the new chunking policy described in this [doc](https://fburl.com/gdoc/gkdwwju1). It has two phases:

**Phase 1 - Memory Pressure Management (shouldChunk)**
The policy monitors total in-memory data size:
*  When memory usage exceeds the maximum threshold, initiates chunking to reduce memory footprint while continuing data ingestion
*  While memory remains above the minimum threshold, continues chunking to further reduce memory usage

**Phase 2 - Storage Size Optimization (shouldFlush)**
 Implements compression-aware stripe size prediction:
*   When chunking fails to reduce memory usage effectively and memory stays above the maximum threshold, forces a full stripe flush to guarantee memory relief
*   Calculates the anticipated final compressed stripe size by applying the estimated compression ratio to unencoded data
*   Triggers stripe flush when the predicted compressed size reaches the target stripe size threshold

`shouldChunk` is also now a separate method required by all flush policies. We updated all previous tests and code references.

Reviewed By: helfman, sdruzkin

Differential Revision: D81516697

fbshipit-source-id: 28e144da2eaca3ca140b4ac02b5f61ae7aec08a0

Author: macvincent

velox/experimental/wave/dwio/nimble/tests/NimbleReaderTestUtil.cpp

@@ -348,7 +348,8 @@ std::vector<std::unique_ptr<StreamLoader>> writeToNimbleAndGetStreamLoaders(
   writerOptions.minStreamChunkRawSize = 0;
   writerOptions.flushPolicyFactory = [] {
     return std::make_unique<LambdaFlushPolicy>(
-        [](const StripeProgress&) { return FlushDecision::Chunk; });
+        /*flushLambda=*/[](const StripeProgress&) { return false; },
+        /*chunkLambda=*/[](const StripeProgress&) { return true; });
   };
 
   std::vector<std::unique_ptr<StreamLoader>> allStreamLoaders;