out_azure_blob: add zstd compression support #11202
Conversation
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughAdds configurable payload compression (gzip and zstd) to the Azure Blob output plugin: replaces a gzip-only flag with a compression enum, integrates compression into payload/URI/header flows, updates HTTP APIs to carry content-encoding, and adds runtime tests for compression behavior. Changes
Sequence Diagram(s)sequenceDiagram
participant Plugin as Azure Blob Plugin
participant Compressor as flb_compression (gzip/zstd)
participant HTTP as HTTP Client
participant Azure as Azure Blob Storage
Note over Plugin,Compressor: Decide compression strategy (config + blob type)
Plugin->>Compressor: compress(payload, algorithm)
alt compression succeeds
Compressor-->>Plugin: compressed_payload
Plugin->>HTTP: create request(payload=compressed_payload, Content-Encoding, Content-Type, URI ext)
else compression fails or disabled
Compressor-->>Plugin: error / no-op
Plugin->>HTTP: create request(payload=uncompressed, default headers, URI ext="")
end
HTTP->>Azure: send PUT/POST (headers + payload)
Azure-->>HTTP: response
HTTP-->>Plugin: response
Note over Plugin,Azure: Plugin logs outcome and updates compression-applied flags
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
plugins/out_azure_blob/azure_blob_blockblob.c (1)
92-92: Minor:const char*assigned tochar*pointer.The function
azb_blob_extensionreturnsconst char*, butextis declared aschar*. While this works, it's technically a const-correctness issue.Consider updating the variable declaration for consistency:
- char *ext; + const char *ext;tests/internal/azure_blob.c (1)
128-143: Clarify test name to match behavior.The test
test_block_blob_extension_gzip_defaultsetscompression = FLB_COMPRESSION_ALGORITHM_NONEbut expects.gz. This correctly tests the fallback behavior inazb_blob_extension()where gzip is the default whencompress_blobis enabled without a specific algorithm. Consider adding a brief comment to clarify this intent.static void test_block_blob_extension_gzip_default() { struct flb_azure_blob ctx; flb_sds_t uri; azure_blob_ctx_init(&ctx); ctx.compress_blob = FLB_TRUE; - ctx.compression = FLB_COMPRESSION_ALGORITHM_NONE; + ctx.compression = FLB_COMPRESSION_ALGORITHM_NONE; /* defaults to gzip when compress_blob is enabled */ uri = azb_block_blob_uri(&ctx, "file", "block", 123, "rand");
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
plugins/out_azure_blob/azure_blob.c(6 hunks)plugins/out_azure_blob/azure_blob.h(2 hunks)plugins/out_azure_blob/azure_blob_blockblob.c(4 hunks)plugins/out_azure_blob/azure_blob_conf.c(2 hunks)plugins/out_azure_blob/azure_blob_http.c(3 hunks)plugins/out_azure_blob/azure_blob_http.h(1 hunks)tests/internal/CMakeLists.txt(1 hunks)tests/internal/azure_blob.c(1 hunks)
🧰 Additional context used
🧠 Learnings (11)
📓 Common learnings
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
plugins/out_azure_blob/azure_blob.hplugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob_conf.ctests/internal/azure_blob.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
plugins/out_azure_blob/azure_blob.hplugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
plugins/out_azure_blob/azure_blob.hplugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
plugins/out_azure_blob/azure_blob.hplugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
plugins/out_azure_blob/azure_blob.hplugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
plugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
plugins/out_azure_blob/azure_blob.c
🧬 Code graph analysis (4)
plugins/out_azure_blob/azure_blob_http.c (1)
src/flb_http_client.c (1)
flb_http_add_header(963-995)
plugins/out_azure_blob/azure_blob_conf.c (1)
src/flb_output.c (1)
flb_output_get_property(1108-1111)
tests/internal/azure_blob.c (2)
plugins/out_azure_blob/azure_blob_blockblob.c (1)
azb_block_blob_uri(72-121)plugins/out_azure_blob/azure_blob_http.c (1)
azb_http_client_setup(299-382)
plugins/out_azure_blob/azure_blob.c (2)
src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_zstd.c (1)
flb_zstd_compress(33-57)
🔇 Additional comments (18)
plugins/out_azure_blob/azure_blob_http.h (1)
31-35: LGTM!The signature change to add
content_encodingparameter is well-designed, cleanly separating content type (payload format) from content encoding (transfer encoding). This aligns with the broader compression refactoring in this PR.tests/internal/CMakeLists.txt (1)
21-21: LGTM!The new
azure_blob.ctest file is correctly added to the unit test list. Based on learnings, ZSTD is always available in Fluent Bit as a core dependency, so no conditional compilation guards are needed.plugins/out_azure_blob/azure_blob_blockblob.c (1)
35-46: LGTM!The
azb_blob_extensionhelper cleanly centralizes extension selection logic. The fallback to.gzwhen compression is enabled but not ZSTD maintains backward compatibility with existing gzip behavior.plugins/out_azure_blob/azure_blob.h (2)
33-39: LGTM!The new
AZURE_BLOB_CT_ZSTDandAZURE_BLOB_CE_ZSTDmacros follow the existing naming conventions and integer value pattern, cleanly extending the compression support.
59-59: LGTM!The field rename from
compress_gziptocompressionis a good generalization that supports the multi-algorithm compression model introduced in this PR.plugins/out_azure_blob/azure_blob_http.c (3)
196-198: LGTM!The ZSTD content encoding branch correctly maps
AZURE_BLOB_CE_ZSTDto the "zstd" encoding string, following the same pattern as the existing GZIP handling.
229-231: LGTM!The ZSTD content type mapping to "application/zstd" is correct and follows the existing pattern for GZIP.
324-339: LGTM!The HTTP header additions for ZSTD content type and content encoding are correctly implemented, mirroring the existing GZIP handling pattern.
plugins/out_azure_blob/azure_blob_conf.c (1)
659-673: LGTM!The compression configuration parsing is well-implemented:
- Uses
FLB_COMPRESSION_ALGORITHM_NONEas the sensible default- Case-insensitive matching improves user experience
- Clear error message listing valid options helps with troubleshooting
- Returning NULL on invalid values prevents running with misconfiguration
tests/internal/azure_blob.c (3)
42-74: LGTM!The fixture initialization/cleanup helpers properly handle resource management with appropriate error paths and cleanup on partial failures.
163-212: LGTM!Good test coverage for ZSTD content encoding header verification. The test correctly validates that
Content-Encoding: zstdis set when usingAZURE_BLOB_CE_ZSTD.
258-265: LGTM!The test list is properly structured with the null terminator. Good coverage of both extension behavior and HTTP header behavior for the new ZSTD support.
plugins/out_azure_blob/azure_blob.c (6)
27-28: LGTM!The new includes for
flb_compression.handflb_zstd.hare correctly added. Based on learnings, ZSTD is a core dependency in Fluent Bit and doesn't require conditional compilation guards.
154-178: LGTM!The helper functions
azure_blob_compress_payloadandazure_blob_compression_nameare well-implemented:
- Clean dispatch to the appropriate compression function
- Safe fallback to "unknown" for invalid algorithms in the name function
- Consistent return values (-1 for failure, 0 for success)
393-428: LGTM!The compression logic is well-structured:
- Properly defaults to GZIP when
compress_blobis enabled without a specific algorithm- Gracefully handles compression failures by falling back to uncompressed with a warning
- Correctly tracks which type of compression was applied (network vs blob)
1853-1861: LGTM!The config map descriptions are well-updated to document the new ZSTD support alongside GZIP, with clear explanations of behavior and compatibility constraints.
432-450: <!-- [scratchpad_start]
[task verify_review_comment_azure_blob_headers]
[observations]
- Web search results show practical Azure Blob Storage examples and patterns
- Key finding from search results [1], [4], [7]: when uploading JSON compressed with gzip:
- Content-Type: application/json + Content-Encoding: gzip (transparent transfer encoding)
- This tells clients the original content is JSON and it's currently gzip encoded
- Pattern from [6]: Content-Type: text/css + Content-Encoding: gzip for CSS files
- Azure semantics confirmed across all sources
- Code logic being reviewed:
- blob_compression_applied → Content-Type only (gzip/zstd)
- network_compression_applied → Content-Encoding (gzip/zstd) + Content-Type: json
[analysis]- CRITICAL ISSUE IDENTIFIED: The code logic appears INCORRECT
- When blob_compression_applied is true, setting Content-Type to application/gzip without Content-Encoding is correct (blob is stored as a gzip file)
- When network_compression_applied is true, setting Content-Encoding + Content-Type: json is correct
- BUT: The code distinguishes these as mutually exclusive paths, not considering the semantic meaning
- The ISSUE: When blob_compression_applied=true, the code sets content_type to AZURE_BLOB_CT_GZIP (application/gzip presumably)
- This means "the blob IS a gzip file" (not auto-decompressed)
- content_encoding = AZURE_BLOB_CE_NONE confirms this (no encoding)
- This is CORRECT for stored blob compression
- When network_compression_applied=true, the code sets:
- Content-Encoding to gzip/zstd
- Content-Type to application/json
- This is CORRECT for transparent transfer encoding
- BUT the review comment question remains: what if you want to store a JSON file and serve it with transfer compression? The code handles this with network_compression_applied, which is correct.
- The code logic actually IS correct according to Azure semantics
[pending]- Verify the actual constant values and header setting logic
- Check if there are any issues in edge cases
[actions]- Search codebase to verify constant definitions and actual header setting
[done]
[scratchpad_start] -->Based on the web search results provided and the code logic shown in the review comment, I can now verify the HTTP header semantics.
When uploading compressed JSON, the pattern shown uses Content-Type: "application/csv" and Content-Encoding: "gzip", which aligns with the network compression logic. For gzip-compressed JSON blobs, setting Content-Type: "application/json" with Content-Encoding: "gzip" is the correct approach.
For example, CSS files should use Content-Type: "text/css" with Content-Encoding: "gzip", which shows the pattern: the original Content-Type paired with the transfer encoding.
The code's HTTP header semantics are correct. The distinction is properly implemented:
- Blob compression (stored format): Content-Type indicates the file format (gzip/zstd), no Content-Encoding
- Network compression (transfer encoding): Content-Encoding indicates the encoding, Content-Type describes the uncompressed content
No issues were found in the verification.
141-143: Verify null-termination behavior when compression fails.The null-termination is skipped when
ctx->compression != NONE, but compression may fail later inhttp_send_blob. If compression fails and the payload is sent uncompressed, the buffer will lack null-termination.This is likely safe because the HTTP client uses the explicit
body_sizerather than relying on null-termination, but it's worth verifying that all downstream consumers handle this correctly.
|
@coderabbitai Fixes for both nitpicks landed in the latest commit a2855d2:
|
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 2392 🏁 Script executed: Length of output: 1040 @nberlee Perfect! I've verified both fixes in commit a2855d2:
Both nitpicks have been properly addressed. Thank you for the quick fixes! 🧠 Learnings used |
cosmo0920
left a comment
cosmo0920
left a comment
There was a problem hiding this comment.
The main logic of out_azure_blob itself is pretty good.
But I found a violation of testing layers.
Related to each of plugins' specific functions, we need to put their test cases into runtime directory of test cases.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tests/runtime/out_azure_blob_compression.c (1)
91-98: Consider adding NULL checks forflb_sds_createreturn values.While unlikely to fail in test environments, the
flb_sds_createcalls could theoretically return NULL on allocation failure. Consider adding error handling for robustness.static void azure_blob_ctx_init(struct flb_azure_blob *ctx) { memset(ctx, 0, sizeof(*ctx)); ctx->base_uri = flb_sds_create("/"); + if (ctx->base_uri == NULL) { + TEST_MSG("flb_sds_create failed for base_uri"); + return; + } ctx->container_name = flb_sds_create("container"); + if (ctx->container_name == NULL) { + TEST_MSG("flb_sds_create failed for container_name"); + flb_sds_destroy(ctx->base_uri); + ctx->base_uri = NULL; + return; + } ctx->btype = AZURE_BLOB_BLOCKBLOB; ctx->atype = AZURE_BLOB_AUTH_KEY; }plugins/out_azure_blob/azure_blob.c (1)
399-456: Approve the compression logic with a minor suggestion.The compression flow correctly handles algorithm selection, execution with graceful fallback, and HTTP header configuration. The logic appropriately compresses the payload once when both network and blob compression are enabled, then prioritizes blob compression for header setting.
Consider adding a comment at line 438 explaining the prioritization:
/* set http header flags */ +/* Note: If both blob and network compression are applied, blob compression + * takes precedence as we cannot set both Content-Type (compressed file) and + * Content-Encoding (transfer compression) simultaneously. */ if (blob_compression_applied == FLB_TRUE) {This would help future maintainers understand why
else ifis used instead of independent conditions.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
plugins/out_azure_blob/azure_blob.c(6 hunks)plugins/out_azure_blob/azure_blob.h(2 hunks)plugins/out_azure_blob/azure_blob_blockblob.c(5 hunks)plugins/out_azure_blob/azure_blob_conf.c(2 hunks)plugins/out_azure_blob/azure_blob_http.c(3 hunks)plugins/out_azure_blob/azure_blob_http.h(1 hunks)tests/runtime/CMakeLists.txt(1 hunks)tests/runtime/out_azure_blob_compression.c(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- plugins/out_azure_blob/azure_blob.h
- plugins/out_azure_blob/azure_blob_http.c
🧰 Additional context used
🧠 Learnings (12)
📓 Common learnings
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob.cplugins/out_azure_blob/azure_blob_conf.ctests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob.cplugins/out_azure_blob/azure_blob_conf.ctests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob.cplugins/out_azure_blob/azure_blob_conf.ctests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob.cplugins/out_azure_blob/azure_blob_conf.ctests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob.cplugins/out_azure_blob/azure_blob_conf.ctests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob.cplugins/out_azure_blob/azure_blob_conf.c
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-12-08T05:21:45.014Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11250
File: src/ripser/flb_ripser_wrapper.cpp:73-106
Timestamp: 2025-12-08T05:21:45.014Z
Learning: In the TDA processor (processor_tda) for Fluent Bit, the dimension limit FLB_RIPSER_MAX_BETTI_DIM is intentionally capped at 3 because the plugin uses embed_dim=3 and delay=1 in practice, and computing higher dimensions (>4) would be computationally prohibitive. The 8-slot betti array allocation is conservative headroom.
Applied to files:
plugins/out_azure_blob/azure_blob_blockblob.c
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
plugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
plugins/out_azure_blob/azure_blob.c
🧬 Code graph analysis (3)
plugins/out_azure_blob/azure_blob.c (2)
src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_zstd.c (1)
flb_zstd_compress(33-57)
plugins/out_azure_blob/azure_blob_conf.c (1)
src/flb_output.c (1)
flb_output_get_property(1108-1111)
tests/runtime/out_azure_blob_compression.c (6)
src/flb_config.c (2)
flb_config_init(233-486)flb_config_exit(488-672)src/flb_upstream.c (1)
flb_upstream_create(290-363)include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)src/flb_sds.c (2)
flb_sds_create(78-90)flb_sds_destroy(389-399)plugins/out_azure_blob/azure_blob_blockblob.c (1)
azb_block_blob_uri(72-121)plugins/out_azure_blob/azure_blob_http.c (1)
azb_http_client_setup(299-382)
🔇 Additional comments (19)
tests/runtime/CMakeLists.txt (1)
207-207: LGTM!The new runtime test entry follows the established pattern and is correctly placed in alphabetical order within the output plugins section.
plugins/out_azure_blob/azure_blob_blockblob.c (3)
26-26: LGTM!The
flb_compression.hinclude is needed forFLB_COMPRESSION_ALGORITHM_ZSTDconstant. Based on learnings, ZSTD is a core dependency in Fluent Bit and doesn't require conditional compilation guards.
35-46: LGTM!The helper function cleanly centralizes the extension logic with proper static linkage. The fallback to
.gzwhencompress_blobis enabled but no specific algorithm is set preserves backward compatibility.
77-77: LGTM!The
const char *type correctly matches the return type ofazb_blob_extension().plugins/out_azure_blob/azure_blob_http.h (1)
34-35: LGTM!The function signature extension to include
content_encodingaligns with the HTTP header handling for compression (gzip/zstd) shown in the implementation.plugins/out_azure_blob/azure_blob_conf.c (2)
23-23: LGTM!The
flb_compression.hinclude is required for theFLB_COMPRESSION_ALGORITHM_*enum constants used in the compression configuration parsing.
659-673: LGTM!The compression property parsing follows the established pattern used for
auth_typeandblob_typein this file, with proper case-insensitive comparison and clear error messaging.tests/runtime/out_azure_blob_compression.c (7)
42-89: LGTM!The fixture initialization and cleanup properly manages the config, upstream, and connection lifecycle with appropriate error handling and cleanup on failure paths.
111-126: LGTM!The test correctly verifies that the
.zstextension is appended to the blob URI when zstd compression is enabled.
128-146: LGTM!The test correctly verifies the legacy gzip fallback behavior when
compress_blobis enabled but no explicit compression algorithm is configured. The inline comment provides helpful clarification.
148-164: LGTM!The test correctly verifies that
compress_blob = FLB_FALSEtakes precedence over the compression algorithm setting, resulting in no extension being added.
166-215: LGTM!The test properly validates that
AZURE_BLOB_CE_ZSTDresults in the correctContent-Encoding: zstdheader whileAZURE_BLOB_CT_JSONsetsContent-Type: application/json. Resource cleanup is handled correctly on all paths.
217-259: LGTM!The test correctly verifies that
AZURE_BLOB_CT_ZSTDsetsContent-Type: application/zstdheader.
261-268: LGTM!All test cases are properly registered in the TEST_LIST with the correct format.
plugins/out_azure_blob/azure_blob.c (5)
27-28: LGTM!The new includes for compression and zstd support are correctly added without conditional compilation guards.
Based on learnings, ZSTD is a core dependency in Fluent Bit and doesn't require
#ifdefguards.
141-143: LGTM!The updated compression check correctly uses the new enum-based
compressionfield. Null termination is appropriately applied only when compression is disabled, which makes sense for JSON text payloads.
159-171: LGTM!The compression dispatcher correctly delegates to the appropriate compression implementation based on the algorithm. The logic is clean and the fallback behavior (returning -1 for unknown algorithms) is appropriate.
174-184: LGTM!The helper function provides clear, human-readable names for compression algorithms used in logging. The fallback to "unknown" is appropriate.
1857-1868: LGTM!The configuration documentation clearly describes the new zstd compression option and explains the relationship between
compressandcompress_blobsettings. The incompatibility warning for appendblob is helpful.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tests/runtime/out_azure_blob_compression.c (1)
91-108: Initialization failure not detectable by caller.When
container_nameallocation fails (lines 100-105), the function cleans upbase_uribut returns without any failure indicator. Callers (test functions) proceed without knowing initialization failed, which could lead to null pointer dereferences.Consider returning an error code or using a sentinel value pattern:
-static void azure_blob_ctx_init(struct flb_azure_blob *ctx) +static int azure_blob_ctx_init(struct flb_azure_blob *ctx) { memset(ctx, 0, sizeof(*ctx)); ctx->base_uri = flb_sds_create("/"); if (!TEST_CHECK(ctx->base_uri != NULL)) { TEST_MSG("flb_sds_create failed for base_uri"); - return; + return -1; } ctx->container_name = flb_sds_create("container"); if (!TEST_CHECK(ctx->container_name != NULL)) { TEST_MSG("flb_sds_create failed for container_name"); flb_sds_destroy(ctx->base_uri); ctx->base_uri = NULL; - return; + return -1; } ctx->btype = AZURE_BLOB_BLOCKBLOB; ctx->atype = AZURE_BLOB_AUTH_KEY; + return 0; }Then update callers to check the return value.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
plugins/out_azure_blob/azure_blob.c(6 hunks)plugins/out_azure_blob/azure_blob.h(2 hunks)plugins/out_azure_blob/azure_blob_blockblob.c(5 hunks)plugins/out_azure_blob/azure_blob_conf.c(2 hunks)plugins/out_azure_blob/azure_blob_http.c(3 hunks)plugins/out_azure_blob/azure_blob_http.h(1 hunks)tests/runtime/CMakeLists.txt(1 hunks)tests/runtime/out_azure_blob_compression.c(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- plugins/out_azure_blob/azure_blob_blockblob.c
- plugins/out_azure_blob/azure_blob_http.h
🧰 Additional context used
🧠 Learnings (11)
📓 Common learnings
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
plugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob.hplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob.ctests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
plugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob.hplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
plugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob.hplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
plugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob.hplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
plugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob.hplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob.ctests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
plugins/out_azure_blob/azure_blob_conf.cplugins/out_azure_blob/azure_blob_http.cplugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
plugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
plugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
plugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
plugins/out_azure_blob/azure_blob.c
🧬 Code graph analysis (3)
plugins/out_azure_blob/azure_blob_conf.c (1)
src/flb_output.c (1)
flb_output_get_property(1108-1111)
plugins/out_azure_blob/azure_blob_http.c (1)
src/flb_http_client.c (1)
flb_http_add_header(963-995)
plugins/out_azure_blob/azure_blob.c (2)
src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_zstd.c (1)
flb_zstd_compress(33-57)
🔇 Additional comments (21)
tests/runtime/CMakeLists.txt (1)
207-207: LGTM!The test registration follows the existing pattern for output plugin tests and is correctly guarded by
FLB_OUT_AZURE_BLOB.plugins/out_azure_blob/azure_blob_http.c (4)
196-198: LGTM!Correctly maps
AZURE_BLOB_CE_ZSTDto the "zstd" encoding string, consistent with the existing gzip pattern.
229-231: LGTM!The
application/zstdcontent type is correctly assigned for zstd-compressed payloads.
324-328: LGTM!Header construction for zstd Content-Type follows the established pattern and uses correct string length (16).
335-339: LGTM!Content-Encoding header for zstd correctly uses "zstd" with proper string length (4).
plugins/out_azure_blob/azure_blob.h (3)
33-33: LGTM!
AZURE_BLOB_CT_ZSTDconstant follows the existing numbering scheme and correctly representsapplication/zstd.
39-39: LGTM!
AZURE_BLOB_CE_ZSTDconstant is correctly defined for the zstd content-encoding.
59-59: LGTM!Renaming from
compress_gziptocompressiongeneralizes the field to support multiple compression algorithms. Theinttype is appropriate for storingFLB_COMPRESSION_ALGORITHMenum values.tests/runtime/out_azure_blob_compression.c (6)
121-136: LGTM!Test correctly validates that
.zstextension is appended to the blob URI when zstd compression is configured.
138-156: LGTM!Test validates the legacy gzip fallback behavior when
compress_blobis enabled but no explicit algorithm is set. The inline comment (lines 147-148) clarifies this intentional backward-compatible behavior, as discussed in the PR.
158-174: LGTM!Test confirms that no compression extension is added when
compress_blobis disabled, regardless of the compression algorithm setting.
176-225: LGTM!Comprehensive test that verifies
Content-Encoding: zstdheader is correctly set while maintaining the expectedContent-Type: application/json. Proper fixture setup/teardown and null checks throughout.
227-269: LGTM!Test validates that
Content-Type: application/zstdis correctly set whenAZURE_BLOB_CT_ZSTDis specified with no content encoding.
271-278: LGTM!Test list properly registers all five test cases with correct naming and null terminator.
plugins/out_azure_blob/azure_blob_conf.c (2)
23-23: LGTM!Correctly includes the compression header for
FLB_COMPRESSION_ALGORITHMenum definitions. Based on learnings, ZSTD is a core dependency in Fluent Bit and doesn't require conditional compilation guards.
659-673: LGTM!The compression configuration parsing is well-implemented:
- Defaults to
FLB_COMPRESSION_ALGORITHM_NONEfor backward compatibility- Supports both "gzip" and "zstd" with case-insensitive matching
- Provides a clear error message listing supported values on invalid input
plugins/out_azure_blob/azure_blob.c (5)
27-28: LGTM!The new includes for compression and zstd support are appropriate. Based on learnings, ZSTD is bundled in Fluent Bit and doesn't require conditional compilation guards.
141-143: LGTM!The generalization from
ctx->compress_gziptoctx->compression == FLB_COMPRESSION_ALGORITHM_NONEcorrectly makes the null-terminator logic algorithm-agnostic.
154-184: LGTM!The helper functions
azure_blob_compress_payloadandazure_blob_compression_nameare well-structured. They correctly dispatch to the appropriate compression backend and handle unknown algorithms gracefully by returning -1 and "unknown" respectively, allowing callers to fall back to uncompressed payloads.
437-459: LGTM with dependency on flag fix.The header-setting logic correctly handles precedence between blob and network compression, with blob compression taking priority. The comment clearly explains the rationale. However, this logic depends on the flag-setting issue (lines 417-422) being fixed to ensure the flags accurately reflect which compression was applied.
1862-1870: LGTM!The configuration documentation clearly explains the new compression options:
compressfor network/transport compressioncompress_blobfor final blob file compression (defaults to gzip, uses thecompresscodec when set)The documentation accurately reflects the implementation and includes the important note about incompatibility with appendblob.
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Signed-off-by: Nico Berlee <[email protected]>
Signed-off-by: Nico Berlee <[email protected]>
|
@codex review |
|
Codex Review: Didn't find any major issues. Breezy! ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
2 participants
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
debug.log
valgrind.log
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
fluent/fluent-bit-docs#2309
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
Bug Fixes / Reliability
Documentation
Tests
✏️ Tip: You can customize this high-level summary in your review settings.