feat(#93): readBlobGranules (FDBReadBlobGranuleContext) and summarizeBlobGranules

CHANGELOG.md

@@ -24,6 +24,27 @@
   (`fdb_transaction_read_blob_granules` + `FDBReadBlobGranuleContext`) and
   the summarize/parse-file family remain unbound (follow-ups on #93).
 
+- [#93] Blob granule API, part 2: granule reads and summaries. Transaction
+  level (`ReadTransaction`): `readBlobGranules()` (backed by
+  `fdb_transaction_read_blob_granules` — a synchronous `FDBResult` call,
+  read via the newly bound `fdb_result_get_keyvalue_array` /
+  `fdb_result_destroy`) with file data fetched through the new
+  `CrazyGoat\FoundationDB\BlobGranuleLoader` interface wired into the
+  `FDBReadBlobGranuleContext` callbacks by
+  `CrazyGoat\FoundationDB\BlobGranuleReadContext` (including the
+  `debugNoMaterialize` test mode and `granuleParallelism`), and
+  `summarizeBlobGranules()` (backed by
+  `fdb_transaction_summarize_blob_granules` + the newly bound
+  `fdb_future_get_granule_summary_array`, with the packed `FDBGranuleSummary`
+  struct declared) returning the new `BlobGranuleSummary` value object via
+  the new `Future\FutureGranuleSummaryArray` future. Integration tests
+  extended in `tests/Integration/BlobGranuleTest.php` (loader validation,
+  debug read request, summary shape; full materialization skips on clusters
+  without a granule blob store); `docs/blob-granules.md` updated. The
+  parse-file family (`fdb_readbg_parse_*`,
+  `fdb_future_readbg_get_descriptions`) remains unbound as a tool-only
+  follow-up on #93.
+
 ### Security
 - [#49] The FoundationDB client library can now be loaded from a pinned
   absolute path via the `FDB_LIBRARY_PATH` environment variable, instead of

docs/blob-granules.md

@@ -14,7 +14,8 @@ FDB's storage tier, and how the storage tier can be relieved of cold data.
 > requires blob-manager workers to be recruited (typically backed by a real
 > blob store). On small test clusters no blob workers are recruited and the
 > blocking call never completes — prefer the non-blocking `blobbifyRange()`
-> in such setups.
+> in such setups. The same cluster capability is what gates
+> `readBlobGranules()` and `summarizeBlobGranules()` — see below.
 
 ## API overview
 
@@ -45,11 +46,47 @@ the `fdb_tenant_*` symbols. Keys are relative to the tenant's prefix.
 | Method | Backed by | Description |
 |--------|-----------|-------------|
 | `getBlobGranuleRanges($begin, $end, $rangeLimit = 0)` | `fdb_transaction_get_blob_granule_ranges` | List the granule boundaries within the range; returns a future of `list<KeyRange>`. |
+| `readBlobGranules($begin, $end, $beginVersion, $loader = null, $readVersion = null, $debugNoMaterialize = false, $granuleParallelism = 1)` | `fdb_transaction_read_blob_granules` | Materialize the granules covering the range; returns `list<KeyValue>` (synchronous `FDBResult` call, not a future). |
+| `summarizeBlobGranules($begin, $end, $summaryVersion = null, $rangeLimit = 100)` | `fdb_transaction_summarize_blob_granules` | Summarize the granules in the range; returns a future of `list<BlobGranuleSummary>`. `rangeLimit` must be >= 1 (the client library asserts `chunkLimit > 0`). |
 
-Direct granule *reads* (`fdb_transaction_read_blob_granules` and the
-`FDBReadBlobGranuleContext` load/free callback machinery) are not yet bound —
-see [issue #93](https://github.com/s2x/fdb-php/issues/93) for the remaining
-scope.
+#### `readBlobGranules()` and the loader
+
+`fdb_transaction_read_blob_granules` fetches granule file data through
+caller-supplied callbacks (the `FDBReadBlobGranuleContext` struct). In PHP
+this is expressed as the `CrazyGoat\FoundationDB\BlobGranuleLoader`
+interface:
+
+```php
+interface BlobGranuleLoader
+{
+    public function startLoad(string $filename, int $offset, int $length, int $fullFileLength): int;
+    public function getLoad(int $loadId): string;
+    public function freeLoad(int $loadId): void;
+}
+```
+
+`startLoad()` begins a load and returns a unique id; `getLoad()` returns the
+bytes for that id (the data must stay valid until `freeLoad()` is called);
+`freeLoad()` releases it. The PHP callbacks are wired into the native struct
+by `CrazyGoat\FoundationDB\BlobGranuleReadContext`, which keeps references to
+the callbacks and to any in-flight data buffers.
+
+With `$debugNoMaterialize = true` the loader is never called and only the
+request to the blob workers is issued (useful for testing). A loader is
+required unless `$debugNoMaterialize` is true — otherwise an
+`InvalidArgumentException` is thrown.
+
+> **Note:** both `readBlobGranules()` and `summarizeBlobGranules()` need a
+> cluster with working blob-manager workers and a real granule blob store.
+> On a cluster without one the C API reports
+> `Operation is not supported` (2108) for reads and
+> "Read version is older than blob granule history supports" (1064) for
+> summaries; the integration tests skip accordingly.
+
+The parse-file family (`fdb_readbg_parse_snapshot_file`,
+`fdb_readbg_parse_delta_file`, `fdb_future_readbg_get_descriptions`) — needed
+only by tools reading granule files directly — remains unbound; see
+[issue #93](https://github.com/s2x/fdb-php/issues/93).
 
 ## KeyRange
 

phpstan.neon

@@ -23,6 +23,8 @@ parameters:
           path: src/Transaction.php
         - identifier: method.notFound
           path: src/ReadTransaction.php
+        - identifier: property.notFound
+          path: src/ReadTransaction.php
         - identifier: method.notFound
           path: src/RangeResult.php
         - identifier: method.notFound
@@ -35,3 +37,9 @@ parameters:
           path: src/Future/*.php
         - identifier: method.notFound
           path: src/FDBException.php
+        - identifier: method.staticCall
+          path: src/BlobGranuleReadContext.php
+        - identifier: property.notFound
+          path: src/BlobGranuleReadContext.php
+        - identifier: argument.type
+          path: src/BlobGranuleReadContext.php

src/BlobGranuleLoader.php

@@ -0,0 +1,39 @@
+<?php
+
+declare(strict_types=1);
+
+namespace CrazyGoat\FoundationDB;
+
+/**
+ * Callback interface used by `ReadTransaction::readBlobGranules()` to fetch
+ * blob granule file data. Mirrors the `FDBReadBlobGranuleContext` function
+ * pointers of `fdb_c.h`.
+ *
+ * `startLoad()` is called (potentially many times, up to
+ * `$granuleParallelism` outstanding loads) to begin reading a slice of a
+ * granule file; it must return a caller-chosen unique load id. `getLoad()`
+ * is then called with that id and must return the requested bytes — the
+ * returned buffer must remain valid until `freeLoad()` is called for the
+ * same id. `freeLoad()` releases the resources held for the id.
+ */
+interface BlobGranuleLoader
+{
+    /**
+     * Begin loading `length` bytes of `filename` starting at `offset`.
+     * `fullFileLength` is the total length of the file.
+     *
+     * @return int A unique load id, used for the subsequent getLoad()/freeLoad() calls.
+     */
+    public function startLoad(string $filename, int $offset, int $length, int $fullFileLength): int;
+
+    /**
+     * Return the bytes for the previously started load. The data must stay
+     * valid until freeLoad() is called with the same load id.
+     */
+    public function getLoad(int $loadId): string;
+
+    /**
+     * Release the resources held for the load. Called exactly once per load id.
+     */
+    public function freeLoad(int $loadId): void;
+}

src/BlobGranuleReadContext.php

@@ -0,0 +1,94 @@
+<?php
+
+declare(strict_types=1);
+
+namespace CrazyGoat\FoundationDB;
+
+use FFI;
+use FFI\CData;
+
+/**
+ * Builds the native `FDBReadBlobGranuleContext` for
+ * `fdb_transaction_read_blob_granules` out of a PHP `BlobGranuleLoader`.
+ *
+ * The PHP callback methods are cast to C function pointers; this object
+ * keeps references to them (and to any in-flight data buffers) so neither
+ * the callbacks nor the buffers are garbage collected while FDB may still
+ * call into them.
+ */
+final class BlobGranuleReadContext
+{
+    private readonly CData $context;
+
+    /** @var array<int, CData> In-flight buffers keyed by load id. */
+    private array $buffers = [];
+
+    public function __construct(
+        private readonly NativeClient $client,
+        private readonly BlobGranuleLoader $loader,
+        int $granuleParallelism = 1,
+    ) {
+        $context = $this->client->fdb->new('FDBReadBlobGranuleContext');
+        $context->user_context = null;
+        $context->start_load_f = $this->startLoad(...);
+        $context->get_load_f = $this->getLoad(...);
+        $context->free_load_f = $this->freeLoad(...);
+        $context->debug_no_materialize = 0;
+        $context->granule_parallelism = $granuleParallelism;
+        $this->context = $context;
+    }
+
+    /** C callback: fdb_c.h start_load_f. */
+    private function startLoad(
+        string $filename,
+        int $filenameLength,
+        int $offset,
+        int $length,
+        int $fullFileLength,
+        ?CData $userContext,
+    ): int {
+        return $this->loader->startLoad(
+            FFI::string($filename, $filenameLength),
+            $offset,
+            $length,
+            $fullFileLength,
+        );
+    }
+
+    /** C callback: fdb_c.h get_load_f. */
+    private function getLoad(int $loadId, ?CData $userContext): ?CData
+    {
+        $data = $this->loader->getLoad($loadId);
+        if ($data === '') {
+            return null;
+        }
+
+        $buffer = $this->client->fdb->new('uint8_t[' . strlen($data) . ']');
+        FFI::memcpy($buffer, $data, strlen($data));
+        $this->buffers[$loadId] = $buffer;
+
+        return FFI::cast('uint8_t*', $buffer);
+    }
+
+    /** C callback: fdb_c.h free_load_f. */
+    private function freeLoad(int $loadId, ?CData $userContext): void
+    {
+        $this->loader->freeLoad($loadId);
+        unset($this->buffers[$loadId]);
+    }
+
+    /**
+     * Disable materialization (only issue the request to the blob workers —
+     * useful for testing). Must be called before the context is passed to
+     * `readBlobGranules()`.
+     */
+    public function setDebugNoMaterialize(bool $debugNoMaterialize = true): void
+    {
+        $this->context->debug_no_materialize = $debugNoMaterialize ? 1 : 0;
+    }
+
+    public function toCData(): CData
+    {
+        return $this->context;
+    }
+}

src/BlobGranuleSummary.php

@@ -0,0 +1,22 @@
+<?php
+
+declare(strict_types=1);
+
+namespace CrazyGoat\FoundationDB;
+
+/**
+ * A single granule summary as returned by
+ * `ReadTransaction::summarizeBlobGranules()` (backed by
+ * `fdb_future_get_granule_summary_array`).
+ */
+final readonly class BlobGranuleSummary
+{
+    public function __construct(
+        public KeyRange $keyRange,
+        public int $snapshotVersion,
+        public int $snapshotSize,
+        public int $deltaVersion,
+        public int $deltaSize,
+    ) {
+    }
+}

src/Future/FutureGranuleSummaryArray.php

@@ -0,0 +1,61 @@
+<?php
+
+declare(strict_types=1);
+
+namespace CrazyGoat\FoundationDB\Future;
+
+use CrazyGoat\FoundationDB\BlobGranuleSummary;
+use CrazyGoat\FoundationDB\KeyRange;
+use FFI;
+
+final class FutureGranuleSummaryArray extends Future
+{
+    /** @var list<BlobGranuleSummary> */
+    private array $cachedResult = [];
+
+    /**
+     * @return list<BlobGranuleSummary>
+     */
+    public function await(): array
+    {
+        if ($this->resolved) {
+            return $this->cachedResult;
+        }
+
+        $this->blockUntilReady();
+
+        $outSummaries = $this->client->fdb->new('FDBGranuleSummary*');
+        $outCount = $this->client->fdb->new('int');
+
+        $this->client->checkError(
+            $this->client->fdb->fdb_future_get_granule_summary_array(
+                $this->fpointer,
+                FFI::addr($outSummaries),
+                FFI::addr($outCount),
+            ),
+        );
+
+        $count = $outCount->cdata;
+        $summaries = [];
+
+        for ($i = 0; $i < $count; $i++) {
+            $summary = $outSummaries[$i];
+            $summaries[] = new BlobGranuleSummary(
+                new KeyRange(
+                    FFI::string($summary->begin_key, $summary->begin_key_length),
+                    FFI::string($summary->end_key, $summary->end_key_length),
+                ),
+                $summary->snapshot_version,
+                $summary->snapshot_size,
+                $summary->delta_version,
+                $summary->delta_size,
+            );
+        }
+
+        $this->cachedResult = $summaries;
+        $this->releaseMemory();
+        $this->resolved = true;
+
+        return $this->cachedResult;
+    }
+}