feat(#115): chunked values — setValueChunked/getValueChunked/deleteValueChunked

CHANGELOG.md

@@ -3,6 +3,27 @@
 ## [Unreleased]
 
 ### Added
+- [#115] Chunked values: transparent handling of single values above the
+  100,000 B per-value limit, split into ordered chunks under a dedicated
+  `\x00`-namespaced key space. `Transaction::setValueChunked(string|KeyConvertible
+  $key, string $value, int $chunkSize = 100000)` (atomic single-transaction
+  write with stale-chunk clearing, capped at 8,000,000 B with the new
+  `CrazyGoat\FoundationDB\ChunkedValueTooLargeException` above the cap),
+  `Database::setValueChunked(..., bool $atomic = true)` (atomic mode via
+  `transact()`, or `atomic: false` — generation scheme: budget-sized chunk
+  groups across multiple transactions plus an atomic metadata swap, no size
+  cap, self-cleaning orphaned chunks), `ReadTransaction::getValueChunked()`
+  (one range read per active generation; `null` for a missing key, `""` for
+  an empty value, the new `CrazyGoat\FoundationDB\ChunkedValueCorruptedException`
+  on malformed metadata or mismatched chunk data; snapshot reads supported)
+  and `Transaction::deleteValueChunked()` / `Database::deleteValueChunked()`
+  (one range clear removes meta, all chunks and all generations). Key-space
+  layout owned by the `@internal Chunk\ChunkKeyCodec` (magic-signed metadata
+  record `FDBCK1` + total length + chunk count + generation). Unit tests
+  (`tests/Unit/ChunkKeyCodecTest.php`) verify the storage-format invariants;
+  integration tests in `tests/Integration/ChunkedValuesTest.php`; documented
+  in `docs/chunked-values.md`.
+
 - [#116] Batch write helpers with up-front mutation-budget accounting:
   `Transaction::setBatch(iterable $pairs)` (queues `[key, value]` pairs —
   `string` or `KeyConvertible` keys — into the current transaction; the

docs/chunked-values.md

@@ -0,0 +1,102 @@
+# Chunked values
+
+FoundationDB enforces a hard limit of **100,000 bytes per value**. Chunked
+values lift that ceiling for a *single logical value* by splitting it into
+ordered chunks under a dedicated, `\x00`-namespaced sub-key space derived
+from the base key. The layout is owned by the `@internal
+Chunk\ChunkKeyCodec`:
+
+- metadata key: `$key . "\x00" . "\x00"` — sorts before every chunk
+- chunk keys:   `$key . "\x00" . Tuple::pack([$generation, $index])`
+- namespace:    `[$key . "\x00", $key . "\x01")`
+
+The `\x00` separator sorts before every tuple type byte, so the namespace
+can never collide with user text suffixes (`$key . "abc"` stays outside) and
+is unreachable through the public `Tuple` / `Subspace` / `Directory` API.
+The metadata record carries a magic signature (`FDBCK1`), the total length,
+the chunk count and the active generation — so foreign or tampered data is
+detected loudly instead of silently mis-read.
+
+## Writing
+
+```php
+// Atomic mode (default): everything in one transaction, capped at 8 MB
+$db->setValueChunked('cache/page/home', $html);
+
+// Non-atomic mode: no size cap, written across multiple transactions
+$db->setValueChunked('cache/page/huge', $enormousHtml, atomic: false);
+
+// Custom chunk size (1..100,000 bytes)
+$db->setValueChunked('cache/blobs', $data, 32768);
+
+// Transaction level (atomic only — it queues into the current transaction)
+$db->transact(function ($tr): void {
+    $tr->setValueChunked('cache/page/home', $html);
+});
+```
+
+### Atomic mode (default)
+
+The namespace clear, all chunk writes and the metadata record happen inside
+one transaction, so readers see either the whole old or the whole new value
+— never a mixture, never a stale tail from a previous larger value. Because
+all mutations share the 10,000,000 B per-transaction budget, the assembled
+value is capped at 8,000,000 B (`MutationBudget::SPLIT_TARGET_BYTES`);
+`ChunkedValueTooLargeException` (public readonly `valueSize` / `maxSize`) is
+thrown at the call site, before any mutation is queued.
+
+### Non-atomic mode (`atomic: false`)
+
+Removes the size cap by splitting the write across multiple transactions
+using **generations**:
+
+1. chunks of the new value are written under `generation + 1` in
+   budget-sized groups, each group committed in its own retried transaction;
+2. a final micro-transaction atomically swaps the metadata record to the new
+   generation and clears everything below it (the previous generation plus
+   any orphaned chunks from interrupted attempts).
+
+Readers always see either the whole old or the whole new value, because
+`getValueChunked()` selects chunks by the generation stored in the metadata.
+A crash between the chunk transactions and the metadata swap leaves only
+orphaned chunks — reads are unaffected, and the next non-atomic write to the
+key cleans them up. Two concurrent non-atomic writers to the same key
+conflict on their chunk ranges, so one retries.
+
+## Reading
+
+```php
+$value = $db->transact(fn ($tr) => $tr->getValueChunked('cache/page/home'));
+```
+
+- key holds no chunked value → `null`
+- key holds an empty chunked value → `""` (not `null`)
+- metadata malformed or chunks missing / wrong total length →
+  `ChunkedValueCorruptedException` (loud failure, never garbage)
+
+Available on `Transaction`, `Tenant` transactions and snapshots
+(`$tr->snapshot()->getValueChunked(...)` — note that snapshot reads create
+no read-conflict ranges).
+
+## Deleting
+
+```php
+$db->deleteValueChunked('cache/page/home');   // or $tr->deleteValueChunked(...)
+```
+
+One range clear removes the metadata record, all chunks and all generations
+at once. Deleting a key that holds no chunked value is a no-op.
+
+## Semantics and caveats
+
+- **Mixing plain `set()` and chunked values on the same base key is
+  unsupported.** The chunk namespace is separate, so the two representations
+  can silently diverge; `getValueChunked()` reads only the namespace, and a
+  raw `set()` under the base key is left as orphaned bytes.
+- `watch()` and atomic operations do not observe chunked values (they
+  operate on single keys, not on the namespace).
+- TTL / expiration is deliberately out of scope for this layer — cache
+  adapters implement it above chunked values.
+- `setBatch()` (#116) writes **many small values**; chunked values handle
+  **one large value**. The two share the `@internal MutationBudget` byte
+  accounting only.

src/Chunk/ChunkKeyCodec.php

@@ -0,0 +1,164 @@
+<?php
+
+declare(strict_types=1);
+
+namespace CrazyGoat\FoundationDB\Chunk;
+
+use CrazyGoat\FoundationDB\Tuple\Tuple;
+
+/**
+ * Key-space codec for chunked values (issue #115).
+ *
+ * A chunked value stored under a base key `$key` lives in a dedicated
+ * `\x00`-namespaced sub-key space:
+ *
+ * - metadata key:  `$key . "\x00" . "\x00"` (sorts before every chunk key)
+ * - chunk keys:    `$key . "\x00" . Tuple::pack([$generation, $index])`
+ * - whole space:   `[$key . "\x00", $key . "\x01")`
+ *
+ * The leading `\x00` byte sorts before every tuple type byte (null is
+ * `0x00`; ints start at `0x0C`, strings and all other types later), so the
+ * namespace can never collide with user text suffixes (e.g. `$key . "abc"`)
+ * and the metadata key always sorts before all chunks. The metadata key and
+ * the packed tuple suffixes are unreachable through the library's public
+ * Tuple / Subspace / Directory API — mixing plain `set()` and
+ * `setValueChunked()` on the same base key is documented as unsupported.
+ *
+ * Metadata record layout (22 bytes):
+ *
+ * ```
+ * "FDBCK1" (6 B magic) | total_length (8 B big-endian)
+ *                      | chunk_count (4 B) | generation (4 B)
+ * ```
+ *
+ * The magic header makes foreign or tampered data detectable: a read
+ * mismatching the magic raises `ChunkedValueCorruptedException` instead of
+ * silently returning garbage. The `generation` field is the active-generation
+ * pointer used by the non-atomic (multi-transaction) write mode: new chunks
+ * are written under `generation + 1` across several transactions and the
+ * final micro-transaction atomically swaps the metadata — readers see either
+ * the whole old or the whole new value.
+ *
+ * This codec is `@internal`: it is private to the chunked layer. Batch
+ * helpers (#116) share `MutationBudget` byte accounting only and must never
+ * touch the `\x00` namespace.
+ *
+ * @internal
+ */
+final class ChunkKeyCodec
+{
+    /** Namespace separator: sorts before every tuple type byte. */
+    private const SEPARATOR = "\x00";
+
+    /** Magic signature stored as the first 6 bytes of the metadata record. */
+    private const META_MAGIC = 'FDBCK1';
+
+    /** Total metadata record length: magic(6) + length(8) + count(4) + generation(4). */
+    public const META_LENGTH = 22;
+
+    /** First byte of the chunk key space (exclusive upper bound of the namespace). */
+    public const NAMESPACE_END = "\x01";
+
+    private function __construct()
+    {
+    }
+
+    public static function metaKey(string $baseKey): string
+    {
+        return $baseKey . self::SEPARATOR . self::SEPARATOR;
+    }
+
+    public static function chunkKey(string $baseKey, int $generation, int $index): string
+    {
+        return $baseKey . self::SEPARATOR . Tuple::pack([$generation, $index]);
+    }
+
+    /**
+     * First key of the whole chunk namespace for a base key (inclusive lower
+     * bound for range operations).
+     */
+    public static function namespaceBegin(string $baseKey): string
+    {
+        return $baseKey . self::SEPARATOR;
+    }
+
+    /**
+     * One-past-the-end of the whole chunk namespace (exclusive upper bound).
+     */
+    public static function namespaceEnd(string $baseKey): string
+    {
+        return $baseKey . self::NAMESPACE_END;
+    }
+
+    /**
+     * Inclusive lower bound of the range holding generation `$generation`'s
+     * chunks — the first byte boundary whose keys all start with the packed
+     * `[generation]` prefix.
+     */
+    public static function generationBegin(string $baseKey, int $generation): string
+    {
+        return $baseKey . self::SEPARATOR . Tuple::pack([$generation]);
+    }
+
+    /**
+     * Exclusive upper bound of generation `$generation`'s chunk range.
+     */
+    public static function generationEnd(string $baseKey, int $generation): string
+    {
+        return $baseKey . self::SEPARATOR . Tuple::pack([$generation + 1]);
+    }
+
+    /**
+     * Exclusive upper bound for clearing every generation strictly below
+     * `$generation` (including the metadata key): chunks of the active
+     * generation start exactly at this boundary, so they are never removed.
+     */
+    public static function clearBelowGenerationEnd(string $baseKey, int $generation): string
+    {
+        return $baseKey . self::SEPARATOR . Tuple::pack([$generation, 0]);
+    }
+
+    /**
+     * @return string the 22-byte metadata record
+     */
+    public static function packMeta(int $totalLength, int $chunkCount, int $generation): string
+    {
+        return pack('a6JNN', self::META_MAGIC, $totalLength, $chunkCount, $generation);
+    }
+
+    /**
+     * Parse a metadata record into `['length' => int, 'count' => int, 'generation' => int]`.
+     *
+     * @return array{length: int, count: int, generation: int}
+     *
+     * @throws \CrazyGoat\FoundationDB\ChunkedValueCorruptedException on a
+     *         malformed record (wrong length or magic)
+     */
+    public static function unpackMeta(string $meta): array
+    {
+        if (strlen($meta) !== self::META_LENGTH) {
+            throw new \CrazyGoat\FoundationDB\ChunkedValueCorruptedException(
+                sprintf(
+                    'Chunked-value metadata record has unexpected length %d (expected %d)',
+                    strlen($meta),
+                    self::META_LENGTH,
+                ),
+            );
+        }
+
+        $unpacked = unpack('a6magic/Jlength/Ncount/Ngeneration', $meta);
+
+        if ($unpacked === false || $unpacked['magic'] !== self::META_MAGIC) {
+            throw new \CrazyGoat\FoundationDB\ChunkedValueCorruptedException(
+                'Chunked-value metadata record has an unknown magic signature — the key was '
+                . 'probably written or overwritten by something other than setValueChunked()',
+            );
+        }
+
+        return [
+            'length' => $unpacked['length'],
+            'count' => $unpacked['count'],
+            'generation' => $unpacked['generation'],
+        ];
+    }
+}

src/ChunkedValueCorruptedException.php

@@ -0,0 +1,20 @@
+<?php
+
+declare(strict_types=1);
+
+namespace CrazyGoat\FoundationDB;
+
+/**
+ * Thrown by `getValueChunked()` when the chunk namespace of a key exists but
+ * cannot be interpreted as a chunked value: the metadata record is missing its
+ * magic signature or has an unexpected length, or the stored chunk count /
+ * total length do not match what the chunk range actually contains.
+ *
+ * This signals data written or overwritten by something other than
+ * `setValueChunked()` (or real corruption) — it is deliberately a loud
+ * exception, not a silent `null`/short read. A *missing* key is not an error:
+ * `getValueChunked()` returns `null` in that case.
+ */
+final class ChunkedValueCorruptedException extends \RuntimeException
+{
+}

src/ChunkedValueTooLargeException.php

@@ -0,0 +1,36 @@
+<?php
+
+declare(strict_types=1);
+
+namespace CrazyGoat\FoundationDB;
+
+/**
+ * Thrown when a value exceeds the per-transaction cap of the atomic chunked
+ * write mode (`Transaction::setValueChunked()` / `Database::setValueChunked()
+ * with atomic: true`).
+ *
+ * The atomic mode must write all chunks plus the metadata record inside a
+ * single transaction, so the assembled value is capped (with a safety margin
+ * below FoundationDB's 10,000,000 B mutation budget, mirroring
+ * `MutationBudget::SPLIT_TARGET_BYTES`). The exception is raised synchronously
+ * at the call site, before any mutation is queued.
+ *
+ * Values above the cap can still be written with
+ * `Database::setValueChunked(..., atomic: false)`, which splits the write
+ * across multiple transactions (per-chunk atomicity, atomic metadata swap).
+ */
+final class ChunkedValueTooLargeException extends \RuntimeException
+{
+    public function __construct(
+        public readonly int $valueSize,
+        public readonly int $maxSize,
+    ) {
+        parent::__construct(sprintf(
+            'Chunked value exceeds the single-transaction cap: %d bytes (limit is %d bytes). '
+            . 'Use Database::setValueChunked(..., atomic: false) to split the write across '
+            . 'multiple transactions.',
+            $valueSize,
+            $maxSize,
+        ));
+    }
+}