Skip to content

Latest commit

 

History

History
359 lines (273 loc) · 13.8 KB

usage.md

File metadata and controls

359 lines (273 loc) · 13.8 KB

Usage

Core Operations

Every Magazine<T> instance exposes these operations:

Method Description
load(T data) Enqueue data into the magazine.
fire() Dequeue and return the next item.
reload(T data) Re-enqueue data (decrements fire counter instead of incrementing load counter).
delete(MagazineData<T>) Remove a specific record from the backend.
deleteAll(Collection<MagazineData<T>>) Remove a batch of records in one round trip.
getMetaData() Retrieve per-shard counters and pointers.
getShards() This magazine's persisted shard count.
peek(Map<Integer, Set<Long>>) Read specific shard/pointer records without consuming.
getMagazineIdentifier() The identifier this magazine is bound to.

Creating a Magazine

IAerospikeClient client = new AerospikeClient("localhost", 3000);

AerospikeStorageConfig config = AerospikeStorageConfig.builder()
        .namespace("test")
        .dataSetName("mag_data")
        .metaSetName("mag_meta")
        .shards(8)
        .recordTtl(30 * 24 * 60 * 60)
        .metaDataTtl(2 * 30 * 24 * 60 * 60)
        .build();

AerospikeStorage<String> storage = AerospikeStorage.<String>builder()
        .aerospikeClient(client)
        .storageConfig(config)
        .enableDeDupe(true)
        .farmId("dc1")
        .clazz(String.class)
        .clientId("my-service")
        .scope(MagazineScope.LOCAL)
        .build();

Magazine<String> magazine = Magazine.<String>builder()
        .baseMagazineStorage(storage)
        .magazineIdentifier("orders")
        .build();

Loading Data

Enqueue data into the magazine. Each successful call advances the load pointer and, after storing the data, increments the load counter for the selected shard.

boolean success = magazine.load("order-12345");

When de-duplication is enabled, the write is issued as a create-only operation. If the data already exists, the call returns true without writing a duplicate.

!!! info "De-duplication support" De-duplication is supported only for String, Long, and Integer data types.

Firing (Consuming) Data

Dequeue the next item. The library picks an active shard and claims a slot with a guarded atomic increment of that shard's fire pointer.

!!! danger "fire() is at-most-once" The claim is applied server-side before the payload reaches you. If the client times out after the server applied the claim, the pointer has already advanced and that record will never be delivered again. Callers that cannot tolerate loss must persist their own idempotency record before acting on the result. See Delivery Semantics.

MagazineData<String> fired = magazine.fire();

String value  = fired.getData();            // the payload
long pointer  = fired.getFirePointer();     // position in shard
Integer shard = fired.getShard();           // shard index
String magId  = fired.getMagazineIdentifier();

!!! info "Two distinct failure modes" fire() throws a MagazineException with one of two error codes, and treating them alike is a bug:

- `NOTHING_TO_FIRE` — the magazine is drained. There is nothing to consume; back off and retry later.
- `RETRIES_EXHAUSTED` — the library gave up while skipping consecutive pointer holes (slots whose pointer was allocated but whose data write failed), bounded by `maxFireHoleSkips`. This is *not* an empty queue: data may still exist. Alerting on it, rather than silently treating it as "empty", is the correct response.

The scan is bounded — there is no retry loop over contention, because the claim is a guarded atomic increment and can never be lost. Exhausted shards are suppressed in the process-local active-shard cache, so loads from another instance can take up to `activeShardRefreshSeconds` (default 5) to become visible.

Reloading Data

Re-enqueue data that was previously fired. This increments the load pointer but decrements the fire counter instead of incrementing the load counter — preserving accurate total-load counts.

boolean reloaded = magazine.reload("order-12345");

Deleting Data

Remove a specific record from the backend. Requires the MagazineData object returned by fire() or peek().

MagazineData<String> fired = magazine.fire();
// Process the data...
magazine.delete(fired);

!!! warning delete() throws a MagazineException with INVALID_CONFIGURATION if the supplied MagazineData belongs to a different magazine than the one you call it on. Do not route records returned by one magazine into another's delete().

`deleteAll()` applies the same rule to every record in the batch, and validates the whole batch
before deleting any of it - so a batch carrying one foreign record is rejected without
retiring the records it did own.

Peeking Data

Read records from specific shards and pointers without modifying any counters or pointers. Useful for inspection, debugging, or replay scenarios.

Map<Integer, Set<Long>> shardPointers = Map.of(
        0, Set.of(1L, 2L, 3L),
        1, Set.of(1L)
);

Set<MagazineData<String>> results = magazine.peek(shardPointers);
results.forEach(md ->
    System.out.printf("Shard %d, Pointer %d → %s%n",
        md.getShard(), md.getFirePointer(), md.getData()));

Getting Metadata

Retrieve per-shard counters and pointers to understand magazine state.

Map<String, MetaData> meta = magazine.getMetaData();

meta.forEach((shard, m) -> {
    System.out.printf("%-20s load=%d/%d  fire=%d/%d%n",
            shard,
            m.getLoadCounter(), m.getLoadPointer(),
            m.getFireCounter(), m.getFirePointer());
});

Example output:

SHARD_0              load=5/5  fire=2/2
SHARD_1              load=3/3  fire=1/1
SHARD_2              load=0/0  fire=0/0
...

Delivery-Time History

Answers "where had each shard's fire pointer reached at time T?" — which is how a caller can tell an abandoned in-flight record from one that was only just claimed, without stamping a timestamp on every message.

Off by default. The checkpoint map rides the shard's pointer record and every claim rewrites that record, so a caller that never asks should not pay for it.

Once on, checkpoints are recorded automatically while the magazine is consumed: at most one per shard per window, none for a shard whose pointer has not moved, and one batch call per magazine. See Fire history for how.

AerospikeStorageConfig.builder()
        .dataSetName("my_data")
        .metaSetName("my_meta")
        .namespace("test")
        .shards(8)
        .fireHistoryEnabled(true)     // default false
        .fireHistoryWindowSeconds(60) // resolution of every answer
        .fireHistoryEntries(32)       // checkpoints retained per shard
        .build();

Asking

Map<String, FireCheckpoint> watermarks =
        magazine.firePointerBefore(Instant.now().minus(Duration.ofMinutes(20)));

watermarks.forEach((shard, checkpoint) ->
        System.out.printf("%s: everything at or below pointer %d was claimed 20+ minutes ago%n",
                shard, checkpoint.firePointer()));

Every slot at or below the returned pointer was claimed at or before that instant. Slots above it might have been claimed a moment ago.

A checkpoint is labelled with the end of the window it was observed in, so it answers questions about instants from one window later onwards. Precision is therefore one to two windows.

!!! note "An absent shard means 'do nothing', not 'empty'" A shard is missing from the result when no checkpoint reaches back that far yet. That is normal for a young magazine. Treat it as "nothing is provably old enough here this round" — never as "this shard has nothing in flight".

Sizing, and the failure you must not swallow

Setting Meaning
fireHistoryWindowSeconds Seconds covered by one checkpoint, and therefore the resolution of every answer. At most one entry per shard per window, whatever the throughput, and none while the shard's pointer does not move. A wider window means fewer writes
fireHistoryEntries Checkpoints retained per shard. Eviction is by count, not age, so an intermittently used magazine reaches further back than window x entries rather than having its history wiped by the first write after an idle period. The map rides the pointer record every claim rewrites, so raise this only as far as you need

Asking about an instant further back than the retained history throws INVALID_REQUEST, and asking at all with the feature disabled throws NOT_ENABLED.

This is deliberate and must not be caught and ignored. Returning an empty map would be indistinguishable from "nothing is old enough", and a caller using this to recover abandoned messages would silently stop recovering them. Size window x entries to comfortably exceed the oldest age you intend to ask about.

try {
    magazine.firePointerBefore(cutoff);
} catch (MagazineException e) {
    if (e.getErrorCode() == ErrorCode.INVALID_REQUEST) {
        // History no longer spans the cutoff. Skip this round loudly - do not treat as "nothing to do".
    }
}

Inspecting the whole history

Map<String, List<FireCheckpoint>> history = magazine.fireHistory(); // newest first per shard

Managing Multiple Magazines

MagazineManager acts as a registry for multiple Magazine instances, potentially of different types.

MagazineManager manager = new MagazineManager("my-service");

Magazine<String> ordersMag = Magazine.<String>builder()
        .baseMagazineStorage(stringStorage)
        .magazineIdentifier("orders")
        .build();

Magazine<Long> idsMag = Magazine.<Long>builder()
        .baseMagazineStorage(longStorage)
        .magazineIdentifier("id-pool")
        .build();

// Register all magazines
manager.replaceAll(List.of(ordersMag, idsMag));

// Retrieve by identifier
Magazine<String> orders = manager.getMagazine("orders");
orders.load("new-order-456");

Magazine<Long> ids = manager.getMagazine("id-pool");
MagazineData<Long> id = ids.fire();

!!! warning "replaceAll evicts" replaceAll() atomically replaces the internal magazine map, so every magazine absent from the list is unregistered. Calling it with a single magazine drops all the others, and the next getMagazine() for one of them throws MAGAZINE_NOT_FOUND.

To add one magazine without disturbing the rest, use `register(magazine)`. On a per-request path use `getOrRegister(identifier, supplier)`: constructing a `Magazine` resolves its persisted shard configuration over the network, and `getOrRegister` skips that entirely when the handle is already held.

If you want to own the miss path yourself, find() is the non-throwing lookup:

Magazine<String> orders = manager.<String>find("orders")
        .orElseGet(() -> {
            Magazine<String> built = Magazine.<String>builder()
                    .baseMagazineStorage(stringStorage)
                    .magazineIdentifier("orders")
                    .build();
            manager.register(built);
            return built;
        });

De-duplication

When enableDeDupe is set to true on the storage, each load() call writes the de-dup marker with a create-only write, so the server — not a client-side lock — resolves concurrent attempts. The data is written only if no marker already exists.

AerospikeStorage<String> storage = AerospikeStorage.<String>builder()
        .aerospikeClient(client)
        .storageConfig(config)
        .enableDeDupe(true)    // ← enable de-duplication
        .farmId("dc1")
        .clazz(String.class)
        .clientId("my-service")
        .scope(MagazineScope.LOCAL)
        .build();

magazine.load("unique-item");
magazine.load("unique-item"); // no-op — already exists

Scoping

Scope Reach Set Name Resolution Status
LOCAL Single data-centre {farmId}_{setName} ✅ Supported
GLOBAL Cross-DC {setName} (no prefix) ❌ Not yet implemented
// Local scope (recommended)
.scope(MagazineScope.LOCAL)

// Global scope — rejected with NOT_IMPLEMENTED while the storage is
// being constructed, so the application fails at boot.
.scope(MagazineScope.GLOBAL)

Complete Lifecycle Example

// ── Setup ──
IAerospikeClient client = new AerospikeClient("localhost", 3000);

AerospikeStorageConfig config = AerospikeStorageConfig.builder()
        .namespace("production")
        .dataSetName("notification_data")
        .metaSetName("notification_meta")
        .shards(32)
        .recordTtl(7 * 24 * 60 * 60)        // 7 days
        .metaDataTtl(30 * 24 * 60 * 60)     // 30 days
        .build();

AerospikeStorage<String> storage = AerospikeStorage.<String>builder()
        .aerospikeClient(client)
        .storageConfig(config)
        .enableDeDupe(true)
        .farmId("dc1")
        .clazz(String.class)
        .clientId("notification-service")
        .scope(MagazineScope.LOCAL)
        .build();

Magazine<String> magazine = Magazine.<String>builder()
        .baseMagazineStorage(storage)
        .magazineIdentifier("push-notifications")
        .build();

// ── Producer ──
magazine.load("user:1001:Welcome to our app!");
magazine.load("user:1002:Your order has shipped.");

// ── Consumer ──
MagazineData<String> notification = magazine.fire();
sendPushNotification(notification.getData());
magazine.delete(notification);  // clean up after processing

// ── Missed delivery? Reload ──
magazine.reload("user:1001:Welcome to our app!");

// ── Monitoring ──
Map<String, MetaData> meta = magazine.getMetaData();
long totalPending = meta.values().stream()
        .mapToLong(m -> m.getLoadPointer() - m.getFirePointer())
        .sum();
System.out.println("Pending notifications: " + totalPending);