Unify lazy remote caching for Caterva2 and fsspec

doc/getting_started/installation.rst

@@ -42,10 +42,10 @@ grouped into *extras* that you opt into with the ``blosc2[extra]`` syntax:
        :doc:`../guides/parquet_to_blosc2`.
    * - ``fsspec``
      - Reading and writing single-file containers through any `fsspec
-       <https://filesystem-spec.readthedocs.io>`_ URL.  The driver for each
-       protocol is a separate install (``s3fs`` for ``s3://``, ``gcsfs`` for
-       ``gs://``, ``adlfs`` for ``abfs://``...), and credentials are configured
-       through the driver, not through blosc2.
+       <https://filesystem-spec.readthedocs.io>`_ URL.  The HTTP(S) driver is
+       included. Other protocol drivers are separate installs (``s3fs`` for
+       ``s3://``, ``gcsfs`` for ``gs://``, ``adlfs`` for ``abfs://``...), and
+       credentials are configured through the driver, not through blosc2.
 
 Install one or more extras by listing them in brackets (quote the
 argument in shells like ``zsh`` that treat brackets specially):
@@ -55,11 +55,12 @@ argument in shells like ``zsh`` that treat brackets specially):
     pip install "blosc2[tui]"             # the b2view terminal browser
     pip install "blosc2[hires]"           # b2view + its high-res view (h key)
     pip install "blosc2[parquet]"         # the Parquet converter
+    pip install "blosc2[fsspec]"          # fsspec URLs, including HTTP(S)
     pip install "blosc2[fsspec]" s3fs     # fsspec URLs, plus the S3 driver
     pip install "blosc2[tui,parquet]"     # several at once
 
 With the ``fsspec`` extra, :func:`blosc2.open` accepts any fsspec URL, chained
-ones included, and reads it whole, through a local cache (``cache_storage=``), or
+ones included, and reads it whole, through a local cache (``cache_dir=``), or
 by fetching only the chunks and blocks a slice touches (``lazy=True``); see
 :func:`blosc2.open` and :ref:`FsspecNDSource` for what each mode supports.
 ``examples/ndarray/rw-fsspec.py`` walks through all three plus the write side,

doc/reference/c2array.rst

@@ -3,16 +3,26 @@
 C2Array
 =======
 
-This is a class for remote arrays. This kind of array can also work as operand on a LazyExpr, LazyUDF or reduction.
+This is a class for one array-like dataset addressed through a Caterva2 server.
+The dataset may be a standalone ``.b2nd`` array, an HDF5 dataset, an NDArray
+leaf inside a ``.b2z`` store, or a lazy/computed array.  A ``C2Array`` does not
+represent or navigate a whole remote ``TreeStore`` or ``DictStore``; use
+Caterva2 to select a leaf and open that leaf's path.  This kind of array can also
+work as an operand on a LazyExpr, LazyUDF or reduction.  :ref:`URLPath` is
+Caterva2-only, including when its ``urlbase`` is omitted and inherited from
+:func:`blosc2.c2context`.
+
+For a comparison with byte-oriented fsspec access, see
+:doc:`Working with Remote Arrays <../guides/remote_arrays>`.
 
 Wrapped in a :ref:`Proxy`, a stored remote array is read at block granularity:
 the proxy asks for the blocks a slice touches rather than the chunks they live
 in, which for a multi-megabyte chunk is a small fraction of the bytes.  That
 rests on the server serving the dataset from a file, ``Range`` header and
 auth cookie both honoured; a dataset it computes instead (a lazy expression, an
-HDF5 leaf) is fetched a whole chunk at a time, as everything was before.  Which
-one this is takes at most one request to find out, and is decided once --
-:meth:`C2Array.block_source` is what answers it.
+HDF5 leaf, or a ``.b2z`` member) is fetched a whole chunk at a time, as
+everything was before.  Which one this is takes at most one request to find out,
+and is decided once -- :meth:`C2Array.block_source` is what answers it.
 
 A stored remote array can also be *filled*, by as many writers at once as it has
 chunks.  The array is laid out first -- ``blosc2.uninit`` writes a couple of

doc/reference/fsspecndsource.rst

@@ -7,6 +7,11 @@ A :ref:`ByteRangeNDSource` that serves the chunks of a Blosc2 frame living
 behind an fsspec URL, reading each one with a range request instead of
 transferring the whole container.  Everything about the frame format, block
 granularity included, lives in the base class; this adds the fsspec transport.
+The URL must name a standalone, contiguous ``.b2nd`` NDArray frame.  It cannot
+name an HDF5 dataset, a member inside a ``.b2z`` store, a sparse directory
+container, or a computed array: fsspec provides bytes, not dataset semantics.
+For the Caterva2 alternative and a capability comparison, see
+:doc:`Working with Remote Arrays <../guides/remote_arrays>`.
 For other sources, see :ref:`ProxyNDSource` and :ref:`ProxySource`.
 
 ``examples/ndarray/rw-fsspec.py`` is a runnable walkthrough of this and the

doc/tutorials/06.remote_proxy.ipynb

@@ -38,7 +38,7 @@
    "metadata": {},
    "source": [
     "## ``C2Array`` class\n",
-    "Before we look at proxies, it is first necessary to understand how to use Blosc2 to work with remote data, via the ``C2Array`` class. The class implements a (limited) version of the NDArray interface of which we have already seen a lot in previous tutorials. However, it is really a local pointer to a remote array (stored e.g. on a remote server). This means that we can refer to the data, access certain attribute information about it, download portions of the data and even define it in computational expressions, without having to download the entire array into local memory or disk. This is particularly useful when working with large datasets that cannot fit into memory or would take far too long to transfer over the network.\n",
+    "Before we look at proxies, it is first necessary to understand how to use Blosc2 to work with remote data, via the ``C2Array`` class. The class implements a (limited) version of the NDArray interface of which we have already seen a lot in previous tutorials. However, it is really a local pointer to a remote array (stored e.g. on a remote server). This means that we can refer to the data, access certain attribute information about it, download portions of the data and even define it in computational expressions, without having to download the entire array into local memory or disk. This is particularly useful when working with large datasets that cannot fit into memory or would take far too long to transfer over the network. The [Working with Remote Arrays](https://www.blosc.org/python-blosc2/guides/remote_arrays.html#choosing-between-fsspec-and-caterva2) guide explains when to use Caterva2's semantic dataset route instead of a byte-oriented fsspec URL.\n",
     "\n",
     "However, one limitation of this approach is that every time one wants to download a slice of the dataset, the data is fetched over the network - even if the same slice has been downloaded before. This can lead to inefficiencies, especially when working with large datasets or when the same data is accessed multiple times. Proxies offer a solution to this, whilst still preserving the low storage requirements of the ``C2Array`` class.\n",
     "\n",

examples/c2array-traffic.py

@@ -11,7 +11,7 @@
 # the whole chunk: on a fast link the two take about as long, and differ by the
 # compression ratio in bytes.  Bytes are also what a metered link and a shared
 # server uplink actually run out of, so they are what `Traffic` counts -- at the
-# transport, so the frame index and block offsets are in the tally too.
+# transport, so metadata, the frame index, and block offsets are in the tally too.
 
 import blosc2
 
@@ -27,9 +27,8 @@ def cost(traffic):
 array = blosc2.C2Array(path, urlbase=urlbase)
 print(f"{path}: shape={array.shape} chunks={array.chunks} blocks={array.blocks}")
 
-# Opening a handle costs one `api/info` call, which is metadata rather than data
-# and is deliberately not counted -- no slice can avoid it, and no choice of
-# granularity changes it.
+# Opening a handle costs one `api/info` call, included so this is a complete
+# account of everything that crossed the wire.
 print(f"after opening:      {array.traffic}")
 
 # -- A proxy reads through the block path, so it pays for what a slice touches.

examples/fsspec-cat2-access.py

@@ -0,0 +1,114 @@
+#######################################################################
+# Copyright (c) 2019-present, Blosc Development Team
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#######################################################################
+
+"""Compare lazy access to the same array through fsspec and Caterva2.
+
+The HTTPS path needs the fsspec extra.  Install it with:
+
+    pip install "blosc2[fsspec]"
+
+By default, caches are kept under ``./fsspec-cat2-cache``. Run the example again to
+see the first data access served by the cache left by the previous process.
+"""
+
+import argparse
+from pathlib import Path
+from time import perf_counter
+
+import numpy as np
+
+import blosc2
+
+# Using the Caterva2 API
+CATERVA2_URL = blosc2.URLPath(
+    "@public/examples/cube-1k-1k-1k.b2nd",
+    urlbase="https://cat2.cloud/demo",
+)
+# ...and also using the fsspec path via fetch URL in Caterva2
+FSSPEC_URL = "https://cat2.cloud/demo/api/fetch/@public/examples/cube-1k-1k-1k.b2nd"
+# The same contents are published in this Backblaze B2 bucket with a ``-2`` suffix.
+# FSSPEC_URL = "https://f001.backblazeb2.com/file/blosc2/cube-1k-1k-1k-2.b2nd"
+
+SLICE = np.s_[100:110, 200:300, 400:500]
+
+
+def traffic_text(traffic: blosc2.Traffic | None) -> str:
+    if traffic is None:
+        return "traffic unavailable"
+    request_word = "request" if traffic.requests == 1 else "requests"
+    return f"{traffic.requests} {request_word}, {traffic.nbytes / 2**20:.3f} MiB"
+
+
+def size_text(size: int) -> str:
+    return f"{size / 2**20:.3f} MiB"
+
+
+def benchmark(label: str, urlpath, cache_dir: Path) -> np.ndarray:
+    start = perf_counter()
+    array = blosc2.open(urlpath, lazy=True, cache_dir=cache_dir)
+    open_time = perf_counter() - start
+    open_traffic = traffic_text(array.traffic)
+
+    metadata = (array.shape, array.dtype, array.chunks, array.blocks)
+    cache_path = Path(array.urlpath).resolve()
+    cache_status = array.cache_status
+
+    array.traffic.reset()
+    start = perf_counter()
+    data = array[SLICE]
+    first_read_time = perf_counter() - start
+    first_traffic = traffic_text(array.traffic)
+    cache_size = cache_path.stat().st_size
+
+    # Open a fresh remote handle over the same on-disk cache. This demonstrates
+    # that cached data survives the Proxy object, not merely one array access.
+    del array
+    start = perf_counter()
+    reopened = blosc2.open(urlpath, lazy=True, cache_dir=cache_dir)
+    reopen_time = perf_counter() - start
+    reopen_traffic = traffic_text(reopened.traffic)
+
+    reopened.traffic.reset()
+    start = perf_counter()
+    cached = reopened[SLICE]
+    cached_read_time = perf_counter() - start
+    cached_traffic = traffic_text(reopened.traffic)
+    np.testing.assert_array_equal(cached, data)
+
+    print(f"\n{label}")
+    print(f"  metadata: shape={metadata[0]}, dtype={metadata[1]}")
+    print(f"            chunks={metadata[2]}, blocks={metadata[3]}")
+    print(f"  persistent cache: {cache_path} ({cache_status})")
+    print(f"  {'open + remote metadata:':<27}{open_time * 1000:.0f} ms ({open_traffic})")
+    print(f"  {'first data slice:':<27}{first_read_time * 1000:.0f} ms ({first_traffic})")
+    print(f"  {'cache after slice:':<27}{size_text(cache_size)}")
+    print(f"  {'reopen + remote metadata:':<27}{reopen_time * 1000:.0f} ms ({reopen_traffic})")
+    print(f"  {'same slice after reopen:':<27}{cached_read_time * 1000:.0f} ms ({cached_traffic})")
+    return data
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--cache-dir",
+        type=Path,
+        default=Path("fsspec-cat2-cache"),
+        help="persistent cache root (default: ./fsspec-cat2-cache)",
+    )
+    args = parser.parse_args()
+    root = args.cache_dir
+
+    print(f"Persistent cache root: {root.resolve()}")
+    print("Run this command again to reuse these cache files.")
+    cat2_data = benchmark("Caterva2", CATERVA2_URL, root / "caterva2")
+    fsspec_data = benchmark("fsspec over HTTPS", FSSPEC_URL, root / "fsspec")
+    np.testing.assert_array_equal(cat2_data, fsspec_data)
+    print("\nBoth services returned identical data.")
+
+
+if __name__ == "__main__":
+    main()

examples/ndarray/rw-fsspec.py

@@ -43,15 +43,15 @@
     # starts from the copy that is already there.  Cached copies are checked
     # against the remote on every open, so a replaced array is never served
     # from a stale cache.
-    c = blosc2.open(urlpath, cache_storage=cachedir, mmap_mode="r")
+    c = blosc2.open(urlpath, cache_dir=cachedir, mmap_mode="r")
     print(f"read cached: {c.shape} (mmapped from {cachedir})")
     np.testing.assert_array_equal(c[:], a[:])
 
     # Read lazily.  Nothing is transferred up front: the array stays where it
     # is and each slice fetches only what it touches -- the chunks it lands in,
     # or just the blocks inside them when the chunks are large enough for that
     # to pay.  This is what you want for an array too big to download.
-    d = blosc2.open(urlpath, lazy=True, cache_storage=cachedir)
+    d = blosc2.open(urlpath, lazy=True, cache_dir=cachedir)
     print(f"read lazy: {type(d).__name__} {d.shape} {d.dtype}")
 
     # Only the two chunks covering rows 15..25 are fetched here

pyproject.toml

@@ -59,10 +59,11 @@ tui = ["textual", "textual-plotext"]
 # Adds the high-res 'h' view on top of [tui], rendering a real matplotlib image
 # (kitty/iTerm2/sixel, or half-cells elsewhere) — matplotlib is the heavy part.
 hires = ["blosc2[tui]", "textual-image", "matplotlib"]
-# Read/write single-file containers through any fsspec URL (s3://, gs://, zip://,
-# memory://...).  The protocol backends (s3fs, gcsfs, adlfs...) are the caller's
-# install: `pip install "blosc2[fsspec]" s3fs`.
-fsspec = ["fsspec"]
+# Read/write single-file containers through any fsspec URL (https://, s3://,
+# gs://, zip://, memory://...).  HTTP support is included; the other protocol
+# backends (s3fs, gcsfs, adlfs...) are the caller's install:
+# `pip install "blosc2[fsspec]" s3fs`.
+fsspec = ["fsspec[http]"]
 
 [project.scripts]
 parquet-to-blosc2 = "blosc2.cli.parquet_to_blosc2:main"

src/blosc2/c2array.py

@@ -264,9 +264,11 @@ def login(username, password, urlbase):
     return "=".join(list(resp.cookies.items())[0])
 
 
-def info(path, urlbase, params=None, headers=None, model=None, auth_token=None):
+def info(path, urlbase, params=None, headers=None, model=None, auth_token=None, traffic=None):
     url = _server_url(urlbase, f"api/info/{path}")
     response = _xget(url, params, headers, auth_token)
+    if traffic is not None:
+        traffic.charge(len(response.content))
     json = response.json()
     return json if model is None else model(**json)
 
@@ -771,10 +773,9 @@ def __init__(self, path: str, /, urlbase: str | None = None, auth_token: str | N
         """Bytes and requests this handle has read off the server; see :ref:`Traffic`.
 
         Cumulative since the array was opened, counted at the transport, so the
-        frame index and the block offsets are in it as well as the data, and the
-        `api/info` call that opened this handle is not.  Whichever endpoint the
-        read used is in it too, and the block source built later is handed this
-        same tally, so one counter answers for the array however it is read.
+        opening `api/info` response, frame index, block offsets, and data are all
+        included. Whichever endpoint serves a read uses this same tally, so one
+        counter answers for the array however it is read.
 
         What a slice cost is the difference between two readings, or one reading
         after :meth:`Traffic.reset`.  `examples/c2array-traffic.py` is a runnable
@@ -784,7 +785,12 @@ def __init__(self, path: str, /, urlbase: str | None = None, auth_token: str | N
 
         # Try to 'open' the remote path
         try:
-            self.meta = info(self.path, self.urlbase, auth_token=self.auth_token)
+            self.meta = info(
+                self.path,
+                self.urlbase,
+                auth_token=self.auth_token,
+                traffic=self.traffic,
+            )
         except _httpx().HTTPStatusError as err:
             # HTTPStatusError only (not the broader HTTPError, which also covers
             # connection-level failures): a 404 means "not found", a connection
@@ -1163,7 +1169,12 @@ def _reread_meta(self) -> None:
         """
         with self._meta_lock:
             seen = self._meta_epoch
-        meta = info(self.path, self.urlbase, auth_token=self.auth_token)
+        meta = info(
+            self.path,
+            self.urlbase,
+            auth_token=self.auth_token,
+            traffic=self.traffic,
+        )
         with self._meta_lock:
             if self._meta_epoch != seen:
                 return
@@ -1670,6 +1681,11 @@ def __init__(self, path: str, /, urlbase: str | None = None, auth_token: str | N
         Create an instance of a remote data file (aka :ref:`C2Array <C2Array>`) urlpath.
         This is meant to be used in the :func:`blosc2.open` function.
 
+        Passing this object to :func:`blosc2.open` returns a :ref:`C2Array`. With
+        ``lazy=True`` it instead returns a :ref:`Proxy`, using an in-memory cache
+        by default or a persistent cache when ``cache_dir`` is provided.
+        Authenticated users sharing a machine must use separate cache directories.
+
         The parameters are the same as for the :meth:`C2Array.__init__`.
 
         """