Unify lazy remote caching for Caterva2 and fsspec

[FrancescAlted]

This PR gives Caterva2 URLPath objects and fsspec URLs a consistent lazy-open and caching interface. Remote arrays can now use an in-memory cache, a managed persistent cache directory, or an explicitly named persistent cache file, while fetching only the chunks or blocks required by each access.

It also makes persistent caches independently reopenable, improves remote traffic accounting and HTTP metadata discovery, documents the two remote access routes in detail, and reduces the runtime of several tests in the default suite.

Main changes

• Extend blosc2.open(URLPath(...), lazy=True) to return a Proxy over a C2Array, matching the existing lazy fsspec behavior.
• Support the same cache controls for lazy Caterva2 and fsspec arrays:
  • no cache argument: process-local in-memory cache
  • cache_dir=...: persistent cache with a derived filename
  • cache_path=...: persistent cache at an exact filename
• Deprecate cache_storage in favor of cache_dir, while retaining it as a warning-producing compatibility alias.
• Validate that cache_storage, cache_dir, and cache_path are mutually exclusive.
• Detect stale persistent caches from the remote source stamp and automatically rebuild them.
• Add Proxy.cache_status, reporting "created", "reused", or "invalidated/rebuilt" for managed persistent caches.
• Record the source kind and location in persistent proxy metadata so fsspec and Caterva2 sources can be reconstructed when a cache file is opened directly.
• Keep Caterva2 authentication out of portable cache metadata; credentials are resolved again from the active Caterva2 context.
• Avoid a redundant Caterva2 metadata refresh when the newly constructed C2Array has already supplied current metadata and its source stamp.
• Optimize lazy HTTP opening to obtain the frame metadata and HTTP validator in one range request. ETag is preferred, with Last-Modified and size as a fallback.
• Include opening metadata responses in C2Array and proxy traffic counters, making request and byte totals cover the full remote interaction.
• Install fsspec's HTTP dependencies through the fsspec optional extra.

Documentation and examples

• Rewrite the remote arrays guide around the distinction between fsspec byte-range access and Caterva2 dataset access.
• Document supported remote object types, cache lifetimes, exact cache paths, independent cache reopening, invalidation, authentication isolation, traffic measurement, block-level reads, and custom transports.
• Expand the C2Array, URLPath, and FsspecNDSource reference documentation.
• Add examples/fsspec-cat2-access.py, which compares persistent lazy access to the same dataset through Caterva2 and fsspec/HTTPS and reports cache status, traffic, timing, and cache size.
• Update existing installation, tutorial, and fsspec examples to use cache_dir.

[Copilot]

🔵 Needs a closer look

It introduces broad behavioral and API changes across core remote opening/caching paths (Caterva2 + fsspec) and merits careful human review of edge cases and compatibility.

Pull request overview

This PR unifies lazy remote opening and caching across Caterva2 URLPath datasets and fsspec URLs, providing consistent in-memory vs persistent cache controls, improving cache reopen/invalidation behavior, and expanding traffic accounting and documentation around remote access.

Changes:

• Extend blosc2.open(URLPath(...), lazy=True) to return a Proxy (matching lazy fsspec behavior), with unified cache_dir / cache_path controls and cache_storage deprecation.
• Improve persistent proxy reopenability by recording source kind/location in proxy metadata, add Proxy.cache_status, and expand remote traffic accounting (including open-time metadata).
• Update tests, docs, and examples to reflect the new API and to reduce runtime in the default test suite (including marking heavier tests as @pytest.mark.heavy).

File summaries

File	Description
tests/test_objectarray.py	Update Caterva2 info() test stub for new traffic parameter.
tests/test_locking.py	Reduce iteration counts to shorten cross-process locking tests.
tests/test_fsspec.py	Switch tests to cache_dir, add coverage for cache_path, cache status, exclusivity, and HTTP stamp behavior.
tests/test_fsspec_s3.py	Rename/cache-arg update for S3 fsspec caching test.
tests/test_ctable_cframe.py	Use CTable.extend() to append rows more efficiently in test.
tests/test_b2objects.py	Update Caterva2 info() test stub for new traffic parameter.
tests/ndarray/test_c2array_blocks.py	Add lazy URLPath proxy tests for memory/persistent caching, reopen, staleness, and option validation; adjust traffic assertions.
tests/ndarray/test_c2array_async.py	Extend fake response to support new traffic accounting via response.content.
tests/ctable/test_utf8.py	Mark a memory-scaling test as heavy.
tests/ctable/test_dictionary_column.py	Simplify/adjust dictionary-column comparison/indexing tests.
tests/ctable/test_ctable_indexing.py	Reduce incremental-summary dataset sizes and mark incremental-summary tests as heavy.
src/blosc2/schunk.py	Implement unified remote cache option parsing, lazy Caterva2 URLPath proxy opening, cache-path support, and persistent proxy source reconstruction.
src/blosc2/proxy.py	Add _refresh_source control, persist richer proxy-source metadata, and expose Proxy.cache_status.
src/blosc2/proxy_source.py	Optimize HTTP lazy-open to capture validators from the first range request; update cache-dir terminology.
src/blosc2/c2array.py	Add traffic charging for api/info and propagate traffic through meta refresh.
pyproject.toml	Install fsspec HTTP dependencies via fsspec[http] extra.
examples/ndarray/rw-fsspec.py	Update example usage from cache_storage to cache_dir.
examples/fsspec-cat2-access.py	New example comparing persistent lazy access via Caterva2 vs fsspec/HTTPS, including cache status and traffic.
examples/c2array-traffic.py	Update example commentary to reflect metadata now being included in traffic.
doc/tutorials/06.remote_proxy.ipynb	Add guidance link clarifying Caterva2 vs fsspec remote routes.
doc/reference/fsspecndsource.rst	Clarify what fsspec byte-range sources can/can’t represent; link to remote arrays guide.
doc/reference/c2array.rst	Expand Caterva2 dataset semantics and link to route comparison.
doc/guides/remote_arrays.md	Rewrite guide around route selection (fsspec vs Caterva2) and unified caching/reopen behavior.
doc/getting_started/installation.rst	Update fsspec extra description and examples to cache_dir and note HTTP(S) support inclusion.

Review details

• Files reviewed: 24/24 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.