Skip to content

Add zarr backend for small-domain forcing extraction - #96

Draft
toakinrele wants to merge 1 commit into
CIROH-UA:mainfrom
toakinrele:zarr-backend-switch
Draft

Add zarr backend for small-domain forcing extraction#96
toakinrele wants to merge 1 commit into
CIROH-UA:mainfrom
toakinrele:zarr-backend-switch

Conversation

@toakinrele

@toakinrele toakinrele commented Jul 23, 2026

Copy link
Copy Markdown

Ref #94

What this does

  • select_forcing_backend() auto-switches netcdf/zarr based on domain size and timestep count (small domain, many timesteps → zarr)
  • zarr_data_extract() uses Dask for lazy, chunked reads from a zarr store
  • Weighting logic is shared between both backends, no duplicated code
  • build_combined_zarr_reference() combines individual kerchunk references into one dataset when given a list of URLs
    Note
    Pins zarr<3 — zarr≥3 breaks reading kerchunk reference filesystems (confirmed on 3.2.1). kerchunk needs zarr≥3 to import, so it's imported lazily inside build_combined_zarr_reference() rather than as a top-level dependency, to avoid the conflict.

Not in this PR

@JordanLaserGit JordanLaserGit added enhancement New feature or request P0 Priority level. P0: Critical, P1: High, P2: Medium, P3: Low speed Related to performance/efficiency/multi-processing/multi-threading. NRDS HBV Related to the HBV datastream deployment in AWS labels Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request NRDS HBV Related to the HBV datastream deployment in AWS P0 Priority level. P0: Critical, P1: High, P2: Medium, P3: Low speed Related to performance/efficiency/multi-processing/multi-threading.

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

3 participants