Skip to content

Unify input file formats to Zarr and generalize dataloaders - #271

Draft
grey-nearing wants to merge 4 commits into
mainfrom
gsnearing-unify-input-file-formats
Draft

Unify input file formats to Zarr and generalize dataloaders#271
grey-nearing wants to merge 4 commits into
mainfrom
gsnearing-unify-input-file-formats

Conversation

@grey-nearing

@grey-nearing grey-nearing commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary of Changes

This PR standardizes input file storage and data loaders across the codebase onto the Zarr format, improves data loader flexibility, and provides seamless migration tooling:

  1. Unified Zarr Input Format & Generalized Dataloaders:
    • Standardized Caravan static attributes (attributes.zarr) and streamflow time series (streamflow.zarr) into consolidated, chunked Zarr stores.
    • Generalized the Multimet dataloader with robust path resolution—supporting single consolidated stores (timeseries.zarr), per-product subdirectories (<product>/timeseries.zarr), and case-insensitive store matching.
    • Removed hardcoded basin naming assumptions (such as basin.split('_')[0]), allowing arbitrary basin identifier naming conventions.
  2. Feature Scaler Migration (scaler.zarr):
    • Migrated feature normalization scalers to Zarr format (scaler.zarr) while retaining backward-compatible fallback to existing scaler.nc files.
  3. Caravan Dataset Conversion Utilities:
    • Introduced googlehydrology.datautils.convert (convert_caravan_attributes, convert_caravan_timeseries, convert_caravan_to_zarr) for converting legacy NetCDF and CSV directories into consolidated Zarr stores.
    • Exposed the conversion tool via the CLI entrypoint: run convert-caravan.
  4. Output Zarr Store Metadata Consolidation:
    • Updated BaseTester.evaluate() to automatically write consolidated metadata for {period}_results.zarr at the completion of inference.
  5. CMAL Sampling & Negative Sample Handling:
    • Extended sample_cmal_deterministic to accept scaler and apply analytical quantile zero-bounding through _handle_negative_values.

Verification & Testing

  • Numerical Parity: Validated identical inference results between legacy NetCDF inputs and converted Zarr stores on both the 5-basin tutorial model and the full 615-catchment FloodHub operational benchmark.
  • Test Suite: All 212 unit and end-to-end integration tests pass successfully (pytest test/).

grey-nearing added a commit that referenced this pull request Aug 12, 2026
@grey-nearing grey-nearing changed the title Gsnearing unify input file formats Unify input file formats to Zarr and generalize dataloaders Aug 12, 2026
@ashermetzger-rgb
ashermetzger-rgb force-pushed the gsnearing-unify-input-file-formats branch from 416c301 to 71affc2 Compare September 1, 2026 09:21
ashermetzger-rgb pushed a commit that referenced this pull request Sep 1, 2026
@ashermetzger-rgb
ashermetzger-rgb force-pushed the gsnearing-unify-input-file-formats branch from 71affc2 to e6160ef Compare September 3, 2026 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant