Add VPU metadata support

[pratikshac15]

This PR recreates the changes from #90 on a branch within the CIROH-UA repository so the authenticated GitHub Actions can run.

This includes the VPU metadata support changes and the latest review updates.

Replaces #90. Resolves #9.

[Copilot]

Pull request overview

Adds VPU-aware handling for multi-GeoPackage / multi-weight forcing runs, including standardized VPU identifiers and new VPU-delineated metadata outputs, while preserving existing global metadata behavior.

Changes:

• Add VPU ID inference/normalization to align naming across weights generation and forcing processing.
• Write new per-VPU metadata outputs (metadata_by_vpu.csv plus catchment stats with a vpu_id column).
• Adjust multiprocessing behavior in weights generation and harden weight-ingestion logging.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/forcingprocessor/weights_hf2ds.py	Adds VPU ID inference and uses a spawn multiprocessing context for ProcessPoolExecutor; updates naming logic used to build VPU/catchment mappings.
src/forcingprocessor/processor.py	Adds VPU ID normalization + config handling and writes VPU-level metadata outputs; fixes local file size accumulation and adjusts grid extraction logic.

Comments suppressed due to low confidence (1)

src/forcingprocessor/processor.py:307

• The new condition treats any dataset with south_north/west_east dims like retrospective-2-1 for timestamp parsing, but many NWM files (e.g. nwm.t00z.short_range.forcing...) use those dims while the filename is not YYYYMMDDHH.... This will raise a ValueError in datetime.strptime. Timestamp extraction should prefer the model_output_valid_time attribute when available, and only fall back to parsing the filename when that attribute is missing.

                if "retrospective-2-1" in nwm_file or ("south_north" in nwm_data.dims and "west_east" in nwm_data.dims):
                    data_allvars[var_dx, :, :] = np.flip(np.squeeze(nwm_data[jvar].isel(west_east=slice(x_min, x_max+1), south_north=slice(shp[1] - (y_max+1), shp[1] - y_min)).values),axis=0)
                    t = datetime.strftime(datetime.strptime(nwm_file.split('/')[-1].split('.')[0],'%Y%m%d%H'),'%Y-%m-%d %H:%M:%S')
                else:
                    data_allvars[var_dx, :, :] = np.flip(np.squeeze(nwm_data[jvar].isel(x=slice(x_min, x_max+1), y=slice(shp[1] - (y_max+1), shp[1] - y_min)).values),axis=0)
                    time_splt = nwm_data.attrs["model_output_valid_time"].split("_")
                    t = time_splt[0] + " " + time_splt[1]

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pratikshac15]

Addressed both suggestions.

Replaced the assert with a ValueError for validating the forcing.vpu_id and forcing.gpkg_file lengths so the check is always enforced.
Updated the multiprocessing logic to avoid issues with concurrent processing and preserve the expected behavior.

[quinnylee]

Crazy week here at AWI! I will finish my review on Monday.

[quinnylee]

Thanks for the PR! I have a couple design comments below. Also, I ran pytests locally and noticed that tests/test_trouterestarts.py::test_restart failed with AssertionError: assert '2024-01-15_13:00:00' == '2024-01-15_12:00:00'

Would probably also be useful to include tests for this new feature

[quinnylee]

This method's only function is to call another method. It would make more sense to delete this method and rename all the calls to infer_vpu_id in processor.py to normalize_vpu_id (or vice versa)

[quinnylee]

What case would lead to i >= len(netcdf_cat_file_sizes_MB)? Not sure what information is in netcdf_cat_file_sizes_MB that wouldn't be in netcdf_size_by_vpu or vice versa.

[quinnylee]

What case would lead to i < len(jcatchment_values)? I am not sure why there is a distinction between vpu_catchment_map and jcatchment_dict

[quinnylee]

This seems to be the same function that is in processor.py. Maybe it would make more sense to move it to utils.py and import it from there, so the code only has to live in one place?