v0.14.4

Changelog

Improvements

• Added GPCP dataset version to 3.3 for the PMP precipitation climatology reference dataset. (#672)
• Updated GPCP dataset version to 3.3 for the PMP precipitation climatology reference dataset. (#684)
• Removed the pandas upper bound and fixed pandas 3 compatibility in catalog and ILAMB result handling. (#708)

Bug Fixes

• Updates the PMP provider data directory (#685)
• Fixed a KeyError: 'branded_variable' when solving diagnostics against CMIP7 datasets.
  The branded_variable facet is now reconstructed when a data catalog is loaded from the database,
  so it is available to data requirement filters. (#712)

Improved Documentation

• Added a Zenodo DOI badge to the README and documentation landing page so users can easily find how to cite the REF. (#634)
• Added a getting-started guide for modelling centres that explains how to evaluate local or pre-publication CMOR-compliant model output with Climate-REF, including deployment options, accessing results, and a suggested first run. (#709)

Trivial/Internal Changes

• #706

Changes

• d26bdce Bump version: 0.14.3 → 0.14.4
• dfee0e3 refactor: declare derived catalog columns on the adapter
• b191910 docs: add changelog fragment for PR 712
• 9d43fee fix: derive branded_variable when loading CMIP7 catalog from DB
• baa358b docs: add Zenodo DOI badge to README and docs landing page
• 673b6fa docs: add changelog fragment for modelling centre guide
• 5206586 docs: add modelling centre onboarding guide
• cf96a14 test: Add tests for the helper
• 42cea87 chore: rename changelog fragment to PR 708
• 182f088 chore: remove pandas upper bound and fix pandas 3 compatibility
• 3b9973c docs(changelog): add fragment for series regression check
• 4995b17 test(core): add series.json regression comparison
• ef7ab30 chore: update the regression.
• aebe346 chore: increment version and fix test
• 22dfaeb feat: add example package to deployment
• b2e1a76 fix: tests
• 7b76f57 fix: test
• 2f41764 docs: Changelog
• 1d17468 fix: correct the data path for the new pmp cache
• 9ab3365 Update GPCP dataset version in changelog
• a69c4c0 Update reference dataset from GPCP-Monthly-3-2 to GPCP-3-3
• 3b6d2e1 chore(deps): bump pymdown-extensions from 10.21 to 10.21.3
• 368334f chore(deps): bump idna from 3.11 to 3.15
• 5734953 Update 672.improvement.md
• 5a5812a add changelog
• 1f72461 Add new climatology dataset entry
• cbbd022 chore(deps-dev): bump the python-dependencies group with 4 updates

v0.14.3

Changelog

Bug Fixes

• Fixed a race condition when trying to create the same ExecutionGroup concurrently.
  Database.get_or_create and Database.update_or_create now wrap the INSERT in a SAVEPOINT
  and re-fetch the winning row on conflict instead of aborting the transaction. (#679)

Changes

• c07e141 Bump version: 0.14.2 → 0.14.3
• 10c9dd0 fix(database): address PR review comments
• 3aae695 docs: changelog
• 9f6d74c docs: changelog for #679
• c60b8ed fix(database): make get_or_create / update_or_create race-safe

v0.14.2

Changelog

Bug Fixes

• Changed the behaviour for register_dataset treatment of files absent from the current ingest slice as kept-in-place with a warning, instead of raising NotImplementedError. (#677)

Changes

• c1d9ce2 Bump version: 0.14.1 → 0.14.2
• 787e62b docs: changelog
• 82fbd00 fix(ingest): keep files absent from current ingest, warn instead of abort

v0.14.1

Changelog

Features

• Added a --chunk-size option to ref datasets ingest (CMIP6 and CMIP7) that streams the catalog in directory-aligned batches instead of loading the whole archive into memory at once.
  Peak memory is now bounded by chunk_size rather than by the total number of files in the input tree. (#674)

Improvements

• Skip the redundant per-dataset validate_data_catalog call inside register_dataset. The production ingest path already validates the catalog (and each streamed chunk) once up-front, so the inner re-validation only duplicated work. Cuts ~20% off ingest wall time. (#675)

Bug Fixes

• Stopped re-ingesting unchanged CMIP6 files on every run.
  Previously, ref datasets ingest reported every file as updated and emitted a "Updating file metadata" warning per file even when nothing on disk had changed, because the file-metadata comparison treated a str loaded from the database as unequal to a freshly parsed cftime.datetime. Re-ingesting an unchanged directory is now a no-op. (#673)

Changes

• aa18732 Bump version: 0.14.0 → 0.14.1
• 761a673 refactor(ingest): raise RefException from the register_dataset guard
• 8f86f10 perf(ingest): keep cheap per-slice metadata-consistency guard
• 3b06df2 feat(ingest): extend streaming ingest to CMIP7
• 190dd2b chore: cleanup
• 29c2013 chore(changelog): rename news fragment to PR number
• 7133baf perf(ingest): drop redundant per-dataset validate_data_catalog call
• e78a3ab test: add test coverage
• f4ea29a docs: address PR review comments
• 8042735 docs: reformat
• baded9b chore(changelog): rename news fragment to PR number
• 9b2438d feat(ingest): stream catalog in chunks to bound ingest memory
• cd00b64 docs: add changelog for PR #673
• 0823e3c fix(ingest): stop re-ingesting unchanged CMIP6 files every run

v0.14.0

Changelog

Features

• Write the diagnostic version to the database.

  Bumping a diagnostic's version now creates a fresh execution group
  (preserving the prior version's group and results) instead of overwriting it,
  and each execution is stamped with the provider version that produced it. (#667)

Improvements

• Reworked the layout of execution output directories.
  New executions now write to <provider>/<diagnostic>/<group_short>/<execution_id>/
  instead of <provider>/<diagnostic>/<dataset_hash>/,
  so reruns of the same diagnostic group no longer overwrite earlier outputs.
  Existing rows on disk continue to resolve through their stored Execution.output_fragment. (#655)
• Added diagnostic versioning read-path foundations.
  New database columns track the diagnostic version for each execution group,
  the promoted version for each diagnostic, and the provider version for each execution.
  Default queries now return only results at the promoted version, and the ref executions stats command reflects the same filter. (#665)

Bug Fixes

• ref datasets ingest now exits with a non-zero status when one or more input directories fail to ingest,
  so cron- and Kubernetes-based deployments can detect failures.
  The CMIP6, CMIP7, and obs4MIPs adapters also now tolerate individual files with missing DRS components:
  those files are logged and skipped instead of aborting the whole batch. (#668)

Changes

• 1d1bfd4 Bump version: 0.13.2 → 0.14.0
• d018f9a fix(ingest): exit non-zero on failure and tolerate per-file DRS errors
• fa30b96 chore: tweak changelog
• ef3eaaf docs: add changelog for PR #667
• 6da4003 feat(versioning): activate diagnostic versioning write path
• 22ed810 chore(deps): bump urllib3 from 2.6.3 to 2.7.0
• ff35ab7 chore: clean up comments
• cc21064 test(versioning): tighten stats filter assertion
• 0fffe26 chore(changelog): add news fragment for PR #665 (diagnostic versioning read-path)
• dfcb0a3 feat(versioning): diagnostic versioning read-path foundations
• 22b3d79 fix(fragment): address PR review feedback on output-fragment helpers
• eebc320 docs(fragment): rename _TEMP_DIAGNOSTIC_VERSION and drop future-PR comment
• 64e2c71 refactor(solver): extract assign_execution_fragment helper and add safety cap
• 6da6249 docs: rename changelog placeholder to 655.improvement.md
• acc8972 refactor(solver): switch execution output layout to //<group_short>/<execution_id>

v0.13.2

Changelog

Features

• Added support for Python 3.14. (#625)

Improvements

• Add extra options for users to ingest options other than base to parsl functions (#651)
• Updated ESMValCore to v2.14.0 and ESMValTool to 2.15.0.dev15+gdead90ca8. (#652)

Bug Fixes

• Fixed ref test-cases fetch aborting the entire run when a single test case could not be parsed (for example because of a PermissionError on a cached CMIP6 file); the failing test case is now logged as a warning and the loop continues. (#639)
• Fixed ref solve periodically reporting executions as never finishing. Stuck executions left in the in-progress state by a crashed worker, walltime kill, or hung diagnostic are now reaped and retried on the next solve, the CLI --timeout default has been raised to 6 hours to match the worker time limit, and per-task timeouts in the local executor cancel hung diagnostics instead of blocking the whole pool. (#641)

Improved Documentation

• Updated CITATION.cff with extra author names and orcid ids. (#636)

Trivial/Internal Changes

• #619, #637, #638

Changes

• e916682 Bump version: 0.13.1 → 0.13.2
• a604ada Revert "Merge pull request #654 from Climate-REF/ci/ilamb-debug-workflow"
• bbd7360 chore(deps): bump mistune from 3.2.0 to 3.2.1
• f6bfd45 chore(deps): bump mako from 1.3.11 to 1.3.12
• 9567a9f chore(deps-dev): bump pip from 26.0.1 to 26.1
• 557738a chore(deps): bump jupyter-server from 2.17.0 to 2.18.0
• 369f4bf docs: add changelog for #654
• 411e3e1 ci: add standalone ILAMB debug workflow
• 97126a8 Add changelog
• 5e04639 Update ESMValCore to v2.14.0 and ESMValTool to 2.15.0.dev15+gdead90ca8
• d9d02ea add changelog
• a654697 add executor.config.extra_* so options other than base can be injected to parsl functions
• 06a745f chore(deps-dev): bump jupyterlab from 4.5.6 to 4.5.7
• 42fd0e8 chore(deps-dev): bump notebook from 7.5.5 to 7.5.6
• dcdef80 Update CITATION.cff
• 8e1cf40 chore: fix trailing line break
• 87ad343 Update CITATION.cff with additional authors and ORCID IDs
• 8d3bd97 docs: address PR review comments
• a69a7ec fix(solver): detach execution before commit to prevent autobegin clash
• eb60774 docs: add changelog entry for #641
• 7ac8da1 refactor(executor): consolidate failure-recording helpers and cache solver lookups
• dab0174 test(executor): add e2e failure-mode coverage for LocalExecutor
• 7df2750 fix(executor): make solve resilient to stuck executions
• 26296ac chore: pin climate_ref_client as known-third-party for ruff isort
• ed4654c docs(changelog): add 639.fix.md
• 8deaf46 fix(test-cases): don't abort fetch loop when one test case fails
• f8f04c8 chore: Use new docker image tags
• d3748d9 docs(changelog): add 638.trivial.md
• f1ba085 test(dataset_registry): isolate cache-dir tests from REF_DATASET_CACHE_DIR
• 332f281 docs: Changelog
• 09b1007 chore: update GHA hooks
• 8843752 chore: update pre-commit versions
• 972c270 Update CITATION.cff to version 1.2.1
• 52525fe chore(deps): bump gitpython from 3.1.46 to 3.1.47
• 739f61a chore(deps): bump python-dotenv from 1.2.1 to 1.2.2
• 5b5f624 chore(deps): bump nbconvert from 7.17.0 to 7.17.1
• 88b67c6 chore(deps-dev): bump the python-dependencies group across 1 directory with 4 updates
• 86f74ef chore(deps): bump mako from 1.3.10 to 1.3.11
• 2d5d9bf chore(deps): bump softprops/action-gh-release
• ef5e3de docs: add changelog for #619
• d8d1885 build(docs): swap mkdocs-typer for mkdocs-typer2
• 0b92945 chore(deps): bump pygments from 2.19.2 to 2.20.0
• 6082685 chore(deps): bump pillow from 12.1.1 to 12.2.0
• 5c7aea6 fix(types): annotate esgf_data_catalog_trimmed result dict
• 020f928 chore(deps-dev): bump the python-dependencies group across 1 directory with 7 updates
• 6eb413a chore: update lock
• 7833bc4 Revert "chore: add Python 3.14 support across packages and CI"
• 576e86b docs: add changelog entry for PR #625
• a086282 chore: add Python 3.14 support across packages and CI
• dd29e46 chore: remove windows from the pypi test

v0.13.1

Changelog

Features

• Added first-class read-only support and a migration-status helper to the Database API.

  Database.from_config(..., read_only=True) rewrites file-based SQLite URLs to read-only URI form and skips migrations,
  and Database.migration_status(config) reports the current/head revisions and state. (#624)

Changes

• cddf9f6 Bump version: 0.13.0 → 0.13.1
• 625f0df docs: Changelog
• 6b57d18 fix(db): address PR review comments
• 86b10fd chore: update the notebook
• a16f701 docs: pin openapi-python-client package name in API guide
• 77f77cb docs: add changelog entry for #624
• b169132 feat(db): add read-only mode and migration_status helper

v0.13.0

Announcements

(#582) reworked how the data cache works. It is recommended to rerun ref providers setup after upgrading.

Changelog

Features

• Added timeseries extraction for ESMValTool regional historical and ozone polar cap diagnostics, and updated file patterns to match the current ESMValTool output directory structure. (#607)
• Added ref executions reingest command to re-ingest existing execution results without re-running diagnostics. Creates a new immutable execution record with a timestamped output fragment, leaving the original execution untouched. (#610)
• Added ref db CLI subcommand group for database management. Includes commands for running migrations, checking schema status, viewing migration history, creating backups, executing SQL queries, and listing tables. (#615)

Improvements

• Fetch ESMValTool recipes when installing the provider. (#582)
• Unify facet filter parsing across CLI commands. --filter in executions list-groups and delete-groups now supports multiple values for the same key with OR semantics (e.g., --filter source_id=A --filter source_id=B), consistent with --dataset-filter in datasets list and solve. (#613)
• Improved solver performance by batching dataset finalisation before grouping, avoiding redundant file I/O when multiple groups share overlapping datasets. (#616)

Bug Fixes

• Fixed CLI test isolation by making the invoke_cli fixture depend on the config fixture, ensuring tests use an isolated database rather than the user's real one. Also marked RLIMIT_AS tests as expected failures on macOS where this resource limit is not supported. (#611)
• Copied the scratch directory for the previous execution when reingesting (#612)

Changes

• 66f8c69 Bump version: 0.12.3 → 0.13.0
• 0fa7bf2 refactor: improve cache migration handling in DatasetRegistryManager
• 30e5a7a feat(core): add cache migration helper for registry layout changes
• 0c7a903 chore(deps): bump pyjwt from 2.11.0 to 2.12.0
• 7fc74f8 chore(deps): bump cryptography from 46.0.5 to 46.0.7
• f3dc942 chore(deps): bump tornado from 6.5.4 to 6.5.5
• f07af91 docs: add changelog for #616
• 4cb8034 perf: batch dataset finalisation before grouping in solver
• fb6105c chore: remove ansi
• b37ae2b fix: address PR review comments
• 88e0104 refactor(db): extract _get_sqlite_path helper for URL parsing
• a176627 test(cli): add tests for db commands without an existing database
• 8d4a1dd docs: add changelog entry for db CLI commands
• b857b79 feat(cli): add db subcommand group for database management
• cfcee3e fix: copy then build
• 450759f docs: add changelog entry for PR #613
• 94ac1dc refactor: unify facet filter parsing across CLI commands
• 48ca342 test: Reorganise tests for reingest
• 12b76f3 refactor: add update_dirty param to handle_execution_result
• 283bcf4 fix: changelog
• 7baf0ac docs: add changelog entry for reingest selective copy fix
• ce3f377 fix(reingest): use handle_execution_result for selective file copying
• 52be560 docs: add changelog entry for PR #611
• 624156b fix(test): isolate CLI tests from user database and xfail RLIMIT_AS on macOS
• 7ec0bb4 fix(reingest): always reingest from oldest (original) execution
• fed124c fix: address PR review comments for reingest
• b1eaba3 refactor(fragment): replace collision-checking with timestamp-based output fragments
• 0480279 fix(solver): build definition early to avoid MagicMock key in tests
• 375ef6c test: add coverage for fragment collision, dataset links, and execution state
• df3350d refactor(reingest): simplify to always-versioned immutable flow
• 9f05ba0 fix(test): handle ANSI escape codes in typer missing option error
• 10a19d1 refactor(executor): extract shared ingest_execution_result and add equivalence tests
• d5a4218 refactor(executor): deduplicate reingest and result_handling ingestion logic
• c02beab fix: clean up finalized->finalised
• 3403780 test(reingest): add CLI and unit test coverage for reingest command
• 303a3fb refactor(test): deduplicate reingest test fixtures and helpers
• 557eee4 test(reingest): improve coverage to 74% with targeted unit tests
• 08d8ddb chore: fix update to environs api
• 7d629f1 fix(executor): remove eager reingest import from executor init
• f977aa1 fix(reingest): address PR review comments
• 484d6bc chore: add changelog entry for PR #610
• 42a9e33 style(cli): remove redundant inline comments in reingest command
• 6ec695a refactor(reingest): consolidate duplicated ingestion functions and add scratch cleanup
• 5f06821 feat(cli): add reingest command with enum-based mode selection
• 8383ae6 chore(deps-dev): bump the python-dependencies group across 1 directory with 16 updates
• 7ff0cec chore(deps): bump the github-actions group across 1 directory with 5 updates
• a2ad979 test(esmvaltool): add tests for byte string index decoding and _region_to_filename
• 9752736 chore: add changelog entry for PR #607
• ba5d941 feat(esmvaltool): add timeseries extraction for regional and ozone diagnostics
• a3a65fd fix: changelog
• 65f9ed9 Put everything under ~/.cache/climate_ref
• 06cf52b Add changelog
• 41d07f5 Fetch recipes in advance

v0.12.3

Announcements

Changelog

Features

• Added ref datasets stats and ref executions stats CLI commands for viewing summary statistics without listing individual records. (#584)

Bug Fixes

• Fixed zero-emission-commitment diagnostic failures caused by space-separated activity_id values creating path mismatches, incorrect parent timerange computation when datasets are split across multiple files, and models with esm-1pctCO2 as parent experiment being incorrectly scheduled (see #586). (#585)
• Fixed experiment selection for computing ZEC. (#589)
• Solve regression tests now use the local default_ignore_datasets.yaml instead of downloading from the main branch on GitHub, ensuring tests reflect the current ignore list. (#606)
• Use flexible time stamp for PMP annual cycle. (#465)

Improvements

• Enable multiple-file-input for PMP's variability modes diagnostics. (#583)-

Improved Documentation

• Added a how-to guide on controlling memory use and parallism during diagnostic execution. (#591)

Changes

• a3831a9 Bump version: 0.12.2 → 0.12.3
• cba52e0 chore(docs): remove --strict flag from mkdocs build
• 653d27b test(pmp): add tests for get_wildcard_pattern and validate common prefix
• 6b94c6c chore: add changelog for #606
• 2d9dbfa fix(tests): use local ignore list in solve regression tests
• c362381 clean up
• 8c3bfb4 clean up
• 01ed2a9 clean up
• d85aa24 clean up
• 2c6943e add change log
• 3ddd098 change input checker, enable multi-file input
• 831c60d docs: cleanups
• 2b53834 docs: rename
• 6789c06 docs: Placeholder QA page
• b60b9a2 Add docs on configuring Dask
• eec3610 Correct experiment selection for computing ZEC
• b42bebf chore: Ignore failing datasets in the sea-ice-sensitivity runs
• 1a8b0ac fix(esmvaltool): exclude ZEC models with esm-1pctCO2 parent experiment
• 7d255d9 chore: add changelog for #585
• 50b4620 fix(esmvaltool): resolve ZEC failures from activity_id mismatch and wrong timerange
• fab8b9c chore: add changelog for #584
• 8a13616 feat(cli): add stats subcommand to datasets and executions

v0.12.2

Changelog

Bug Fixes

• Improved conda error logging by capturing stderr in solve logs and avoiding unnecessary stacktraces. (#580)

Trivial/Internal Changes

• #580

Changes

• 7bc2bff Bump version: 0.12.1 → 0.12.2
• 02c2e0c chore: add second EOF
• 78e84db Bump version: 0.12.0 → 0.12.1
• fac866b chore: support mip_id
• d09556c fix: update test to expect CondaCommandError instead of CalledProcessError
• ba662d2 docs: add changelog entries for PR #580
• 29bba46 fix: Log conda stderr to solve logs
• 7d269b6 chore: add extra cvs