Add the Ensemble Copula Coupling "transformation" approach by gavinevans · Pull Request #2303 · metoppv/improver

gavinevans · 2026-02-11T15:20:09Z

Addresses https://github.com/metoppv/mo-blue-team/issues/952

Description
This PR adds the ECC-transformation (ECC-T) capability, largely based on Schefzik et al., 2013. Rather than sampling at equally spaced quantiles, the ECC-T fits a distribution based on data from a cube defining the intensities (e.g. of precipitation rate), at each grid point separately. This can allow for the regeneration of realizations that are more similar to the "raw" realizations than when sampling at equally spaced quantiles.

The number of lines changed in this PR have been vastly increased by conversions from unittest to pytest for the ConvertProbabilitiesToPercentiles, EnsembleReordering and RebadgePercentilesAsRealizations.

An outline of the files changed:

doc/source/examples/generate_realizations_simple_example.py –
Provides a worked example of the ECC-Transformation approach (largely following Schefzik et al., 2013).
emos_calibration.py – Update to EnsembleReordering plugin interface.
generate_realizations.py – Update CLI to include a sampling option and a ensure_evenly_spaced_realizations option, which will be passed to the ConvertProbabilitiesToPercentiles plugin.
ensemble_copula_coupling.py –
1. Rearrange RebadgePercentilesAsRealizations plugin and add an ensure_evenly_spaced_percentiles argument.
2. Update ConvertProbabilitiesToPercentiles plugin to handle the transformation sampling option. Currently only a gamma distribution is supported. In this particular plugin, the changes are related to handling a multi-dimensional array of percentiles to interpolate to, rather than a 1D array. More arguments are passed to the choose_set_of_percentiles.
3. Support the passing through of the ensure_evenly_spaced_percentiles argument from the EnsembleReordering plugin to the RebadgePercentilesAsRealizations plugin. Some rearrangement of arguments.
numba_utilities.py –
Add a separate function (fast_interp_same_y_2d) to perform linear interpolation using numba when the input is 2D. This must be a separate function with a static function signature for compatibility with numba. A small function, fast_interp_same_y_nd, is used to decide which function should be called.
utilities.py –
1. Modify the existing choose_set_of_percentiles function to take the arguments required to support ECC-Transformation.
2. A CalculatePercentilesFromIntensityDistribution plugin that adds the core ECC-T functionality.
3. A create_cube_with_percentile_index function is added, which is very similar to the existing create_cube_with_percentiles function, but handles a percentile_index coordinate instead.
4. Add a slow_interp_same_y_2d function (where numba is not available), and update interpolate_multiple_rows_same_y.
SHA256SUMS – checksum update.
test_generate_realizations.py –
Update the generate_realizations acceptance tests.
test_ConvertProbabilitiesToPercentiles.py –
Most of the changes in this file are a conversion from unittest to pytest. The only new unit test is the test_process_transformation_sampling_3d unit test at the bottom of the file.
test_EnsembleReordering.py –
These are almost entirely a conversion from unittest-based tests to pytest. The only new test is test_process_percentile_index.
test_RebadgePercentilesAsRealizations.py –
These are almost entirely a conversion from unittest-based tests to pytest. The only new test is test_ensure_evenly_spaced_percentiles.
test_utilities.py –
The choose_set_of_percentiles tests in this file have been converted from unittest-based tests to pytest. The test_choose_set_of_percentiles_transformation unit test is newly added.
New unit tests have been added for the new CalculatePercentilesFromIntensityDistribution class.
Update Test_interpolate_multiple_rows_same_y to cover testing of slow_interp_same_y_2d and fast_interp_same_y_nd.
test_probabilistic.py –
I’ve added a new unit test for the find_percentile_coordinate function to show the a “percentile_index” coordinate is supported.

Testing:

Ran tests and they passed OK
Added new tests for the new feature(s)

CLA

If a new developer, signed up to CLA

…n approach.

…ToPercentiles plugin.

…used.

…sampling option.

…acceptance test data used.

maxwhitemet

Thanks @gavinevans. I have added some comments - almost all documentation based rather than functionality concerns.

maxwhitemet · 2026-03-02T13:45:38Z

                random_ordering=randomise,
                random_seed=random_seed,
+            ).process(
+                percentiles,


Although extraneous to this PR's aims, the percentiles variable could be renamed to percentiles_forecast_cube to clarify the variable is a cube.

maxwhitemet · 2026-03-02T13:48:02Z

+and then reorder these percentiles to generate ensemble realizations. The example
+illustrates both the "quantile" and "transformation" sampling options available
+within ECC, highlighting the differences in the resulting percentiles and
+realizations produced by each method


Suggested change

realizations produced by each method

realizations produced by each method.

maxwhitemet · 2026-03-02T14:13:04Z

+# the bottom left grid point has a probability of 0.8 of exceeding 1 mm/h, so the
+# quantile sampling at the 25th percentile gives a value above 1 mm/h. For
+# transformation without scaling, the 0.103 percentile maps to a precipitation rate
+# below 1 mm/h, whilst the scaling, scales the percentiles to be within the


Suggested change

# below 1 mm/h, whilst the scaling, scales the percentiles to be within the

# below 1 mm/h, whilst the scaling scales the percentiles to be within the

Amended these sentences related to another comment.

maxwhitemet · 2026-03-02T14:20:01Z

+=======================================================================
+Example: Convert probabilities to percentiles and perform reordering to generate realizations
+=======================================================================


Although the docs build, I get the error 'Title underline too short' for many titles. E.g.

improver/doc/source/auto_examples/generate_realizations_simple_example.rst:485: WARNING: Title underline too short.

These warnings have now been removed as seen in https://app.readthedocs.org/projects/improver/builds/31690884/#309424762--686

…sformation * upstream/master: EPPT3121 Add clip option to fine fuel moisture content calculation (metoppv#2317) Edit CLI docstring. (metoppv#2318) Update WeightAndBlend.process() to use *cubes and as_cubelist (metoppv#2314) Lighting in vicinity cell_method alignment for master (metoppv#2296) Add quantile mapping and associated tests (metoppv#2264) MOBT-1069: Using adjacent validity times in SAMOS (metoppv#2310) Mobt991 ApplySAMOS plugin distribution fix (metoppv#2311) Add attribute to realization clustering to record source realizations (metoppv#2305) MOBT-1070: Time-zone safe handling of time point extraction in SAMOS calibration (metoppv#2306) Set START_DATE_DICT to outside largest lag period to silence warnings (metoppv#2304) Eppt2408 implement initialisation process for fire severity iterative parameters (metoppv#2302) FireWeatherIndexBase plugin - change Error to Warning for input values outside expected ranges (metoppv#2301)

maxwhitemet

Thanks @gavinevans. Happy with the changes made. Approved 👍

mo-jbeaver

Some comments added but all the tests ran successfully.

mo-jbeaver · 2026-04-28T11:10:41Z

+        cube.coord(percentile_coord_name).points = realization_numbers
        cube.coord(percentile_coord_name).rename("realization")


Suggested change

cube.coord(percentile_coord_name).points = realization_numbers

cube.coord(percentile_coord_name).rename("realization")

cube.coord(percentile_coord_name).rename("realization")

cube.coord("realization").points = realization_numbers

mo-jbeaver · 2026-04-28T11:14:01Z

        cube.coord("realization").points = cube.coord("realization").points.astype(
            np.int32
        )


Could this be combined or linked with the line above:
cube.coord(percentile_coord_name).points = realization_numbers

mo-jbeaver · 2026-04-28T12:14:41Z

+        distribution: str = "gamma",
+        nan_mask_value: float = 0.0,
+        scale_percentiles_to_probability_lower_bound: bool = False,
    ) -> None:


Some of the __init__ methods don't return anything but this one returns a None. Is there a reason for the difference?

mo-jbeaver · 2026-04-28T12:16:52Z

+            distribution:
+                Valid if the "transformation" option is selected for sampling
+                the probability distribution. Type of distribution to fit
+                (currently only 'gamma' is supported).


If so, does there need to be a check for if something other than "gamma" is passed?

mo-jbeaver · 2026-04-28T12:43:25Z

+                values generated are not reproducible.
+            tie_break:
+                The method of tie breaking to use when the first ordering method
+                contains ties. The available methods are "random", to tie-break


Suggested change

contains ties. The available methods are "random", to tie-break

contains ties. The available methods are the default "random", to tie-break

mo-jbeaver · 2026-04-28T14:10:50Z

        ignore_ecc_bounds_exceedance (bool):
            If True where percentiles (calculated as an intermediate output
            before realization) exceed the ECC bounds range, raises a
            warning rather than an exception.


Suggested change

warning rather than an exception. False is the default.

mo-jbeaver · 2026-04-28T14:12:08Z

@@ -55,10 +86,36 @@
            interpolation from the nearest available percentile, rather than using
            linear interpolation between the nearest available percentile and
            the ECC bound.


Suggested change

the ECC bound.

the ECC bound. The default is false.

mo-jbeaver · 2026-04-29T09:36:45Z

@@ -5,12 +5,10 @@
 """
 Unit tests for the


Suggested change

Unit tests for the

Pytest-based tests for the

mo-jbeaver · 2026-04-29T10:04:28Z

+def test_rank_ecc_preserves_order(cubes_for_rank):
+    """Test that rank_ecc preserves order when input is already sorted."""
+    cube, _ = cubes_for_rank
+    arr = np.ones_like(cube.data)
+    for i in range(arr.shape[0]):
+        arr[i] *= i + 1
+    raw_cube = cube.copy(data=arr)
+    cal_cube = cube.copy(data=arr)
+    result = Plugin().rank_ecc(cal_cube, raw_cube)
+    np.testing.assert_array_almost_equal(result.data, arr)


This test is the same as the one before but just testing something different something different. Could they be combined, testing it is a cube and the order is persevered?

mo-jbeaver · 2026-04-29T10:35:41Z

 # This file is part of 'IMPROVER' and is released under the BSD 3-Clause license.
 # See LICENSE in the root of the repository for full licensing details.
 """
 Unit tests for the


Suggested change

Pytest-based tests for the

gavinevans added 16 commits February 11, 2026 13:16

Initial changes to set up the capability to use the ECC-transformatio…

747b02e

…n approach.

Add support for ECC-transformation option within ConvertProbabilities…

586d488

…ToPercentiles plugin.

Conversion of tests from unittest to pytest style using Copilot.

a6e5d35

Latest edits.

1c40580

Output a percentile_index coordinate when transformation sampling is …

cd90e0e

…used.

Refactor EnsembleReordering plugin.

cec517a

Add argument.

a4999fc

Convert RebadgePercentilesAsRealizations to use pytest-style unit tests.

b1b1367

Update unit tests for EnsembleReordering.py.

ad8b921

Updates so tests pass.

dd210f9

Modifications to generate_realizations CLI to support transformation …

fd50f22

…sampling option.

Add slow_interp_same_y_2d capability.

fe86944

Further edits including adding a worked example.

a04443a

Add slow_interp_same_y_nd function.

bdd2d09

Minor edits.

c8e4305

Support additional arguments in generate_realizations CLI and change …

5bd3bb6

…acceptance test data used.

gavinevans mentioned this pull request Feb 11, 2026

Add the Ensemble Copula Coupling "transformation" approach metoppv/improver_test_data#123

Open

Revert accidental edit.

d3b102c

gavinevans mentioned this pull request Feb 20, 2026

Mobt991 ApplySAMOS plugin distribution fix #2311

Merged

2 tasks

maxwhitemet self-requested a review March 2, 2026 13:14

maxwhitemet reviewed Mar 5, 2026

View reviewed changes

gavinevans added 3 commits March 5, 2026 09:53

Edits following review.

9721775

Modify spacing.

6a7125d

maxwhitemet approved these changes Mar 10, 2026

View reviewed changes

mo-jbeaver requested changes Apr 30, 2026

View reviewed changes

	realizations produced by each method
	realizations produced by each method.

	# below 1 mm/h, whilst the scaling, scales the percentiles to be within the
	# below 1 mm/h, whilst the scaling scales the percentiles to be within the

		cube.coord(percentile_coord_name).points = realization_numbers
		cube.coord(percentile_coord_name).rename("realization")

	contains ties. The available methods are "random", to tie-break
	contains ties. The available methods are the default "random", to tie-break

Conversation

gavinevans commented Feb 11, 2026

Uh oh!

maxwhitemet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gavinevans Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxwhitemet left a comment

Choose a reason for hiding this comment

Uh oh!

mo-jbeaver left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gavinevans Mar 6, 2026 •

edited

Loading