CMIP7 compliance issues in examples

[mauzey1]

I've been experimenting with checking CMOR's output with the IOOS Compliance Checker using cc-plugin-wcrp. My first experiments have so far been with the Python examples. The command compliance-checker -t wcrp_cmip7 was used to test against CMIP7 compliance checks. Below are the issues that were found in the output of the examples.

I've attached the report that was created with Codex to help find these compliance issues: cmip7_compliance_report.md

Mandatory Issues

Consolidated Metadata

This was an issue that we tried to tackle when making CMOR's output compliant with cmip7_repack (see issue #880). We were able to get CMOR to handle the chunking but not the metadata consolidation. We settled on having that part be done post-CMORization.

We might explore having this performed in CMOR in the future to avoid needing extra steps in the publication pipeline.

Time Squareness

The compliance checker took issue with the monthly time axis using bounds of [0, 31, 60] and time values of [15.5, 45.5] with the 360_day calendar. It expected the time axis to have bounds of [0, 30, 60] and time values of [15, 45].

You can still create compliant files with CMOR as long as you enter compliant time values. However, CMOR currently doesn't check time values if they meet the checker's compliance. The CMIP7 checker validates time values against the filename's start date, frequency, calendar, and time-related cell_methods.

Having CMOR be more rigorous with the time bounds and midpoint checks would be helpful.

standard_hybrid_sigma Coordinate Monotonicity

This is specific to the file generated by example_05_model_levels.py. There was an issue with the values and bounds for the standard_hybrid_sigma axis not being stored in increasing order. It didn't matter which order you gave the values in the example code as CMOR would store them in decreasing order.

The fix was to change the stored_direction attribute of standard_hybrid_sigma to increasing in CMIP7_coordinate.json.

        "standard_hybrid_sigma": {
            "stored_direction": "increasing",

This either requires a change to the standard_hybrid_sigma entry in the CMIP7 tables, or a change in the compliance checker if it is wrong.

Warning Issues

Missing cell_measures

This was an issue caused by not using the CMIP7_cell_measures.json file to set the cell_measures variable attribute. Using it got rid of all of the warnings for this except for htovgyre. In the case of the htovgyre variable, the compliance checker was expecting its cell_measures to have an empty string. However, CMOR removes attributes that have emtpy string values.

Should CMOR allow attributes with empty strings, or should the compliance checker consider attributes without values or are empty strings to be unnecessary or invalid?

long_name Mismatches

Similar to the cell_measures issue, this could be resolved by using the CMIP7_long_name_overrides.json file to override the long_name attribute of variables to those that are compliant. However, the current file only has a few overrides that don't cover cases for tas or hfls variables in the examples.

The solution should be to expand the long_name overrides to include more cases.

Missing Parent and Sub-Experiment Global Attributes

parent_experiment_id and sub_experiment_id were considered missing required attributes by the compliance checker. However, we have already established that experiments without a parent should not have parent attributes and that subexperiments are not a part of CMIP7. This appears to be a clear mismatch between compliance checker expectations and CMIP7 standards.

Plan

• The most actionable one on CMOR's end that we can start working on is the time axis check. We should make it so that it takes into account the same dataset information as the compliance checker when checking the bounds and midpoint values for time.
• For the consolidated metadata, we could either stick with just doing the repack step after CMORizing or incorporating it into CMOR so that it is performed when writing a file.
• The long_name and cell_measures issues will require updating our CMIP7 tests and demos to use CMIP7_long_name_overrides.json and CMIP7_cell_measures.json, but it will also require some changes in the cmip7-cmor-trables repo to include more long_name overrides.
• standard_hybrid_sigma may need to have its stored_direction value changed. This might include other coordinates with the stored_direction attribute.
• The rules for parent attributes and subexperiments will need to be changed in the compliance checker.

@durack1 @taylor13 @matthew-mizielinski

[taylor13]

Thanks for doing these checks and analaysis. Here is some feedback:

The most actionable one on CMOR's end that we can start working on is the time axis check. We should make it so that it takes into account the same dataset information as the compliance checker when checking the bounds and midpoint values for time.

I would not make this a priority. Groups will be running the compliance checker, so let's rely on that for now.

For the consolidated metadata, we could either stick with just doing the repack step after CMORizing or incorporating it into CMOR so that it is performed when writing a file.

I vote for no immediate action and keep repacking separate from cmorization (and therefore independently controlled by the user).

The long_name and cell_measures issues will require updating our CMIP7 tests and demos to use CMIP7_long_name_overrides.json and CMIP7_cell_measures.json, but it will also require some changes in the cmip7-cmor-trables repo to include more long_name overrides.

I don't think any action should be taken. The long_names and cell_measures do not need to be consistent with the data request. The compliance checker can raise a warning, but users should be able to ignore that.

standard_hybrid_sigma may need to have its stored_direction value changed. This might include other coordinates with the stored_direction attribute.

The correct specification for this coordinate is:
"stored_direction": "increasing". [This was corrected below after this comment was posted; the direction should be "decreasing".]
and
"positive": "down"

So I think CMOR wrote the data correctly, but the checker has a bug. I think this is the issue raised in ESGF/cc-plugin-wcrp#45, which is being addressed. (Right, @glevava?)

The rules for parent attributes and subexperiments will need to be changed in the compliance checker.

Yes, unless we're missing some piece of information. I am surprised that the checker raised errors here. [Are you accessing this cmor-cvs.json file?] Maybe @glevava can say if this is a bug or not in the IOOS Compliance Checker.

[mauzey1]

@taylor13

The correct specification for this coordinate is:
"stored_direction": "increasing"
and
"positive": "down"

So I think CMOR wrote the data correctly, but the checker has a bug.

CMOR is working correctly, but the issue is that the standard_hybrid_sigma doesn't have the correct specifications for stored_direction.

[taylor13]

Sorry, my assertion above is probably wrong. Atmospheric model level data should be stored from the lowest atmospheric layer (near the surface) to the highest atmospheric layer (near the top of the atmosphere). The standard_hybrid_sigma coordinate has a value of 1 at the surface and a value of 0 at the top of the atmosphere, so the stored_direction is "decreasing" (from about 1 to about 0).

So I think the table was in fact correct and the checker likely wrong.

[glevava]

Many thanks @mauzey1 for this feedback. Here are my comments and recommendations.

Regarding the mandatory (high-severity) errors:

Consolidated Metadata

We have the same issue with XIOS, which is our other standardisation tool. The QA/QC still reports an error on repacking because of the metadata. The decision was made to handle this afterwards with the cmip7_repack tool. So on this error, I agree with @taylor13 , and I would therefore keep the repacking step separate of CMORization.

Time Squareness

This is one of the examples I often use to explain that, although CMOR makes an impressive standardisation effort and remains very close to the Data Request and CV requirements, errors are still possible and this cannot replace QA/QC.

Regarding your plan for this error, I would disagree with @taylor13 if the proposal is to do nothing for now. Otherwise, all groups using CMOR to standardise their data may end up with erroneous files that will not pass the QA/QC when the calendar is 360_day. I do not know how many groups this affects, but in my view this is a critical error to fix as high-priority.

standard_hybrid_sigma coordinate monotonicity

This is indeed a known QA/QC error, and I confirm what @taylor13 said. See the issue here:
ESGF/cc-plugin-wcrp#45
We are currently fixing the problem, but until the next QA/QC release, I recommend disabling this check by simply commenting out the following block in the corresponding TOML file:
https://github.com/ESGF/cc-plugin-wcrp/blob/c195812aa990bb264f30878218f1391225aef0f0/plugins/cmip7/config/wcrp/coordinate_variables.toml#L916

Regarding warning (medium-severity) errors — and I remind here that none of these warnings block ESGF publication, in line with what was agreed with the WIP:

cell_measures

Indeed, the QA/QC currently always expects to find a cell_measures attribute consistent with what is defined in the Data Request. However, since the DR sometimes specifies an empty field, if it is understood that the attribute may then be absent, we need to change the is_required parameter to false in the QA/QC TOML file:
https://github.com/ESGF/cc-plugin-wcrp/blob/c195812aa990bb264f30878218f1391225aef0f0/plugins/cmip7/config/wcrp/geophysical_variable.toml#L54-L55

Here, I do not think you need to do anything on your side. We need to validate with the CV-TT and/or the WIP that this attribute may be absent when empty, and the error should resolve itself in the next release. @taylor13 what do you think?

long_name

For this issue, as @taylor13 said I would do nothing for now, but not for the same reason. We are currently restructuring the CV source of information that aggregates variable-related metadata, including long_name. This will include the latest DR changes as well as the dictionary in CMIP7_long_name_overrides.json.
No need a new release of QA/QC here but a new CMIP7 CV release. Nothing to do on your side.

parent_experiment_id and sub_experiment_id global attributes

Here, I do not understand either. sub_experiment_id is no longer part of the CMIP7 global attributes checked by the QA/QC, in accordance with what was agreed with the CV-TT. It does not appear in the TOML configuration file:
https://github.com/ESGF/cc-plugin-wcrp/blob/master/plugins/cmip7/config/wcrp/global_attributes.toml

As for parent_experiment_id, it is clearly marked as is_required = false:
https://github.com/ESGF/cc-plugin-wcrp/blob/c195812aa990bb264f30878218f1391225aef0f0/plugins/cmip7/config/wcrp/global_attributes.toml#L184
Its absence should therefore not return an error.
Which version of cc-plugin-wcrp are you using? Could you check these files locally? If you have the same configuration on your side then we will dive into the code to investigate any bug on our side (but it's strange that this only affects the parent_experiment_id attribute as all parent_* attributes follow the same configuration).

[mauzey1]

I used cc-plugin-wcrp 2.3.2 with compliance-checker 6.1.0 installed with pip. Looking at venv/lib/python3.13/site-packages/plugins/cmip7/config/wcrp/global_attributes.toml, I see is_required = false for parent_experiment_id.

[global.attributes.parent_experiment_id]
# Checks ATTR001 ATTR002 ATTR003 ATTR004
severity = "H"
value_type = "str"
is_required = false
cv_source_collection = "experiment"

[taylor13]

@mauzey1 Do you know which of the "examples" threw the parent_experiment_id error? Also, where are the example_0*_input.json files located?
thanks,
Karl

[glevava]

hi @mauzey1 @taylor13 it seems the issue related to sub_experiment_id and parent_experiment_id has also been encountered here ESGF/cc-plugin-wcrp#44

We are investigating it.

[mauzey1]

@taylor13 The parent_experiment_id issue occurred for every example.

The user input is created by the configure function in each example.

cmor/examples/python/example_01_usual_2d_field.py

Lines 18 to 44 in c003a73

	def configure(output_dir: Path) -> None:
	output_dir.mkdir(parents=True, exist_ok=True)
	user_input = {
	"_AXIS_ENTRY_FILE": "CMIP7_coordinate.json",
	"_FORMULA_VAR_FILE": "CMIP7_formula_terms.json",
	"_cmip7_option": 1,
	"_controlled_vocabulary_file": str(CMOR_CV_PATH),
	"activity_id": "CMIP",
	"calendar": "360_day",
	"experiment_id": "amip",
	"forcing_index": "f1",
	"frequency": "mon",
	"grid_label": "g999",
	"initialization_index": "i1",
	"institution_id": "MOHC",
	"license_id": "CC-BY-4.0",
	"nominal_resolution": "100 km",
	"outpath": str(output_dir),
	"physics_index": "p1",
	"realization_index": "r1",
	"region": "glb",
	"source_id": "DUMMY-MODEL",
	}
	input_path = output_dir / "example_01_input.json"
	input_path.write_text(json.dumps(user_input, indent=2, sort_keys=True))
	cmor.setup(inpath=str(TABLES_PATH), netcdf_file_action=cmor.CMOR_REPLACE)
	cmor.dataset_json(str(input_path))

[taylor13]

thanks! I think we just have to wait for @glevava and colleagues to debug the problem.

[taylor13]

@glevava : perhaps not urgent, but do we have a time-frame for fixing this?