Specs approach for AMOF files

[joshua-hampton]

As with issue #16, I have also made some progress on the idea of using specs to check AMOF files. In this approach, I have created a number of spec files, one that looks at land variables, and one that does common global attributes.

• land variables also includes a dimension and a variable that isn't real, just as part of checking it's working
• common global attributes refers to a new vocab file amof.json, although I think the info in here might also be in the AMF CVs, I didn't look because I did this first.

To accommodate this, I made some changes to generic.py, where the functions run by the spec checker live

• lines 31, 40, 50-54, function check_global_attrs: added regex checks
• lines 59-86: added two new functions - check_var_exists and check_dim_exists. Quite self-explanatory

This approach doesn't (currently) check the values of variable attributes, just that they exist. This is better suited for checking valid_min (compare with discussion in issue #16), but not for checking units, for example.

I think this approach will also work well when all variables have the same attributes; however, that's not the case (e.g. quality control variables have flag_values and flag_meanings, which don't exist in other variables). I guess that the specs can be written to do checks on QC variables separately to other variables.

Example of running check against these spec files:

checksit check -t off --specs=amof-land-common,amof-global-attrs-common /gws/nopw/j04/ncas_obs/public/cdao/ncas-radar-wind-profiler-1/ncas-radar-wind-profiler-1_cdao_20221020_snr-winds_low-mode_15min_v1.0.nc

Running with:
	Template: OFF
	Datafile: /gws/nopw/j04/ncas_obs/public/cdao/ncas-radar-wind-profiler-1/ncas-radar-wind-profiler-1_cdao_20221020_snr-winds_low-mode_15min_v1.0.nc


---------------- Running checks ------------------

[WARNING] Template checks switched off!
[FAILED] with 3 errors:

	01. [variable**************:cows]: Does not exist in file.
	02. [dimension**************:fields]: Does not exist in file.
	03. [global-attributes:******:instrument_serial_number]: '' does not match regex pattern '.{3,}'.

[joshua-hampton]

Update through commit 65cdf5c:

• added make_specs.py - I imagine this will probably live in amf-check-writer but I've added it here for now so it is somewhere. This takes (most of) the json CV files created by amf-check-writer and creates spec files that can be used by checksit.
  • currently this creates a product-specific spec file with variables and dimensions, and a start of doing common global attributes, although this still needs a bit of work
  • still to do:
    • complete common global attributes
    • common land/sea/air/trajectory variables and dimensions
    • product-specific global attributes
• added function check_var to generic.py and added rules_attrs keyword argument to check_global_attrs function to help with global attribute spec check

In general, these product specific spec checks work, e.g.

$ checksit check -t off --specs=amof-snr-winds ncas-radar-wind-profiler-1_cdao_20221017_snr-winds_high-mode_15min_v1.0.nc 

Running with:
	Template: OFF
	Datafile: ncas-radar-wind-profiler-1_cdao_20221017_snr-winds_high-mode_15min_v1.0.nc


---------------- Running checks ------------------

[WARNING] Template checks switched off!
[INFO] File is compliant!

and

$ checksit check -t off --specs=amof-surface-met ncas-aws-10_iao_20220301_surface-met_v1.0.nc 

Running with:
	Template: OFF
	Datafile: ncas-aws-10_iao_20220301_surface-met_v1.0.nc


---------------- Running checks ------------------

[WARNING] Template checks switched off!
[FAILED] with 8 errors:

	01. [variable**************:downwelling_longwave_flux_in_air]: Does not exist in file.
	02. [variable**************:downwelling_shortwave_flux_in_air]: Does not exist in file.
	03. [variable**************:downwelling_total_irradiance]: Does not exist in file.
	04. [variable**************:net_total_irradiance]: Does not exist in file.
	05. [variable**************:qc_flag_radiation]: Does not exist in file.
	06. [variable**************:qc_flag_downwelling_total_irradiance]: Does not exist in file.
	07. [variable**************:qc_flag_net_total_irradiance]: Does not exist in file.
	08. [dimension**************:index]: Does not exist in file.

In the second case, the variables that checksit reports as not existing do not exist in the file but should not necessarily be in the file - while they are valid variables for that data product, the ncas-aws-10 instrument does not measure radiation, and so the product-specific variables relating to radiation are omitted from the file, rather than being left in empty.

The index dimension should not appear in this spec - while in the middle of writing this I have found the error in make_specs.py and will correct this shortly.

[joshua-hampton]

make_specs.py fixed in commit d527361, with updated spec file for the surface-met product.

[joshua-hampton]

Commit c72106f adds making spec files for deployment modes, e.g. for land

[joshua-hampton]

Added section to check.py which, if template is auto (default), checks to see if the file given is a netCDF file, if so checks Conventions global attribute against known AMOF standard Conventions vocab. If so, gets spec files for that file based on data product and deployment mode, sets template to off, and runs checksit using those specs. For example:

$ checksit check ncas-radar-wind-profiler-1_cdao_20221017_snr-winds_high-mode_15min_v1.0.nc 

AMOF file detected, finding correct spec files

Running with:
	Template: OFF
	Datafile: ncas-radar-wind-profiler-1_cdao_20221017_snr-winds_high-mode_15min_v1.0.nc


---------------- Running checks ------------------

[WARNING] Template checks switched off!
[INFO] File is compliant!

Doesn't currently add in the global attributes spec file because that has not been created correctly/in a useful manner. Also still need to work on product-specific attributes.

I've not tested this against a non-netCDF file, but I have tested against a non-AMOF netCDF file, which is correctly ignored in the AMOF-checking process.

[joshua-hampton]

One issue with global attributes spec was the licence global attribute, which includes a URL. The check code is splitting the string based on a colon, which meant half of the attribute was not included. This is fixed in commit defa3b1, joining the list with a colon, and only applies to the regex checks.

[joshua-hampton]

In the general global attributes, there are two that have vocab checks - source and platform.

• platform is taken from the AMF_platfrom CV file, with any of the locations being acceptable. In vocab speak - __vocabs__:AMF_CVs/AMF_platform:platform:* where * represents any
• source is from the AMF_ncas_instrument (or AMF_community_instrument) CV file, however this then depends on instrument name, i.e. __vocabs__:AMF_CVs/AMF_ncas_instrument:ncas_instrument:<instrument_name>:description

To help toward this, commit 6c5b8a0 introduces the idea of wildcard to cvs.py. The wildcard can be used in specifying the vocab check, however at the moment it will only work if the wildcard is the last argument in the check. This means that platform will work, but source will not. The reason for this is that the vocab check is going through dictionaries until it gets to the required value(s) at the end, either a single value or a list of values. The wildcard returns a list of values. If the wildcard is used before the end, the vocab check will get confused at the next stage trying to take a key of a list. I think I would too.

[joshua-hampton]

Commit 1d75c78 allows for wildcard to appear anywhere in specs vocab check.

• When a wildcard is encountered by a vocab check, checksit looks to see if the wildcard is the last key in the vocab lookup.
  • If it is, the wildcard returns a list of all keys of the dictionary being checked.
  • If it isn't, the wildcard returns a list of all the values within the object being checked, but without the keys to those values. When this list then gets checked at the next stage of the vocab check, each element in the list will be checked independently, with a new list being returned.

A (hopefully not too confusing) generic example to help (mostly me) understand, considering the very generic dictionary

{'a' : {'b' : 'c', 'd' : {'e' : 'f', 'g' : 'h'} },
 'i' : {'b' : 'j', 'd' : {'e' : 'k', 'g' : 'l'} },
 'm' : {'b' : 'n', 'd' : {'e' : 'o', 'g' : 'p'} } }

The following checks get the (also) following results:

"__all__" => ['a', 'i', 'm']
"a:__all__" => ['b', 'd']
"a:d:__all__" => ['e', 'g']
"__all__:b" => ['c', 'j', 'n']
"__all__:d" => [{'e' : 'f', 'g' : 'h'}, {'e': 'k', 'g' : 'l'}, {'e' : 'o', 'g' : 'p'}]
"__all__:d:e => ['f', 'k', 'o']

[joshua-hampton]

Final issue that needs dealing with for AMOF spec checks (I think) - how to deal with product-specific variables that are not in the netCDF file? The AMOF standard allows for these variables to not be there, but at the moment checksit expects all possible variables to be present.
One option - create a function similar to checksit.generic.check_var that returns info or warning rather than error if variable is not present?

[agstephens]

Great work @joshua-hampton,

I wonder if we should have a way of specifying that any component (i.e. a dimension, variable or an attribute) could be optional. If so, we just need a nice clean way of identifying them.

[joshua-hampton]

Couple of thoughts on how we could do that:

1. Have the option to append names with something like __optional__, then within the checksit.generic.check_XXX function check to see if that string is present before running check on that value.
2. Add an extra parameter to the checksit.generic.check_XXX functions called required which takes a boolean value for each variable/dimension/attribute name being passed to the function (or the inverse parameter called optional)

[joshua-hampton]

Another thought - global attributes sampling_interval and averaging_interval have a Compliance Checking Rule in the spreadsheets that they should be a string with at least two characters - do we want to add an extra check that they should have units of time in them?

[joshua-hampton]

Idea on applying optional status to things in spec checks (commit c4fe779). When making spec files for data products, variables and dimensions being checked have :__OPTIONAL__ appended to them (e.g. see amof-surface-met.yml). The functions in generic.py then check the variables and dimensions to see if :__OPTIONAL__ is in each one. If it is, the check still happens, but (currently) no error is reported if the variable/dimension is not present. If :__OPTIONAL__ is not in the variable/dimension name, then the check happens as it previously did.

[agstephens]

@joshua-hampton: sounds good 👍

[joshua-hampton]

Commit 31a0209 - Added "warnings" as a returned parameter from all functions in generic.py. When optional product-specific dimensions or variables are missing, a warning is added, which passes back up the chain through specs.py to check.py, where they are printed out along with errors/file compliance, e.g.

(checksit_env) (base) [earjham@rsg1-202206011551 checksit]$ checksit check ncas-aws-10_iao_20220301_surface-met_v1.0.nc

AMOF file detected, finding correct spec files

Running with:
	Template: OFF
	Spec Files: ['amof-common-land', 'amof-surface-met', 'amof-global-attrs']
	Datafile: ncas-aws-10_iao_20220301_surface-met_v1.0.nc


---------------- Running checks ------------------

[WARNING] Template checks switched off!
[FAILED] with 5 errors:

	01. [global-attributes:******:creator_email]*** Value 'CHANGE: E-mail address of the person who made the file. Valid email' does not match regex rule: 'valid-email'.
	02. [global-attributes:******:creator_url]*** Value 'CHANGE: ORCID URL of the person who made the file. Valid URL' does not match regex rule: 'valid-url'.
	03. [global-attributes:******:calibration_certification_date]*** Value 'CHANGE: when bought' does not match regex rule: 'datetime-or-na'.
	04. [global-attributes:******:processing_level]*** Value '1' is not of required type: 'integer'.
	05. [global-attributes:******:platform_altitude]*** Value 'CHANGE: Altitude of the deployment position above MSL. Enter Not Applicable for air. Exact match: <number> m' does not match regular expression: '^\d+\.?\d*\sm$'.

[WARNING] 7 warnings about file:

	01. [variable**************:downwelling_longwave_flux_in_air]: Optional variable does not exist in file.
	02. [variable**************:downwelling_shortwave_flux_in_air]: Optional variable does not exist in file.
	03. [variable**************:downwelling_total_irradiance]: Optional variable does not exist in file.
	04. [variable**************:net_total_irradiance]: Optional variable does not exist in file.
	05. [variable**************:qc_flag_radiation]: Optional variable does not exist in file.
	06. [variable**************:qc_flag_downwelling_total_irradiance]: Optional variable does not exist in file.
	07. [variable**************:qc_flag_net_total_irradiance]: Optional variable does not exist in file.

I've also added a flag to the check command in cli.py - -w/--ignore-warnings which means the warnings don't print out, e.g.

(checksit_env) (base) [earjham@rsg1-202206011551 checksit]$ checksit check -w ncas-aws-10_iao_20220301_surface-met_v1.0.nc

AMOF file detected, finding correct spec files

Running with:
	Template: OFF
	Spec Files: ['amof-common-land', 'amof-surface-met', 'amof-global-attrs']
	Datafile: ncas-aws-10_iao_20220301_surface-met_v1.0.nc


---------------- Running checks ------------------

[WARNING] Template checks switched off!
[FAILED] with 5 errors:

	01. [global-attributes:******:creator_email]*** Value 'CHANGE: E-mail address of the person who made the file. Valid email' does not match regex rule: 'valid-email'.
	02. [global-attributes:******:creator_url]*** Value 'CHANGE: ORCID URL of the person who made the file. Valid URL' does not match regex rule: 'valid-url'.
	03. [global-attributes:******:calibration_certification_date]*** Value 'CHANGE: when bought' does not match regex rule: 'datetime-or-na'.
	04. [global-attributes:******:processing_level]*** Value '1' is not of required type: 'integer'.
	05. [global-attributes:******:platform_altitude]*** Value 'CHANGE: Altitude of the deployment position above MSL. Enter Not Applicable for air. Exact match: <number> m' does not match regular expression: '^\d+\.?\d*\sm$'.

[agstephens]

Another thought - global attributes sampling_interval and averaging_interval have a Compliance Checking Rule in the spreadsheets that they should be a string with at least two characters - do we want to add an extra check that they should have units of time in them?

@joshua-hampton: I think all structural rules like this should be referred to Barb.