Model evaluation

[lehors]

The current code doing the evaluation has been through various rounds of patching and is due for an overhaul. Before we get into the actual code it is however important that we define the rules clearly. We are currently lacking clarity regarding what data should be entered by users and how it is to be interpreted during the evaluation process.

For instance, users can currently enter "license not specified" for a given component but the evaluation will then count that component as missing, when it is typically interpreted by users as meaning: "the component is included but no license for that component was specified". There is also the question of what users should enter, when the model distribution only contains a single license. Should that license be entered for all components that are included (which may not be valid for some types of components) or enter the license as not specified?

I think we are dealing with different dimensions that are currently being conflated and need to pulled apart, to support something like this:

1. is the component included in the distribution? If not, the component is missing.
2. does the component have a license? If not, the license is missing.
3. is the license an open license? If not, the license is invalid.
4. is the license a type-appropriate? If not, the license is not appropriate.
   otherwise the component is included, has a type-appropriate open license.

From a data entry point of view, we should document that in case of a single license it applies to all components that are included. Of course if we add global licenses per proposal #72 this could be implemented in different ways so it might be worth figuring this out first.

[SachinVarghese]

Thanks for raising this. And I 100% agree that we should define the component states and license states more clearly also as part of the spec. Global license is an ideal state that may take a while before adoption, so we should improve the current issue as a short term goal.

[lehors]

Having thought about this some more, I think the following approach would work.

Assuming per proposal #72 we are adding 4 different "global/distribution-wide" licenses:

Global license - this could eventually be Matt's OpenMDW license but for now it's often just something like Apache-2.0.
Global license for code
Global license for doc
Global license for data

The evaluation of a component then goes like this:

Is the component included?
If yes, is there a license attached to the component? If yes, use that.
If not, is there a type-specific global license for the component? If yes, use that.
If not, is there a global license? If yes, use that.
If not, there is no license.

Then:
Is the license a type-appropriate open license? If yes, ok.
If not, is the license an open license? If yes, ok but display warning.
If not, the license is not open. (no license is not open btw)

We also need to take into account a few special cases:

If paper is included, tech report is optional
The paper may be replaced by a detailed tech report (easiest is probably to direct user to claim paper is included in this case)
Evaluation results may be included in tech report, research paper, or model card (in any of those cases, evaluation results should be marked as included)
If model card contains data info, data card is optional (easiest is probably to direct user to claim data card is included in this case)

One final note: to identify components not included we currently use the license name set as "Component not included". I think we should simply add a separate field: included true/false.

[SachinVarghese]

I would propose a "License Unknown" option to cover the edge cases whena license is not provided - rather than leaving empty

[lehors]

As discussed yesterday on the call I can't really think of a case where we would actually need this. If the license isn't specified it is inherited as above or not set at all in which case it's not just unknown, there is just not one and we should process it accordingly.
If it's just a way to put some kind of marker that this info might need further investigation I would rather have some kind of marker such as "unknown" or "tentative", but we then should have this on other fields too and I'm not sure this is worth the additional complexity at this point.

[SachinVarghese]

Yes, I wasn't suggesting to have an additional marker, that would be an overhead/complexity best avoided as you pointed out. We are in agreement there.

I'm just opposing the field value to be empty in any case - it is a protocol best paractice in to specify the definition of even the empty value using a "unspecified" or "unknown" tag in a way that logically makes sense for us. This helps in future versioning changes too as these definitions evolve.

[lehors]

Interesting different point of view.
I'd actually argue we should free the yaml file of all the fields that have no value/an empty value. I feel like all these empty fields are just adding clutter that makes reading and maintaining the yaml files more complicated than it ought to be. So, I'd be in favor of going further the other way. :-)

[SachinVarghese]

Ok to clarify my perspective, I'm not opposed to optional values, just that empty values are not great in my opinion. We can have optional values that defaults to a certain association. This will remove the clutter for human readability but all values should have a well defined association.

[SachinVarghese]

I have created a flow chart to describle this using https://mermaid.live/.

flowchart TD
    A(Start) -->C{Global License Available?}
    C -->|Yes| D{Appropriate Global License?}
    C -->|No| E{Other Licenses Available?}
    D -->|Yes| H[Select Global License]
    D -->|No| I[Inappropriate License Warning]
    E -->|No| G["fa:fa-check License Selection Complete"]
    E -->|Yes| F[Select MOF Component]
    F --> J{Is Component Included?}
    J -->|Yes| K{License Attached to Component?}
    J -->|No| L[Display Warning]
    L --> E
    K -->|No| N{Type specific Global License Available?}
    K -->|Yes| M{Type appropriate Component License?}
    M -->|No| Q[Inappropriate License Warning]
    M -->|Yes| R[Select Component License]
    N -->|No| O["fa:fa-xmark No Component License Available"]
    N -->|Yes| P{Type appropriate Global License?}
    P -->|No| S[Inappropriate License Warning]
    P -->|Yes| T[Select Type Global License]
    H --> E
    I --> E
    O --> E
    Q --> E
    R --> E
    S --> E
    T --> E
    G --> U(End)

[matthew-d-white]

This looks great, my only addition would be to distinguish between "license available" and "open license available".

In the MOF if the license is not an open license globally or per component(s), then the component(s) covered under that license fail to qualify(even if they are included.)