completeness updates via claude

[mshadbolt]

Enlisted claude to try to provide updated completeness statistics using the latest crtieria from MOHCCN.

Overall I believe it did a pretty good job and I don't think it affected any current functionality.

Summary statitistics are saved into the map.json file like this:

"completeness_summary": {
            "total_donors": 13,
            "tier_a_min_clinical_complete": 3,
            "tier_b_min_clinical_complete": 3,
            "incomplete_min_donors": 7,
            "tier_a_full_clinical_complete": 2,
            "tier_b_full_clinical_complete": 2,
            "incomplete_full_donors": 9
        }

And a summary of incomplete donors is saved into <foldername>_completeness_failures.json and provides a summary for each failed donor and why it failed like:

{
            "donor_id": "CMPLT_BM",
            "tier": "B",
            "minimal_complete": true,
            "fulsome_complete": false,
            "reasons": [
                "Fails fulsome clinical completeness: 2 required/conditionally-required field(s) missing"
            ],
            "sample_counts": {
                "normal_dna": 1,
                "tumour_dna": 1
            },
            "minimal_unmet": [],
            "fulsome_unmet": [
                "donors[CMPLT_BM] > primary_diagnoses[PD_BM] > specimens[SPEC_BM_T].specimen_storage",
                "missing required object: treatments (found 0, need >= 1)"
            ]
        },

I got it to add new test data to the test csvs to test the conditions and it updated existing tests where needed

[kcranston]

Was the general strategy to ask Claude to move the code / logic from the candig-analytics to this repo?

[mshadbolt]

No, I asked it to add the completeness checks but didn't give it the context of the other repo. All the logic that is required is already in clinical etl so it wouldn't have made sense to get it from the other repo. I think the way it did it was fairly logical and I think it will be easy to maintain since it works off the validation checks we already do in the mohv#schema files

[daisieh]

I didn't look over the code extremely thoroughly, but if it works for you, it looks fine to me.