Devise/Improve validation methods for `SpatialData` and other methods

[Artur-man]

Validation in scverse/spatialdata is performed for each SpatialElement type individually:
https://spatialdata.scverse.org/en/latest/tutorials/notebooks/notebooks/examples/models2.html#data-models-validation-and-parsing

e.g.

https://spatialdata.scverse.org/en/stable/api/models.html#spatialdata.models.PointsModel

This can be performed with either s4 or s7 validators:
https://cran.r-project.org/web/packages/S7/vignettes/classes-objects.html

[Bisaloo]

Posting here to ensure this doesn't get lost: https://github.com/huber-group-embl/SpatialData.Validate.

It doesn't have to be a standalone package and can be merged into SpatialData if you wish. I don't have a strong preference myself.

This is only a partial solution at this point because:

• it only checks the images folder. The schema could be extended to include all folders in a SpatialData object (easy improvement).
• it only checks conformance to a schema and there is "dynamic check". E.g., that the number elements in a specific slot match the number of the same elements in another slot. This would be a more complex improvement because it cannot be checked with the schema.

[Artur-man]

Thanks @Bisaloo, I think we can slowly start implementing these into SpatialData but I am also ok with if you wanna keep them under SpatialData.validate() until you are confident with the functionality, however is more comfortable with you.

We need to discuss a few things:

• schemas do not exist in the original spatialdata implementation @LucaMarconato, thus we gonna need to write all of em ourselves for each version. Formatting has been scripted here if you need more information: https://github.com/scverse/spatialdata/blob/d024c0591bb925c81642702db781f113102b9ece/src/spatialdata/_io/format.py

• There is another limitation which @Bisaloo also described, schema cannot check if, e.g., the number of dimensions match with the ones in the arrays. One can first go through the schema, then apply additional checks. This is pretty much the approach taken in most OME-ZARR implementations (some do not even use schema at all).

• We already have some validation here https://github.com/HelenaLC/SpatialData/blob/main/R/validity.R, but these are really basic and was mostly written based on examples v2 and v3 datasets in the package.

I am co-assigning this to @Bisaloo.

[LucaMarconato]

Thanks for the update. The schemas could also be useful for the Python version. One option could be to use them as the source of truth for the SpatialData format and complement them with natural language when the schema is insufficient. This approach was discussed by the NGFF community, and multiple people agreed on it. However, some NGFF members (me included) expressed a preference for natural language as the source of truth for the specification; I'll expand on this next.

The :natural language as source of truth approach" would consist in replacing https://github.com/scverse/spatialdata/blob/main/docs/design_doc.md with a better versioned natural language specification for the SpatialData format (which will not be long, since the heavy lifting is done by NGFF 0.6). Then, we could use an LLM CI job to ensure consistency between the natural language spec and the schema. This way, we still use the schemas as potential additional aids, but we keep natural language as the entry point.

Finally, another option: @melonora might suggest going directly with LinkML—it may work, or it may be overkill, but we can try both.

[melonora]

The LinkMl path would also connect it to the semantic web, I believe @joshmoore also has an interest in that. It would take longer to establish though.

[Artur-man]

May not be related but while I was working on #182 and #185, I realized that we still do not validate/compare names of the zarr arrays under the multiscale image/label zarr group together with ome-zarr/sd attributes (e.g. how do we know the number of arrays match with # of datasets in attr).

Anyways something to consider. I know some OME-ZARR implementations to these multi-object validations but can't remember now.

[LucaMarconato]

A good list of existing validators is here: https://github.com/imaging-formats/yaozarrs#existing-projects