feat(schema): metadata field for dataFiles

[abdellah257]

Description

a metadata field for dataFile, to include file related variables

[
  {
    "dataFileList": [
      {
        "path": "string",
        "size": 0,
        "time": "2026-04-14T12:05:20.052Z",
        "chk": "string",
        "uid": "string",
        "gid": "string",
        "perm": "string",
        "type": "string",
        "metadata": {}
      }
    ]
  }
]

Motivation

In case of large number of files in a dataset it's very useful to collect file specific metadata to help seperate between them.
The following is a use case example:

Changes:

additional field metadata in the dataFile DTO, schema and interface

Tests included

• Included for each change/fix?
• Passing?

Documentation

• swagger documentation updated (required for API changes)
• official documentation updated

[HayenNico]

This feature was already proposed last year in this PR: #1967. For the same reason as that time, I think we should not allow arbitrary metadata in multiple places to avoid ambiguity between the responsibilities of Dataset and DataFile.

[abdellah257]

This feature was already proposed last year in this PR: #1967. For the same reason as that time, I think we should not allow arbitrary metadata in multiple places to avoid ambiguity between the responsibilities of Dataset and DataFile.

I see, thank you for referencing that issue, it helps bringing back that discussion, as I can get to the same point my predecessor got to at the end. I' am very open to other suggestion, but as of now it's the only option I can see.
I can also argue that the point of conflicting a Dataset and a DataFile is not clear especially at ILL, due to how general a Dataset is, it can have up to 64 000 files, with a set of these having a very specific function for later processing ....
Including this information in a Dataset as a map<file, metadata> at scientific_metadata is also not a viable option due the sizes of our datasets, and also conflicts with the purpose of scientific_metadata in the first place.

[Junjiequan]

The changes look fine to me. The concern around unclear responsibilities between dataset metadata and datafile metadata makes sense, but I think it's mainly an issue if we assume a 1:1 relationship between datasets and datafiles, which I dont think is the case for everyone. To me, datafile metadata feels like a good fit for file specific details.

That being said, I'm not ignoring risks of users mixing the responsiblities between dataset and dataFile. I just think there's more practical way to address it without restrcting the change, we could:

• Add a clear field description that indicates datafile metadata is not a substitute for dataset metadata
• Document use case boundries for each

I'd love to discuss more about the potential risks, but my main point is that optional dataFile metadata is a good addition

[nitrosx]

I understand the worries to have metadata both in datasets and files, but I think that there is value to have metadata associated with a specific file, specifically when your datasets contains multiples files.
A clear documentation with examples should be able to address all the questions from how to use it, to data curation.
Also, we clearly have a use case from a supporting facility and it will not impact existing deployments if the feature is not used or, even better, disabled.

I see ESS using this feature in the near future, specifically for derived dataset.
I support the feature. Of course, I will be more than happy to discuss further the topic, if that help clarifying and dissipate the worries expressed above.

[Junjiequan]

If possible, please wait for the merge until the next scicat meeting. There might be different opinions and suggestions