Alternate strategy for cross requesting epochs with wfcatalog database

[jschaeff]

For now, there is a background worker that creates on a daily basis another database in mongodb with connections to the metadata epochs as provided by ws station.

This has some drawbacks in a production system:

• if a worker fails, weird things can happer
• there is a 1 day lag on metadata changes
• the resulting database may be not well optimized for some requests

The expected work is just to understand the actual system and imagine better ways to do the job.

[NikolaosSokos]

Proposed plan:
Old: cron $\rightarrow$ mongosh $\rightarrow$ main.js (blind execution/hardcoded dates).
New: Docker $\rightarrow$ Python $\rightarrow$ PyMongo + Sentry (monitored/state-driven sync).

[jschaeff]

Another option would be to schedule refresh with celery jobs

[petrrr]

Hi @jschaeff!

We also had a look a this "two stage process" and are not really happy with the way this is currently implemented.

Our starting point here is that each file should "inserted" or "checked" by a controlled process (policy) into the archive, i.e. the file is not only copied into the archive but also processed and related metadata are updated. The current process breaks this logic. We manage the WFcatalog update by a policy on archiving. However, the avalability update is currently done by an independent cron job or it needs to be trigered manually for file which are reprocessed.

Our idea here was to actually update also the availability collection along with the WFcatalog collections. This way the analisis of files would still be executed only once, but the insert to all collections (WF and avaliablity) would be done by the python process.

Do you see any disadvantage with this approach?

UPDATE:
Just to make clear, it would actually be the WF collector to do the complete operation.

[jschaeff]

Look also for a way to schedule the refresh task from inside of the app.

For instance: https://github.com/dbader/schedule or https://apscheduler.readthedocs.io/en/latest/