Track Active Observation

[westei]

For tracking expected data (#353) we need to persist actual observation instances for study participants.

The currently modeled observations are configurations that can be used to calculate what actual observations each participant of a study is expected to perform. This information is sufficient for the study manager and the participant application to materialize observations on demand, but as we start to track information for actual observation instances like the data health and need to perform actions on those information like sending alerts, we need a way to persist information for observation instances.

Additionally if studies are paused and re-configured one could end up in a situation, where existing observation data does no longer confirm with the changed configuration.

Provide Storage and Access for ParticipantObservations

A ParticipantObservation is an occurrence of an observation for a study of an study participant. It is

• linked to a study (direct or indirect)
• to an participant of that study
• to an configured observation of that study
• it has an starting timestamp that is <= the current time
• it has an end timestamp
• it has an data state (missing, partial, incomplete, invalid, complete)
• it has an temporal state (active, complete) - transient - calculated based on start/end timestamps
• it can store or link to actions performed (e.g. Notifications)

ParticipantObservationService to:

• CRUD ParticipantObservations
• Upsert by ObservationId, ParticipantId, startTimestamp and endTimestamp
• Query for ParticipantObservation by data state
• Update the data state of a ParticipantObservation

Periodic Job to create ParticipantObservation

We need to periodically create ParticipantObservation instances for Participants based on the current time. For that one needs to iterate over all participants of an study, materialize observations based on the current configuration and create ParticipantObservation instances for active Observations that do not already exist. The ParticipantObservationService shall provide a special upsert method for this task.

Periodic Job to validate the Data Health of ParticipantObservation

We need to periodically validate data health for ParticipantObservation that are not yet in the data states completed or invalid. For such ParticipantObservations this component needs to

• retrieve data from the elastic index
• if no data are available set the data state to missing
• use the Observation module to check the validity of the present data. This module can set the state to
  • partial: if some data are missing (e.g. if temporal data started late
  • invalid: if the available data are not as expected
  • incomplete: if all data are available and valid, but the observation is ongoing and therefore additional data are expected
  • complete: if the available data is valid and complete

NOTE: The validation of received data is the responsibility of the Observation Module. The Observation code needs to provide a functionality to validate received data. A default implementation that simply validates the existence of any data shall be provided as a reasonable default for modules that do not provide this functionality

In addition this component needs to create Notifications for ParticipantObservation that do not have valid data. For that a configuration shall be created that allows to configure after what timespan notifications are triggered.

TODO: We might store Notifications in an own database table - especially when we want to track the status of notifications of if we want to remove unread notifications if data does become available after a notification was already created.

[westei]

Update: Final name is OccurredObservation - an observation that occurred in a study. The identification (PK) is based on the studyId, observationId, participantId and start timestamp of the Observation. The end is not part of the primary key, as if an study is paused during an ongoing observation and a configuration change results in the end to be updated one wants to have the possibility to update the end time without loosing state information.

Every OccurredObservation has

• dataValid this starts with true and can be set to false if it is detected that the received data are not valid
• dataState with the possible values
  • missing ... no data available (initial state)
  • incomplete ... some data is already recoded, but further data is expected (e.g. the observation is still ongoing
  • partial ... the data is only partially available, but the observation has already completed - not sure if we need this state or if incomplete is sufficient
  • complete all the expected data is available
• properties allows to store any JSON data with the OccurredObservation. This might be used to represent observation type specific state

[westei]

NOTES:

it can store or link to actions performed (e.g. Notifications)

This part was not implemented. The Participant Endpoint does however provide a data health indicator (Green, Orange and Red). The new OccurredObservation endpoint allows to query for OccurredQbservations based on Study, Participant and/or Observation.

Those services endpoints are used by the current data health UI/UX. This might be extended by later issues