Provenance Metadata Pattern: Recommendations for PR #2711

[turbomam]

PR:

• 2710 add slot to biosample to track metadata origin #2711

Issue:

• Add slot to Biosample to track metadata origin #2710

What's Good About the Current PR

✅ SourceSystemEnum (basic_classes.yaml)

• Correct use of controlled vocabulary
• Good descriptions with see_also links
• Extensible (can add more systems later)

✅ source_system_of_record slot (basic_classes.yaml)

• Clear description
• Appropriate range constraint
• Solves the immediate need from issue Add slot to Biosample to track metadata origin #2710

✅ provenance_metadata slot (basic_classes.yaml)

• Follows existing pattern (like has_credit_associations)
• Simple, reusable across classes

✅ Biosample integration (core.yaml)

• Minimal, non-disruptive addition
• Sets foundation for extending to other classes

✅ Test data (Biosample-invalid-source-system.yaml)

• Correctly validates enum constraints

---

Suggested Enhancement: Add slot_usage to ProvenanceMetadata

Why?

The PR reuses existing slots (git_url, version) which currently have descriptions specific to WorkflowExecution. Without slot_usage, readers may be confused:

• Does git_url point to workflow code or translator code?
• Is version a workflow release or a translator release?

This is the same challenge NMDC faced with add_date - it was defined generically but only used for GOLD, causing confusion (see issue #8).

Recommendation: Use slot_usage Pattern

The schema already uses this pattern in WorkflowExecution (basic_classes.yaml:491-510):

WorkflowExecution:
  slots:
    - git_url
  slot_usage:
    git_url:
      required: true  # contextual refinement

Proposed Addition to ProvenanceMetadata Class

Add to basic_classes.yaml (lines 512-519):

  ProvenanceMetadata:
    description: Metadata pertaining to how a record was created.
    class_uri: nmdc:ProvenanceMetadata
    comments:
      - This class tracks the provenance of record creation, including which source system contributed the data and what translation/processing software was used.
      - This is distinct from WorkflowExecution provenance, which tracks bioinformatics analysis workflows that process data.
      - The ProvenanceMetadata pattern can be applied to any class by adding the provenance_metadata slot.
      - See issue #2710 for background on the need to track metadata origin for curation workflows.
    slots:
      - git_url
      - version
      - source_system_of_record
      - type
    slot_usage:
      git_url:
        description: The URL pointing to the code repository and version of the software that translated or created this record.
        comments:
          - For records created by NMDC translators, this points to the translator code repository and file path
          - This is distinct from WorkflowExecution.git_url, which points to bioinformatics workflow code (e.g., metaG assembly pipeline)
          - Should point to a specific commit, tag, or file path when possible for reproducibility
        examples:
          - value: "https://github.com/microbiomedata/nmdc-runtime/blob/main/nmdc_runtime/site/translation/submission_portal_translator.py"
          - value: "https://github.com/microbiomedata/nmdc-runtime/tree/v2.12.0"
      version:
        description: The version tag of the software that translated or created this record.
        comments:
          - For NMDC translators, this is typically the nmdc-runtime release tag
          - For external systems (GOLD, NEON), this represents the source system's version or export date
          - This is distinct from WorkflowExecution.version, which tracks the bioinformatics workflow release (e.g., "v1.0.5" for the assembly workflow)
        examples:
          - value: "v2.12.0"
          - value: "2024-12-15"
      source_system_of_record:
        required: true
        description: The originating system from which this record's metadata was obtained.
        comments:
          - Required to enable data curation workflows that need to track metadata provenance
          - Identifies the authoritative source for this metadata (see issue #2710)
          - This is distinct from processing_institution, which identifies who performed bioinformatics analysis

---

Why slot_usage Instead of New Slots?

❌ Don't Do This (slot proliferation):

slots:
  translator_git_url:
    description: ...
  workflow_git_url:
    description: ...
  pipeline_git_url:
    description: ...

✅ Do This (slot_usage refinement):

ProvenanceMetadata:
  slots:
    - git_url  # generic, reusable
  slot_usage:
    git_url:
      comments:
        - This is distinct from WorkflowExecution.git_url...

Benefits:

1. Avoids dozens of near-identical slots
2. Follows LinkML best practice (slot_usage is designed for this)
3. Aligns with OBO philosophy (general predicates, contextual usage)
4. Already used in NMDC schema (see WorkflowExecution)

---

The "This is Distinct From" Comment Pattern

These comments prevent semantic confusion when slots are reused:

git_url:
  comments:
    - This is distinct from WorkflowExecution.git_url, which points to bioinformatics workflow code

source_system_of_record:
  comments:
    - This is distinct from processing_institution, which identifies who performed bioinformatics analysis

Why it matters:

• Developers understand contextual differences
• Auto-generated docs inherit distinctions
• Validation guidance is clearer

---

Summary of Suggested Changes

File	Current	Suggestion	Why
basic_classes.yaml	ProvenanceMetadata has slots only	Add slot_usage block + comments	Clarifies semantic context
core.yaml	✅ Good as-is	No change	Minimal, correct integration
basic_classes.yaml	✅ SourceSystemEnum	No change	Well-structured enum
Test data	✅ Good coverage	No change	Validates constraints

Impact: Establishes a clear, reusable pattern that demonstrates best practices for future schema development.

---

Questions for Discussion

1. Do the "this is distinct from" comments add clarity without being too verbose?

2. Should we extend ProvenanceMetadata to other classes now (Study, DataGeneration) or wait?

3. Is nmdc:ProvenanceMetadata the right class_uri, or should we search harder for OBO mappings?

---

Related Issues/PRs

• #8 - Historical discussion about GOLD add_date/mod_date
• #884 - Datetime field consistency
• #1889 - Tracking provenance of schema artifacts
• #1348 - Fixed "provenance" spelling

[turbomam]

Existing Provenance Patterns

Pattern 1: WorkflowExecution (Computational Provenance)

Location: basic_classes.yaml:446-511

Purpose: Track bioinformatics workflow execution

Key slots:

• started_at_time, ended_at_time - Temporal bounds
• git_url, version - Workflow code provenance
• execution_resource - Computing facility (NERSC, etc.)
• processing_institution - Organization that ran workflow
• was_informed_by - Input data that triggered workflow

Example: MetagenomeAssembly tracking when/where/how reads were assembled

Distinction: WorkflowExecution tracks data processing, ProvenanceMetadata tracks metadata record creation

---

Pattern 2: CreditAssociation (Human Agent Provenance)

Location: basic_classes.yaml:160-175

Purpose: Link human contributors to Studies using CRediT taxonomy

Structure:

CreditAssociation:
  class_uri: prov:Association
  slots:
    - applies_to_person
    - applied_roles

Study:
  slots:
    - has_credit_associations  # multivalued, inlined

Example: Tracking PIs, data curators, and other contributors

Distinction: CreditAssociation tracks human contributions, ProvenanceMetadata tracks system/software

---

Pattern 3: GOLD Legacy (Deprecated Temporal Provenance)

Location: basic_slots.yaml:333-339

Slots: add_date, mod_date (both untyped strings)

Applied to: Biosample, DataGeneration

Problems (see issues #8, #884):

• Untyped strings vs. datetime values
• GOLD-specific, not generalizable
• No indication of who or what system
• Inconsistent with other datetime fields

Why this matters: Shows the pain of not using slot_usage. These slots are defined generically but only make sense in GOLD context, causing confusion.

---

Complete Provenance Coverage by Class

Class	Creation Time	Source System	Version/Git	Human Agent	Coverage
WorkflowExecution	✅ started_at_time	✅ processing_institution	✅ git_url, version	❌	~70%
Biosample (with PR #2711)	❌	✅ source_system_of_record	✅ git_url, version	❌	~40%
Biosample (GOLD legacy)	⚠️ add_date (string)	❌	❌	❌	~15%
DataGeneration	⚠️ add_date (string)	❌	❌	❌	~15%
Study	❌	❌	❌	✅ has_credit_associations	~20%
DataObject	❌	❌	✅ was_generated_by	❌	~20%

What's Still Missing (Future Work)

1. Modification tracking (who/when/what changed)
2. Agent attribution (human vs. automated)
3. Change history / audit log
4. Database-level provenance

Not blockers for this PR - just noting the gaps.

---

NMDC's Move Away from PROV-O

Historical Context

The schema originally used W3C PROV-O vocabulary (prov:wasGeneratedBy, prov:Association). NMDC is shifting toward OBO Foundry ontologies (BFO, OBI, IAO).

This PR's Approach

• Uses class_uri: nmdc:ProvenanceMetadata (not prov:)
• No forced PROV-O mappings
• Appropriate for NMDC-specific semantics

Why not force OBO mappings?

• IAO/OBI don't have exact equivalents for "metadata record creation provenance"
• NMDC-specific concepts can legitimately use nmdc: namespace
• Could add see_also to conceptually related OBO terms later

---

Future Extensibility

Short-term (Next 1-2 PRs)

Extend to other classes:

Study:
  slots:
    - provenance_metadata

DataGeneration:
  slots:
    - provenance_metadata

Medium-term

Deprecate GOLD legacy:

add_date:
  deprecated: true
  deprecated_element_has_exact_replacement: created_at

Long-term

Add modification tracking:

Biosample:
  slots:
    - provenance_metadata        # creation (single)
    - modification_history        # changes (multivalued)

---

References

Issues

• #2710 - Add slot to Biosample to track metadata origin
• #8 - Do we keep GOLD add_date and mod_date?
• #884 - Consistency among all datetime fields
• #1889 - Ensure origin of assets files can be determined

Pull Requests

• #2711 - Current PR
• #1348 - Fix spelling of "provenance"

Documentation

• LinkML slot_usage
• W3C PROV-O
• OBO Foundry
• CRediT taxonomy