Deterministic agent-guided scarf analysis

[Gautam8387]

Summary

This PR introduces a bounded, agent-guided workflow for one RNA assay. analyze_rna() combines study context with measured dataset evidence to:

• ingest and inspect the dataset;
• establish QC and experimental-design constraints;
• select registered preprocessing choices;
• compare HVG, PCA, neighborhood, clustering, and optional Harmony alternatives;
• validate selected settings on the full retained cohort;
• persist exact artifacts, decisions, and limitations;
• return the final UMAP, clusters, markers, and an evidence-based HTML report.

The model cannot generate arbitrary analysis code or call unrestricted DataStore operations. It selects from registered choices whose execution and validation remain controlled by Scarf.

Architecture at a glance

flowchart LR
    API["analyze_rna()"] --> O["AgentOrchestrator"]

    O --> C["Bounded model control plane"]
    C --> V["Typed outputs<br/>closed choices<br/>validation gates"]
    V --> E["Scarf scientific executors"]

    O --> J["Append-only workflow journal"]
    E --> A["Immutable Zarr artifacts"]
    J --> A

    A --> R["AutomatedWorkflowResult"]
    R --> U["Final UMAP"]
    R --> M["Markers"]
    R --> H["HTML report<br/>no new model calls"]

Loading

What deterministic means

"Deterministic" describes the execution and replay boundary around the model:

• Numerical operations use fixed seeds and frozen cell and feature selections.
• Requests, configuration, input data, model identity, evidence, and decisions are checksummed.
• Agents choose from closed option sets. Deterministic validators reject unsupported or insufficiently evidenced choices.
• Scientific results are stored as immutable, provenance-addressed artifacts.
• Write-once journal checkpoints make committed decisions and work resumable.
• Repeating an identical call against the same destination reuses or resumes the committed workflow.

This does not guarantee identical LLM output on a new workflow. Temperature zero and seed 4444 are requested from the provider, but providers may still produce different valid responses. A new destination can therefore follow a different valid branch. The strongest claim is deterministic execution and replay after a decision has been committed, not bit-identical LLM reasoning.

Agent and model inventory

Scarf does not select or construct a provider model. The caller supplies one Pydantic AI model, which is shared across the workflow. Each model invocation creates a short-lived Pydantic AI agent with bounded requests, tools, tokens, retries, and timeout. Thinking is disabled and tool calls are sequential by default.

The automated workflow uses:

• Conditional ingest decision: resolves ambiguous assay modality.
• DataEnrichmentAgent: inspects species, feature families, controls, and assay roles.
• ExperimentalContextAgent: establishes captures, independent units, covariates, confounding, QC evidence, and whether Harmony is scientifically licensed.
• RNA decision agents: select registered QC grouping and cell-quality policies.
• RNA assessment agents: review screening and full-cohort alternatives using quantitative evidence and, when supported, diagnostic images.

AgentOrchestrator and most of RnaTuningRun are controllers and executors, not LLM agents.

Two domain agents remain standalone and are not part of analyze_rna():

• ParameterTuningAgent
• BiologicalInterpretationAgent

Agent hierarchy

flowchart TB
    M["Caller-supplied Pydantic AI model"] --> O["AgentOrchestrator<br/>deterministic stage controller"]

    O --> I["Conditional ingest decision"]
    O --> D["DataEnrichmentAgent"]
    O --> X["ExperimentalContextAgent"]
    O --> Q["QC grouping and cell-quality decisions"]
    O --> T["RnaTuningRun"]

    T --> S0["Screening assessment"]
    T --> S1["Optional enlarged-screen assessment"]
    T --> F["Full-cohort assessment"]

    I --> K["Shared model execution and validation"]
    D --> K
    X --> K
    Q --> K
    S0 --> K
    S1 --> K
    F --> K

    subgraph Standalone["Standalone APIs"]
        P["ParameterTuningAgent"]
        B["BiologicalInterpretationAgent"]
    end

Loading

Automated workflow

flowchart TD
    A["Source, model, study context, objective"] --> B{"Matching journal exists?"}
    B -->|Yes| R["Resume or reuse committed work"]
    B -->|No| I["Validate and ingest source"]

    I --> E["Data enrichment<br/>model-guided"]
    E --> Q["RNA QC metrics<br/>deterministic"]
    Q --> C["Experimental context<br/>design and correction license"]
    C --> P["Preprocessing decisions and execution"]
    P --> S["Seeded screening cohort<br/>10%, bounded to 10k-100k"]

    S --> T["Compare registered alternatives<br/>HVG, PCA, neighbors, partitions"]
    T --> D{"Agent assessment"}
    D -->|Combine or experiment| T
    D -->|Enlarge| S2["One larger nested screen"]
    S2 --> T
    D -->|Defer| N["Pause or fail with unresolved evidence"]
    D -->|Accept| F["Full-cohort validation"]

    F --> H["Matched native/Harmony evaluation<br/>when correction is licensed"]
    H --> Z["Finalize graph, clusters, UMAP, and markers"]
    Z --> G["Generate report from saved evidence"]

Loading

Screening compares 1,000, 2,000, and 4,000 variable genes; 10, 21, and 30 PCs; 11, 21, and 41 neighbors; and four Leiden resolutions. Selected settings are then executed and assessed on the complete QC-retained cohort.

Import: XXX cells
└─ Enrichment and experimental-context agents: completed
   └─ QC decisions: completed
      └─ Retained cohort: YYY cells
         ├─ Full-cohort gene statistics
         │  └─ Each of N libraries
         │     └─ Select its cells → calculate gene variability
         │        └─ Aggregate batch-aware ranking
         └─ Screening population: (10,000 <= 10% of XXX <= 100,000) cells
            └─ 12 candidate analyses: completed
               ├─ PCA, neighbors and clustering
               ├─ Markers, stability and covariate diagnostics
               └─ Tuning agent assessment: CURRENT
                  └─ Combined settings → full-cohort validation → report

Current limitations

• Doublets: score_doublets=False skips scoring when Harmony is unavailable or prohibited. Harmony-eligible workflows still calculate matched doublet evidence. Scores are advisory and never remove cells.
• RNA only: one RNA assay is analyzed. Paired modalities, WNN/SNN integration, and multimodal tuning are outside this workflow.
• No automatic HTO demultiplexing or assignment.
• Descriptive analysis only: markers and population structure are produced, but differential expression, significance testing, abundance inference, treatment-effect estimation, and causal claims are excluded.
• No final cell-type annotation: the standalone biological interpretation agent is not wired into the automated workflow.
• Harmony is conditional: confounded or insufficiently characterized designs cannot license correction and remain native or unresolved.
• Bounded search: screening, graph, partition, repair, model-request, and tool-call limits can cause the workflow to defer rather than force a result.
• Rare populations may be underrepresented during screening. One enlarged screen and one targeted full-cohort repair are available, but search is intentionally not unbounded.
• Image review depends on model capability. Unsupported providers fall back to structured evidence and record that limitation.

Future doublet-removal path

A removal workflow should:

1. Resolve the physical capture identity and define an explicit capture-aware removal policy.
2. Persist the removal decision and supporting score artifacts.
3. Create a new immutable singlet cell selection rather than mutating live metadata.
4. Re-run normalization, graph construction, tuning, and finalization on that selection.
5. Recalculate matched native and Harmony evidence on the same post-removal cells.
6. Report removed and retained counts by capture and preserve the original unfiltered branch.

A global threshold should not be applied silently because score distributions and expected doublet rates depend on capture conditions.

Accuracy notes found during review:

• The API reference says the default screen is 50,000 cells, but the implementation uses 10% bounded to 10,000 through 100,000.
• The notebook says doublet scoring is enabled, but its analyze_rna() call leaves score_doublets=False; scoring is therefore conditional on Harmony eligibility.
• The notebook mentions execution_status.json, but Scarf itself does not create that file.

[codecov]

Codecov Report

❌ Patch coverage is 97.18488% with 111 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...carf/agent/biological_interpretation/validation.py	92.01%	19 Missing ⚠️
scarf/agent/cell_quality/profiles.py	96.07%	13 Missing ⚠️
scarf/agent/decisions/kernel.py	96.03%	13 Missing ⚠️
scarf/agent/experimental_context/comparisons.py	95.18%	13 Missing ⚠️
scarf/agent/cell_quality/execution.py	96.80%	8 Missing ⚠️
scarf/agent/data_enrichment/contracts.py	96.59%	8 Missing ⚠️
scarf/agent/experimental_context/contracts.py	98.62%	7 Missing ⚠️
scarf/agent/biological_interpretation/tools.py	97.36%	6 Missing ⚠️
scarf/agent/experimental_context/agent.py	96.15%	5 Missing ⚠️
scarf/agent/biological_interpretation/agent.py	90.47%	4 Missing ⚠️
... and 7 more

📢 Thoughts on this report? Let us know!