Skip to content

Fix inspection mapping quick index flow - #463

Draft
DylanWelzel wants to merge 1 commit into
1.2.xfrom
agent/fix-inspection-quick-index
Draft

DylanWelzel wants to merge 1 commit into
1.2.xfrom
agent/fix-inspection-quick-index

Conversation

@DylanWelzel

@DylanWelzel DylanWelzel commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Problem

During 1.2.x testing on free-threaded Python 3.14t, source mapping inspection completed its batch workers, but the subsequent PUT /flatten_inspection_data request failed with:

KeyError: 'inspect'

After inspecting and saving the generated mapping, a quick index that had previously worked then failed with:

No such builder for 'geneinfo_configuration_<timestamp>_<random>'

These are two related state-handling problems. The inspection failure leaves the mapping workflow in a bad state, while saving a mapping can expose incomplete source-master metadata used by quick index. The reported quick-index error is also misleading because the temporary builder was created successfully.

Is inspection itself unsafe in free-threaded mode?

There is no evidence in this trace that the free-threaded workers corrupt or concurrently merge inspection data.

Each worker computes an isolated batch result. batch_inspected() awaits that result and merges it into the aggregate on the asyncio event-loop thread. Those merge callbacks therefore remain serialized even when defer_to_process() is backed by the free-threaded ThreadPoolExecutor. The surrounding TaskGroup is awaited before final Elasticsearch mapping generation and before the inspection status/results are persisted.

The Remove PID file ... FTWorker_* messages only mean that each worker function has returned. They do not mean that event-loop aggregation, mapping generation, metadata conversion, or the final status write has completed. Consequently, a flatten request may still observe an inspection job in an intermediate state with a status such as inspecting but without inspect.results yet.

The same incomplete state can also exist after a failed/canceled inspection or in another subsource job stored in the same source document. flatten() iterates all of those jobs and previously assumed every entry had a completed result. Free-threaded execution makes this timing easier to encounter, but the unsafe assumption is not specific to the free-threaded executor or to TaskGroup.

This PR does not change worker scheduling, asyncio aggregation, or inspection calculation. It makes the consumer tolerate valid non-complete job states while preserving completed results.

Inspection fix

For source inspection, InspectorManager.flatten() now:

  • reads inspect.results defensively;
  • omits jobs that have no completed result instead of raising KeyError;
  • logs the skipped source job and its current status at debug level; and
  • continues returning completed results from other jobs in the same source document.

Build inspection is unaffected. It already reads a single build result with .get(..., {}) rather than iterating source job records.

No empty or fabricated mapping is returned for an in-progress/failed job; that job is simply absent from the flattened response until a completed result exists.

Why quick index fails only after the mapping workflow

Quick index creates a temporary build configuration and asks DataBuilder.resolve_sources() to resolve each configured source through src_master.

A normal upload creates a complete master document containing both _id and name, so quick index works before inspection. However, when a generated inspection mapping is saved to dest="master" and no master document exists yet, SourceManager.save_mapping() previously created only:

{"_id": subsource, "mapping": mapping}

resolve_sources() then accessed master["name"], raising KeyError. BuilderManager.merge() caught that unrelated KeyError and rewrote it as No such builder for <temporary configuration>, even though the temporary builder was present. This is why the visible error names the generated build configuration rather than the missing source metadata field.

Quick-index coverage

Flow Result with this PR
Quick index before mapping inspection Unchanged; continues to use the complete uploader-created master document.
Save an inspected mapping into a new master record, then quick index Fixed; new records now include name.
Quick index with an existing legacy/minimal master record lacking name Fixed; source resolution falls back to _id.
Quick index for a main source Fixed through the same source-resolution path.
Quick index with an explicit subsource Fixed through the same source-resolution path.
Regex-based source master with _id != name Unchanged; the explicit name regex is still used.
Save mapping only to dest="inspect" Unchanged; this destination does not create or modify the source-master record.

The fallback assumes the minimal record's _id is the exact collection name, which is how save_mapping() derives the id for a non-regex source. Existing regex-aware records retain and use their explicit name.

Changes

  1. Skip source inspection jobs that do not yet contain completed inspect.results when flattening.
  2. Include name when saving a mapping creates a source-master record for the first time.
  3. Fall back from missing src_master.name to src_master._id, so records created by older versions are also usable without migration.
  4. Restrict the No such builder translation to the actual builder registry lookup. A KeyError raised while constructing or starting a merge now retains its real context.

Validation

  • pytest tests/hub/datainspect tests/hub/databuild tests/hub/dataload — 50 passed
  • pytest tests/utils/test_manager.py — 12 passed, including the existing free-threaded executor opt-in and distinct worker tracking-file coverage
  • black --check --fast biothings/hub/datainspect/inspector.py biothings/hub/dataload/source.py biothings/hub/databuild/builder.py — passed

The exact reported dataset/workflow was not reproduced end-to-end under a local 3.14t runtime. Validation covers the affected managers and existing free-threaded JobManager behavior. No new test files are included in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant