Fix inspection mapping quick index flow - #463
Draft
DylanWelzel wants to merge 1 commit into
Draft
DylanWelzel wants to merge 1 commit into
DylanWelzel wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
During 1.2.x testing on free-threaded Python 3.14t, source mapping inspection completed its batch workers, but the subsequent
PUT /flatten_inspection_datarequest failed with:After inspecting and saving the generated mapping, a quick index that had previously worked then failed with:
These are two related state-handling problems. The inspection failure leaves the mapping workflow in a bad state, while saving a mapping can expose incomplete source-master metadata used by quick index. The reported quick-index error is also misleading because the temporary builder was created successfully.
Is inspection itself unsafe in free-threaded mode?
There is no evidence in this trace that the free-threaded workers corrupt or concurrently merge inspection data.
Each worker computes an isolated batch result.
batch_inspected()awaits that result and merges it into the aggregate on the asyncio event-loop thread. Those merge callbacks therefore remain serialized even whendefer_to_process()is backed by the free-threadedThreadPoolExecutor. The surroundingTaskGroupis awaited before final Elasticsearch mapping generation and before the inspection status/results are persisted.The
Remove PID file ... FTWorker_*messages only mean that each worker function has returned. They do not mean that event-loop aggregation, mapping generation, metadata conversion, or the final status write has completed. Consequently, a flatten request may still observe an inspection job in an intermediate state with a status such asinspectingbut withoutinspect.resultsyet.The same incomplete state can also exist after a failed/canceled inspection or in another subsource job stored in the same source document.
flatten()iterates all of those jobs and previously assumed every entry had a completed result. Free-threaded execution makes this timing easier to encounter, but the unsafe assumption is not specific to the free-threaded executor or toTaskGroup.This PR does not change worker scheduling, asyncio aggregation, or inspection calculation. It makes the consumer tolerate valid non-complete job states while preserving completed results.
Inspection fix
For source inspection,
InspectorManager.flatten()now:inspect.resultsdefensively;KeyError;Build inspection is unaffected. It already reads a single build result with
.get(..., {})rather than iterating source job records.No empty or fabricated mapping is returned for an in-progress/failed job; that job is simply absent from the flattened response until a completed result exists.
Why quick index fails only after the mapping workflow
Quick index creates a temporary build configuration and asks
DataBuilder.resolve_sources()to resolve each configured source throughsrc_master.A normal upload creates a complete master document containing both
_idandname, so quick index works before inspection. However, when a generated inspection mapping is saved todest="master"and no master document exists yet,SourceManager.save_mapping()previously created only:{"_id": subsource, "mapping": mapping}resolve_sources()then accessedmaster["name"], raisingKeyError.BuilderManager.merge()caught that unrelatedKeyErrorand rewrote it asNo such builder for <temporary configuration>, even though the temporary builder was present. This is why the visible error names the generated build configuration rather than the missing source metadata field.Quick-index coverage
name.name_id.subsource_id != namenameregex is still used.dest="inspect"The fallback assumes the minimal record's
_idis the exact collection name, which is howsave_mapping()derives the id for a non-regex source. Existing regex-aware records retain and use their explicitname.Changes
inspect.resultswhen flattening.namewhen saving a mapping creates a source-master record for the first time.src_master.nametosrc_master._id, so records created by older versions are also usable without migration.No such buildertranslation to the actual builder registry lookup. AKeyErrorraised while constructing or starting a merge now retains its real context.Validation
pytest tests/hub/datainspect tests/hub/databuild tests/hub/dataload— 50 passedpytest tests/utils/test_manager.py— 12 passed, including the existing free-threaded executor opt-in and distinct worker tracking-file coverageblack --check --fast biothings/hub/datainspect/inspector.py biothings/hub/dataload/source.py biothings/hub/databuild/builder.py— passedThe exact reported dataset/workflow was not reproduced end-to-end under a local 3.14t runtime. Validation covers the affected managers and existing free-threaded JobManager behavior. No new test files are included in this PR.