Parser plugin rewrite: streaming API, benchmarks, and 24 fixed bugs - #461
Parser plugin rewrite: streaming API, benchmarks, and 24 fixed bugs#461kozlov721 wants to merge 20 commits into
Conversation
Parsers produced a `ParsedDataset` whose `files` had to be complete before its `records` could be consumed, so every parser walked its source twice and threw away everything but the file paths. `ParseResult` instead streams `(split, record)` pairs and the importer collects the files as they go by; `detect()` returns the `Layout` it discovered and `parse()` is handed it, rather than both rediscovering the same splits. The parser suite goes 46.7s -> 15.0s over ~100k records, with peak allocation 348 MiB -> 170 MiB. Output is byte-identical: `tools/profile_parsers.py` hashes the ordered file list, splits, skeletons, reported issues and every record for all 18 dataset types, and every step of this change was gated on it reporting `identical`. Also fixes 15 bugs found while reviewing the rewrite, seven of them regressions against the previous parser implementation - dropped background images, a hard error on the standard Ultralytics layout, silently ignored COCO keypoint options, `KeyError` and orphaned datasets from count-based split ratios, an import-time `KeyError` from a colliding plugin, and a `LuxonisParser` that re-downloaded its source on every `parse()`. Adds a `-m benchmark` suite that builds a large synthetic dataset for each parser covering every feature it supports, and reorganizes the parser tests into one module per dataset type under synthetic/, real_world/ and benchmarks/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughThis PR replaces the legacy parser flow with plugin-based streaming imports, adds source preparation and issue handling, and introduces parser benchmarks with comparison tooling and CI automation. It also updates documentation and expands parser, import, and integration tests. ChangesParser import architecture
Benchmark infrastructure
Validation and documentation
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
`image_size` was assigned `img.shape[:2]`, whose element type depends on which numpy stubs are installed. Where that slice resolves to `Any`, the assignment does not narrow the declared `tuple[int, ...] | None`, so pyright in CI rejected the following unpack as `"None" is not iterable` while the same version passed locally. Unpacking into an explicit pair of `int`s narrows definitely under either set of stubs. Parser output is unchanged - the digest reports `identical` for all three YOLOv8 dataset types. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The benchmarks reported a table nobody could act on: one run of one ref, with no way to tell a real slowdown from a busy runner. They now measure two refs the same way and fail the release when one of them regressed. The release pull request benchmarks its head and its base as a matrix, so both are measured on the same runner image, and a third job compares the two, posts the table as a single edited comment and exits non-zero past the threshold. `workflow_dispatch` takes both refs, the threshold, the scale, the repeat count and the aggregate; leaving the baseline empty benchmarks one ref and compares it to nothing. Telling a regression from noise needed measuring the noise first. Two runs of identical code, same machine, differ by a median of 2.8% and up to 11.5%, so the old table's few-percent movements said nothing. Three things come out of that: - `min` is the default reduction, not `mean`. Noise here is one-sided - a busy machine only ever makes a parse slower - so the fastest of N timings is the most reproducible summary. Measured over ten repeats: min varies 1.7% run to run at the median, mean 2.0%, median 2.8%. - Every timing is kept, and its scatter reported. `clsdir` and `fiftyone-classification` parse in hundredths of a second and scatter by 7% and 77%; no repeat count fixes that. - A parser counts as regressed only when it is both past the threshold and more than twice the two runs' combined scatter, which is what lets those two stay in the suite without failing a release on jitter. Peak allocation is reported but never gates. It is reproducible within a run of the suite and not across differently measured ones: the suite generates its datasets in-process, which grows CPython's interned-string table before anything is traced, and the same parse measured in a fresh process pays for that growth inside the traced window and reports ~5 MiB more. Defaults are 2000 images per split doubled, ten repeats and a 25% threshold, held there by the runner rather than by patience - scale 3 writes 1.7 GB across 290k files, which is not something to hand a hosted runner that also has to finish inside an hour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running the benchmarks by hand turns the whole run red for two reasons
that have nothing to do with the benchmarks.
`auto-assigner` wants a pull request payload and errors out without one
("Can't get payload"), so every `workflow_dispatch` run failed on it. It
now runs only for pull requests, which is the only event it can serve.
`test_tensorflow_csv_file_list_does_not_replay_the_records` compared the
parsed records in order, and that order is the order the split was
globbed in — the file system's, not the parser's. The runner listed the
unannotated image first and the assertion failed there while passing
here. The assertion two lines above it already concedes this and sorts;
this one now does too, which still pins what the test is about: one
record per annotation row plus one for the unannotated image.
Verified by forcing the reverse of the local glob order, which fails
without this and passes with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both compared records in the order the parser emitted them, and that is the order `_list_images` globbed the split in - the file system's, not the parser's. They pass here and would fail on a machine that lists the two images the other way round, which is how the tensorflow-csv test next to them was already failing in CI. Neither test is about ordering. The tensorflow-csv one is about each spelling of a filename landing on the image it names, four rows apiece; the yolov6 one is about both images being present exactly once, and it already looks the individual records up by name. Sorting both sides keeps what they check and drops what they cannot rely on. Found by running the parser suite with `_list_images` reversed, which is the cheapest stand-in for a different file system. With these two fixed, nothing under tests/test_data fails only under that reversal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage measures `tests` as well as `luxonis_ml`, and the run that measures it deselects the benchmarks with `-m "not benchmark"`. The dataset builders and the timing harness therefore reported as 20-39% covered - not because anything is untested, but because that run cannot execute them at all. Between them they contributed 1089 statements, 751 of which were unreachable by construction. `comparison.py` and its tests stay measured. They decide whether a release is allowed through and they do run in the default suite, so their coverage is worth something; the rest of the directory's is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every fix ships a regression test verified to fail without it, and `tools/profile_parsers.py compare` reports `identical` for all 18 dataset types, so none of the parser changes move a record. Silent data loss and destroyed datasets: - `BBoxAnnotation`, `KeypointAnnotation` and `SegmentationAnnotation` still edited the caller's own annotation dict in place. Dropping the `deepcopy` from `DatasetRecord.validate_files` was only made safe for masks and polylines, so a producer reusing one annotation dict across files had it clipped by the first record and wrote the clipped values for every record after it, warning once. - The failed-import cleanup caught `BaseException`, so a Ctrl-C halfway through a long import deleted everything it had already parsed and uploaded. Only `Exception` triggers it now. - That cleanup also read `delete_local=True` as proof the import owned the dataset. For remote storage the constructor only clears the local cache, so a part-way failure deleted a pre-existing bucket dataset the import had merely appended to. - Percentage `split_ratios` were validated only by `make_splits`, after every record was written and every image uploaded, and the cleanup then deleted the whole import. Counts were already checked up front; ratios now are too. - `UltralyticsNDJSONParser.enumerate_files` counted images the parse skips, so count-based `split_ratios` quietly under-filled the splits. Wrong or lost results: - `YOLOv8Parser` picked the alphabetically first YAML as its class file, so a `data.yaml` next to any earlier-sorting YAML lost to it and every box was labelled with another dataset's classes. `data.yaml` wins now. - `COCOParser.parse` dropped `**kwargs` on the split-directory path, so a mistyped parser argument raised `TypeError` for a single-split source and was silently ignored for the common layout. - `LuxonisParser.get_parser_issue_messages` read through the dataset a failed parse never returns, so it answered `[]` for exactly the case it exists to explain. - A class directory skipped for having a reserved name is now logged rather than silently dropped. Also: the benchmark regression threshold fell back to 10% on release pull requests, below the ~12% run-to-run noise its own tool documents, against the 25% every other default uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pair of benchmark legs runs in parallel with the test matrix and
finishes inside the slowest shard, so measuring every pull request
instead of only `release/*` ones costs no wall-clock time. The baseline
is the branch the pull request merges into.
The report was already written to the job summary and to a single
edited pull request comment; neither had ever been seen on an ordinary
pull request, because the job never ran for one. Making it run exposed
three ways it breaks:
- `download-artifact` was given a `pattern`, which unpacks into the
workspace root rather than a per-artifact directory when only one
artifact matches - exactly what a failed matrix leg leaves behind.
Every path in the report job then missed, and run 30622969650 failed
on `jq: Could not open parser-benchmarks-current/benchmark-results.json`
after a successful 12-minute benchmark. Each artifact is now fetched
by name into a directory named after it.
- A baseline older than the benchmark suite - every ref on `main` until
this merges - has nothing to measure. That leg now exits green with a
notice, uploads nothing, and the report says the baseline is missing
instead of the job going red on the pull request's behalf.
- With no results the report step ran `cat report.md` on a file no
earlier step had written. A missing measurement now produces a report
saying so, and `Fail on a regression` no longer claims a regression
for what may have been an empty comparison.
Refs reach the shell through the environment rather than `${{ }}`
interpolation. A pull request branch is named by whoever opened it, and
this job now runs for all of them. Commenting is `continue-on-error`,
since a fork's token is read-only and cannot comment at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Parser benchmarks
|
Each row of the comparison table carries a mark: `❗` past the threshold, `⚠️ ` past half of it, `✅` otherwise, and the change is coloured to match, so a reader does not have to hold the threshold in their head while scanning. Dataset types are monospaced and the numbers are set as math. Two rules govern where the math can go, both found by rendering through GitHub and reading the result: - A closing `$` followed by a letter is not parsed as math at all - `$0.39$s` renders as literal text, dollar signs included. So no unit trails a delimiter; the seconds columns say so in the header instead. - A `%` inside the math is worse, because it renders. Markdown strips the backslash from `\%` before MathJax sees it, and the bare `%` left behind opens a comment that swallows the rest of the expression. The percent signs stay outside. A test asserts both rules over every math cell the renderer emits, rather than over the two columns that happen to have units today. Colours are the xcolor names rather than hex, which GitHub's MathJax leaves unrendered. Neither math nor colour renders in a job summary, which would print the markup itself, so `render_markdown` takes `rich` and the tool writes both forms: `--output` for the pull request comment, `--plain-output` for the summary. The plain form keeps the marks and the monospaced names, which render everywhere. The single-ref table moves out of the `jq` block in the workflow and into `render_single_markdown`, so both tables share the formatting and the second one is testable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #461 +/- ##
==========================================
+ Coverage 93.97% 96.78% +2.80%
==========================================
Files 164 190 +26
Lines 12794 16447 +3653
==========================================
+ Hits 12023 15918 +3895
+ Misses 771 529 -242
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
Parser plugin rewrite: streaming API, benchmarks, and 24 fixed bugs
Replaces
BaseParserwith a plugin architecture, moves dataset constructionout of the parsers and into
BaseDataset.import_dataset, adds a benchmarksuite, and makes every parser walk its source exactly once.
Parsing runs 49.7s → 15.0s (3.3x) over 984k records, with byte-identical
output at every step of the rewrite. Peak allocation is roughly flat in
aggregate (186 MiB → 180 MiB across the suite) with one deliberate regression;
per-parser numbers and the memory discussion are in Benchmarks.
Breaking changes
BaseParserParserPlugin/SplitParserPlugin, registered viaregister_parser_pluginor aparser_pluginsentry pointParsedDataset(records, skeletons, files, splits)ParseResult(records, skeletons), whererecordsyields(split_name, record)ParserPlugin.supports(source) -> boolParserPlugin.detect(source) -> Layout | NoneSplitParserPlugin.discover_splits()detect(); theLayoutis passed toparse()SplitParserPlugin._parse_split()_split_records(), returning records onlycombine_split_outputs,ParserPlugin._get_added_imagesLuxonisParserLuxonisDataset.import_dataset, still presentA third-party parser needs:
detect()returning aLayout,parse()(or_split_records()) yielding records, and optionally_split_files()/enumerate_files()so count-basedsplit_ratiosstay cheap.Bugs fixed
Found by a review of the rewrite; each has a regression test that fails
without its fix. Items marked regression behaved correctly before the
rewrite and would have shipped broken.
Silent data loss and hard failures
_prepare_import_recordsdropped annotation-less images (background imagesin YOLOv6/v8/COCO/VOC/Darknet) until the first annotated record — a string
task_namebecame an emptydefaultdict, so the fan-out set was empty.Order-dependent loss. Regression.
images/train+images/vallayout matched bothYOLOv6 and YOLOv8 and hard-errored as ambiguous. Resolved by split
coverage; genuine ties still raise. Regression.
use_keypoint_ann/keypoint_ann_paths,so pose models could be trained on keypoint-free datasets. Now raises.
Regression.
split_ratiosraised a bareKeyErrorafterthe dataset was created on disk.
make_splitsand left an orphanedregistered dataset. Validated before construction now.
Plugin and API surface
dataset_typescollided with a built-in madeplain
import luxonis_ml.dataraiseKeyError, breaking every downstreamconsumer. Now warns and skips; registration is atomic.
LuxonisParserdeferred source acquisition and dataset construction intoparse(), so construction-time errors stopped firing and eachparse()re-downloaded the source and returned a new dataset. Regression.
Pre-existing bugs fixed along the way
_list_imageswas case-sensitive (listed.WebPbut not.JPG), souppercase-extension datasets were rejected or silently truncated.
UnboundLocalErroron a label line with fewer than 5 values;now skipped via
ParserIssue.MALFORMED_ANNOTATION./on the source produced dataset name"", writing storagedirectories into the datasets root and merging into prior imports.
ClassificationDirectoryParser.validate_splitexcludeddata/raw/masksbut
_parse_splitingested them as classes.Performance defects
first yield, so multi-GB exports would OOM.
model_copy(deep=True)for every annotated record.list (see below).
Second review round
A later review of the finished branch found nine more, all in code this PR
adds or changes. Same standard: a regression test each, verified to fail
without its fix, and
profile_parsers comparereportsidenticalfor all18 dataset types, so none of them moves a record.
BBoxAnnotation,KeypointAnnotationandSegmentationAnnotationstilledited the caller's annotation dict in place — dropping the
deepcopy(below) was only made safe for masks and polylines. A generator that
reuses one annotation dict across files, the documented way to add a
dataset, had it clipped by the first record and wrote the clipped values
for every record after it, warning exactly once.
BaseException, so a Ctrl-C halfwaythrough a long import deleted everything already parsed and uploaded.
delete_local=Trueas proof the import owned thedataset. For remote storage the constructor only clears the local cache,
so a part-way failure deleted a pre-existing bucket dataset the import
had merely appended to.
split_ratioswere validated only bymake_splits, afterevery record was written, and the cleanup then deleted the whole import.
Counts were already checked up front (bugs 4 and 5); ratios now are too.
UltralyticsNDJSONParser.enumerate_filescounted images the parse skips,so count-based
split_ratiosquietly under-filled the splits — theenumeration and the parse have to agree for counts to mean anything.
YOLOv8Parsertook the alphabetically first YAML as its class file, so adata.yamlnext to any earlier-sorting YAML lost to it and every box waslabelled with another dataset's classes.
data.yamlwins now.COCOParser.parsedropped**kwargson the split-directory path, so amistyped parser argument raised
TypeErrorfor a single-split source andwas silently ignored for the common layout — the other half of bug 3.
LuxonisParser.get_parser_issue_messagesread through the dataset that afailed parse never returns, so it answered
[]for exactly the case itexists to explain.
logged instead of vanishing from the import silently.
The two systemic performance problems
Every parser parsed its source twice.
ParsedDatasetrequiredfilesbefore
recordscould be consumed, so each parser built a throwaway generatorand discarded everything but the paths.
ParseResulttags records with theirsplit and the importer collects files as they stream, so the second walk is
gone everywhere.
Detection ran twice.
get_parser_plugincalledsupports()andparse()rediscovered the same splits — a second directory listing per split, and for
COCO/SOLO/FiftyOne a second decode of every annotation file.
Plus per-parser work: YOLOv8 decoded each image once per polygon; COCO's
clean_annotationshit pure-Python JSON encoding; path resolution ran perannotation on directories that never change. See
luxonis_ml/data/parsers/OPTIMIZATIONS.md.How "no behaviour change" was verified
tools/profile_parsers.py digests/comparehashes the ordered file list,the splits, the skeletons, the reported issues and every record, in order, for
all 18 dataset types. Every optimization and the whole API port were gated on
it reporting
identical.That gate covers what the benchmark datasets contain, so it is not sufficient
alone — it missed two eager-failure regressions that were caught by running the
original pre-refactor test modules against the new code.
Benchmarks
pytest -m benchmark tests/test_data/parsers/benchmarksbuilds a largesynthetic dataset for every parser covering all the features it supports, times
a full parse and records peak memory. Excluded from the default run
(
addoptscarries-m "not benchmark"), so a plainpytestnever pays forit.
CI runs it on every pull request, against the branch the pull request
merges into. The two refs are measured on separate runners in parallel with
the test matrix: a leg takes ~13 minutes against a slowest
testsshard thathas run 14-19 minutes on recent pushes, so the pair finishes inside it and
the workflow's wall clock is unchanged — though not by much on a fast day,
since shard times swing widely.
The table goes to the job summary and to a single pull request comment,
edited in place on each push rather than appended. A parser more than 25%
slower than the base branch fails the job; two runs of identical code differ
by up to ~12%, so that leaves room for noise.
Until this merges,
mainhas no benchmark suite to run as a baseline. Thatleg exits green with a notice and the comment reports the current ref alone;
comparisons start once the suite is on the base branch.
Benchmarks also assert exact record, file and issue counts, so they double as
correctness tests at a scale the unit tests never reach.
mainvs this PR2000 images per split, best of three parses per side, one process per dataset
type (see the memory note below for why that matters). Both sides measure the
parser alone — producing records, nothing ingested into a dataset. On
mainthat includes the second pass every parser ran to build
added_images, becausethere that pass was part of parsing. Record counts are identical on both
sides for every row except
clsdir.main(s)clsdir²segmasktfcsvyolov8instancesegmentationnative¹yolov4yolov8keypointsyolov6darknetcocofiftyone-classificationyolov8createmlvocultralytics-ndjson-instancesegmentationultralytics-ndjsonsoloultralytics-ndjson-keypointsTotal: 49.7s → 15.0s (3.3x) over 983,566 records.
Speedup is green above$1.5\times$ and orange below it. $5$ MiB, the size of the measurement
peak MiBiscoloured only when it moves by more than
artifact described below — smaller moves are the method, not the parser.
¹
main'sNativeParserraisesTypeError: 'NoneType' object is not subscriptableon a record whoseannotationisnull, so it cannot parse thebenchmark dataset at all. To get a comparison anyway, this row uses a copy of
the dataset with only those records removed — 600 of 54,600, 1.1% — leaving
every other feature it exercises (sub-detections, sample metadata, multi-source
records, path resolution, masks, depth) intact. Both sides parse the same
54,000 records from the same image bytes. On the full dataset this PR takes
1.10s at 45.0 MiB peak and
maincannot run at all.²
mainyields 4,236 records here against this PR's 6,000: its_list_imagesis case-sensitive and silently skips every
.JPGimage (bug 8 above).mainisdoing ~29% less work in that row, so 17.1x overstates it.
Why some parsers now peak higher
Only one of these is a real change in what a parser holds:
coco, 42.0 → 70.6 MiB.detect()decodes each split's annotation JSONinto the
Layoutandparse()reuses it instead of decoding a second time.Measured directly: detection alone leaves 64.4 MiB resident, all of it
attributed to
json/decoder.pyscan_once.maindecoded one split at atime inside
from_splitand released it before the next, so it never heldmore than the largest split at once. This is what buys the 3.3x, on the same
index the parse would otherwise decode twice.
yolov6keeps its split's image listing in theLayout(
images: list[2000]per split) so the parse does not re-list the directory,where
mainlisted the same images during validation and threw the listaway. Measured at ~1.8 MiB — the same trade as
coco, three orders ofmagnitude smaller, and small enough that it does not show as a net
regression.
Every other row moves by a few MiB in either direction and should not be
read as parser behaviour. Measuring a parse outside the benchmark suite — a
fresh process parsing an already-built dataset, which is the only way
maincan be measured, since the suite does not exist there — puts a one-time resize
of CPython's interned-string table inside the traced window, worth exactly
5.00 MiB on path-heavy parsers. The suite never pays it, because it
generates the dataset in the same process first: on identical data
yolov6reports 2.4 MiB under
pytest -m benchmarkand 7.5 MiB measuredstandalone.
So the peak column above is trustworthy exactly where the two methods agree —
coco70.6,native45.0,tfcsv8.4, the NDJSON trio at 0.1 — and thesmall rows are inside the artifact. Within the suite the column is stable:
identical run to run, and identical whether a type runs alone or alongside the
other seventeen.
Tests
tests/test_data/parsers/is now one module per dataset type, split intosynthetic/(builds every input, needs no credentials),real_world/(cloudfixtures) and
benchmarks/. 214 synthetic tests, against 50 in the singletest_parsers.pymodule onmain.Also in here
pyproject.tomlused the inert[tool.pytest]key, sotestpaths,addoptsanddoctest_optionflagshad never taken effect. Fixed to[tool.pytest.ini_options]; the 48 doctests now actually run.tests/conftest.pyno longer replacesbuiltins.printwith rich's, whichwas corrupting doctest output and one assertion.
import_datasetnow deletes the dataset it created instead ofleaving a half-populated one registered.
DatasetRecordvalidators no longerdeepcopyevery incoming record; theyrebuild without mutating, which is the single largest cost in the import
path. (Bug 16 above: three of them were still mutating until the second
review round. Rebuilding instead costs nothing measurable — 20k bbox
records validate in 487-494ms rebuilding against 512-518ms mutating.)
base_tempdiris process-scoped, so two pytest sessions in one worktree stopdeleting each other's fixtures.
input and
compare_benchmarks.pyboth default to 25%. Two runs of identicalcode differ by up to ~12%, so pull requests would have gone red on noise.
reporting broke, none of which had ever run on an ordinary pull request:
download-artifactwas given apattern, which unpacks into the workspaceroot rather than a per-artifact directory when only one artifact matches —
what a failed matrix leg leaves behind — so every path in the report job
missed; a report step ran
cat report.mdon a file no earlier step hadwritten when there were no results; and
Fail on a regressionclaimed aregression for what may have been an empty comparison. Refs now reach the
shell through the environment rather than
${{ }}interpolation, since apull request branch is named by whoever opened it and this job now runs for
all of them.
Known trade-offs
SOLOParsercan no longer fail eagerly on an undecodable mask — the passthat checked is gone by design. The failed-import cleanup above covers the
user-visible consequence.
yolov4,yolov8instancesegmentation,segmask) are 6-17%slower than in the intermediate round: the streaming contract requires
interleaving work that could previously be batched per split. All three
remain far faster than before the work started.
cocoholds every split's decoded annotation index for the whole parse, soit peaks at 70.6 MiB against
main's 42.0 MiB; seeBenchmarks.
parse()is only 4-7% of animport_dataset. The remaining time is progressrendering and pydantic validation on the ingestion side, deliberately left
out of scope.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Bug Fixes
Tests
Refactor