Skip to content

Parser plugin rewrite: streaming API, benchmarks, and 24 fixed bugs - #461

Draft
kozlov721 wants to merge 20 commits into
mainfrom
feat/parsers-refactor
Draft

Parser plugin rewrite: streaming API, benchmarks, and 24 fixed bugs#461
kozlov721 wants to merge 20 commits into
mainfrom
feat/parsers-refactor

Conversation

@kozlov721

@kozlov721 kozlov721 commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Parser plugin rewrite: streaming API, benchmarks, and 24 fixed bugs

Replaces BaseParser with a plugin architecture, moves dataset construction
out of the parsers and into BaseDataset.import_dataset, adds a benchmark
suite, and makes every parser walk its source exactly once.

Parsing runs 49.7s → 15.0s (3.3x) over 984k records, with byte-identical
output at every step of the rewrite. Peak allocation is roughly flat in
aggregate (186 MiB → 180 MiB across the suite) with one deliberate regression;
per-parser numbers and the memory discussion are in Benchmarks.

Breaking changes

Removed Replacement
BaseParser ParserPlugin / SplitParserPlugin, registered via register_parser_plugin or a parser_plugins entry point
ParsedDataset(records, skeletons, files, splits) ParseResult(records, skeletons), where records yields (split_name, record)
ParserPlugin.supports(source) -> bool ParserPlugin.detect(source) -> Layout | None
SplitParserPlugin.discover_splits() folded into detect(); the Layout is passed to parse()
SplitParserPlugin._parse_split() _split_records(), returning records only
combine_split_outputs, ParserPlugin._get_added_images no longer needed
LuxonisParser deprecated wrapper over LuxonisDataset.import_dataset, still present

A third-party parser needs: detect() returning a Layout, parse() (or
_split_records()) yielding records, and optionally _split_files() /
enumerate_files() so count-based split_ratios stay cheap.

Bugs fixed

Found by a review of the rewrite; each has a regression test that fails
without its fix. Items marked regression behaved correctly before the
rewrite and would have shipped broken.

Silent data loss and hard failures

  1. _prepare_import_records dropped annotation-less images (background images
    in YOLOv6/v8/COCO/VOC/Darknet) until the first annotated record — a string
    task_name became an empty defaultdict, so the fan-out set was empty.
    Order-dependent loss. Regression.
  2. The standard Ultralytics images/train + images/val layout matched both
    YOLOv6 and YOLOv8 and hard-errored as ambiguous. Resolved by split
    coverage; genuine ties still raise. Regression.
  3. Single-split COCO silently dropped use_keypoint_ann / keypoint_ann_paths,
    so pose models could be trained on keypoint-free datasets. Now raises.
    Regression.
  4. Partial or empty count-based split_ratios raised a bare KeyError after
    the dataset was created on disk.
  5. Zero-selecting count ratios crashed in make_splits and left an orphaned
    registered dataset. Validated before construction now.

Plugin and API surface

  1. An entry-point plugin whose dataset_types collided with a built-in made
    plain import luxonis_ml.data raise KeyError, breaking every downstream
    consumer. Now warns and skips; registration is atomic.
  2. LuxonisParser deferred source acquisition and dataset construction into
    parse(), so construction-time errors stopped firing and each parse()
    re-downloaded the source and returned a new dataset. Regression.

Pre-existing bugs fixed along the way

  1. _list_images was case-sensitive (listed .WebP but not .JPG), so
    uppercase-extension datasets were rejected or silently truncated.
  2. YOLOv8 raised UnboundLocalError on a label line with fewer than 5 values;
    now skipped via ParserIssue.MALFORMED_ANNOTATION.
  3. A trailing / on the source produced dataset name "", writing storage
    directories into the datasets root and merging into prior imports.
  4. ClassificationDirectoryParser.validate_split excluded data/raw/masks
    but _parse_split ingested them as classes.

Performance defects

  1. NDJSON parsing stopped streaming — the whole file was decoded before the
    first yield, so multi-GB exports would OOM.
  2. Per-record model_copy(deep=True) for every annotated record.
  3. SOLO walked the dataset twice, decoding every mask PNG twice.
  4. Every parser ran its full parse a second time purely to collect the file
    list (see below).

Second review round

A later review of the finished branch found nine more, all in code this PR
adds or changes. Same standard: a regression test each, verified to fail
without its fix, and profile_parsers compare reports identical for all
18 dataset types, so none of them moves a record.

  1. BBoxAnnotation, KeypointAnnotation and SegmentationAnnotation still
    edited the caller's annotation dict in place — dropping the deepcopy
    (below) was only made safe for masks and polylines. A generator that
    reuses one annotation dict across files, the documented way to add a
    dataset, had it clipped by the first record and wrote the clipped values
    for every record after it, warning exactly once.
  2. The failed-import cleanup caught BaseException, so a Ctrl-C halfway
    through a long import deleted everything already parsed and uploaded.
  3. That cleanup also read delete_local=True as proof the import owned the
    dataset. For remote storage the constructor only clears the local cache,
    so a part-way failure deleted a pre-existing bucket dataset the import
    had merely appended to.
  4. Percentage split_ratios were validated only by make_splits, after
    every record was written, and the cleanup then deleted the whole import.
    Counts were already checked up front (bugs 4 and 5); ratios now are too.
  5. UltralyticsNDJSONParser.enumerate_files counted images the parse skips,
    so count-based split_ratios quietly under-filled the splits — the
    enumeration and the parse have to agree for counts to mean anything.
  6. YOLOv8Parser took the alphabetically first YAML as its class file, so a
    data.yaml next to any earlier-sorting YAML lost to it and every box was
    labelled with another dataset's classes. data.yaml wins now.
  7. COCOParser.parse dropped **kwargs on the split-directory path, so a
    mistyped parser argument raised TypeError for a single-split source and
    was silently ignored for the common layout — the other half of bug 3.
  8. LuxonisParser.get_parser_issue_messages read through the dataset that a
    failed parse never returns, so it answered [] for exactly the case it
    exists to explain.
  9. A class directory skipped for carrying a reserved name (bug 11) is now
    logged instead of vanishing from the import silently.

The two systemic performance problems

Every parser parsed its source twice. ParsedDataset required files
before records could be consumed, so each parser built a throwaway generator
and discarded everything but the paths. ParseResult tags records with their
split and the importer collects files as they stream, so the second walk is
gone everywhere.

Detection ran twice. get_parser_plugin called supports() and parse()
rediscovered the same splits — a second directory listing per split, and for
COCO/SOLO/FiftyOne a second decode of every annotation file.

Plus per-parser work: YOLOv8 decoded each image once per polygon; COCO's
clean_annotations hit pure-Python JSON encoding; path resolution ran per
annotation on directories that never change. See
luxonis_ml/data/parsers/OPTIMIZATIONS.md.

How "no behaviour change" was verified

tools/profile_parsers.py digests / compare hashes the ordered file list,
the splits, the skeletons, the reported issues and every record, in order, for
all 18 dataset types. Every optimization and the whole API port were gated on
it reporting identical.

That gate covers what the benchmark datasets contain, so it is not sufficient
alone — it missed two eager-failure regressions that were caught by running the
original pre-refactor test modules against the new code.

Benchmarks

pytest -m benchmark tests/test_data/parsers/benchmarks builds a large
synthetic dataset for every parser covering all the features it supports, times
a full parse and records peak memory. Excluded from the default run
(addopts carries -m "not benchmark"), so a plain pytest never pays for
it.

CI runs it on every pull request, against the branch the pull request
merges into. The two refs are measured on separate runners in parallel with
the test matrix: a leg takes ~13 minutes against a slowest tests shard that
has run 14-19 minutes on recent pushes, so the pair finishes inside it and
the workflow's wall clock is unchanged — though not by much on a fast day,
since shard times swing widely.

The table goes to the job summary and to a single pull request comment,
edited in place on each push rather than appended. A parser more than 25%
slower than the base branch fails the job; two runs of identical code differ
by up to ~12%, so that leaves room for noise.

Until this merges, main has no benchmark suite to run as a baseline. That
leg exits green with a notice and the comment reports the current ref alone;
comparisons start once the suite is on the base branch.

Benchmarks also assert exact record, file and issue counts, so they double as
correctness tests at a scale the unit tests never reach.

main vs this PR

2000 images per split, best of three parses per side, one process per dataset
type
(see the memory note below for why that matters). Both sides measure the
parser alone — producing records, nothing ingested into a dataset. On main
that includes the second pass every parser ran to build added_images, because
there that pass was part of parsing. Record counts are identical on both
sides for every row except clsdir.

dataset type records main (s) this PR (s) speedup peak MiB
clsdir² $6{,}000$ $0.39$ $0.02$ $\textcolor{green}{\mathbf{17.1\times}}$ $1.7 \to 0.1$
segmask $27{,}150$ $2.25$ $0.18$ $\textcolor{green}{\mathbf{12.3\times}}$ $\textcolor{green}{6.4 \to 0.7}$
tfcsv $54{,}600$ $4.05$ $0.37$ $\textcolor{green}{\mathbf{11.0\times}}$ $\textcolor{green}{16.3 \to 8.4}$
yolov8instancesegmentation $60{,}000$ $8.22$ $0.87$ $\textcolor{green}{\mathbf{9.5\times}}$ $4.4 \to 1.6$
native¹ $54{,}000$ $12.01$ $1.30$ $\textcolor{green}{\mathbf{9.2\times}}$ $49.1 \to 44.8$
yolov4 $49{,}200$ $2.82$ $0.70$ $\textcolor{green}{\mathbf{4.0\times}}$ $4.1 \to 7.0$
yolov8keypoints $60{,}000$ $2.13$ $0.59$ $\textcolor{green}{\mathbf{3.6\times}}$ $3.0 \to 6.0$
yolov6 $55{,}680$ $0.76$ $0.22$ $\textcolor{green}{\mathbf{3.5\times}}$ $2.9 \to 7.5$
darknet $54{,}600$ $0.71$ $0.21$ $\textcolor{green}{\mathbf{3.4\times}}$ $3.4 \to 2.6$
coco $40{,}200$ $3.81$ $1.17$ $\textcolor{green}{\mathbf{3.3\times}}$ $\textcolor{red}{42.0 \to 70.6}$
fiftyone-classification $5{,}940$ $0.06$ $0.02$ $\textcolor{green}{\mathbf{2.5\times}}$ $\textcolor{green}{8.2 \to 1.7}$
yolov8 $60{,}000$ $0.84$ $0.38$ $\textcolor{green}{\mathbf{2.2\times}}$ $2.9 \to 1.0$
createml $56{,}400$ $1.43$ $0.78$ $\textcolor{green}{\mathbf{1.8\times}}$ $19.8 \to 19.2$
voc $43{,}800$ $1.89$ $1.14$ $\textcolor{green}{\mathbf{1.7\times}}$ $8.5 \to 6.4$
ultralytics-ndjson-instancesegmentation $54{,}622$ $1.26$ $0.89$ $\textcolor{orange}{\mathbf{1.4\times}}$ $3.9 \to 0.1$
ultralytics-ndjson $162{,}752$ $1.91$ $1.49$ $\textcolor{orange}{\mathbf{1.3\times}}$ $3.8 \to 0.1$
solo $84{,}000$ $4.20$ $3.80$ $\textcolor{orange}{\mathbf{1.1\times}}$ $2.1 \to 2.1$
ultralytics-ndjson-keypoints $54{,}622$ $0.96$ $0.89$ $\textcolor{orange}{\mathbf{1.1\times}}$ $3.8 \to 0.1$

Total: 49.7s → 15.0s (3.3x) over 983,566 records.

Speedup is green above $1.5\times$ and orange below it. peak MiB is
coloured only when it moves by more than $5$ MiB, the size of the measurement
artifact described below — smaller moves are the method, not the parser.

¹ main's NativeParser raises TypeError: 'NoneType' object is not subscriptable on a record whose annotation is null, so it cannot parse the
benchmark dataset at all. To get a comparison anyway, this row uses a copy of
the dataset with only those records removed — 600 of 54,600, 1.1% — leaving
every other feature it exercises (sub-detections, sample metadata, multi-source
records, path resolution, masks, depth) intact. Both sides parse the same
54,000 records from the same image bytes. On the full dataset this PR takes
1.10s at 45.0 MiB peak and main cannot run at all.

² main yields 4,236 records here against this PR's 6,000: its _list_images
is case-sensitive and silently skips every .JPG image (bug 8 above). main is
doing ~29% less work in that row, so 17.1x overstates it.

Why some parsers now peak higher

Only one of these is a real change in what a parser holds:

  • coco, 42.0 → 70.6 MiB. detect() decodes each split's annotation JSON
    into the Layout and parse() reuses it instead of decoding a second time.
    Measured directly: detection alone leaves 64.4 MiB resident, all of it
    attributed to json/decoder.py scan_once. main decoded one split at a
    time inside from_split and released it before the next, so it never held
    more than the largest split at once. This is what buys the 3.3x, on the same
    index the parse would otherwise decode twice.
  • yolov6 keeps its split's image listing in the Layout
    (images: list[2000] per split) so the parse does not re-list the directory,
    where main listed the same images during validation and threw the list
    away. Measured at ~1.8 MiB — the same trade as coco, three orders of
    magnitude smaller, and small enough that it does not show as a net
    regression.

Every other row moves by a few MiB in either direction and should not be
read as parser behaviour. Measuring a parse outside the benchmark suite — a
fresh process parsing an already-built dataset, which is the only way main
can be measured, since the suite does not exist there — puts a one-time resize
of CPython's interned-string table inside the traced window, worth exactly
5.00 MiB on path-heavy parsers. The suite never pays it, because it
generates the dataset in the same process first: on identical data yolov6
reports 2.4 MiB under pytest -m benchmark and 7.5 MiB measured
standalone.

So the peak column above is trustworthy exactly where the two methods agree —
coco 70.6, native 45.0, tfcsv 8.4, the NDJSON trio at 0.1 — and the
small rows are inside the artifact. Within the suite the column is stable:
identical run to run, and identical whether a type runs alone or alongside the
other seventeen.

Tests

tests/test_data/parsers/ is now one module per dataset type, split into
synthetic/ (builds every input, needs no credentials), real_world/ (cloud
fixtures) and benchmarks/. 214 synthetic tests, against 50 in the single
test_parsers.py module on main.

Also in here

  • pyproject.toml used the inert [tool.pytest] key, so testpaths,
    addopts and doctest_optionflags had never taken effect. Fixed to
    [tool.pytest.ini_options]; the 48 doctests now actually run.
  • tests/conftest.py no longer replaces builtins.print with rich's, which
    was corrupting doctest output and one assertion.
  • A failed import_dataset now deletes the dataset it created instead of
    leaving a half-populated one registered.
  • DatasetRecord validators no longer deepcopy every incoming record; they
    rebuild without mutating, which is the single largest cost in the import
    path. (Bug 16 above: three of them were still mutating until the second
    review round. Rebuilding instead costs nothing measurable — 20k bbox
    records validate in 487-494ms rebuilding against 512-518ms mutating.)
  • base_tempdir is process-scoped, so two pytest sessions in one worktree stop
    deleting each other's fixtures.
  • The benchmark job's regression threshold fell back to 10% while the workflow
    input and compare_benchmarks.py both default to 25%. Two runs of identical
    code differ by up to ~12%, so pull requests would have gone red on noise.
  • Opening the benchmark job up to every pull request exposed three ways its
    reporting broke, none of which had ever run on an ordinary pull request:
    download-artifact was given a pattern, which unpacks into the workspace
    root rather than a per-artifact directory when only one artifact matches —
    what a failed matrix leg leaves behind — so every path in the report job
    missed; a report step ran cat report.md on a file no earlier step had
    written when there were no results; and Fail on a regression claimed a
    regression for what may have been an empty comparison. Refs now reach the
    shell through the environment rather than ${{ }} interpolation, since a
    pull request branch is named by whoever opened it and this job now runs for
    all of them.

Known trade-offs

  • SOLOParser can no longer fail eagerly on an undecodable mask — the pass
    that checked is gone by design. The failed-import cleanup above covers the
    user-visible consequence.
  • Three parsers (yolov4, yolov8instancesegmentation, segmask) are 6-17%
    slower than in the intermediate round: the streaming contract requires
    interleaving work that could previously be batched per split. All three
    remain far faster than before the work started.
  • coco holds every split's decoded annotation index for the whole parse, so
    it peaks at 70.6 MiB against main's 42.0 MiB; see
    Benchmarks.
  • parse() is only 4-7% of an import_dataset. The remaining time is progress
    rendering and pydantic validation on the ingestion side, deliberately left
    out of scope.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Unified dataset import API with automatic format detection and split-aware importing
    • Support for local, ZIP, Roboflow, and Ultralytics dataset sources
    • Extensible parser plugins with streaming imports, sampling, and parser issue reporting
    • Added benchmark reporting with regression detection and pull request summaries
  • Documentation

    • Added parser optimization, benchmarking, and contributor guidance
  • Bug Fixes

    • Improved annotation validation without modifying caller-provided data
  • Tests

    • Expanded synthetic, integration, and benchmark coverage
  • Refactor

    • Deprecated the legacy parser interface in favor of the unified import API

kozlov721 and others added 4 commits July 30, 2026 01:20
Parsers produced a `ParsedDataset` whose `files` had to be complete before
its `records` could be consumed, so every parser walked its source twice and
threw away everything but the file paths. `ParseResult` instead streams
`(split, record)` pairs and the importer collects the files as they go by;
`detect()` returns the `Layout` it discovered and `parse()` is handed it,
rather than both rediscovering the same splits.

The parser suite goes 46.7s -> 15.0s over ~100k records, with peak allocation
348 MiB -> 170 MiB. Output is byte-identical: `tools/profile_parsers.py`
hashes the ordered file list, splits, skeletons, reported issues and every
record for all 18 dataset types, and every step of this change was gated on
it reporting `identical`.

Also fixes 15 bugs found while reviewing the rewrite, seven of them
regressions against the previous parser implementation - dropped background
images, a hard error on the standard Ultralytics layout, silently ignored
COCO keypoint options, `KeyError` and orphaned datasets from count-based
split ratios, an import-time `KeyError` from a colliding plugin, and a
`LuxonisParser` that re-downloaded its source on every `parse()`.

Adds a `-m benchmark` suite that builds a large synthetic dataset for each
parser covering every feature it supports, and reorganizes the parser tests
into one module per dataset type under synthetic/, real_world/ and
benchmarks/.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation enhancement New feature or request tests Adding or changing tests data Changes affecting luxonis_ml.data subpackage utils Changes affecting luxonis_ml.utils subpackage CLI Changes affecting the CLI DevOps Changes related to DevOps labels Jul 30, 2026
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6e5a9e7a-523e-49cf-b952-1b484f98a99e

📥 Commits

Reviewing files that changed from the base of the PR and between 01e53d6 and 6624b7c.

📒 Files selected for processing (5)
  • luxonis_ml/data/parsers/coco_parser.py
  • luxonis_ml/data/parsers/fiftyone_classification_parser.py
  • tests/test_data/parsers/real_world/test_dir_parser.py
  • tests/test_data/parsers/synthetic/test_coco.py
  • tests/test_data/parsers/synthetic/test_fiftyone_classification.py
🚧 Files skipped from review as they are similar to previous changes (3)
  • tests/test_data/parsers/real_world/test_dir_parser.py
  • luxonis_ml/data/parsers/fiftyone_classification_parser.py
  • luxonis_ml/data/parsers/coco_parser.py

📝 Walkthrough

Walkthrough

This PR replaces the legacy parser flow with plugin-based streaming imports, adds source preparation and issue handling, and introduces parser benchmarks with comparison tooling and CI automation. It also updates documentation and expands parser, import, and integration tests.

Changes

Parser import architecture

Layer / File(s) Summary
Plugin contracts and registration
luxonis_ml/data/parsers/parser_plugin.py, luxonis_ml/data/parsers/__init__.py, luxonis_ml/data/__init__.py
Adds parser detection, streaming results, split handling, issue collection, registration, parser resolution, and public exports.
Dataset import orchestration
luxonis_ml/data/datasets/base_dataset.py, luxonis_ml/data/parsers/luxonis_parser.py, luxonis_ml/data/__main__.py
Routes imports through BaseDataset.import_dataset, supports file selection and task routing, preserves parser issues, and cleans up failed imports.
Built-in parser migration
luxonis_ml/data/parsers/*_parser.py
Migrates built-in parsers to plugin interfaces with streaming records, split-aware detection, and file enumeration.
Source handling and annotation validation
luxonis_ml/data/parsers/source.py, luxonis_ml/data/datasets/annotation.py
Adds local and remote source preparation and avoids mutation of caller-owned annotation data.

Benchmark infrastructure

Layer / File(s) Summary
Benchmark harness and synthetic datasets
tests/test_data/parsers/benchmarks/*
Adds deterministic benchmark generation, parser measurements, memory tracking, output digests, report comparison, and format-specific cases.
Benchmark commands and CI
.github/workflows/ci.yaml, tools/compare_benchmarks.py, tools/profile_parsers.py
Adds configurable benchmark execution, ref comparison, Markdown reporting, profiling, digest checks, PR comments, and regression failures.

Validation and documentation

Layer / File(s) Summary
Parser and import tests
tests/test_data/parsers/synthetic/*, tests/test_data/test_annotations.py, tests/test_data/test_zip_layout_equivalence.py
Adds coverage for streaming, split preservation, parser resolution, path handling, issue reporting, cleanup, annotation immutability, and wrapped ZIP imports.
Integration tests
tests/test_data/parsers/real_world/*, tests/test_data/parsers/benchmarks/test_*
Adds real-source import coverage and benchmark comparison tests.
Documentation and configuration
CONTRIBUTING.md, luxonis_ml/data/README.md, luxonis_ml/data/parsers/README.md, pyproject.toml, requirements-dev.txt
Documents the import API, parser plugins, benchmarks, profiling commands, and benchmark test configuration.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

Suggested labels: fix

Suggested reviewers: klemen1999, conorsim, tersekmatija

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 63.41% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main parser rewrite, streaming API, and benchmark changes.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/parsers-refactor

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

kozlov721 and others added 8 commits July 30, 2026 17:36
`image_size` was assigned `img.shape[:2]`, whose element type depends on
which numpy stubs are installed. Where that slice resolves to `Any`, the
assignment does not narrow the declared `tuple[int, ...] | None`, so pyright
in CI rejected the following unpack as `"None" is not iterable` while the
same version passed locally.

Unpacking into an explicit pair of `int`s narrows definitely under either
set of stubs. Parser output is unchanged - the digest reports `identical`
for all three YOLOv8 dataset types.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The benchmarks reported a table nobody could act on: one run of one ref,
with no way to tell a real slowdown from a busy runner. They now measure
two refs the same way and fail the release when one of them regressed.

The release pull request benchmarks its head and its base as a matrix, so
both are measured on the same runner image, and a third job compares the
two, posts the table as a single edited comment and exits non-zero past
the threshold. `workflow_dispatch` takes both refs, the threshold, the
scale, the repeat count and the aggregate; leaving the baseline empty
benchmarks one ref and compares it to nothing.

Telling a regression from noise needed measuring the noise first. Two
runs of identical code, same machine, differ by a median of 2.8% and up
to 11.5%, so the old table's few-percent movements said nothing. Three
things come out of that:

- `min` is the default reduction, not `mean`. Noise here is one-sided - a
  busy machine only ever makes a parse slower - so the fastest of N
  timings is the most reproducible summary. Measured over ten repeats:
  min varies 1.7% run to run at the median, mean 2.0%, median 2.8%.
- Every timing is kept, and its scatter reported. `clsdir` and
  `fiftyone-classification` parse in hundredths of a second and scatter
  by 7% and 77%; no repeat count fixes that.
- A parser counts as regressed only when it is both past the threshold
  and more than twice the two runs' combined scatter, which is what lets
  those two stay in the suite without failing a release on jitter.

Peak allocation is reported but never gates. It is reproducible within a
run of the suite and not across differently measured ones: the suite
generates its datasets in-process, which grows CPython's interned-string
table before anything is traced, and the same parse measured in a fresh
process pays for that growth inside the traced window and reports ~5 MiB
more.

Defaults are 2000 images per split doubled, ten repeats and a 25%
threshold, held there by the runner rather than by patience - scale 3
writes 1.7 GB across 290k files, which is not something to hand a hosted
runner that also has to finish inside an hour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running the benchmarks by hand turns the whole run red for two reasons
that have nothing to do with the benchmarks.

`auto-assigner` wants a pull request payload and errors out without one
("Can't get payload"), so every `workflow_dispatch` run failed on it. It
now runs only for pull requests, which is the only event it can serve.

`test_tensorflow_csv_file_list_does_not_replay_the_records` compared the
parsed records in order, and that order is the order the split was
globbed in — the file system's, not the parser's. The runner listed the
unannotated image first and the assertion failed there while passing
here. The assertion two lines above it already concedes this and sorts;
this one now does too, which still pins what the test is about: one
record per annotation row plus one for the unannotated image.

Verified by forcing the reverse of the local glob order, which fails
without this and passes with it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both compared records in the order the parser emitted them, and that is
the order `_list_images` globbed the split in - the file system's, not
the parser's. They pass here and would fail on a machine that lists the
two images the other way round, which is how the tensorflow-csv test
next to them was already failing in CI.

Neither test is about ordering. The tensorflow-csv one is about each
spelling of a filename landing on the image it names, four rows apiece;
the yolov6 one is about both images being present exactly once, and it
already looks the individual records up by name. Sorting both sides
keeps what they check and drops what they cannot rely on.

Found by running the parser suite with `_list_images` reversed, which is
the cheapest stand-in for a different file system. With these two fixed,
nothing under tests/test_data fails only under that reversal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage measures `tests` as well as `luxonis_ml`, and the run that
measures it deselects the benchmarks with `-m "not benchmark"`. The
dataset builders and the timing harness therefore reported as 20-39%
covered - not because anything is untested, but because that run cannot
execute them at all. Between them they contributed 1089 statements, 751
of which were unreachable by construction.

`comparison.py` and its tests stay measured. They decide whether a
release is allowed through and they do run in the default suite, so
their coverage is worth something; the rest of the directory's is not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every fix ships a regression test verified to fail without it, and
`tools/profile_parsers.py compare` reports `identical` for all 18
dataset types, so none of the parser changes move a record.

Silent data loss and destroyed datasets:

- `BBoxAnnotation`, `KeypointAnnotation` and `SegmentationAnnotation`
  still edited the caller's own annotation dict in place. Dropping the
  `deepcopy` from `DatasetRecord.validate_files` was only made safe for
  masks and polylines, so a producer reusing one annotation dict across
  files had it clipped by the first record and wrote the clipped values
  for every record after it, warning once.
- The failed-import cleanup caught `BaseException`, so a Ctrl-C halfway
  through a long import deleted everything it had already parsed and
  uploaded. Only `Exception` triggers it now.
- That cleanup also read `delete_local=True` as proof the import owned
  the dataset. For remote storage the constructor only clears the local
  cache, so a part-way failure deleted a pre-existing bucket dataset the
  import had merely appended to.
- Percentage `split_ratios` were validated only by `make_splits`, after
  every record was written and every image uploaded, and the cleanup
  then deleted the whole import. Counts were already checked up front;
  ratios now are too.
- `UltralyticsNDJSONParser.enumerate_files` counted images the parse
  skips, so count-based `split_ratios` quietly under-filled the splits.

Wrong or lost results:

- `YOLOv8Parser` picked the alphabetically first YAML as its class file,
  so a `data.yaml` next to any earlier-sorting YAML lost to it and every
  box was labelled with another dataset's classes. `data.yaml` wins now.
- `COCOParser.parse` dropped `**kwargs` on the split-directory path, so
  a mistyped parser argument raised `TypeError` for a single-split
  source and was silently ignored for the common layout.
- `LuxonisParser.get_parser_issue_messages` read through the dataset a
  failed parse never returns, so it answered `[]` for exactly the case
  it exists to explain.
- A class directory skipped for having a reserved name is now logged
  rather than silently dropped.

Also: the benchmark regression threshold fell back to 10% on release
pull requests, below the ~12% run-to-run noise its own tool documents,
against the 25% every other default uses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kozlov721 kozlov721 changed the title Parser plugin rewrite: streaming API, benchmarks, and 15 fixed bugs Parser plugin rewrite: streaming API, benchmarks, and 24 fixed bugs Jul 31, 2026
The pair of benchmark legs runs in parallel with the test matrix and
finishes inside the slowest shard, so measuring every pull request
instead of only `release/*` ones costs no wall-clock time. The baseline
is the branch the pull request merges into.

The report was already written to the job summary and to a single
edited pull request comment; neither had ever been seen on an ordinary
pull request, because the job never ran for one. Making it run exposed
three ways it breaks:

- `download-artifact` was given a `pattern`, which unpacks into the
  workspace root rather than a per-artifact directory when only one
  artifact matches - exactly what a failed matrix leg leaves behind.
  Every path in the report job then missed, and run 30622969650 failed
  on `jq: Could not open parser-benchmarks-current/benchmark-results.json`
  after a successful 12-minute benchmark. Each artifact is now fetched
  by name into a directory named after it.
- A baseline older than the benchmark suite - every ref on `main` until
  this merges - has nothing to measure. That leg now exits green with a
  notice, uploads nothing, and the report says the baseline is missing
  instead of the job going red on the pull request's behalf.
- With no results the report step ran `cat report.md` on a file no
  earlier step had written. A missing measurement now produces a report
  saying so, and `Fail on a regression` no longer claims a regression
  for what may have been an empty comparison.

Refs reach the shell through the environment rather than `${{ }}`
interpolation. A pull request branch is named by whoever opened it, and
this job now runs for all of them. Commenting is `continue-on-error`,
since a fork's token is read-only and cannot comment at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Parser benchmarks

feat/parsers-refactor. No results for main - it has no benchmark suite to run, so there is nothing to compare against.

dataset type images records seconds records/s peak MiB
clsdir $12{,}000$ $12{,}000$ $0.052$ $229{,}276$ $0.2$
coco $12{,}001$ $80{,}400$ $2.586$ $31{,}085$ $141.3$
createml $11{,}760$ $112{,}800$ $1.362$ $82{,}802$ $28.4$
darknet $12{,}000$ $109{,}200$ $0.358$ $304{,}646$ $4.9$
fiftyone-classification $12{,}000$ $11{,}880$ $0.046$ $257{,}234$ $3.2$
native $12{,}000$ $109{,}200$ $1.967$ $55{,}528$ $95.1$
segmask $3{,}000$ $54{,}300$ $0.318$ $171{,}006$ $1.7$
solo $13{,}200$ $168{,}000$ $6.661$ $25{,}222$ $0.5$
tfcsv $12{,}000$ $109{,}200$ $0.655$ $166{,}711$ $16.4$
ultralytics-ndjson $12{,}000$ $325{,}352$ $3.069$ $106{,}001$ $0.1$
ultralytics-ndjson-instancesegmentation $12{,}000$ $109{,}222$ $1.790$ $61{,}005$ $0.1$
ultralytics-ndjson-keypoints $12{,}000$ $109{,}222$ $1.778$ $61{,}413$ $5.1$
voc $12{,}000$ $87{,}600$ $2.196$ $39{,}882$ $2.8$
yolov4 $12{,}000$ $98{,}400$ $1.191$ $82{,}597$ $3.4$
yolov6 $12{,}000$ $111{,}360$ $0.360$ $309{,}636$ $4.8$
yolov8 $12{,}000$ $120{,}000$ $0.581$ $206{,}713$ $6.6$
yolov8instancesegmentation $12{,}000$ $120{,}000$ $1.473$ $81{,}470$ $7.0$
yolov8keypoints $12{,}000$ $120{,}000$ $1.206$ $99{,}471$ $2.0$

kozlov721 and others added 2 commits July 31, 2026 15:36
Each row of the comparison table carries a mark: `❗` past the
threshold, `⚠️` past half of it, `✅` otherwise, and the change is
coloured to match, so a reader does not have to hold the threshold in
their head while scanning. Dataset types are monospaced and the numbers
are set as math.

Two rules govern where the math can go, both found by rendering through
GitHub and reading the result:

- A closing `$` followed by a letter is not parsed as math at all -
  `$0.39$s` renders as literal text, dollar signs included. So no unit
  trails a delimiter; the seconds columns say so in the header instead.
- A `%` inside the math is worse, because it renders. Markdown strips
  the backslash from `\%` before MathJax sees it, and the bare `%` left
  behind opens a comment that swallows the rest of the expression. The
  percent signs stay outside.

A test asserts both rules over every math cell the renderer emits,
rather than over the two columns that happen to have units today.

Colours are the xcolor names rather than hex, which GitHub's MathJax
leaves unrendered.

Neither math nor colour renders in a job summary, which would print the
markup itself, so `render_markdown` takes `rich` and the tool writes
both forms: `--output` for the pull request comment, `--plain-output`
for the summary. The plain form keeps the marks and the monospaced
names, which render everywhere.

The single-ref table moves out of the `jq` block in the workflow and
into `render_single_markdown`, so both tables share the formatting and
the second one is testable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@luxonis luxonis deleted a comment from codecov Bot Jul 31, 2026
@kozlov721
kozlov721 marked this pull request as ready for review July 31, 2026 14:33
@kozlov721
kozlov721 requested a review from a team as a code owner July 31, 2026 14:34
@kozlov721
kozlov721 removed the request for review from a team July 31, 2026 14:34
@kozlov721
kozlov721 requested a review from klemen1999 July 31, 2026 14:34
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.15832% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.78%. Comparing base (f45cf41) to head (6624b7c).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
tests/test_data/parsers/benchmarks/comparison.py 87.82% 14 Missing ⚠️
luxonis_ml/data/datasets/base_dataset.py 91.15% 13 Missing ⚠️
tests/test_data/parsers/helpers.py 87.50% 7 Missing ⚠️
...data/parsers/segmentation_mask_directory_parser.py 91.93% 5 Missing ⚠️
...est_data/parsers/synthetic/test_plugin_registry.py 96.75% 5 Missing ⚠️
luxonis_ml/data/parsers/coco_parser.py 98.23% 4 Missing ⚠️
luxonis_ml/data/parsers/parser_plugin.py 98.91% 2 Missing ⚠️
luxonis_ml/data/parsers/yolov8_parser.py 98.36% 2 Missing ⚠️
luxonis_ml/data/datasets/annotation.py 97.14% 1 Missing ⚠️
...ml/data/parsers/classification_directory_parser.py 97.14% 1 Missing ⚠️
... and 3 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #461      +/-   ##
==========================================
+ Coverage   93.97%   96.78%   +2.80%     
==========================================
  Files         164      190      +26     
  Lines       12794    16447    +3653     
==========================================
+ Hits        12023    15918    +3895     
+ Misses        771      529     -242     
Flag Coverage Δ
pytest-ubuntu-latest 96.78% <98.15%> (+2.80%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

coderabbitai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLI Changes affecting the CLI data Changes affecting luxonis_ml.data subpackage DevOps Changes related to DevOps documentation Improvements or additions to documentation enhancement New feature or request tests Adding or changing tests utils Changes affecting luxonis_ml.utils subpackage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant