Move parsing into BaseDataset.import_dataset and replace BaseParser with a plugin API

[kozlov721]

Part 1 of 2, split out of #461.

#461 bundles two separable changes: a public API change, and a performance
rewrite of the parsers. This branch is the API change on its own. The
streaming rewrite, the benchmark suite and its CI job, and a second round of
review fixes stay on the follow-up branch feat/parser-optimizations, which
is stacked on this one.

The split is exact: merging this branch and the follow-up into main
reproduces #461's tree byte for byte (b3cb1907), with an empty diff against
feat/parsers-refactor.

What changes

A parser no longer owns the dataset it fills. It used to construct a
LuxonisDataset, write records into it, and hand it back. Now it is a
data-only plugin that yields records, and BaseDataset.import_dataset builds
the dataset around them.

# before
dataset = LuxonisParser("path/to/source", dataset_name="cats").parse()

# after
dataset = LuxonisDataset.import_dataset("path/to/source", dataset_name="cats")

import_dataset is a classmethod on BaseDataset, so every registered
dataset implementation inherits it and gets back an instance of the class it
was called on.

LuxonisParser still works and still returns the same dataset. It is now a
thin wrapper over import_dataset that raises a DeprecationWarning on
construction.

Breaking changes

Removed	Replacement
BaseParser	ParserPlugin / SplitParserPlugin
ParserOutput (a bare 3-tuple)	ParsedDataset(records, skeletons, files, splits)
BaseParser.from_dir() / from_split()	ParserPlugin.parse() / SplitParserPlugin._parse_split()
BaseParser.validate()	ParserPlugin.supports()
BaseParser.discover_dir_splits()	SplitParserPlugin.discover_splits()
BaseParser.parse_dir() / parse_split()	BaseDataset.import_dataset()
LuxonisParser	deprecated wrapper over BaseDataset.import_dataset, still present

A parser is registered with register_parser_plugin, or by publishing the
class through a parser_plugins entry point. The type names it answers to
are its dataset_types class variable.

So a third-party parser needs: supports(), parse() — or _parse_split()
plus validate_split() for split layouts — and a dataset_types entry.

Bugs fixed

Each has a regression test that fails without its fix.

Silent data loss

1. _prepare_import_records dropped annotation-less images — background
   images in YOLOv6/v8, COCO, VOC and Darknet — until the first annotated
   record. A string task_name became a defaultdict, which is only
   populated on lookup, so the fan-out set for unannotated records was empty.
   Order-dependent loss.
2. The standard Ultralytics images/train + images/val layout matched both
   YOLOv6 and YOLOv8 and hard-errored as ambiguous. Resolved by split
   coverage; genuine ties still raise.
3. Single-split COCO silently dropped use_keypoint_ann and
   keypoint_ann_paths, so a pose model could be trained on a keypoint-free
   dataset. It raises now.

Failures that landed after the dataset was already on disk

4. Partial or empty count-based split_ratios raised a bare KeyError after
   the dataset had been created.
5. Count ratios selecting zero samples crashed in make_splits and left an
   orphaned registered dataset.

Both are validated before construction now.

Plugin and API surface

6. An entry-point plugin whose dataset_types collided with a built-in made
   plain import luxonis_ml.data raise KeyError, breaking every downstream
   consumer. It warns and skips now, and registration is atomic — a collision
   on a later type no longer leaves the earlier ones registered.
7. LuxonisParser deferred source acquisition into parse(), so
   construction-time errors stopped firing and every parse() call
   re-downloaded the source and returned a new dataset. The source is
   acquired and its format resolved in __init__ again.

Pre-existing, fixed along the way

8. _list_images was case-sensitive — it listed .WebP but not .JPG — so
   datasets with uppercase extensions were rejected or silently truncated.
9. YOLOv8 raised UnboundLocalError on a label line with fewer than five
   values; it is skipped via ParserIssue.MALFORMED_ANNOTATION now.
10. A trailing / on the source produced the dataset name "", writing
    storage directories into the datasets root and merging into prior imports.
11. ClassificationDirectoryParser.validate_split excluded data, raw and
    masks, but the parse then ingested them as classes.

Deliberately not in this branch

These are in feat/parser-optimizations, and reviewing this branch as if they
were missing oversights would be reading it wrong:

• the single-walk rewrite (enumerate_files) — every parser here still walks
  its source twice, once for records and once for the file list
• removal of the per-record model_copy(deep=True)
• ParseResult and Layout; this branch still has ParsedDataset and
  combine_split_outputs
• the annotation-validator in-place-mutation fixes
• the benchmark suite, tools/profile_parsers.py,
  tools/compare_benchmarks.py, and the CI benchmark job

Testing

tests/test_data/test_parser_plugins.py is new — 61 credential-free tests
covering registration, detection, split discovery and the plugin base — and
test_parsers.py is extended alongside it. ruff check and ruff format --check are clean.

I ran the parser suite locally. The only failures were the tests that download
from the test bucket, which fail on expired GCS credentials in my environment
rather than on anything in this change; the credential-free tests all pass.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 657ff7f9-ef1e-4b92-a1b6-0d53263dd11d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 99.36232% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.94%. Comparing base (4763b22) to head (e7ebf70).

Files with missing lines	Patch %	Lines
luxonis_ml/data/datasets/base_dataset.py	94.62%	5 Missing ⚠️
tests/test_data/test_parser_plugins.py	99.60%	3 Missing ⚠️
luxonis_ml/data/parsers/parser_plugin.py	99.47%	1 Missing ⚠️
luxonis_ml/data/parsers/solo_parser.py	95.83%	1 Missing ⚠️
tests/test_data/test_parsers.py	99.67%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #478      +/-   ##
==========================================
+ Coverage   95.03%   96.94%   +1.90%     
==========================================
  Files         177      179       +2     
  Lines       14557    15500     +943     
==========================================
+ Hits        13834    15026    +1192     
+ Misses        723      474     -249     

Flag	Coverage Δ
pytest-ubuntu-latest	96.94% <99.36%> (+1.90%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.