process_s3_file (datacontract/engines/fastjsonschema/check_jsonschema.py:202-217) validates only the last object matched by the location glob. Every other file is downloaded and then discarded unvalidated, so an invalid record in any file but the last goes unreported and the run still appends the passed check "All JSON entries are valid." for the model.
Cause: the loop body only assigns a lazy generator over the current file's content to json_stream, overwriting the previous file's generator before anything consumes it; validate_json_stream is then called once, after the loop.
Applies whenever location matches more than one object — e.g. the documented s3://my-bucket/orders/*.json, or a {model} pattern expanding onto several files. Secondary effect: the full dataset is downloaded (and paid for in egress) even though only one file is checked.
Fix direction: validate inside the loop and accumulate the exceptions across files.
Where to start: the fix is in process_s3_file — call validate_json_stream per file inside the loop and collect the exceptions from all files before process_exceptions.
How to test: tests/test_test_s3_json_multiple_models.py shows the MinIO/testcontainers setup (needs Docker). A regression test uploads two JSON objects matched by one glob, with the invalid record in the first one, and asserts the run fails.
process_s3_file(datacontract/engines/fastjsonschema/check_jsonschema.py:202-217) validates only the last object matched by thelocationglob. Every other file is downloaded and then discarded unvalidated, so an invalid record in any file but the last goes unreported and the run still appends the passed check "All JSON entries are valid." for the model.Cause: the loop body only assigns a lazy generator over the current file's content to
json_stream, overwriting the previous file's generator before anything consumes it;validate_json_streamis then called once, after the loop.Applies whenever
locationmatches more than one object — e.g. the documenteds3://my-bucket/orders/*.json, or a{model}pattern expanding onto several files. Secondary effect: the full dataset is downloaded (and paid for in egress) even though only one file is checked.Fix direction: validate inside the loop and accumulate the exceptions across files.
Where to start: the fix is in
process_s3_file— callvalidate_json_streamper file inside the loop and collect the exceptions from all files beforeprocess_exceptions.How to test:
tests/test_test_s3_json_multiple_models.pyshows the MinIO/testcontainerssetup (needs Docker). A regression test uploads two JSON objects matched by one glob, with the invalid record in the first one, and asserts the run fails.