Skip to content

Vectorize check_conditional_requirement (~127x speedup)#266

Merged
sstruzik merged 5 commits into
mainfrom
enhance/validation_CR
May 19, 2026
Merged

Vectorize check_conditional_requirement (~127x speedup)#266
sstruzik merged 5 commits into
mainfrom
enhance/validation_CR

Conversation

@sstruzik
Copy link
Copy Markdown
Contributor

@sstruzik sstruzik commented May 6, 2026

Summary

  • Replace the per-row df.apply(check_cr, axis=1) in Validator.check_conditional_requirement with column-wise boolean masks. Per trigger column, a single vectorized (~blank) & non_default & truthy mask is computed once; per required field, the OR of the masks of triggers that demand it is intersected with that field's blank mask.
  • The remaining row-wise apply runs only over the subset of rows that actually have a missing CR field, just to build the comma-joined label string.
  • Behavior parity preserved: numeric 0 does not trigger, blank cells do not trigger, cells equal to the schema Default do not trigger, and a required field whose column is entirely absent is reported by its field name (matching the original).

Performance

Measured against exposure (location 9501x80, account 221x41):

Step Before After
check_conditional_requirement 7.239 s 0.057 s
TOTAL checks 7.866 s 0.756 s

Test plan

  • pytest tests/test_ods_package.py -v (50 passed)
  • Run ods_tools check against an exposure that intentionally trips conditional-requirement errors and diff the messages against the prior implementation row-by-row

🤖 Generated with Claude Code

@sstruzik sstruzik requested a review from SkylordA May 6, 2026 12:26
@sstruzik sstruzik self-assigned this May 7, 2026
@sstruzik sstruzik moved this to Waiting for Review in Oasis Dev Team Tasks May 7, 2026
Comment thread ods_tools/oed/validator.py Outdated
Comment thread ods_tools/oed/validator.py Outdated
Comment thread tests/test_ods_package.py Outdated
Comment thread ods_tools/oed/validator.py Outdated
@SkylordA SkylordA self-requested a review May 19, 2026 06:53
@sstruzik sstruzik merged commit 1907e28 into main May 19, 2026
15 checks passed
@github-project-automation github-project-automation Bot moved this from Waiting for Review to Done in Oasis Dev Team Tasks May 19, 2026
@sambles sambles deleted the enhance/validation_CR branch May 28, 2026 08:38
@github-actions github-actions Bot mentioned this pull request May 28, 2026
@awsbuild awsbuild added this to the 5.0.5 milestone Jun 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants