fix: scope DUPLICATE_COUNT under --filter to the keys the window contains

[gkhnelbstn]

The bug

--filter narrows every check to the rows it selects. For uniqueness that answers "are today's rows unique among themselves", which misses a duplicate whose other half arrived outside the window.

We hit this on real data: a windowed field_unique passed every day for 45 days. The same contract read unfiltered against the same table reported duplicate_count(order_id) was 8 -- 8 duplicate primary keys on a column declared primaryKey: true and unique: true, invisible the whole time.

Filed as #1593.

The fix

Under --filter, DUPLICATE_COUNT reads every row of the table -- selected or not -- whose key occurs among the selected rows. A semi-join, as suggested in review:

SELECT key, COUNT(*) FROM t
WHERE EXISTS (SELECT 1 FROM t AS w WHERE <filter> AND w.key IS NOT DISTINCT FROM t.key)
GROUP BY key HAVING COUNT(*) > 1

That catches a key repeated within the window and one repeated across its edge, and skips keys the window does not contain, whose duplicates belong to another window. Without a filter nothing changes.

The change is in _run_model and a new _duplicate_scope() in datacontract/engines/ibis/ibis_check_execute.py, built with ibis's semi_join. On duckdb it compiles to:

FROM "data" AS "t1"
SEMI JOIN (SELECT DISTINCT "t0"."id" FROM "data" AS "t0" WHERE "t0"."batch" = 5) AS "t3"
  ON "t1"."id" IS NOT DISTINCT FROM "t3"."id"

The join is null-safe on purpose. GROUP BY puts NULL keys in one group, so the unfiltered check counts two NULL ids as a duplicate. A plain key IN (...) never matches NULL, so the same rows would pass when filtered. IS NOT DISTINCT FROM (ibis identical_to) keeps the two answers the same. Whether NULL keys should count as duplicates is a separate question; this only keeps filtered and unfiltered from disagreeing about it.

Two consequences, both keeping the check's behaviour from before this PR:

• A percent threshold is of the filtered row count, as for every other filtered check. The count is of the window's keys, so the window's rows are the matching denominator.
• A --filter that does not compile errors DUPLICATE_COUNT like every other check that reads rows. The check needs the window's keys to know which rows to read.

--include-failed-samples reads the same scoped rows as the count, so the samples and the number agree.

Testing

tests/test_row_filter_duplicate_count.py, against tests/fixtures/row-filter-duplicate/, whose batches hold: a key repeated across batches 1 and 2, a key repeated within batch 3, a batch-4 key that never repeats, and two NULL ids in batch 5.

case	filter	result
unfiltered	--	failed
duplicate across the window edge	batch = 1	failed
duplicate within the window	batch = 3	failed
duplicate wholly outside the window	batch = 4	passed
NULL key	batch = 5	failed, as unfiltered
sibling row_count still narrows	batch = 1	passed
filter that does not compile	no_such_column = 1	error

Two of these pin the semantics rather than only the happy path. Replacing identical_to with == fails the NULL case; reading the whole table instead of the semi-join fails the wholly-outside case. Both mutations were run.

Locally against duckdb: the 7 tests pass, the 13 in test_test_row_filter.py pass, the 59 matching unique, duplicate or samples across the suite pass, and ruff check / ruff format --check are clean.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[jschoedl]

Hi @gkhnelbstn. thank you for creating the issue and the PR!

Right now you check for duplicates in the whole table. But since a filter is set, it would be enough if we do the check (a) within the selected rows and (b) between selected and unselected rows - i.e., not within the unselected rows.

In other words, I'd propose to save costs by using a semi-join between the filtered rows and the whole table, conceptually like this:

SELECT key, COUNT(*) FROM my_table
WHERE key IN (SELECT DISTINCT key FROM my_table WHERE <filter>)
GROUP BY key HAVING COUNT(*) > 1

[gkhnelbstn]

Thanks @jschoedl, that is a better shape and it's done in ceb6f6e. Under --filter the check now reads the rows of the whole table whose key occurs among the selected rows, so it covers (a) and (b) and skips duplicates wholly outside the window. There's a test for each of the three, and the one for the wholly-outside case passes a window whose only key never repeats, even though the table has duplicates elsewhere.

One deliberate difference from the SQL you sketched: the join is IS NOT DISTINCT FROM (ibis identical_to), not IN. GROUP BY puts NULL keys in one group, so the unfiltered check counts two NULL ids as a duplicate. key IN (...) never matches NULL, so the same rows would pass when filtered. The null-safe join keeps the filtered and unfiltered answers the same. On duckdb it compiles to a SEMI JOIN ... ON a.id IS NOT DISTINCT FROM b.id.

If you'd rather NULL keys not count as duplicates at all, that's a reasonable call, but it would change the unfiltered check too, so I've left it out of this PR.

Two things went back to how they were before this PR, since the check now depends on the filter:

• a percent threshold is of the filtered row count again, like every other filtered check;
• a --filter that doesn't compile errors DUPLICATE_COUNT like the other row-reading checks, because the check needs the window's keys.

I've rewritten the PR description to match.