The problem
A row filter is all-or-nothing: --filter narrows every check in the run.
But checks divide cleanly into two kinds, and they want opposite windows.
Row-level checks are about the rows that arrived. Scoping
columnValuesToBeNotNull to today answers "did today's load bring nulls",
which is what an operator acts on. Unscoped, a null that arrived three weeks
ago keeps the check red forever and today's load is invisible.
Table-level invariants are about the whole table. Uniqueness is the clear
case: scoped to one day, field_unique only sees duplicates that arrive within
the same batch. A row that duplicates a key loaded last week passes.
We hit this with real data. Our windowed uniqueness check passed every day for
45 days; running the same contract unfiltered reported
duplicate_count(order_id) was 8. Eight duplicate primary keys, invisible for
six weeks, on a check declared primaryKey: true and unique: true.
Referential integrity has the same shape — the child rows are today's, the
parent table is all of it — and that one is expressible today by writing the
join in SQL. Uniqueness derived from unique: true is not, because the check
is generated rather than written.
What would fix it
A per-rule scope, so a contract can say which window a check means. Something
like:
properties:
- name: order_id
unique: true
scope: table # ignore --filter for this check
- name: customer_id
required: true # default: follows --filter
or the same idea on the quality rules themselves. The name matters less than
the ability to say it; today the only way to get both is two runs of the same
contract against two servers, and then the results are two documents that have
to be merged by whoever consumes them.
This is arguably an ODCS question rather than a CLI one — if so I am happy to
take it to bitol-io. Raising it here first because the CLI is where the window
exists at all.
Related
--filter is currently unusable on postgres for an unrelated reason:
#1592
The problem
A row filter is all-or-nothing:
--filternarrows every check in the run.But checks divide cleanly into two kinds, and they want opposite windows.
Row-level checks are about the rows that arrived. Scoping
columnValuesToBeNotNullto today answers "did today's load bring nulls",which is what an operator acts on. Unscoped, a null that arrived three weeks
ago keeps the check red forever and today's load is invisible.
Table-level invariants are about the whole table. Uniqueness is the clear
case: scoped to one day,
field_uniqueonly sees duplicates that arrive withinthe same batch. A row that duplicates a key loaded last week passes.
We hit this with real data. Our windowed uniqueness check passed every day for
45 days; running the same contract unfiltered reported
duplicate_count(order_id) was 8. Eight duplicate primary keys, invisible forsix weeks, on a check declared
primaryKey: trueandunique: true.Referential integrity has the same shape — the child rows are today's, the
parent table is all of it — and that one is expressible today by writing the
join in SQL. Uniqueness derived from
unique: trueis not, because the checkis generated rather than written.
What would fix it
A per-rule scope, so a contract can say which window a check means. Something
like:
or the same idea on the quality rules themselves. The name matters less than
the ability to say it; today the only way to get both is two runs of the same
contract against two servers, and then the results are two documents that have
to be merged by whoever consumes them.
This is arguably an ODCS question rather than a CLI one — if so I am happy to
take it to bitol-io. Raising it here first because the CLI is where the window
exists at all.
Related
--filteris currently unusable on postgres for an unrelated reason:#1592