Skip to content

Row filter is all-or-nothing: table-level invariants like uniqueness need to opt out of --filter #1593

Description

@gkhnelbstn

The problem

A row filter is all-or-nothing: --filter narrows every check in the run.
But checks divide cleanly into two kinds, and they want opposite windows.

Row-level checks are about the rows that arrived. Scoping
columnValuesToBeNotNull to today answers "did today's load bring nulls",
which is what an operator acts on. Unscoped, a null that arrived three weeks
ago keeps the check red forever and today's load is invisible.

Table-level invariants are about the whole table. Uniqueness is the clear
case: scoped to one day, field_unique only sees duplicates that arrive within
the same batch. A row that duplicates a key loaded last week passes.

We hit this with real data. Our windowed uniqueness check passed every day for
45 days; running the same contract unfiltered reported
duplicate_count(order_id) was 8. Eight duplicate primary keys, invisible for
six weeks, on a check declared primaryKey: true and unique: true.

Referential integrity has the same shape — the child rows are today's, the
parent table is all of it — and that one is expressible today by writing the
join in SQL. Uniqueness derived from unique: true is not, because the check
is generated rather than written.

What would fix it

A per-rule scope, so a contract can say which window a check means. Something
like:

properties:
  - name: order_id
    unique: true
    scope: table        # ignore --filter for this check
  - name: customer_id
    required: true      # default: follows --filter

or the same idea on the quality rules themselves. The name matters less than
the ability to say it; today the only way to get both is two runs of the same
contract against two servers, and then the results are two documents that have
to be merged by whoever consumes them.

This is arguably an ODCS question rather than a CLI one — if so I am happy to
take it to bitol-io. Raising it here first because the CLI is where the window
exists at all.

Related

--filter is currently unusable on postgres for an unrelated reason:
#1592

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions