Feature Request: Validations that return Warnings instead of Failures

[dylan-zeno]

I would like to be able to somehow "tier" certain schema issues. For instance, if a row is missing an ID, then that is an issue that needs to be addressed and should present as an error when the validation is run. But if some ancillary field is null then perhaps that is not strictly an issue to the degree of lacking an ID, and should simply present as a Warning, or some other category of issue.

[AndreasAlbertQC]

hey @dylan-zeno, interesting question. I like that schemas currently are quite straightforward to understand and if I tell you that you get a dy.DataFrame[SomeSchema], you can just assume that all constraints in SomeSchema are fully OK on the passed data. Loosening this setup to permit warning-only rules would make this harder. I also wonder if throwing a warning is truly a good way of handling mildly-wrong data, because after all your code afterwards needs to be able to handle that data anyway, so why not handle it explicitly right away?

That being said, with current functionality, one way to accomplish something like what you describe would be to just implement two schemas, where a stricter version inherits from a looser version. You then only validate the loose version, and use filter on the strict version to get your warning rows:

import polars as pl

import dataframely as dy

class BaseSchema(dy.Schema):
    important_id = dy.Int64(primary_key=True)
    ancillary_field = dy.String()

class WarningSchema(BaseSchema):
    """
    no new columns defined here,
    we just add one or multiple rules
    that we want to use for warnigns
    """

    @dy.rule
    def ancillary_not_null() -> pl.Expr:
        return pl.col("ancillary_field").is_not_null()
    

def main():
    mydf: pl.DataFrame

    # Will error if BaseSchema is violated
    validated = BaseSchema.validate(mydf)

   # .filter does not raise, just splits good and bad rows
    validated_and_no_warnings, failures = WarningSchema.filter(
        validated
    )
    your_warning_code(failures)