Add handling for corrupted rows e.g. enum property onBadLines with options ERROR, WARN or SKIP to all ToIon plugins e.g. CsvToIon #80
Description
Feature description
in pandas, there is a "on_bad_lines" property:
on_bad_lines{‘error’, ‘warn’, ‘skip’}, default ‘error’
Specifies what to do upon encountering a bad line (a line with too many fields). Allowed values are :
'`error`', raise an Exception when a bad line is encountered.
'`warn`', raise a warning when a bad line is encountered and skip that line.
'`skip`', skip bad lines without raising or warning when they are encountered.
It's worth adding an enum property onBadLines with options ERROR, WARN or SKIP to all ToIon tasks to allow a more configurable handling of bad lines:
