Skip to content

fix(regression): MAPE near_zero_values counts rows, not DataFrame cells - #1933

Open
feiiiiii5 wants to merge 1 commit into
evidentlyai:mainfrom
feiiiiii5:r13b-mape-near-zero-count
Open

feiiiiii5 wants to merge 1 commit into
evidentlyai:mainfrom
feiiiiii5:r13b-mape-near-zero-count

Conversation

@feiiiiii5

Copy link
Copy Markdown

Problem

The MAPE metric tells the user how many rows were affected by near-zero targets, and that number is always twice the real count.

User-visible symptom (insight rendered next to the MAPE percentage-error plot, src/evidently/metrics/regression.py:356-366):

Near-zero values detected (|target| ≤ epsilon). Applied zero_handling='replace'. Affected rows — current: 6, reference: 4.

for a dataset that actually has 3 near-zero rows in current and 2 in reference.

The same doubled number is persisted in report snapshots as MoreRegressionMetrics.near_zero_values, so it is also part of the serialized API surface, not just the HTML text.

Root cause

src/evidently/legacy/calculations/regression_performance.py, _calculate_quality_metrics:

data = dataset[[prediction_column, target_column]]      # 2 columns
...
epsilon_values = data[~(abs(data[target_column]) > epsilon)]
...
"near_zero_values": epsilon_values.size,

epsilon_values is a DataFrame with two columns, and pandas.DataFrame.size is the number of cells (len(df) * len(df.columns)), not the number of rows. The filter always selects exactly the [prediction, target] pair, so .size is 2 * <affected rows>.

The counter was introduced in #1835 ("Add methods to handle near-zero values for MAPE metric"). RegressionPerformanceMetrics.near_zero_values is annotated int and every consumer treats it as a row count:

  • src/evidently/metrics/regression.py:351 — if cur_near_zero_values + ref_near_zero_values > 0
  • src/evidently/metrics/regression.py:362 — f" Affected rows — current: {cur_near_zero_values}"

Nothing depends on the doubled value, and no existing test asserted it.

Fix

Report rows instead of cells:

"near_zero_values": len(epsilon_values),

The other use of .size in the same function (epsilon_values.size > 0, a pure emptiness check) is left untouched — it is correct either way and changing it would be churn.

Reproduction / verification

Environment: macOS, Python 3.11.15, pytest 7.4.4, pandas 3.0.5, numpy 2.4.6, evidently installed editable from this branch.

Minimal trigger (calculate_regression_performance with 3 rows whose target is exactly 0.0):

expected near-zero rows: 3
reported near_zero_values: 6     # before the fix
reported near_zero_values: 3     # after the fix

New regression test tests/calculations/test_regression_performance.py.

Base commit 34771fe41c08ca074d96a52aaf4a45cd640cbaf5, test file added, fix not applied:

python -m pytest tests/calculations/test_regression_performance.py -v
...
E       AssertionError: assert 6 == 3
5 failed, 1 passed

Same command with the fix applied:

python -m pytest tests/calculations/test_regression_performance.py -v
...
6 passed

Control runs with an identical command, to separate pre-existing failures:

python -m pytest tests/calculations tests/tests/test_regression_performance_tests.py tests/utils tests/report -q
tree command result
base commit (new test not present) the 4-path control command 178 passed
base commit + new test (fix line mutated back) the 4-path control command 5 failed, 179 passed
this branch (fix + new test) the 4-path control command 184 passed

The mutation run above is the base-code control for the new tests: on this branch I reverted only the fixed line back to epsilon_values.size and re-ran the same command — exactly the 5 count-sensitive tests fail and nothing else.

ruff format --diff reports both changed files as already formatted.

Scope and limitations

  • Single-line behaviour fix plus a new test module. No API, schema or serialization format change; only the value written to the existing near_zero_values field becomes correct.
  • Snapshots produced by earlier releases keep the doubled value they were written with; this PR does not rewrite existing snapshots.
  • Local run used PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 because an unrelated third-party pytest plugin in the shared environment failed to import. The full suite was not run locally (it needs the bikes/scipy datasets and the [llm,spark] extras that CI downloads), so the control runs above are limited to tests/calculations, tests/tests/test_regression_performance_tests.py, tests/utils and tests/report.
  • My local pandas/numpy are newer than the requirements.min.txt pins (pandas 3.0.5 / numpy 2.4.6 vs 1.3.5 / 1.23.0). The fix is a len() vs .size distinction, which is stable across those versions, but the matrix runs in CI are the authoritative check.
  • Separately noticed while reading this code, not changed here: the MAPE.zero_handling docstring in src/evidently/metrics/regression.py says rows where "the target or prediction" is within ±epsilon are handled, while the implementation and the insight text only consider the target. That is an independent documentation/behaviour question and is left to the maintainers.

`_calculate_quality_metrics` selects the prediction and target columns into
a two-column DataFrame before detecting near-zero targets, then reports the
count of affected rows with `epsilon_values.size`. For a DataFrame, `.size`
is the number of cells (rows x columns), so `near_zero_values` is always
twice the real number of rows.

The value is exposed to users as `MoreRegressionMetrics.near_zero_values` in
report snapshots and rendered into the MAPE insight "Affected rows -
current: N". Use `len(epsilon_values)` so the counter reports rows.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant