Conversation
`_calculate_quality_metrics` selects the prediction and target columns into a two-column DataFrame before detecting near-zero targets, then reports the count of affected rows with `epsilon_values.size`. For a DataFrame, `.size` is the number of cells (rows x columns), so `near_zero_values` is always twice the real number of rows. The value is exposed to users as `MoreRegressionMetrics.near_zero_values` in report snapshots and rendered into the MAPE insight "Affected rows - current: N". Use `len(epsilon_values)` so the counter reports rows.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The MAPE metric tells the user how many rows were affected by near-zero targets, and that number is always twice the real count.
User-visible symptom (insight rendered next to the MAPE percentage-error plot,
src/evidently/metrics/regression.py:356-366):for a dataset that actually has 3 near-zero rows in
currentand 2 inreference.The same doubled number is persisted in report snapshots as
MoreRegressionMetrics.near_zero_values, so it is also part of the serialized API surface, not just the HTML text.Root cause
src/evidently/legacy/calculations/regression_performance.py,_calculate_quality_metrics:epsilon_valuesis a DataFrame with two columns, andpandas.DataFrame.sizeis the number of cells (len(df) * len(df.columns)), not the number of rows. The filter always selects exactly the[prediction, target]pair, so.sizeis2 * <affected rows>.The counter was introduced in #1835 ("Add methods to handle near-zero values for MAPE metric").
RegressionPerformanceMetrics.near_zero_valuesis annotatedintand every consumer treats it as a row count:src/evidently/metrics/regression.py:351—if cur_near_zero_values + ref_near_zero_values > 0src/evidently/metrics/regression.py:362—f" Affected rows — current: {cur_near_zero_values}"Nothing depends on the doubled value, and no existing test asserted it.
Fix
Report rows instead of cells:
The other use of
.sizein the same function (epsilon_values.size > 0, a pure emptiness check) is left untouched — it is correct either way and changing it would be churn.Reproduction / verification
Environment: macOS, Python 3.11.15, pytest 7.4.4, pandas 3.0.5, numpy 2.4.6,
evidentlyinstalled editable from this branch.Minimal trigger (
calculate_regression_performancewith 3 rows whose target is exactly 0.0):New regression test
tests/calculations/test_regression_performance.py.Base commit
34771fe41c08ca074d96a52aaf4a45cd640cbaf5, test file added, fix not applied:Same command with the fix applied:
Control runs with an identical command, to separate pre-existing failures:
178 passed5 failed, 179 passed184 passedThe mutation run above is the base-code control for the new tests: on this branch I reverted only the fixed line back to
epsilon_values.sizeand re-ran the same command — exactly the 5 count-sensitive tests fail and nothing else.ruff format --diffreports both changed files as already formatted.Scope and limitations
near_zero_valuesfield becomes correct.PYTEST_DISABLE_PLUGIN_AUTOLOAD=1because an unrelated third-party pytest plugin in the shared environment failed to import. The full suite was not run locally (it needs the bikes/scipy datasets and the[llm,spark]extras that CI downloads), so the control runs above are limited totests/calculations,tests/tests/test_regression_performance_tests.py,tests/utilsandtests/report.requirements.min.txtpins (pandas 3.0.5 / numpy 2.4.6 vs 1.3.5 / 1.23.0). The fix is alen()vs.sizedistinction, which is stable across those versions, but the matrix runs in CI are the authoritative check.MAPE.zero_handlingdocstring insrc/evidently/metrics/regression.pysays rows where "the target or prediction" is within ±epsilon are handled, while the implementation and the insight text only consider the target. That is an independent documentation/behaviour question and is left to the maintainers.