Skip to content

chore: parallelize testing in ci with xdist #3906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bcmyguest wants to merge 6 commits into
base: main
Choose a base branch
from
+17 −7
Open

chore: parallelize testing in ci with xdist #3906

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
af472fd
chore: parallelize testing in ci with xdist
bcmyguest Aug 26, 2025
8070b8b
fix: coverage not working with xdist
bcmyguest Aug 26, 2025
236c797
chore: remove accidentally committed file
bcmyguest Aug 27, 2025
18e9493
chore: Add --cov-branch option to default pytest args
bcmyguest Aug 27, 2025
32ac995
chore: Update default pytest arguments in action.yml
bcmyguest Aug 27, 2025
8e17a84
Merge branch 'main' into parallel-tests
bcmyguest Feb 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions .github/actions/run-tests/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ inputs:
pytest-args:
description: 'Additional arguments to pass to pytest'
required: false
default: '--mpl -W error::metpy.deprecation.MetpyDeprecationWarning'
default: '--mpl -W error::metpy.deprecation.MetpyDeprecationWarning -n auto --cov --cov-report=xml'
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description mentions concerns about matplotlib not being thread-safe, which could make tests flaky with parallel execution. Consider adding pytest-xdist configuration to use 'loadscope' or 'loadfile' distribution mode instead of the default 'load' mode to reduce potential race conditions. This can be configured in pyproject.toml under [tool.pytest.ini_options] by adding 'addopts = "-n auto --dist=loadfile"' or similar. The '--dist=loadfile' mode ensures all tests in a file run on the same worker, which can help with matplotlib's thread safety issues.

Copilot uses AI. Check for mistakes.
runs:
using: composite
steps:
Expand All @@ -30,10 +30,8 @@ runs:
# By running coverage in "parallel" mode and "combining", we can clean up the path names
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment about running coverage in "parallel" mode and "combining" is now outdated since coverage is being run via pytest-cov plugin rather than directly. This comment should be updated or removed to reflect the new approach.

Suggested change
# By running coverage in "parallel" mode and "combining", we can clean up the path names
# Coverage is collected via pytest-cov as configured in the pytest-args input

Copilot uses AI. Check for mistakes.
run: |
set -e -o pipefail
python -m coverage run -p -m pytest ${{ inputs.pytest-args }} tests/ 2>&1 | tee tests-${{ inputs.key }}.log
python -m coverage combine
python -m coverage report
python -m coverage xml
python -m pytest ${{ inputs.pytest-args }} tests/ 2>&1 | tee tests-${{ inputs.key }}.log


- name: Run doctests
if: ${{ inputs.run-doctests == 'true' }}
Expand Down
9 changes: 8 additions & 1 deletion CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,14 @@ To run these tests, use:

```sh
pytest --mpl
````
```

To run tests in parallel, use `-n <number_of_processes>` or `-n auto` (using `pytest-xdist`):

```sh
pytest --mpl -n auto
pytest --mpl -n 2
```

When adding new image comparison tests, start by creating the baseline images for the tests:

Expand Down
5 changes: 4 additions & 1 deletion ci-dev/test_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@

Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This blank line at the top of the file is unnecessary and should be removed for consistency with the previous version of the file.

Suggested change

Copilot uses AI. Check for mistakes.
packaging==26.0
pytest==9.0.0
pytest-cov==6.2.1
pytest-mpl==0.18.0
pytest-xdist==3.8.0
coverage==7.13.0
vcrpy==8.1.0
vcrpy==8.1.0
2 changes: 2 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ test = [
"netCDF4",
"packaging>=21.0",
"pytest>=7.0",
"pytest-xdist",
"pytest-cov",
Comment on lines +61 to +62
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When using pytest-xdist with pytest-cov, it's recommended to configure coverage for parallel execution. While pytest-cov should handle this automatically in recent versions, explicitly setting 'parallel = true' in the [run] section of .coveragerc (or adding [tool.coverage.run] with parallel = true in pyproject.toml) ensures accurate coverage collection across parallel workers. This is particularly important for consistent coverage reporting.

Copilot uses AI. Check for mistakes.
"pytest-mpl",
"vcrpy>=4.3.1"
]
Expand Down
Loading