Skip to content

Commit 1a48ae7

Browse files
sadym-chromiumsadym-chromium
authored
test: measure screenshot performance (#3902)
Step 1: just upload performance test results to artifacts. Analogue of Selenium performance tests. ### Out of scope * Results presentation. * Exclude performance tests from normal e2e runs.
1 parent b536b0c commit 1a48ae7

File tree

7 files changed

+393
-7
lines changed

7 files changed

+393
-7
lines changed

‎.github/workflows/performance.yml‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# This workflow will run measure performance in different environments.
2+
3+
name: Measure performance
4+
5+
# Declare default permissions as read only.
6+
permissions: read-all
7+
8+
env:
9+
DEBUG: 'bidi:server:*,bidi:mapper:*'
10+
DEBUG_DEPTH: 10
11+
FORCE_COLOR: 3
12+
PIP_DISABLE_PIP_VERSION_CHECK: 1
13+
14+
on:
15+
merge_group:
16+
pull_request:
17+
push:
18+
branches: 'main'
19+
workflow_dispatch:
20+
inputs:
21+
verbose:
22+
description: Verbose logging
23+
default: false
24+
required: false
25+
type: boolean
26+
27+
concurrency:
28+
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
29+
cancel-in-progress: true
30+
31+
jobs:
32+
performance_metric:
33+
name: ${{ matrix.kind }}-${{ matrix.os }}-${{ matrix.head }}
34+
strategy:
35+
fail-fast: false
36+
matrix:
37+
# TODO(#876): Add Windows CI.
38+
os: [ubuntu-latest, macos-latest]
39+
head: [headful, 'new-headless', 'old-headless']
40+
# `cd` runs e2e via `chromedriver`. `node` runs tests using `NodeJS` runner.
41+
kind: [cd, node]
42+
runs-on: ${{ matrix.os }}
43+
steps:
44+
- name: Checkout
45+
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
46+
- name: Set up Node.js
47+
uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
48+
with:
49+
node-version-file: '.nvmrc'
50+
cache: npm
51+
- name: Disable AppArmor
52+
if: ${{ matrix.os == 'ubuntu-latest' }}
53+
# https://chromium.googlesource.com/chromium/src/+/main/docs/security/apparmor-userns-restrictions.md
54+
run: echo 0 | sudo tee /proc/sys/kernel/apparmor_restrict_unprivileged_userns
55+
- uses: google/wireit@f21db1f3a6a4db31f42787a958cf2a18308effed # setup-github-actions-caching/v2.0.3
56+
- name: Install and build npm dependencies
57+
run: npm ci
58+
# Install chrome, chromedriver and headless shell is required to keep them cached.
59+
- name: Install all chrome binaries if needed
60+
uses: ./.github/actions/setup-chrome-binaries
61+
- name: Set up Python
62+
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
63+
with:
64+
python-version: '3.11'
65+
cache: pipenv
66+
- name: Install pipenv
67+
run: pip install pipenv
68+
- name: Install python dependencies
69+
run: pipenv install
70+
- name: Run E2E performance tests with xvfb-run
71+
if: matrix.os == 'ubuntu-latest' && matrix.head == 'headful'
72+
timeout-minutes: 20
73+
run: >
74+
xvfb-run --auto-servernum
75+
npm run e2e:${{ matrix.head }}
76+
--
77+
-k test_performance_
78+
-s
79+
| tee tests_output.txt
80+
env:
81+
VERBOSE: ${{ github.event.inputs.verbose }}
82+
CHROMEDRIVER: ${{ matrix.kind == 'cd' }}
83+
- name: Run E2E performance tests
84+
if: matrix.os != 'ubuntu-latest' || matrix.head != 'headful'
85+
timeout-minutes: 20
86+
run: >
87+
npm run e2e:${{ matrix.head }}
88+
--
89+
-k test_performance_
90+
-s
91+
| tee tests_output.txt
92+
env:
93+
VERBOSE: ${{ github.event.inputs.verbose }}
94+
CHROMEDRIVER: ${{ matrix.kind == 'cd' }}
95+
- name: Extract and store performance metrics
96+
id: extract_metrics
97+
run: |
98+
grep 'PERF_METRIC:' tests_output.txt | sed 's/^PERF_METRIC://' | sed "s/^/${{ matrix.os }}-${{ matrix.head }}-${{ matrix.kind }}:/" > performance_metrics.txt
99+
echo "Extracted performance metrics:"
100+
cat performance_metrics.txt
101+
- name: Upload performance result as artifact
102+
uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
103+
with:
104+
name: performance-metrics-${{ matrix.os }}-${{ matrix.head }}-${{ matrix.kind }}
105+
path: performance_metrics.txt
106+
# Do not store these artifacts for long, as they will be present in the `all-performance-metrics`.
107+
retention-days: 1
108+
combine_metrics:
109+
name: Combine performance metrics
110+
runs-on: ubuntu-latest
111+
needs: performance_metric
112+
steps:
113+
- name: Download all performance metrics artifacts
114+
uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6.0.0
115+
with:
116+
path: artifacts
117+
- name: Combine metrics into a single file
118+
run: |
119+
find artifacts -name "performance_metrics.txt" -exec cat {} + > all_performance_metrics.txt
120+
echo "Combined performance metrics:"
121+
sort -o all_performance_metrics.txt all_performance_metrics.txt
122+
cat all_performance_metrics.txt
123+
- name: Upload combined performance metrics
124+
uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
125+
with:
126+
name: all-performance-metrics
127+
path: all_performance_metrics.txt

‎tests/conftest.py‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,15 @@ async def capabilities(request):
121121

122122

123123
@pytest_asyncio.fixture
124-
async def websocket(test_headless_mode, capabilities, request):
124+
async def current_test_name(request):
125+
if request and request.node and request.node.name:
126+
# Add test name to ease log analyse.
127+
return request.node.name
128+
return "::NO_TEST"
129+
130+
131+
@pytest_asyncio.fixture
132+
async def websocket(test_headless_mode, capabilities, current_test_name):
125133
"""Connects to endpoint, creates a session and returns a websocket connection."""
126134
async def create_session(connection):
127135
"""
@@ -144,12 +152,10 @@ async def create_session(connection):
144152
"--disable-background-timer-throttling",
145153
"--disable-backgrounding-occluded-windows",
146154
]
147-
}
148-
}
149-
150-
if request and request.node and request.node.name:
155+
},
151156
# Add test name to ease log analyse.
152-
default_capabilities["goog:pytest_name"] = request.node.name
157+
"goog:pytest_name": current_test_name
158+
}
153159

154160
maybe_browser_bin = os.getenv("BROWSER_BIN")
155161
if maybe_browser_bin:

0 commit comments

Comments
 (0)