chore(e2e): run Playwright visual tests in Docker (#DS-5311) by lskramarov · Pull Request #1845 · koobiq/angular-components

lskramarov · 2026-08-05T10:12:11Z

No description provided.

github-actions · 2026-08-05T10:16:57Z

🚨 E2E tests failed

Review the report for details.

💡 Comment /approve-snapshots to approve snapshot changes.

github-actions · 2026-08-05T10:16:58Z

Visit the preview URL for this PR (updated for commit 789cb91):

https://koobiq-next--prs-1845-1ikpol4e.web.app

_{(expires Thu, 13 Aug 2026 12:00:19 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: c9e37e518febda70d0317d07e8ceb35ac43c534c}

artembelik · 2026-08-05T10:26:35Z

+yarn run e2e:docker:update-snapshots    # Run E2E tests in Docker and update the baselines
 ```

+The committed baselines under `__screenshots__` are compared with `threshold: 0` and have no


это не нужно в AGENTS.md, дубль из packages/e2e/README.md

Одно для человека другое для агента, так же ? и тут скорее из ридми нужно выбрасывать и для агента больше писать..

github-actions · 2026-08-05T11:03:02Z

🚨 E2E tests failed

Review the report for details.

💡 Comment /approve-snapshots to approve snapshot changes.

lskramarov · 2026-08-05T12:11:42Z

/approve-snapshots

github-actions · 2026-08-05T12:12:02Z

🔄 Updating snapshots.

github-actions · 2026-08-05T12:18:35Z

⚠️ No snapshot changes detected.

NikGurev · 2026-08-05T13:19:34Z

Two things worth resolving before merge:

1. assert-browsers.js doesn't validate webkit

tools/e2e/assert-browsers.js only checks for chromium and chromium-headless-shell against the image's /ms-playwright contents. But e2e:setup in package.json installs both chromium and webkit — if any visual spec runs against webkit, a stale image digest would silently ship a mismatched webkit build with no build-time failure, unlike chromium which is caught. Could you either add webkit to the required array, or confirm webkit isn't exercised by this suite and drop a comment explaining why it's excluded?

2. No layer caching for the Docker build in CI

tools/e2e/run.js always runs docker compose run --rm --build, and I don't see actions/cache, a BuildKit registry cache (--cache-from/--cache-to), or a prebuilt/pushed image anywhere in the workflows. GitHub-hosted runners don't carry Docker layer cache between jobs, so every CI run seems to pay the full cost of the Node install, yarn install --immutable, and the font layer from scratch. Given the goal here was partly to cut CI time (removing the ~174MB Playwright browser download), it'd be good to confirm actual before/after wall-clock numbers, since a full image rebuild every run could offset that savings.

Copilot

Pull request overview

This PR makes Playwright visual regression testing reproducible by running the component E2E suite inside a Docker image that matches CI’s OS/font stack and the exact @playwright/test version. It also updates CI to run the same Docker-based workflow and documents the supported local/CI snapshot update path.

Changes:

Add a Docker image + docker compose runner (e2e:docker) to run Playwright component tests in a CI-matching environment and safely update committed baselines.
Update GitHub Actions E2E workflows to execute the suite (and snapshot approval flow) via the Docker runner instead of installing browsers on the runner.
Add worker-count override validation in playwright.config.ts, plus documentation and repo attributes/ignore rules to support reliable Docker builds and screenshot handling.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tools/e2e/run.js	Wrapper that validates prerequisites, derives Playwright version from `package.json`, and runs the suite via Compose (with an update-snapshots overlay when needed).
tools/e2e/Dockerfile	Digest-pinned Playwright base image with deterministic fonts, Yarn 4 shim, Node version aligned to `.nvmrc`, and a browser-revision assertion.
tools/e2e/docker-compose.yml	Compose service for running the suite with sensible defaults (platform pin, worker cap, bind-mounted outputs).
tools/e2e/docker-compose.update.yml	Overlay that mounts `packages/components` only when updating snapshots, to write baselines back to the working tree.
tools/e2e/assert-browsers.js	Build-time check ensuring the image’s baked browsers match what `playwright-core` expects for the installed version.
playwright.config.ts	Adds validated `PLAYWRIGHT_WORKERS` override handling to prevent silent “0 tests ran” success cases.
packages/e2e/README.md	Documents Docker as the supported path for visual tests and snapshot regeneration.
package.json	Adds `e2e:docker` and `e2e:docker:update-snapshots` scripts.
docs/guides/06-testing.md	Expands testing docs with Docker-based visual testing guidance and worker-count rationale.
AGENTS.md	Adds the Docker E2E commands and guidance to avoid non-Linux baseline regeneration.
.github/workflows/e2e.yml	Runs component E2E tests via `npm run e2e:docker` (container builds/installs internally).
.github/workflows/e2e-approve-snapshots.yml	Regenerates baselines via `npm run e2e:docker:update-snapshots` using the same container as the comparator job.
.gitattributes	Forces LF for `tools/e2e/**` and marks PNGs as binary to protect baseline bytes.
.dockerignore	Limits Docker build context size and excludes transient/local outputs while keeping committed baselines included.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

artembelik · 2026-08-06T13:37:47Z

+# library on the Angular build's hot path, which costs about 5 seconds per run across the host
+# filesystem boundary (measured: 27s against 32s for the same spec). Updating snapshots is rare and
+# can afford both; a plain run should pay for neither.
+services:


обновление скриншотов и сами тесты запускается на разных platform (смотри) , могут быть расхождения, когда падает определенный скриншот тест, а при обновлении обновятся N других

в дата-грид используется единый docker-compose для обоих сценариев, мб и здесь поступим также?

Про разные платформы — проверил, не подтверждается: --file base --file update мержится, а не заменяется, platform из базового файла сохраняется. Вывод docker compose --file tools/e2e/docker-compose.yml --file tools/e2e/docker-compose.update.yml config:

platform: linux/amd64
volumes: playwright-report, test-results, packages/components
Про единый файл: в дата-гриде монтируются две конкретные папки (screenshots, snapshots), у нас baseline'ы лежат внутри каждого компонента (packages/components/*/screenshots), поэтому единый файл = монтировать всю библиотеку на запись при каждом прогоне: +~5 с на прогон (27 с против 32 с на одной спеке) и право писать в рабочее дерево у обычного yarn run e2e:docker. Оставил overlay, а в шапку docker-compose.update.yml дописал, что platform наследуется, и как это проверить.

artembelik · 2026-08-06T13:49:53Z

+    # by mount target in some versions and replaced them wholesale in others; spelling all three
+    # out is correct either way, and a repeated identical target is a no-op when they are merged.
+    volumes:
+      - ../../packages/components:/app/packages/components


в доке есть е2е тесты, но пока нет скриншотов, я к тому, что артефакты из документации тоже можно возвращать (это минор, можно будет при необходимости добавить)

Согласен, но сейчас монтировать нечего: job docs в e2e.yml гоняется на раннере, а не в контейнере (свой playwright.docs.config.ts и playwright-report-docs). Как только у доки появятся скриншоты — переведём её на этот же образ, тогда и маунт добавится.

artembelik · 2026-08-06T14:01:27Z

удобный подход, когда ты игнорируешь все по умолчанию, и кладешь в образ только то, что действительно необходимо, пример:

# Exclude everything by default, then explicitly allow only what is needed. ** # Yarn Berry binary and configuration !.yarn/releases/ !.yarn/releases/** !.yarnrc.yml !package.json !yarn.lock # Application and test source code !packages/components/

я к тому что, в данной реализации в докер образ попадают например packages/components-dev, packages/docs-examples и тд

сам файл можно положить рядом с Dockerfile в tools/e2e/

Поправил. Файл переехал в tools/e2e/Dockerfile.dockerignore — BuildKit ищет .dockerignore раньше, чем /.dockerignore, а Compose v2 всегда собирает через BuildKit, так что он лежит рядом с Dockerfile и при этом реально читается (обычный .dockerignore в tools/e2e/ не прочитал бы никто).

Переписал на allowlist: контекст 57 MB → 28.1 MB, apps/docs, packages/docs-examples, packages/schematics больше не попадают. Из components-dev оставил ровно один файл — theme-toggle.ts, его тянет packages/e2e/module.ts; без него ng serve dev-e2e падает с NG1010, что я и словил при проверке.

Это как раз то, чем allowlist неприятен — он теряет файлы молча, — поэтому после COPY . . добавил ассерт: если ignore-файл не применился, сборка падает с внятным сообщением, а не собирается «на всё дерево». Проверено полной сборкой + прогоном button/button-toggle в контейнере, 49 тестов зелёные.

artembelik · 2026-08-06T14:03:25Z

@@ -17,7 +60,7 @@ export default defineConfig({
    fullyParallel: true,
    forbidOnly: isCI,
    retries: isCI ? 2 : 0,


в идеале конечно отключить retries в 0, чтобы исключить нестабильные тесты на этапе разработки, но лучше это отдельно сделать

Согласен. Причём это шире, чем кажется: retries: isCI ? 2 : 0, а контейнер выставляет CI=true ради паритета с раннером — то есть локальный docker-прогон тоже ретраит. Развязывать стоит вместе с forbidOnly, отдельной задачей.

artembelik · 2026-08-06T14:08:06Z

+# The two fc-match calls are the assertion, not a demonstration: they are what stops this layer
+# from being "cleaned up" later on the reasonable-sounding grounds that the fonts are bundled. A
+# regression here is otherwise silent until six screenshots disagree for no visible reason.
+RUN apt-get update \


в случае дата-грид помогла установка runs-on: ubuntu-24.04-arm в воркфлоу и platform: linux/arm64 для компоуза, но раз работает давай оставим

lskramarov · 2026-08-10T10:55:18Z

Two things worth resolving before merge:

1. assert-browsers.js doesn't validate webkit

tools/e2e/assert-browsers.js only checks for chromium and chromium-headless-shell against the image's /ms-playwright contents. But e2e:setup in package.json installs both chromium and webkit — if any visual spec runs against webkit, a stale image digest would silently ship a mismatched webkit build with no build-time failure, unlike chromium which is caught. Could you either add webkit to the required array, or confirm webkit isn't exercised by this suite and drop a comment explaining why it's excluded?

2. No layer caching for the Docker build in CI

tools/e2e/run.js always runs docker compose run --rm --build, and I don't see actions/cache, a BuildKit registry cache (--cache-from/--cache-to), or a prebuilt/pushed image anywhere in the workflows. GitHub-hosted runners don't carry Docker layer cache between jobs, so every CI run seems to pay the full cost of the Node install, yarn install --immutable, and the font layer from scratch. Given the goal here was partly to cut CI time (removing the ~174MB Playwright browser download), it'd be good to confirm actual before/after wall-clock numbers, since a full image rebuild every run could offset that savings.

Поправил: список больше не хардкодится, берётся из playwright-core/browsers.json по installByDefault. WebKit туда попадает — его выбирают scrollbar и sidepanel через test.use({ browserName: 'webkit' }) — и любой браузер, который добавят завтра, тоже.
Дописал в e2e.yml, что это не выигрыш по времени, а воспроизводимость: у GitHub-раннеров нет кеша слоёв между запусками, образ пересобирается каждый раз. cache-to: type=gha положил бы больше гигабайта слоёв в тот же 10 GB кеш, за который конкурируют yarn-кеши остальных джоб; если время станет проблемой — правильный ответ это готовый образ из GHCR по тегу.

github-actions · 2026-08-10T11:03:47Z

🚨 E2E tests failed

Review the report for details.

💡 Comment /approve-snapshots to approve snapshot changes.

Moves .dockerignore to tools/e2e/Dockerfile.dockerignore, next to the Dockerfile it belongs to — BuildKit reads `<dockerfile>.dockerignore` in preference to the one at the context root, and Compose v2 always builds through BuildKit — and rewrites it as an allowlist. The context drops from 57 MB to 28.1 MB: apps/docs, packages/docs-examples and packages/schematics no longer reach the image, and packages/components-dev contributes only theme-toggle.ts, which packages/e2e/module.ts puts in the root component's imports. An allowlist loses files silently, so the Dockerfile now asserts that the file was applied at all. CHANGELOG.md is the sentinel because no e2e run will ever want it, which a directory that might be allowlisted later cannot promise. Also documents two things review asked about: why assert-browsers.js exists when the base image already ships browsers (`FROM` carries both a tag and a digest, and the digest wins, so bumping @playwright/test alone leaves the old browsers in place), and that docker-compose.update.yml is an overlay whose `platform` is inherited from the base file rather than a second, diverging configuration.

…311) @playwright/test moved to 1.62.1 on main in #1850, which this branch is now rebased onto. The `FROM` digest still resolved the v1.55.0-noble image, so playwright-core asked for chromium-1234, firefox-1538 and webkit-2336 while the image carried 1187, 1490 and 2203. Nothing silently degraded: tools/e2e/assert-browsers.js failed the build with that exact list, which is what it is there for. The digest below is what its own hint prints, `docker buildx imagetools inspect mcr.microsoft.com/playwright:v1.62.1-noble`. The baselines came along with the rebase — #1850 regenerated all 68 of them in the same commit as the version bump.

lskramarov requested a review from artembelik August 5, 2026 10:12

lskramarov self-assigned this Aug 5, 2026

lskramarov requested a review from NikGurev as a code owner August 5, 2026 10:12

lskramarov added enhancement New feature or request github_actions Pull requests that update GitHub Actions code labels Aug 5, 2026

lskramarov marked this pull request as draft August 5, 2026 10:21

artembelik reviewed Aug 5, 2026

View reviewed changes

lskramarov marked this pull request as ready for review August 5, 2026 12:16

lskramarov force-pushed the chore/DS-5311 branch from c9f88ee to 63f1c91 Compare August 5, 2026 12:33

NikGurev approved these changes Aug 5, 2026

View reviewed changes

lskramarov force-pushed the chore/DS-5311 branch from faa3740 to b49c949 Compare August 5, 2026 15:42

artembelik requested a lite review from Copilot August 5, 2026 19:06

Copilot started reviewing on behalf of artembelik August 5, 2026 19:06 View session

Copilot AI reviewed Aug 5, 2026

View reviewed changes

artembelik reviewed Aug 6, 2026

View reviewed changes

Comment thread tools/e2e/assert-browsers.js

artembelik reviewed Aug 6, 2026

View reviewed changes

artembelik approved these changes Aug 6, 2026

View reviewed changes

lskramarov added 4 commits August 10, 2026 14:43

chore(e2e): run Playwright visual tests in Docker (#DS-5311)

d95f0ab

fix: after review

ad3b425

lskramarov force-pushed the chore/DS-5311 branch from 55c773d to 789cb91 Compare August 10, 2026 11:56

lskramarov merged commit ce0990f into main Aug 10, 2026
11 checks passed

lskramarov deleted the chore/DS-5311 branch August 10, 2026 12:06

Uh oh!

Conversation

lskramarov commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

🚨 E2E tests failed

Uh oh!

github-actions Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 5, 2026

🚨 E2E tests failed

Uh oh!

lskramarov commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

NikGurev commented Aug 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lskramarov commented Aug 10, 2026

Uh oh!

github-actions Bot commented Aug 10, 2026

🚨 E2E tests failed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Aug 5, 2026 •

edited

Loading