Key validation example workarounds per test in config.json by bkaradzic-microsoft · Pull Request #1920 · BabylonJS/BabylonNative

bkaradzic-microsoft · 2026-10-08T02:06:17Z

validation_native.js has workarounds for bugs in the examples (not in Babylon Native), and today they run for every test. This gives each one an ID and runs it only for tests that list it in config.json:

"workarounds": ["wait-for-readiness"]

The workarounds lists double as the list of examples to fix.

ID	Example bug	Origin	Enabled tests
`wait-for-readiness`	Renders before utility scenes, GUI images, or material effects are ready	#1890	15
`undeclared-name`	Reads the browser-only global `name`	#1823	0
`prime-effect-layers`	Captures effect-layer output on the first frame	#1913	9
`leak-cleanup`	Leaves scenes, textures, or Draco state behind	#1830 / #1838	2
`wait-for-import`	Starts a model load in `createScene` without awaiting it (was a `replace` patch)	#1872	1
`opaque-clear-alpha`	Clears opaque but leaves alpha < 1 in the image (was an unconditional gate)	#1913	9

Without wait-for-readiness, the runner matches the browser harness: main-scene executeWhenReady, then renderCount frames.
Harness resets the browser also does stay unconditional (stencil, scissor, reverse depth, releaseEffects, hardware scaling, OnPluginActivatedObservable.clear()).
Without opaque-clear-alpha, translucent pixels are always composited over the canvas background, as in the browser.
Unknown IDs fail config load (exit 1).
--list prints the IDs as a fifth TSV column.
The runner logs Workaround '<id>' applied for '<title>' whenever a workaround changes what it does.
Docs: AddingNewValidationTests.md and the debugging instructions.

How tests were assigned

I ran the 371 enabled tests three times (V8/D3D11) with every workaround on for every test, and assigned each test the workarounds that triggered for it. Every assignment triggered in 3/3 runs except "GUI Images in Grid" (1/3).

Notes:

With every workaround off, all 371 still passed locally, so on this machine the assignments are precautionary. Other engines or graphics APIs may genuinely need them.
undeclared-name has no current users. Excluded tests were also checked with --include-excluded, and none raised name is not defined. The ID is kept for future use.

wait-for-import and opaque-clear-alpha were assigned the same way: the replace patch's test, and the 9 tests that failed with the alpha gate removed.

Validation

V8/D3D11 RelWithDebInfo, final config: ran=371 passed=371 failed=0.
A bogus ID makes config load fail as expected.

validation_native.js carried four workarounds for bugs in the examples rather than in Babylon Native. They ran for every test. Each now has an ID and runs only for tests that list it in config.json "workarounds", so the lists double as the set of examples to fix: - wait-for-readiness: wait for utility scenes, GUI textures, and material effects (BabylonJS#1890). Without it the runner matches the browser: main-scene executeWhenReady, then renderCount frames. - undeclared-name: declare `name` for the example's direct eval (BabylonJS#1823). - prime-effect-layers: one extra frame before capture (BabylonJS#1913). - leak-cleanup: dispose stray scenes, release leaked textures, reset Draco state after the test. Unknown IDs fail config load. --list prints the IDs as a fifth column. The runner logs "Workaround '<id>' applied for '<title>'" when one changes what it does. Assignments come from three all-on runs of the 371 enabled tests (V8/D3D11), taking every test that triggered a workaround. With the final config all 371 pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c46f0590-2707-466b-93c7-bb74316f385a

Copilot

🟡 Changes recommended

The undeclared-name workaround does not emit its required application log.

1 open finding

Record applied status for undeclared-name workaround · New

What changed in this PR

Scopes validation-runner workarounds to tests configured in the catalog.

Changes:

Adds four validated workaround IDs with application logging.
Assigns readiness, effect-layer, and cleanup workarounds per test.
Updates --list output and documentation.

File	Description
`Documentation/AddingNewValidationTests.md`	Documents workaround configuration and behavior.
`Apps/Playground/Shared/CommandLine.cpp`	Documents the fifth TSV column.
`Apps/Playground/Scripts/validation_native.js`	Implements scoped workarounds and validation.
`Apps/Playground/Scripts/config.json`	Assigns workarounds to affected tests.
`.github/instructions/babylon-native-debugging.instructions.md`	Updates debugging guidance.

🧠 Review effort: Balanced

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

- wait-for-import replaces the text `replace` patch on "OpenPBR Transmission Roughness vs IOR - Analytic Lights": the runner waits for loader promises (ImportMeshAsync, AppendSceneAsync, LoadAssetContainerAsync) started by createScene. - opaque-clear-alpha keeps the old clear-alpha compositing gate for the 9 tests that write alpha < 1 after an opaque clear; every other test is composited over the canvas background like the browser. - undeclared-name now logs when it is applied. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c46f0590-2707-466b-93c7-bb74316f385a

Copilot

🟡 Changes recommended

Tracked import rejections are swallowed, allowing rendering to continue after failed model loads.

1 open finding

Loader rejection is swallowed, allowing rendering after import failure · New

1 resolved since last review

Record applied status for undeclared-name workaround

🧠 Review effort: Balanced

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

A rejected tracked import now rejects the createScene wait, so the test takes the existing failure path instead of rendering after a failed load. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c46f0590-2707-466b-93c7-bb74316f385a

Follow-up to #1920, addressing #1920 (comment). #1920 moved stray-scene disposal behind the `leak-cleanup` workaround. If a test fails before `currentScene` is assigned (for example, `wait-for-import` rejects for the OpenPBR test), the scene it created stayed registered on the reused engine and could affect later tests. - `cleanupAfterTest` disposes `engine.scenes` after every test again, as it did before #1920. - `leak-cleanup` now covers only the texture-cache release and the Draco reset. Stray scenes no longer count toward its "applied" log, because that disposal is now unconditional. - Updated the workaround description in `validation_native.js` and `Documentation/AddingNewValidationTests.md`. Validated locally on Win32 D3D11 with `Draco Mesh Compression (fallback)`, `GLTF Serializer KHR gpu instancing`, and `OpenPBR Transmission Roughness vs IOR - Analytic Lights`. Co-authored-by: Branimir Karadzic <bkaradzic@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 97f1ddbf-b571-4a6c-8cb3-411b37e0d15a

Retain the accepted 720-scene catalog with 715 enabled scenes and five exclusions. Preserve every fixture, reference, render count and numerical gate from the pre-merge support stack. Assign validation_native.js workarounds per test as BabylonJS#1920 does: every test ran with every workaround enabled, and each test lists the ones that triggered for it, plus the assignments already upstream. 126 tests list workarounds (wait-for-import 54, wait-for-readiness 34, leak-cleanup 19, prime-effect-layers 15, opaque-clear-alpha 9); undeclared-name is unused. Scene 567 uses wait-for-import instead of its old replace patch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60c2ec68-6de1-445d-9fc9-b699db737eae

bkaradzic-microsoft requested review from bghgary and a balanced review from Copilot October 8, 2026 02:06

Copilot started reviewing on behalf of bkaradzic-microsoft October 8, 2026 02:12 View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Comment thread Apps/Playground/Scripts/validation_native.js

bkaradzic-microsoft requested a balanced review from Copilot October 8, 2026 02:37

Copilot started reviewing on behalf of bkaradzic-microsoft October 8, 2026 02:47 View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Comment thread Apps/Playground/Scripts/validation_native.js Outdated

bkaradzic-microsoft enabled auto-merge (squash) October 8, 2026 21:41

bghgary approved these changes Oct 8, 2026

View reviewed changes

Comment thread Apps/Playground/Scripts/validation_native.js

bkaradzic-microsoft merged commit 3bf6bf5 into BabylonJS:master Oct 8, 2026
35 checks passed

bkaradzic-microsoft mentioned this pull request Oct 8, 2026

Dispose stray validation scenes after every test #1923

Merged

Navigation Menu

Repository navigation

Key validation example workarounds per test in config.json - #1920

Key validation example workarounds per test in config.json#1920
bkaradzic-microsoft merged 3 commits into
BabylonJS:masterfrom
bkaradzic-microsoft:pr/validation-workarounds

bkaradzic-microsoft commented Oct 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

bkaradzic-microsoft commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How tests were assigned

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bkaradzic-microsoft commented Oct 8, 2026 •

edited

Loading