Skip to content

Key validation example workarounds per test in config.json #1920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bkaradzic-microsoft merged 3 commits into from
Oct 8, 2026
Merged

Key validation example workarounds per test in config.json #1920

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0783487
Key validation example workarounds per test in config.json
bkaradzic-microsoft Oct 8, 2026
18ca467
Add wait-for-import and opaque-clear-alpha workaround IDs
bkaradzic-microsoft Oct 8, 2026
705b4e7
Propagate loader failures through wait-for-import
bkaradzic-microsoft Oct 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/instructions/babylon-native-debugging.instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ For pure RenderDoc CLI usage (any app, not BN-specific), see
| `Apps/Playground/Shared/Diagnostics.{h,cpp}` | Crash handler, `DumpFailure`, finish-line, exit-code tracking. |
| `Apps/Playground/Win32/App.cpp` (and the other per-host `App.*`) | Wires the `RuntimeOptions::log` callback into `DumpFailure` (`JS CONSOLE ERROR` for `LogLevel::Error`, `UNCAUGHT JS ERROR` for `LogLevel::Fatal`) and injects `_playgroundOptions` into JS via `Runtime::RunOnJsThread`. |
| `Apps/Playground/Scripts/validation_native.js` | Test runner. Reads `_playgroundOptions`, picks tests, calls `TestUtils.captureNextFrame()`. Reference-image load failures arrive via `BABYLON.Tools.LoadFile`'s `onLoadFileError` and are tagged with `MISSING_REFERENCE_IMAGE:`. |
| `Apps/Playground/Scripts/config.json` | Test catalog. Each entry has `title`, `playgroundId`/`scriptToRun`, `referenceImage`, optional `excludeFromAutomaticTesting`/`reason`/`onlyVisual`/`renderCount`/`capture`/`threshold`/`errorRatio`. |
| `Apps/Playground/Scripts/config.json` | Test catalog. Each entry has `title`, `playgroundId`/`scriptToRun`, `referenceImage`, optional `excludeFromAutomaticTesting`/`reason`/`onlyVisual`/`renderCount`/`capture`/`threshold`/`errorRatio`/`workarounds`. |
| `Plugins/TestUtils/Source/TestUtils.cpp` | Native side of `TestUtils.captureNextFrame()` -- calls `m_deviceContext.RequestCaptureNextFrame()`. |
| `Core/Graphics/Source/BgfxCallback.cpp` | bgfx trace/fatal sink. Routes bgfx output to stdout in headless mode and to `OutputDebugString` always. |

Expand Down Expand Up @@ -194,7 +194,7 @@ debugging.
```powershell
.\Playground.exe --list
```
Prints TSV with index, title, referenceImage, exclusionReason. Pipe to
Prints TSV with index, title, referenceImage, exclusionReason, workarounds. Pipe to
`findstr` / `Select-String` for filtering.

### Pixel-diff investigation
Expand Down Expand Up @@ -280,6 +280,7 @@ the assert.
| `errorRatio: F` | % of pixels allowed to differ (default 2.5) |
| `replace: "src,dst,..."` | String-replace pairs applied to the playground source before `eval` |
| `specificRoot: "..."` | Override `BABYLON.Tools.BaseUrl` for asset loads |
| `workarounds: ["id", ...]` | Enable runner workarounds for bugs in this example (see `Documentation/AddingNewValidationTests.md`). Unknown IDs fail config load. Only tests that list an ID get it; the list doubles as the to-fix list for examples. |

## Adding new diagnostic output -- guidelines

Expand Down
109 changes: 72 additions & 37 deletions Apps/Playground/Scripts/config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,8 @@
"title": "GLTF Extension KHR_materials_volume with attenuation",
"playgroundId": "#YG3BBF#18",
"referenceImage": "gltfExtensionKhrMaterialsVolumeAttenuation.png",
"renderCount": 10
"renderCount": 10,
"workarounds": ["wait-for-readiness"]
},
{
"title": "GLTF ClearCoat",
Expand Down Expand Up @@ -130,13 +131,15 @@
"title": "Glow layer and LODs",
"playgroundId": "#UNS6ZV#2",
"renderCount": 50,
"referenceImage": "glowlayerandlods.png"
"referenceImage": "glowlayerandlods.png",
"workarounds": ["prime-effect-layers"]
},
{
"title": "Nested BBG",
"playgroundId": "#ZG0C8B#6",
"renderCount": 10,
"referenceImage": "nested_BBG.png"
"referenceImage": "nested_BBG.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "Dynamic Texture context clip",
Expand All @@ -147,7 +150,8 @@
"title": "GUI3D SpherePanel",
"playgroundId": "#HB4C01#9",
"renderCount": 50,
"referenceImage": "spherepanel.png"
"referenceImage": "spherepanel.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "Kernel Blur",
Expand Down Expand Up @@ -225,14 +229,16 @@
"title": "Ribbon morphing",
"playgroundId": "#ACKC2#1",
"renderCount": 50,
"referenceImage": "ribbon morphing.png"
"referenceImage": "ribbon morphing.png",
"workarounds": ["opaque-clear-alpha"]
},
{
"title": "Solid particle system",
"playgroundId": "#WCDZS#531",
"renderCount": 150,
"referenceImage": "sps.png",
"errorRatio": 6.0
"errorRatio": 6.0,
"workarounds": ["opaque-clear-alpha"]
},
{
"title": "LightFalloff",
Expand Down Expand Up @@ -628,7 +634,8 @@
{
"title": "RH billboard",
"playgroundId": "#PDO1L6#1",
"referenceImage": "rh-billboard.png"
"referenceImage": "rh-billboard.png",
"workarounds": ["opaque-clear-alpha"]
},
{
"title": "RH billboard2",
Expand Down Expand Up @@ -666,7 +673,8 @@
{
"title": "Attractors",
"playgroundId": "#DEZ79M#57",
"referenceImage": "attractors.png"
"referenceImage": "attractors.png",
"workarounds": ["prime-effect-layers"]
},
{
"title": "PBR in mirror",
Expand Down Expand Up @@ -1011,7 +1019,8 @@
{
"title": "Nested BBG",
"playgroundId": "#ZG0C8B#12",
"referenceImage": "nested-BBG.png"
"referenceImage": "nested-BBG.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "Weighted animations",
Expand Down Expand Up @@ -1058,7 +1067,8 @@
"title": "GUI Near Menu",
"playgroundId": "#2YZFA0#302",
"renderCount": 60,
"referenceImage": "guiNearMenu.png"
"referenceImage": "guiNearMenu.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "Camera rig",
Expand Down Expand Up @@ -1100,13 +1110,15 @@
"title": "Gizmos",
"playgroundId": "#8GY6J8#196",
"renderCount": 50,
"referenceImage": "Gizmos.png"
"referenceImage": "Gizmos.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "Particle Helper",
"playgroundId": "#1VGT5D#2",
"renderCount": 50,
"referenceImage": "particle-helper.png"
"referenceImage": "particle-helper.png",
"workarounds": ["opaque-clear-alpha"]
},
{
"title": "Chibi Rex",
Expand Down Expand Up @@ -1149,7 +1161,8 @@
"playgroundId": "#T7FXR8#67",
"excludedGraphicsApis": ["OpenGL"],
"reason": "Renders black on Linux desktop OpenGL (area-light LTC path unsupported); passes on Win32 D3D11.",
"referenceImage": "areaLightsStandardMaterial.png"
"referenceImage": "areaLightsStandardMaterial.png",
"workarounds": ["opaque-clear-alpha"]
},
{
"title": "Textured - Area Lights - Standard Material",
Expand All @@ -1170,7 +1183,8 @@
"playgroundId": "#T7FXR8#12",
"excludedGraphicsApis": ["OpenGL"],
"reason": "Renders black on Linux desktop OpenGL (area-light LTC path unsupported); passes on Win32 D3D11.",
"referenceImage": "areaLightsPBRMaterial.png"
"referenceImage": "areaLightsPBRMaterial.png",
"workarounds": ["opaque-clear-alpha"]
},
{
"title": "Bones",
Expand Down Expand Up @@ -1248,7 +1262,8 @@
"title": "Draco Mesh Compression (fallback)",
"playgroundId": "#22MFU2#75",
"replace": "numWorkers = 0, numWorkers = 1",
"referenceImage": "draco.png"
"referenceImage": "draco.png",
"workarounds": ["leak-cleanup"]
},
{
"title": "Draco Decoder",
Expand Down Expand Up @@ -1334,19 +1349,22 @@
"playgroundId": "#1Q2BWN#10",
"excludedGraphicsApis": ["OpenGL"],
"reason": "Pixel diff 3.4% on OpenGL backend (Linux JSC); passes on D3D11.",
"referenceImage": "glTFSerializerKhrGpuInstancing.png"
"referenceImage": "glTFSerializerKhrGpuInstancing.png",
"workarounds": ["leak-cleanup"]
},
{
"title": "GLTF Serializer KHR punctual light, left-handed",
"playgroundId": "#FLXW8B#27",
"replace": "//options//, roundtrip = true; useRightHandedSystem = false;",
"referenceImage": "glTFSerializerKHRPunctualLightLH.png"
"referenceImage": "glTFSerializerKHRPunctualLightLH.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "GLTF Serializer KHR punctual light, right-handed",
"playgroundId": "#FLXW8B#27",
"replace": "//options//, roundtrip = true; useRightHandedSystem = true;",
"referenceImage": "glTFSerializerKHRPunctualLightRH.png"
"referenceImage": "glTFSerializerKHRPunctualLightRH.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "GLTF Serializer Camera, left-handed",
Expand Down Expand Up @@ -1476,7 +1494,8 @@
"title": "Enable disable post process",
"playgroundId": "#1VI6WV#19",
"renderCount": 50,
"referenceImage": "enableDisablePostProcess.png"
"referenceImage": "enableDisablePostProcess.png",
"workarounds": ["opaque-clear-alpha"]
},
{
"title": "Depth of field",
Expand Down Expand Up @@ -1515,7 +1534,8 @@
"title": "Particles",
"playgroundId": "#G3ZYFU#7",
"renderCount": 100,
"referenceImage": "particles.png"
"referenceImage": "particles.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "PBRMetallicRoughnessMaterial",
Expand Down Expand Up @@ -1570,7 +1590,8 @@
"title": "MultiSample render targets",
"playgroundId": "#12MKMN#0",
"renderCount": 20,
"referenceImage": "MultiSample-render-targets.png"
"referenceImage": "MultiSample-render-targets.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "Default rendering pipeline",
Expand Down Expand Up @@ -1758,7 +1779,8 @@
{
"title": "Glow layer and LODs",
"playgroundId": "#UNS6ZV#13",
"referenceImage": "glowlayerandlods.png"
"referenceImage": "glowlayerandlods.png",
"workarounds": ["prime-effect-layers"]
},
{
"title": "Shadows with instances in left handed system",
Expand All @@ -1783,7 +1805,8 @@
"renderCount": 50,
"referenceImage": "prepass-ssao-particles.png",
"excludedGraphicsApis": ["OpenGL"],
"reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES"
"reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES",
"workarounds": ["wait-for-readiness"]
},
{
"title": "Prepass SSAO + instances",
Expand Down Expand Up @@ -1902,7 +1925,8 @@
"renderCount": 10,
"referenceImage": "prepass-ssao-highlight-layer.png",
"excludedGraphicsApis": ["OpenGL"],
"reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES"
"reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES",
"workarounds": ["prime-effect-layers"]
},
{
"title": "Prepass SSAO + point light",
Expand All @@ -1917,7 +1941,8 @@
"renderCount": 10,
"referenceImage": "prepass-ssao-on-off-pp.png",
"excludedGraphicsApis": ["OpenGL"],
"reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES"
"reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES",
"workarounds": ["opaque-clear-alpha"]
},
{
"title": "Prepass SSAO + thin instances",
Expand Down Expand Up @@ -2026,7 +2051,8 @@
"title": "Thin instances + motion blur",
"playgroundId": "#HJGC2G#60",
"renderCount": 30,
"referenceImage": "thin-instances-mb.png"
"referenceImage": "thin-instances-mb.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "Thin instances + motion blur + manual",
Expand All @@ -2050,7 +2076,8 @@
"title": "Thin instances + render self motion blur",
"playgroundId": "#217750#218",
"renderCount": 10,
"referenceImage": "thin-instances-renderself-mb.png"
"referenceImage": "thin-instances-renderself-mb.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "Reverse depth buffer and shadows",
Expand Down Expand Up @@ -2127,7 +2154,8 @@
"title": "Bones and morphs computation order",
"playgroundId": "#WGZLGJ#4040",
"renderCount": 40,
"referenceImage": "bonesMorphsCompute.png"
"referenceImage": "bonesMorphsCompute.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "DepthRenderer rhs",
Expand Down Expand Up @@ -2233,7 +2261,8 @@
{
"title": "GUI Images in Grid",
"playgroundId": "#8Y0XJL#8",
"referenceImage": "GUI-Images-in-Grid.png"
"referenceImage": "GUI-Images-in-Grid.png",
"workarounds": ["wait-for-readiness"]
},
{
"title": "GUI Force Resize Width",
Expand Down Expand Up @@ -2463,7 +2492,8 @@
"excludedGraphicsApis": ["OpenGL"],
"reason": "OpenGL: pixel diff out of tolerance (90900 px) -- likely glow-layer + custom-shader interaction differs from D3D11. Tracked for OpenGL backend follow-up.",
"renderCount": 5,
"referenceImage": "sphere-with-custom-shader-to-display-wireframe-using-glow-layer.png"
"referenceImage": "sphere-with-custom-shader-to-display-wireframe-using-glow-layer.png",
"workarounds": ["prime-effect-layers"]
},
{
"title": "blur-cube-with-the-effect-renderer",
Expand Down Expand Up @@ -2546,7 +2576,8 @@
"title": "dissolve-effect-with-node-material-and-glow-layer",
"playgroundId": "#F6PGMC#12",
"renderCount": 2,
"referenceImage": "dissolve-effect-with-node-material-and-glow-layer.png"
"referenceImage": "dissolve-effect-with-node-material-and-glow-layer.png",
"workarounds": ["prime-effect-layers"]
},
{
"title": "cascaded-shadow-maps-and-flying-saucers",
Expand Down Expand Up @@ -2679,7 +2710,8 @@
{
"title": "Glow layer with vertex alpha",
"playgroundId": "#GWAX2G#7",
"referenceImage": "Glow-layer-with-vertex-alpha.png"
"referenceImage": "Glow-layer-with-vertex-alpha.png",
"workarounds": ["prime-effect-layers"]
},
{
"title": "IBL Voxel Shadowing Right-Handed",
Expand Down Expand Up @@ -3662,8 +3694,8 @@
{
"title": "OpenPBR Transmission Roughness vs IOR - Analytic Lights",
"playgroundId": "#GRQHVV#134",
"replace": "BABYLON.ImportMeshAsync, var meshLoad = BABYLON.ImportMeshAsync, return scene;, return Promise.resolve(meshLoad).then(function () { return scene; });",
"referenceImage": "OpenPBR-Transmission-Roughness-vs-IOR---Analytic-Lights.png"
"referenceImage": "OpenPBR-Transmission-Roughness-vs-IOR---Analytic-Lights.png",
"workarounds": ["wait-for-import"]
},
{
"title": "OpenPBR Transmission Anisotropy Roughness vs IOR - IBL",
Expand Down Expand Up @@ -4255,7 +4287,8 @@
"playgroundId": "#A97UQ6#5",
"renderCount": 120,
"referenceImage": "particles-helper-sun.png",
"errorRatio": 8
"errorRatio": 8,
"workarounds": ["opaque-clear-alpha"]
},
{
"title": "Particles - Helper - Smoke",
Expand All @@ -4276,7 +4309,8 @@
{
"title": "Particles - Attractors",
"playgroundId": "#PUWWM3#0",
"referenceImage": "particles-attractors.png"
"referenceImage": "particles-attractors.png",
"workarounds": ["prime-effect-layers"]
},
{
"title": "Particles - Flowmaps",
Expand All @@ -4300,7 +4334,8 @@
{
"title": "Selection outline layer with instances and LOD",
"playgroundId": "#8I487I#0",
"referenceImage": "Selection-outline-layer-with-instances-and-LOD.png"
"referenceImage": "Selection-outline-layer-with-instances-and-LOD.png",
"workarounds": ["prime-effect-layers"]
},
{
"title": "Vertex Pulling - Normals UVs Colors Tangents",
Expand Down
Loading
Loading