diff --git a/.github/instructions/babylon-native-debugging.instructions.md b/.github/instructions/babylon-native-debugging.instructions.md index b182dc6dd..21e376765 100644 --- a/.github/instructions/babylon-native-debugging.instructions.md +++ b/.github/instructions/babylon-native-debugging.instructions.md @@ -26,7 +26,7 @@ For pure RenderDoc CLI usage (any app, not BN-specific), see | `Apps/Playground/Shared/Diagnostics.{h,cpp}` | Crash handler, `DumpFailure`, finish-line, exit-code tracking. | | `Apps/Playground/Win32/App.cpp` (and the other per-host `App.*`) | Wires the `RuntimeOptions::log` callback into `DumpFailure` (`JS CONSOLE ERROR` for `LogLevel::Error`, `UNCAUGHT JS ERROR` for `LogLevel::Fatal`) and injects `_playgroundOptions` into JS via `Runtime::RunOnJsThread`. | | `Apps/Playground/Scripts/validation_native.js` | Test runner. Reads `_playgroundOptions`, picks tests, calls `TestUtils.captureNextFrame()`. Reference-image load failures arrive via `BABYLON.Tools.LoadFile`'s `onLoadFileError` and are tagged with `MISSING_REFERENCE_IMAGE:`. | -| `Apps/Playground/Scripts/config.json` | Test catalog. Each entry has `title`, `playgroundId`/`scriptToRun`, `referenceImage`, optional `excludeFromAutomaticTesting`/`reason`/`onlyVisual`/`renderCount`/`capture`/`threshold`/`errorRatio`. | +| `Apps/Playground/Scripts/config.json` | Test catalog. Each entry has `title`, `playgroundId`/`scriptToRun`, `referenceImage`, optional `excludeFromAutomaticTesting`/`reason`/`onlyVisual`/`renderCount`/`capture`/`threshold`/`errorRatio`/`workarounds`. | | `Plugins/TestUtils/Source/TestUtils.cpp` | Native side of `TestUtils.captureNextFrame()` -- calls `m_deviceContext.RequestCaptureNextFrame()`. | | `Core/Graphics/Source/BgfxCallback.cpp` | bgfx trace/fatal sink. Routes bgfx output to stdout in headless mode and to `OutputDebugString` always. | @@ -194,7 +194,7 @@ debugging. ```powershell .\Playground.exe --list ``` -Prints TSV with index, title, referenceImage, exclusionReason. Pipe to +Prints TSV with index, title, referenceImage, exclusionReason, workarounds. Pipe to `findstr` / `Select-String` for filtering. ### Pixel-diff investigation @@ -280,6 +280,7 @@ the assert. | `errorRatio: F` | % of pixels allowed to differ (default 2.5) | | `replace: "src,dst,..."` | String-replace pairs applied to the playground source before `eval` | | `specificRoot: "..."` | Override `BABYLON.Tools.BaseUrl` for asset loads | +| `workarounds: ["id", ...]` | Enable runner workarounds for bugs in this example (see `Documentation/AddingNewValidationTests.md`). Unknown IDs fail config load. Only tests that list an ID get it; the list doubles as the to-fix list for examples. | ## Adding new diagnostic output -- guidelines diff --git a/Apps/Playground/Scripts/config.json b/Apps/Playground/Scripts/config.json index 75c3cbf88..e78c09f7b 100644 --- a/Apps/Playground/Scripts/config.json +++ b/Apps/Playground/Scripts/config.json @@ -56,7 +56,8 @@ "title": "GLTF Extension KHR_materials_volume with attenuation", "playgroundId": "#YG3BBF#18", "referenceImage": "gltfExtensionKhrMaterialsVolumeAttenuation.png", - "renderCount": 10 + "renderCount": 10, + "workarounds": ["wait-for-readiness"] }, { "title": "GLTF ClearCoat", @@ -130,13 +131,15 @@ "title": "Glow layer and LODs", "playgroundId": "#UNS6ZV#2", "renderCount": 50, - "referenceImage": "glowlayerandlods.png" + "referenceImage": "glowlayerandlods.png", + "workarounds": ["prime-effect-layers"] }, { "title": "Nested BBG", "playgroundId": "#ZG0C8B#6", "renderCount": 10, - "referenceImage": "nested_BBG.png" + "referenceImage": "nested_BBG.png", + "workarounds": ["wait-for-readiness"] }, { "title": "Dynamic Texture context clip", @@ -147,7 +150,8 @@ "title": "GUI3D SpherePanel", "playgroundId": "#HB4C01#9", "renderCount": 50, - "referenceImage": "spherepanel.png" + "referenceImage": "spherepanel.png", + "workarounds": ["wait-for-readiness"] }, { "title": "Kernel Blur", @@ -225,14 +229,16 @@ "title": "Ribbon morphing", "playgroundId": "#ACKC2#1", "renderCount": 50, - "referenceImage": "ribbon morphing.png" + "referenceImage": "ribbon morphing.png", + "workarounds": ["opaque-clear-alpha"] }, { "title": "Solid particle system", "playgroundId": "#WCDZS#531", "renderCount": 150, "referenceImage": "sps.png", - "errorRatio": 6.0 + "errorRatio": 6.0, + "workarounds": ["opaque-clear-alpha"] }, { "title": "LightFalloff", @@ -628,7 +634,8 @@ { "title": "RH billboard", "playgroundId": "#PDO1L6#1", - "referenceImage": "rh-billboard.png" + "referenceImage": "rh-billboard.png", + "workarounds": ["opaque-clear-alpha"] }, { "title": "RH billboard2", @@ -666,7 +673,8 @@ { "title": "Attractors", "playgroundId": "#DEZ79M#57", - "referenceImage": "attractors.png" + "referenceImage": "attractors.png", + "workarounds": ["prime-effect-layers"] }, { "title": "PBR in mirror", @@ -1011,7 +1019,8 @@ { "title": "Nested BBG", "playgroundId": "#ZG0C8B#12", - "referenceImage": "nested-BBG.png" + "referenceImage": "nested-BBG.png", + "workarounds": ["wait-for-readiness"] }, { "title": "Weighted animations", @@ -1058,7 +1067,8 @@ "title": "GUI Near Menu", "playgroundId": "#2YZFA0#302", "renderCount": 60, - "referenceImage": "guiNearMenu.png" + "referenceImage": "guiNearMenu.png", + "workarounds": ["wait-for-readiness"] }, { "title": "Camera rig", @@ -1100,13 +1110,15 @@ "title": "Gizmos", "playgroundId": "#8GY6J8#196", "renderCount": 50, - "referenceImage": "Gizmos.png" + "referenceImage": "Gizmos.png", + "workarounds": ["wait-for-readiness"] }, { "title": "Particle Helper", "playgroundId": "#1VGT5D#2", "renderCount": 50, - "referenceImage": "particle-helper.png" + "referenceImage": "particle-helper.png", + "workarounds": ["opaque-clear-alpha"] }, { "title": "Chibi Rex", @@ -1149,7 +1161,8 @@ "playgroundId": "#T7FXR8#67", "excludedGraphicsApis": ["OpenGL"], "reason": "Renders black on Linux desktop OpenGL (area-light LTC path unsupported); passes on Win32 D3D11.", - "referenceImage": "areaLightsStandardMaterial.png" + "referenceImage": "areaLightsStandardMaterial.png", + "workarounds": ["opaque-clear-alpha"] }, { "title": "Textured - Area Lights - Standard Material", @@ -1170,7 +1183,8 @@ "playgroundId": "#T7FXR8#12", "excludedGraphicsApis": ["OpenGL"], "reason": "Renders black on Linux desktop OpenGL (area-light LTC path unsupported); passes on Win32 D3D11.", - "referenceImage": "areaLightsPBRMaterial.png" + "referenceImage": "areaLightsPBRMaterial.png", + "workarounds": ["opaque-clear-alpha"] }, { "title": "Bones", @@ -1248,7 +1262,8 @@ "title": "Draco Mesh Compression (fallback)", "playgroundId": "#22MFU2#75", "replace": "numWorkers = 0, numWorkers = 1", - "referenceImage": "draco.png" + "referenceImage": "draco.png", + "workarounds": ["leak-cleanup"] }, { "title": "Draco Decoder", @@ -1334,19 +1349,22 @@ "playgroundId": "#1Q2BWN#10", "excludedGraphicsApis": ["OpenGL"], "reason": "Pixel diff 3.4% on OpenGL backend (Linux JSC); passes on D3D11.", - "referenceImage": "glTFSerializerKhrGpuInstancing.png" + "referenceImage": "glTFSerializerKhrGpuInstancing.png", + "workarounds": ["leak-cleanup"] }, { "title": "GLTF Serializer KHR punctual light, left-handed", "playgroundId": "#FLXW8B#27", "replace": "//options//, roundtrip = true; useRightHandedSystem = false;", - "referenceImage": "glTFSerializerKHRPunctualLightLH.png" + "referenceImage": "glTFSerializerKHRPunctualLightLH.png", + "workarounds": ["wait-for-readiness"] }, { "title": "GLTF Serializer KHR punctual light, right-handed", "playgroundId": "#FLXW8B#27", "replace": "//options//, roundtrip = true; useRightHandedSystem = true;", - "referenceImage": "glTFSerializerKHRPunctualLightRH.png" + "referenceImage": "glTFSerializerKHRPunctualLightRH.png", + "workarounds": ["wait-for-readiness"] }, { "title": "GLTF Serializer Camera, left-handed", @@ -1476,7 +1494,8 @@ "title": "Enable disable post process", "playgroundId": "#1VI6WV#19", "renderCount": 50, - "referenceImage": "enableDisablePostProcess.png" + "referenceImage": "enableDisablePostProcess.png", + "workarounds": ["opaque-clear-alpha"] }, { "title": "Depth of field", @@ -1515,7 +1534,8 @@ "title": "Particles", "playgroundId": "#G3ZYFU#7", "renderCount": 100, - "referenceImage": "particles.png" + "referenceImage": "particles.png", + "workarounds": ["wait-for-readiness"] }, { "title": "PBRMetallicRoughnessMaterial", @@ -1570,7 +1590,8 @@ "title": "MultiSample render targets", "playgroundId": "#12MKMN#0", "renderCount": 20, - "referenceImage": "MultiSample-render-targets.png" + "referenceImage": "MultiSample-render-targets.png", + "workarounds": ["wait-for-readiness"] }, { "title": "Default rendering pipeline", @@ -1758,7 +1779,8 @@ { "title": "Glow layer and LODs", "playgroundId": "#UNS6ZV#13", - "referenceImage": "glowlayerandlods.png" + "referenceImage": "glowlayerandlods.png", + "workarounds": ["prime-effect-layers"] }, { "title": "Shadows with instances in left handed system", @@ -1783,7 +1805,8 @@ "renderCount": 50, "referenceImage": "prepass-ssao-particles.png", "excludedGraphicsApis": ["OpenGL"], - "reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES" + "reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES", + "workarounds": ["wait-for-readiness"] }, { "title": "Prepass SSAO + instances", @@ -1902,7 +1925,8 @@ "renderCount": 10, "referenceImage": "prepass-ssao-highlight-layer.png", "excludedGraphicsApis": ["OpenGL"], - "reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES" + "reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES", + "workarounds": ["prime-effect-layers"] }, { "title": "Prepass SSAO + point light", @@ -1917,7 +1941,8 @@ "renderCount": 10, "referenceImage": "prepass-ssao-on-off-pp.png", "excludedGraphicsApis": ["OpenGL"], - "reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES" + "reason": "llvmpipe on the Ubuntu CI runner aborts the process with LLVM ERROR: Cannot emit physreg copy instruction while JIT-compiling the prepass SSAO shaders (Mesa/LLVM register allocator bug, thrown before pixel comparison, not a pixel-diff); the test passes on D3D11 and on ANGLE/GLES", + "workarounds": ["opaque-clear-alpha"] }, { "title": "Prepass SSAO + thin instances", @@ -2026,7 +2051,8 @@ "title": "Thin instances + motion blur", "playgroundId": "#HJGC2G#60", "renderCount": 30, - "referenceImage": "thin-instances-mb.png" + "referenceImage": "thin-instances-mb.png", + "workarounds": ["wait-for-readiness"] }, { "title": "Thin instances + motion blur + manual", @@ -2050,7 +2076,8 @@ "title": "Thin instances + render self motion blur", "playgroundId": "#217750#218", "renderCount": 10, - "referenceImage": "thin-instances-renderself-mb.png" + "referenceImage": "thin-instances-renderself-mb.png", + "workarounds": ["wait-for-readiness"] }, { "title": "Reverse depth buffer and shadows", @@ -2127,7 +2154,8 @@ "title": "Bones and morphs computation order", "playgroundId": "#WGZLGJ#4040", "renderCount": 40, - "referenceImage": "bonesMorphsCompute.png" + "referenceImage": "bonesMorphsCompute.png", + "workarounds": ["wait-for-readiness"] }, { "title": "DepthRenderer rhs", @@ -2233,7 +2261,8 @@ { "title": "GUI Images in Grid", "playgroundId": "#8Y0XJL#8", - "referenceImage": "GUI-Images-in-Grid.png" + "referenceImage": "GUI-Images-in-Grid.png", + "workarounds": ["wait-for-readiness"] }, { "title": "GUI Force Resize Width", @@ -2463,7 +2492,8 @@ "excludedGraphicsApis": ["OpenGL"], "reason": "OpenGL: pixel diff out of tolerance (90900 px) -- likely glow-layer + custom-shader interaction differs from D3D11. Tracked for OpenGL backend follow-up.", "renderCount": 5, - "referenceImage": "sphere-with-custom-shader-to-display-wireframe-using-glow-layer.png" + "referenceImage": "sphere-with-custom-shader-to-display-wireframe-using-glow-layer.png", + "workarounds": ["prime-effect-layers"] }, { "title": "blur-cube-with-the-effect-renderer", @@ -2546,7 +2576,8 @@ "title": "dissolve-effect-with-node-material-and-glow-layer", "playgroundId": "#F6PGMC#12", "renderCount": 2, - "referenceImage": "dissolve-effect-with-node-material-and-glow-layer.png" + "referenceImage": "dissolve-effect-with-node-material-and-glow-layer.png", + "workarounds": ["prime-effect-layers"] }, { "title": "cascaded-shadow-maps-and-flying-saucers", @@ -2679,7 +2710,8 @@ { "title": "Glow layer with vertex alpha", "playgroundId": "#GWAX2G#7", - "referenceImage": "Glow-layer-with-vertex-alpha.png" + "referenceImage": "Glow-layer-with-vertex-alpha.png", + "workarounds": ["prime-effect-layers"] }, { "title": "IBL Voxel Shadowing Right-Handed", @@ -3662,8 +3694,8 @@ { "title": "OpenPBR Transmission Roughness vs IOR - Analytic Lights", "playgroundId": "#GRQHVV#134", - "replace": "BABYLON.ImportMeshAsync, var meshLoad = BABYLON.ImportMeshAsync, return scene;, return Promise.resolve(meshLoad).then(function () { return scene; });", - "referenceImage": "OpenPBR-Transmission-Roughness-vs-IOR---Analytic-Lights.png" + "referenceImage": "OpenPBR-Transmission-Roughness-vs-IOR---Analytic-Lights.png", + "workarounds": ["wait-for-import"] }, { "title": "OpenPBR Transmission Anisotropy Roughness vs IOR - IBL", @@ -4255,7 +4287,8 @@ "playgroundId": "#A97UQ6#5", "renderCount": 120, "referenceImage": "particles-helper-sun.png", - "errorRatio": 8 + "errorRatio": 8, + "workarounds": ["opaque-clear-alpha"] }, { "title": "Particles - Helper - Smoke", @@ -4276,7 +4309,8 @@ { "title": "Particles - Attractors", "playgroundId": "#PUWWM3#0", - "referenceImage": "particles-attractors.png" + "referenceImage": "particles-attractors.png", + "workarounds": ["prime-effect-layers"] }, { "title": "Particles - Flowmaps", @@ -4300,7 +4334,8 @@ { "title": "Selection outline layer with instances and LOD", "playgroundId": "#8I487I#0", - "referenceImage": "Selection-outline-layer-with-instances-and-LOD.png" + "referenceImage": "Selection-outline-layer-with-instances-and-LOD.png", + "workarounds": ["prime-effect-layers"] }, { "title": "Vertex Pulling - Normals UVs Colors Tangents", diff --git a/Apps/Playground/Scripts/validation_native.js b/Apps/Playground/Scripts/validation_native.js index 4e6b3ac9c..c928876a8 100644 --- a/Apps/Playground/Scripts/validation_native.js +++ b/Apps/Playground/Scripts/validation_native.js @@ -48,10 +48,59 @@ const cliCaptureFrame = (typeof opts.captureFrame === "number" && opts.captureFrame > 0) ? (opts.captureFrame | 0) : 0; // Frames after the trigger to let RenderDoc finalize the .rdc. const POST_CAPTURE_FRAMES = 5; - // Stopgap so native validation can pass. Examples should wait for their own - // scene, material, GUI, and utility-scene resources; that belongs in the - // examples, not this harness. Waiting here only because fixing each test - // individually is a much larger task. + // Workarounds for bugs in the examples, not in Babylon Native. A test opts in + // by listing IDs in its config.json "workarounds" array, so that list is also + // the list of examples to fix. + const WORKAROUNDS = { + // Example renders before its utility scenes, GUI images, or material effects are ready. + "wait-for-readiness": "wait for utility scenes, GUI textures, and material effects before rendering", + // Example reads the browser-only global `name` (window.name). + "undeclared-name": "declare `name` for the example code", + // Example captures effect-layer output before its render targets were submitted once. + "prime-effect-layers": "render one extra frame before capture when effect layers exist", + // Example leaves scenes, textures, or Draco state behind for later tests. + "leak-cleanup": "dispose leaked scenes and textures and reset Draco after the test", + // Example starts an async model load in createScene without awaiting it. + "wait-for-import": "wait for loader promises started by createScene before rendering", + // Example clears opaque but writes alpha < 1 that the browser would composite. + "opaque-clear-alpha": "skip canvas-background compositing when the scene clears opaque", + }; + let appliedWorkarounds = []; + + function hasWorkaround(test, id) { + return !!test && Array.isArray(test.workarounds) && test.workarounds.indexOf(id) !== -1; + } + + // Logged only when the workaround changed what the runner did for the test. + function noteWorkaround(test, id) { + if (appliedWorkarounds.indexOf(id) !== -1) { + return; + } + appliedWorkarounds.push(id); + console.log("Workaround '" + id + "' applied for '" + ((test && test.title) || "(unnamed)") + "'"); + } + + function validateWorkarounds(tests) { + const errors = []; + for (let i = 0; i < tests.length; ++i) { + const ids = tests[i].workarounds; + if (ids === undefined) { + continue; + } + if (!Array.isArray(ids)) { + errors.push("test " + i + " '" + (tests[i].title || "") + "': workarounds must be an array"); + continue; + } + for (let j = 0; j < ids.length; ++j) { + if (!Object.prototype.hasOwnProperty.call(WORKAROUNDS, ids[j])) { + errors.push("test " + i + " '" + (tests[i].title || "") + "': unknown workaround '" + ids[j] + "'"); + } + } + } + return errors; + } + + // wait-for-readiness limits. const MAX_CONVERGENCE_TICKS = 240; const INITIAL_READINESS_TIMEOUT_MS = 10 * 60 * 1000; const READINESS_RECONCILE_INTERVAL_MS = 100; @@ -62,8 +111,8 @@ const dracoDecoderModule = globalThis.DracoDecoderModule; const dracoEncoderModule = globalThis.DracoEncoderModule; - // UtilityLayerRenderer exposes both sides of the association, but its utility - // scene can have no active camera when the layer is created before the main camera. + // Used by wait-for-readiness. UtilityLayerRenderer exposes both sides of the association, + // but its utility scene can have no active camera when the layer is created before the main camera. const updateUtilityLayerCamera = BABYLON.UtilityLayerRenderer.prototype._updateCamera; BABYLON.UtilityLayerRenderer.prototype._updateCamera = function () { const result = updateUtilityLayerCamera.apply(this, arguments); @@ -111,6 +160,62 @@ return true; } + // undeclared-name: browsers resolve a bare `name` to window.name (""). A direct eval + // here sees this local, without a real global (which breaks the UMD bundles) or + // rewriting the example text (which would shift line numbers and "use strict"). + function evalWithName(code) { + // eslint-disable-next-line no-unused-vars + var name = ""; + return eval(code); + } + + function evalExample(test, code) { + if (hasWorkaround(test, "undeclared-name")) { + noteWorkaround(test, "undeclared-name"); + return evalWithName(code); + } + return eval(code); + } + + // wait-for-import: while a tracker is active, loader promises are recorded so the + // runner can wait for loads the example did not await. + let importTracker = null; + function wrapImportFunction(owner, name) { + const original = owner && owner[name]; + if (typeof original !== "function") { + return; + } + owner[name] = function () { + const promise = original.apply(this, arguments); + if (importTracker && promise && typeof promise.then === "function") { + const entry = { settled: false }; + entry.promise = promise.then( + function () { entry.settled = true; }, + function (error) { entry.settled = true; throw error; }); + // Keep abandoned trackers quiet; waitForTrackedImports still sees the rejection. + entry.promise.catch(function () { }); + importTracker.push(entry); + } + return promise; + }; + } + ["ImportMeshAsync", "AppendSceneAsync", "LoadAssetContainerAsync"].forEach(function (name) { + wrapImportFunction(BABYLON, name); + }); + ["ImportMeshAsync", "AppendAsync", "LoadAssetContainerAsync"].forEach(function (name) { + wrapImportFunction(BABYLON.SceneLoader, name); + }); + + function waitForTrackedImports(test, scene) { + const tracked = importTracker || []; + importTracker = null; + if (tracked.some(function (entry) { return !entry.settled; })) { + noteWorkaround(test, "wait-for-import"); + } + return Promise.all(tracked.map(function (entry) { return entry.promise; })) + .then(function () { return scene; }); + } + function failTest(done) { if (breakOnFail) { // eslint-disable-next-line no-debugger @@ -119,25 +224,73 @@ done(false); } + function hasDracoStateChanged() { + function configChanged(current, saved) { + const keys = Object.keys(Object.assign({}, current, saved)); + for (let i = 0; i < keys.length; ++i) { + if (current[keys[i]] !== saved[keys[i]]) { + return true; + } + } + return false; + } + return BABYLON.DracoCompression.DefaultNumWorkers !== dracoDefaultNumWorkers || + configChanged(BABYLON.DracoDecoder.DefaultConfiguration, dracoDecoderConfiguration) || + configChanged(BABYLON.DracoEncoder.DefaultConfiguration, dracoEncoderConfiguration) || + globalThis.DracoDecoderModule !== dracoDecoderModule || + globalThis.DracoEncoderModule !== dracoEncoderModule; + } + + // leak-cleanup: release state the example left behind so it cannot change later tests. + function cleanupLeaks(test) { + let leaked = false; + + // Async loads may leave additional scenes registered on the engine. + const strayScenes = engine.scenes.slice(); + leaked = leaked || strayScenes.length > 0; + for (let i = 0; i < strayScenes.length; ++i) { + try { strayScenes[i].dispose(); } catch (e) { console.error(e); } + } + + // Cache keys omit load-time options; leaked textures can change later tests. + const leakedTextures = engine.getLoadedTexturesCache(); + leaked = leaked || leakedTextures.length > 0; + for (let i = leakedTextures.length - 1; i >= 0; --i) { + engine._releaseTexture(leakedTextures[i]); + } + engine.clearInternalTexturesCache(); + + // Each scene must load matching Draco JS/WASM modules, not reuse another scene's factory. + leaked = leaked || hasDracoStateChanged(); + BABYLON.DracoCompression.ResetDefault(); + BABYLON.DracoDecoder.ResetDefault(); + BABYLON.DracoEncoder.ResetDefault(); + BABYLON.DracoCompression.DefaultNumWorkers = dracoDefaultNumWorkers; + BABYLON.DracoDecoder.DefaultConfiguration = Object.assign({}, dracoDecoderConfiguration); + BABYLON.DracoEncoder.DefaultConfiguration = Object.assign({}, dracoEncoderConfiguration); + globalThis.DracoDecoderModule = dracoDecoderModule; + globalThis.DracoEncoderModule = dracoEncoderModule; + + if (leaked) { + noteWorkaround(test, "leak-cleanup"); + } + } + // Reset the reused engine between tests. - function cleanupAfterTest() { + function cleanupAfterTest(test) { if (currentScene) { try { currentScene.dispose(); } catch (e) { console.error(e); } currentScene = null; } - // Async loads may leave additional scenes registered on the engine. - if (engine && engine.scenes) { - const strayScenes = engine.scenes.slice(); - for (let i = 0; i < strayScenes.length; ++i) { - try { strayScenes[i].dispose(); } catch (e) { console.error(e); } - } - } - if (!engine) { return; } + if (hasWorkaround(test, "leak-cleanup")) { + cleanupLeaks(test); + } + engine.setHardwareScalingLevel(1); engine.setStencilBuffer(false); @@ -148,29 +301,12 @@ // This is necessary because of https://github.com/BabylonJS/Babylon.js/pull/15217 so that each test starts fresh. engine.releaseEffects(); - // Cache keys omit load-time options; leaked textures can change later tests. - const leakedTextures = engine.getLoadedTexturesCache(); - for (let i = leakedTextures.length - 1; i >= 0; --i) { - engine._releaseTexture(leakedTextures[i]); - } - engine.clearInternalTexturesCache(); - // Global loader observers outlive scenes and can leak settings into later tests. BABYLON.SceneLoader.OnPluginActivatedObservable.clear(); - - // Each scene must load matching Draco JS/WASM modules, not reuse another scene's factory. - BABYLON.DracoCompression.ResetDefault(); - BABYLON.DracoDecoder.ResetDefault(); - BABYLON.DracoEncoder.ResetDefault(); - BABYLON.DracoCompression.DefaultNumWorkers = dracoDefaultNumWorkers; - BABYLON.DracoDecoder.DefaultConfiguration = Object.assign({}, dracoDecoderConfiguration); - BABYLON.DracoEncoder.DefaultConfiguration = Object.assign({}, dracoEncoderConfiguration); - globalThis.DracoDecoderModule = dracoDecoderModule; - globalThis.DracoEncoderModule = dracoEncoderModule; } // Every completion path must stop rendering and clean up exactly once. - function makeTestDone(outerDone) { + function makeTestDone(outerDone, test) { let finished = false; return function (status) { if (finished) { @@ -184,7 +320,7 @@ } catch (e) { console.error(e); } - cleanupAfterTest(); + cleanupAfterTest(test); outerDone(status); }; } @@ -432,8 +568,6 @@ } // Match browser screenshots: composite over the CSS canvas background, then the white page. - // FrameGraph may copy transparent attachments regardless of scene.clearColor. - // Legacy paths retain the clear-alpha gate because some write invalid alpha after opaque clears. const CANVAS_BACKGROUND = [173, 255, 47]; function compositeOverCanvasBackground(data, canvasBackgroundColor) { @@ -459,10 +593,27 @@ return data; } + function hasTranslucentPixel(data) { + for (let index = 3; index < data.length; index += 4) { + if (data[index] !== 255) { + return true; + } + } + return false; + } + function evaluateScreenshot(test, screenshot, referenceImage, done, compareFunction) { let testRes = true; - if (test.canvasBackgroundColor || (currentScene && (currentScene.frameGraph || (currentScene.clearColor && currentScene.clearColor.a < 1)))) { + // opaque-clear-alpha: the browser always composites; skip it when the scene clears opaque + // (FrameGraph may copy transparent attachments regardless of scene.clearColor). + const opaqueClear = !test.canvasBackgroundColor && currentScene && !currentScene.frameGraph && + !(currentScene.clearColor && currentScene.clearColor.a < 1); + if (opaqueClear && hasWorkaround(test, "opaque-clear-alpha")) { + if (hasTranslucentPixel(screenshot)) { + noteWorkaround(test, "opaque-clear-alpha"); + } + } else { compositeOverCanvasBackground(screenshot, test.canvasBackgroundColor); } @@ -580,6 +731,12 @@ let readinessReconcileTimer = null; let readinessTimeoutTimer = null; let waitingForReadiness = true; + const waitForReadiness = hasWorkaround(test, "wait-for-readiness"); + // Without wait-for-readiness, match the browser: main-scene executeWhenReady only. + const getReadinessScenes = waitForReadiness + ? getConvergenceScenes + : function (scene) { return [scene]; }; + const primeEffectLayers = hasWorkaround(test, "prime-effect-layers"); // Effect-layer RTTs need one submitted frame before composition, even with renderCount=1. let effectLayerPrimed = false; @@ -632,8 +789,9 @@ } // Recompute because utility layers can be attached or disposed while // convergence is pending, updating the engine's virtual-scene list. - const convergenceScenes = getConvergenceScenes(currentScene); + const convergenceScenes = waitForReadiness ? getConvergenceScenes(currentScene) : []; if (!convergenceScenes.every(isSceneConverged)) { + noteWorkaround(test, "wait-for-readiness"); if (convergenceTicks >= MAX_CONVERGENCE_TICKS) { stopped = true; evaluated = true; @@ -650,8 +808,9 @@ return; } - if (!effectLayerPrimed && currentScene.effectLayers && currentScene.effectLayers.length > 0) { + if (primeEffectLayers && !effectLayerPrimed && currentScene.effectLayers && currentScene.effectLayers.length > 0) { effectLayerPrimed = true; + noteWorkaround(test, "prime-effect-layers"); currentScene.render(); return; } @@ -716,7 +875,7 @@ readinessReconcileTimer = null; } - const scenes = getConvergenceScenes(currentScene); + const scenes = getReadinessScenes(currentScene); const newScenes = []; for (let i = 0; i < scenes.length; i++) { if (readinessScenes.indexOf(scenes[i]) === -1) { @@ -766,11 +925,16 @@ let allReady = readinessScenes.length > 0; for (let i = 0; i < readinessScenes.length; i++) { // GUI image loads are not included in Scene.executeWhenReady. - if (readyScenes.indexOf(readinessScenes[i]) === -1 || !areGuiTexturesReady(readinessScenes[i])) { + if (readyScenes.indexOf(readinessScenes[i]) === -1 || + (waitForReadiness && !areGuiTexturesReady(readinessScenes[i]))) { allReady = false; break; } } + if (!allReady && readyScenes.indexOf(currentScene) !== -1) { + // The browser would have started rendering here. + noteWorkaround(test, "wait-for-readiness"); + } if (allReady) { startRendering(); } else if (readinessReconcileTimer === null) { @@ -911,8 +1075,6 @@ if (finished) { return; } - // eslint-disable-next-line no-unused-vars - var name = ""; // see the note on the scriptToRun eval below try { if (test.requiresHavok) { await initializeHavokAsync(); @@ -921,7 +1083,13 @@ } } - let createdScene = eval(pgCode); + importTracker = hasWorkaround(test, "wait-for-import") ? [] : null; + let createdScene = evalExample(test, pgCode); + if (importTracker) { + createdScene = Promise.resolve(createdScene).then(function (scene) { + return waitForTrackedImports(test, scene); + }); + } if (createdScene && createdScene.then) { // Bound async creation before scene readiness begins; native blocking needs an external timeout. @@ -1028,11 +1196,8 @@ if (finished) { return; } - // Direct eval sees the browser's default name through this local binding. - // eslint-disable-next-line no-unused-vars - var name = ""; try { - currentScene = eval(scriptCode); + currentScene = evalExample(test, scriptCode); stopSceneProcessing = processCurrentScene(test, referenceImage, done, compareFunction); } catch (e) { @@ -1056,13 +1221,14 @@ } } function runTest(index, outerDone) { - const done = makeTestDone(outerDone); + const test = config.tests[index]; + const done = makeTestDone(outerDone, test); + appliedWorkarounds = []; if (index >= config.tests.length) { done(false); return; } - const test = config.tests[index]; const testInfo = "Running " + test.title; console.log(testInfo); TestUtils.setTitle(testInfo); @@ -1206,12 +1372,23 @@ if (xhr.status === 200) { config = JSON.parse(xhr.responseText); + const workaroundErrors = validateWorkarounds(config.tests); + if (workaroundErrors.length > 0) { + for (let i = 0; i < workaroundErrors.length; ++i) { + console.error("config.json: " + workaroundErrors[i]); + } + console.error("Known workarounds: " + Object.keys(WORKAROUNDS).join(", ")); + TestUtils.exit(1); + return; + } + if (listTests) { // TSV exclusions reflect the catalog, regardless of --include-excluded. for (let i = 0; i < config.tests.length; ++i) { const t = config.tests[i]; const reason = getExclusionReason(t) || ""; - console.log(i + "\t" + (t.title || "") + "\t" + (t.referenceImage || "") + "\t" + reason); + const workarounds = Array.isArray(t.workarounds) ? t.workarounds.join(",") : ""; + console.log(i + "\t" + (t.title || "") + "\t" + (t.referenceImage || "") + "\t" + reason + "\t" + workarounds); } TestUtils.exit(0); return; diff --git a/Apps/Playground/Shared/CommandLine.cpp b/Apps/Playground/Shared/CommandLine.cpp index ac077df7f..a1c54b724 100644 --- a/Apps/Playground/Shared/CommandLine.cpp +++ b/Apps/Playground/Shared/CommandLine.cpp @@ -141,8 +141,9 @@ namespace FlagSpec{"--list", "-l", FlagKind::Boolean, "", "List configured tests as TSV and exit (0).", " Columns: index, title, referenceImage,\n" - " exclusionReason. exclusionReason reflects\n" - " config state (ignores --include-excluded).\n", + " exclusionReason, workarounds. exclusionReason\n" + " reflects config state (ignores\n" + " --include-excluded).\n", [](PlaygroundOptions& o, std::string_view, std::string&) { o.ListTests = true; }}, FlagSpec{"--headless", "", FlagKind::Boolean, "", diff --git a/Documentation/AddingNewValidationTests.md b/Documentation/AddingNewValidationTests.md index f9876d618..74e08d40a 100644 --- a/Documentation/AddingNewValidationTests.md +++ b/Documentation/AddingNewValidationTests.md @@ -28,9 +28,44 @@ In order to add a new test scene, first thing to do is to add a few lines in `Ap `playgroundId` : the snippet id of the playground you want to test `referenceImage` : the reference image name you want to compare to. You don't have a reference yet, so choose a self-explanatory name with .png extension. -# Readiness and deterministic capture +# Example workarounds -The Native runner waits for scene readiness, GUI image readiness, and clean material +Some examples have bugs that only show up in the Native runner, for example rendering +before their resources are ready. Fix these in the example. Until an example is fixed, +its test can enable runner workarounds by ID in `config.json`: + +```json +{ + "title": "GUI slider", + "playgroundId": "#XXXXXX#1", + "referenceImage": "gui-slider.png", + "workarounds": ["wait-for-readiness"] +} +``` + +Only tests that list a workaround get it. Unknown IDs fail config loading. The +`workarounds` lists are the list of examples to fix. `--list` prints them in its last +TSV column. When a workaround changes what the runner does for a test, the runner logs +`Workaround '' applied for ''`. + +| ID | Example bug | Runner behavior | +|---|---|---| +| `wait-for-readiness` | Renders before utility-layer scenes, GUI images, or material effects are ready | Waits for those as described below | +| `undeclared-name` | Reads the browser-only global `name` (`window.name`) | Declares `var name = ""` for the example code | +| `prime-effect-layers` | Captures effect-layer (glow/highlight) output on the first frame | Renders one extra frame before counting frames | +| `leak-cleanup` | Leaves scenes, textures, or Draco state behind after disposal | Disposes stray scenes, releases leaked textures, and resets Draco after the test | +| `wait-for-import` | Starts `ImportMeshAsync`/`AppendSceneAsync`/`LoadAssetContainerAsync` in `createScene` without awaiting it | Waits for loader promises started before the scene is returned | +| `opaque-clear-alpha` | Clears opaque but leaves alpha < 1 in the final image | Skips canvas-background compositing unless the test sets `canvasBackgroundColor`, the scene uses a frame graph, or `clearColor.a < 1` | + +Without `opaque-clear-alpha`, the runner matches the browser screenshot and always composites +translucent pixels over the canvas background (`canvasBackgroundColor`, or green-yellow by default). + +Without `wait-for-readiness`, the runner matches the browser harness: it waits for the +main scene's `executeWhenReady` (10-minute timeout), then renders `renderCount` frames. + +## wait-for-readiness + +The runner waits for scene readiness, GUI image readiness, and clean material defines with ready effects in the active camera's render pass. Utility scenes using the main scene's camera participate in the same check. Inspection restores the previous render pass, including on errors. @@ -43,9 +78,12 @@ Readiness polling does not render extra frames or consume `renderCount`. It refr scene render IDs so material readiness is checked again on the next tick. A scene that still has not converged after 240 waiting render-loop ticks fails explicitly and follows normal once-only cleanup and suite continuation. Screenshot and -RenderDoc capture indices still count rendered frames only. Failures invalidate -pending screenshot callbacks so they cannot evaluate after cleanup or during the -next scene. +RenderDoc capture indices still count rendered frames only. + +# Deterministic capture + +Failures invalidate pending screenshot callbacks so they cannot evaluate after +cleanup or during the next scene. Each test restores both the seeded `Math.random` function and its seed, so a snippet that replaces `Math.random` cannot change the sequence used by the next test.