Wrong computation of error bounds in nanite.cpp

[Axel-Reactor]

Hey Arseny,

I was playing around with clustered LODs and by visual inspection the result never looked like it was reducing triangles enough with distance while using a formula similar to what you have in nanite.cpp.

meshoptimizer/demo/nanite.cpp

Line 31 in d40efb0

return bounds.error / (d > camera_znear ? d : camera_znear) * (camera_proj * 0.5f);

I believe this is incorrect: Area in screen space reduces by distance^2, not distance, so it really should be dividing by d * d. This visually looks much closer to what I would expect too.

[zeux]

Area in screen space reduces by distance^2

Sure, but this isn't meant to be computing area or scaling with it - all quantities involved here are linear (e.g. error is approximating distance between the simplified mesh and the original mesh, so the computation here approximates the distance in pixel-related units). In traditional QEM, the error produced by quadrics scales as a square but that is not the case for any of the meshopt functions.

[Axel-Reactor]

I mean visually it just looks wrong this way. The tri density in screen space should be approximately the same at any distance, no? It gets way higher than that with just error / d very quickly.

[zeux]

I would not necessarily expect the triangle density to be the same/similar at every distance: if the camera is close to the mesh and the source mesh isn't tessellated too aggressively, you'd expect to have source triangles but as the camera gets further away, eventually triangles will get smaller (presumably approaching the size of individual pixels), as the details need to be preserved. In well tessellated but more planar areas of the source mesh you will get larger triangles pretty quickly, whereas in areas with complex geometry the triangles will tend towards pixel sized to preserve silhouettes, as well as lighting if normals are factored in.

In other words, I'd guess it depends on the mesh but triangle density isn't really explicitly controlled here, as the goal is to select a DAG cut where the error is imperceptible (which doesn't always imply uniform triangle density).

I haven't tried to use d * d, the above is just the expectations based on how the math is supposed to work here. I'd guess that using d * d results in visible loss of quality (as in, switching between original geometry & simplified geometry will be much more apparent), but I can't test this easily for the next few weeks.

[Axel-Reactor]

They are way way smaller than a pixel though, it gets really alias-y and flickery. Because of the 1/x curve at some distance essentially nothing changes anymore, you can't even reach the terminal clusters unless the object is absolutely tiny on screen. I don't have it running right now, I can provide some screenshots later, that will illustrate it better.

[zeux]

There might be cases where the underlying error is an over-estimate of the real error, unsure - hard to tell without the actual content. It might be an interesting idea to have an optional error limit on each cluster computed from the triangle size, to prevent cases where sub-pixel triangles aren't removed at all. (of course, you would expect aliasing & flickering still, Nanite pretty much requires a TAA solution to denoise). For more lenient transitions you'd have to compare to a higher threshold (e.g. comparing to 1/screen height results in subpixel triangles but ideally comparing to 2/screen height will result in roughly pixel sized triangles)

Also note https://github.com/nvpro-samples/vk_lod_clusters/blob/main/shaders/traversal.glsl#L53-L63 is using a linear measure, I'm very confident that part is correct but the error is much more difficult to reason about and much more of an approximation. I've converted this to a discussion and will follow up in December when I can test this :)

[zeux]

I looked into this a little further and tested on some assets. I plan to take another look on a different scene once it is released which might not happen this year, so I will update this in the future again but here's notes so far:

The linear scale is definitely correct and how the cutoff should be established more or less. There are some variations on how exactly the scale is computed - e.g. whether the distance is radial or along the view direction; whether the thresholding takes perspective distortion into account or not; etc. These are all mostly relevant close up.

vk_lod_clusters uses a slightly different formula for thresholding, which is err / distance <= sin(atan(tan(fov * 0.5) * errorSizeInPixels / resolution)), whereas meshopt sample uses an equivalent of err / distance * (0.5 / tan(fov * 0.5)) <= errorSizeInPixels / resolution = err / distance <= (2 * tan(fov * 0.5)) * errorSizeInPixels / resolution. These are approximately 2x off, with meshopt thresholding being more aggressive: meshopt threshold is the size of one pixel whereas vk_lod_clusters thresholding is half a pixel I think?

There might be other not-super-obvious constant factors involved here. With the default meshopt sample thresholding, assuming everything else works perfectly, you should select clusters with error <= 1px; if that leads to selection of triangles <= 1px, then by definition a lot of triangles will be sub-pixel (because a triangle <= 1px can easily cover no pixels!). Maybe a better default errorSizeInPixels is 2 with this in mind. Obviously per prior comments, even if the triangles are 1-2px in size, you will need to anti-alias the result.

Additionally, because hierarchical clusterization switches clusters in groups, if a single cluster has a somewhat higher error this could contribute to the delayed transitions (which is likely to also be a constant-time factor that's difficult to estimate). This could in theory be amplified when using the default configuration; I mostly tested all of this with vk_lod_clusters default settings which uses the RT setup. That said my expectation is that since we're talking about behavior in the distance, both configurations would perform similarly...

Note that since a terminal group with a single cluster of 128 triangles is, approximately, a 8x8 quad patch, if the mesh on-screen is larger than ~10x10 pixels, then it would legitimately be too early to switch in certain cases. Obviously setting a higher errorSizeInPixels cutoff will mitigate that.

Finally, in theory, there could be cases where depending on the attribute weights and values used, the quadric error overestimates the actual visual impact of edge collapses, which could result in delayed switching. I was able to trigger this by artificially setting normal weights to be quite high; I don't know to what extent this can happen naturally, as I test on either isolated meshes (where it's more difficult to understand the impact), or on NV Zorah scene (which currently has no vertex attributes). To mitigate this, I added an optional experimental edge limit factor - setting this to e.g. 1.f or 0.5f will limit the error and could sometimes allow for closer transitions. Testing on Zorah produces a ~1% improvement in streamed geometry sizes, so this problem clearly doesn't affect that scene significantly - but I'll test more scenes in the future and add more data here if present. I plan to merge the above for now regardless as it can be helpful to validate whether quadric errors are indeed a limiting factors or no.