Object Centric Mesh

[gloganfalzarano]

Hello,

Do you have any thoughts on adapting your work to an object centric scenario? I'm interested in extracting a texture mesh for a single object. This work proposes some loss adaptations which I've been able to use to get good mesh results using vanilla 2DGS, but I think that your work which additionally utilizes depth will provide better results for objects with featureless texture. Will I additionally need to mask upstream parts of the network like depth estimation or will masking during mesh extraction allow for getting the mesh for a single object?

Thanks for you great work!

[Anttwo]

Hi @gloganfalzarano ,

Thanks for the nice words!

In practice, leaving the background really helps both MASt3R-SfM and DepthAnythingV2 to get better camera poses and better depth maps. Depth maps are also pretty important for getting optimal geometry.

As a consequence, I would say that, in my opinion, the best strategy would be to optimize MAtCha the usual way, leaving the background as it is in the training images.

And then, as you mentioned, you could segment the background during mesh extraction:

1. You could either first extract the mesh, and then identify the triangles belonging to the background (by rasterizing the mesh and using segmentation masks) and remove these triangles
2. Or you could also remove Gaussians belonging to the background, and then apply the mesh extraction only on these Gaussians. This would be more precise, in my opinion.

For option 2, you first need to identify which Gaussians belong to the background. One elegant way to do that would be to "uplift" the 2D segmentation masks into the 3D space:

• For instance, let's say that for each image, you have segmentation masks where the background is black and the foreground is white.
• After (or during) training the Gaussians, you could equip each Gaussian with an additional optimizable value between 0 and 1 (i.e. a single channel, gray-scale color).
• You can rasterize and render Gaussians with this gray-scale color value, which would produce a gray-scale image
• You can then compute a loss comparing this gray-scale image with the segmentation mask, for instance a L1 loss or binary cross-entropy loss
• You can optimize these gray-scale color values by repeating this for N iterations. N does not need to be big, so this should be pretty short
• After optimizing these gray-scale color values, Gaussians belonging to the foreground should have a white color (value > 0.5), and Gaussians belonging to the background should have a black color (value < 0.5).
• So you can finally segment the foreground Gaussians in 3D by just removing the Gaussians with a color value < 0.5.

This is a simple way to "uplift" the 2D segmentation masks into the 3D space.

You do this uplifting (= optimizing the gray-scale color values) after training the Gaussians with MAtCha; In this context, as we just want Gaussians to be segmented and not to be modified, you should freeze all the other parameters of the Gaussians, and optimize only the gray-scale color values. As a consequence, this optimization should be really short.

[gloganfalzarano]

Thanks for your response! The process you explained for lifting the 2D segmentation mask into 3D space makes sense.

I'm working in a robotic setting where the background is static and the object is rotated in front of the camera. I understand that Mast3r won't be able to be used for camera pose calculation here, but I already have decent camera pose initializations and I was hoping I that I could use these in place of the Mast3R poses while still utilizing this atlas of charts approach you've designed. I can leave the background in to help the pre-trained monocular depth estimation, but I'm concerned that the static background may confuse reconstruction. Do you have anything thoughts about using your approach in this static-background scenario?

Thanks again,
Logan