Git submodules for the libraries

[SharkWipf]

First of all, nice work on this project, it looks very nice. Once Gsplat support is in it should work in fully FOSS environments too (if you cut out the MAST3R-SFM step at least), which is great.

Just a heads-up though, you seem to have included the complete mast3r-sfm, dust3r, inria 2dgs and depth-anything-v2 projects in this repo. Unless you've directly made modifications to these projects, the recommended way to handle external projects is via git submodules that point to the external projects, instead of including the projects themselves whole. That way, links to external projects remain preserved, and the focus of your repo remains on just your own code, making things easier to review.

[Anttwo]

Hello @SharkWipf,
Thanks for the nice words!

You are absolutely right.
The reason for this is that we actually modified the code of 2DGS and MASt3R-SfM/DUSt3R for MAtCha, so we decided to include our modified versions of their code within the repo.

If you take a close look at the code included in this repo, you'll see that it differs from the original code (differences are not that big, but they are here):

• For 2DGS, we modified the base Gaussian model class and added some utilities here and there, such as the mipnerf antialiasing filter and some mesh extraction functions.
• For MASt3R-SfM, we did slightly modify the model to make it compatible with predefined camera poses, for scenarios where posed images are available.

DepthAnythingV2 (DAV2) remained untouched though. We still included a version of their code within our repo because, initially, we considered slightly modifying it (but ultimately didn't); this approach was also consistent with what we did with the other works.
But I totally agree that including DAV2 as a submodule could have been better.

Of course, while including modified versions of these codes, we wanted to give proper credit to the original authors; We tried to properly reference the original authors through the LICENSE section in the README.md file, but I agree this might not be very clear.

[SharkWipf]

Understood! It wasn't clear if anything was actually modified in those repos, since there's no git history, so no easy way to check exactly what changed short of comparing with upstream side-by-side.

Am I understanding correctly that even pre-posed datasets (i.e. Colmap) go through MAST3R-SFM currently? That might make the code license on your project less straightforward (and through that, incompatible with open source projects like Gsplat if you plan on using the same implementation), due to the "viral" nature of the license Naver uses:

If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

If you're building on Dust3r/Mast3r/-sfm, that would effectively override your chosen MIT license.

(Edit: and the "virality" of the license Naver/MAST3R-SfM uses might also make your project incompatible with the Inria-licensed 2DGS. Inria allows relicensing your own code, but if you're modifying 2DGS as part of your code, which automatically inherits the Naver license from MAST3R, I am not sure how/if that would work legally)

[chris-aeviator]

I encountered some minor issues when using your versions, this is not complete but hints at some things to be done if a long term support is desired:

• simpleknn won't build until I add a pyproject.toml

[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "torch"]

• cmake was installed as v4 and can't build the submodule unless installing cmake<4 via conda-forge
• support for pytorch>=2.6 & CUDA 12.8 requires changes to torch.load(), all (or some) ckpt file loading torch.load's
  will need a weights_only=False

I can confirm that running this project with pytorch>2 & Cuda 12.8 works and produces a valid mesh.

edit

also this fix (from CUT3R/CUT3R#7 ) needs to be applied:

--- a/src/croco/models/curope/kernels.cu
+++ b/src/croco/models/curope/kernels.cu
@@ -98,7 +98,7 @@ void rope_2d_cuda( torch::Tensor tokens, const torch::Tensor pos, const float ba
     const int N_BLOCKS = B * N; // each block takes care of H*D values
     const int SHARED_MEM = sizeof(float) * (D + D/4);

-    AT_DISPATCH_FLOATING_TYPES_AND_HALF(tokens.type(), "rope_2d_cuda", ([&] {
+    AT_DISPATCH_FLOATING_TYPES_AND_HALF(tokens.scalar_type(), "rope_2d_cuda", ([&] {
         rope_2d_cuda_kernel<scalar_t> <<<N_BLOCKS, THREADS_PER_BLOCK, SHARED_MEM>>> (
             //tokens.data_ptr<scalar_t>(),
             tokens.packed_accessor32<scalar_t,4,torch::RestrictPtrTraits>(),

to support newser versions of pytorch/cuda

[Anttwo]

@SharkWipf
Thank you for all the details!
In practice, MASt3R-SfM is always applied at initialization, so I suppose this clearly overrides the MIT license even for our own scripts. Thank you for pointing that out, I'm going to change the license file.
I must admit that I'm not very knowledgeable about licensing details.

@chris-aeviator
Thanks a lot for your testing!
This is really valuable, thanks.

I ran the installation script on two different machines, both equipped with CUDA 11.8; it worked without any issues, but I did not experiment with more recent versions of CUDA.

Could you tell me if you used the installation script (which is tailored for CUDA 11.8) or followed the detailed instructions for setting up the environment?
Indeed, I forgot to specify some versions in the detailed instructions, which might explain why the wrong version of cmake was installed.
I'm surprised by the fact that simpleknn won't build without adding a pyproject.toml though... Do you have a clue why?

Again, big thanks to both of you for your comments, this is really helpful!