Relevance Check - Further Insights on Nix for ROS Deployment

[Magnusgaertner]

Hi @mikepurvis,

I hope this message finds you well! I'm curious to know if the insights and recommendations shared during the talk are still relevant today. Could you please shed some light on the following:

• Deployment challenges with Nix? Since you mentioned that Nix was not used during deployment at the time, is that still the case?
• Other unforeseen challenges and maintenance effort when using Nix?

Looking forward to hearing from you!

Best regards,
Magnus Gaertner

[mikepurvis]

Hello! Thanks for the reach-out— since the talk OTTO have shipped two major releases using the Nix methodology, with another very soon. It's overall been pretty great:

• It wasn't practical for us to jump straight to fully declarative NixOS system images; we have staged things by continuing to install the Nix workspaces on a standard Ubuntu base. This allowed peripheral portions of the system to continue to be managed using things like Debian packages, Ansible scripts, and the like. However, it also meant system interoperability challenges, some of which are addressed by tools that Numtide helped us with, like nix-gl-host and system-manager.
• For containers/k8s, we initially did imperative Dockerfiles with RUN nix build <whatever>, but are now moving toward declarative containers created by Nix itself using nix2container. This is a really neat tool and I highly recommend it. It's great for how it allows fine-grained layer control, but also that the images are assembled entirely offline means no dependency on overlayfs, SYS_ADMIN, etc. We also contributed a change (Pull image layers from a supplied manifest. nlewo/nix2container#77) that permits easily pulling the layers of an "underlay" image, allowing us to continue using Ubuntu-underlaid images in the container domain also.
• Overall developers are happy with the system, especially the ability to quickly upgrade and switch between different iterations and slices of the system. There are certainly some quirks to it, and some packaging problems (particularly around Python) can be pretty hairy to sort out— looking initially simple but ending up requiring our handful of Nix gurus to all put our heads together and recommend a solution. Partly this is all-around maturity that there isn't just One Way to do things that should be standard, but also it's a reality that Nix is very powerful and lets you do things a lot of different ways— so it can be challenging to understand the nuance of different approaches.

Stuff that isn't so great:

• Coming from the rock solid world of Debian tooling, it can be frustrating how academic/rough Nix can sometimes feel. Weird bugs (eg) sometimes crop up for which there is no easy or obvious answer and it can be hard to be an advocate for it in situations like that.
• Further to that, the ongoing fracture in the Nix core community between flakes and non-flakes is deeply frustrating and at times embarrassing— a lot of online documentation refers primarily or exclusively to the "legacy" methods of doing things, which are in fact the currently supported methods, since officially all of Flakes and CLI 2.0 is still an experimental beta feature.
• Some systems integrate better with Nix's strict sandbox approach than others; despite the complaint above, poetry2nix is actually a really solid solution for Python and it's frustrating that nothing comparable seems to exist for the rust, npm, or bazel ecosystems. For rust and bazel, the build tool basically downloads a giant "deps" blob into the store, and then you manually bump the hash of that asset when your deps change— so nothing is built or cached incrementally, which means things like Tensorflow have to be rebuilt from scratch each time.
• Going with an internal Hydra instance as our choice of build system has turned out to be unfortunate. Hydra is very rough in places, and although you can get it "up" relatively quickly using the NixOS module, it's a pain to manage or customize, the API coverage is sparse, and upgrading it can be scary. It's still basically the only game in town for something on-prem, but if I were rebooting my efforts, I would go with a fully managed solution like nixbuild.net— that is set up from the get-go to be remote-controlled from other CI systems, and all the store management, GC side, all that nonsense is taken care of.

On the whole though, Nix has been a huge asset to us, and it's hard to imagine going back to a world of omni-debs for our core product deliverables.

[ruffsl]

@mikepurvis how are you handling the recent changes of vendered packages dynamically downloading source at build time?

• Don't build the samples or tests. ros2/mimick_vendor#26 (comment)

I conducting a viability assessment in adopting nix to benefit from the deterministic builds and shares binary caching across our developer team and CI, so I've been digging through the historical tickets and references in the comments on this repo. However, this change with ament_cmake_vendor_package struck me as a rather tricky blocker from the ROS ecosystem side. If we must rebuild the ROS world and all vendered dependencies, I'm hesitant of also needing to patch them as well to maintain nix like builds.

[ruffsl]

BTW, @mikepurvis did you post a public draft of the related colcon REP you prospected over on ROS Discourse?

• https://discourse.ros.org/t/proposal-for-gnuinstalldirs-use-to-enable-multiple-installspaces/29182

[mikepurvis]

@ruffsl For the vendor packages that download source I'm still just aping what @lopsided98 does in nix-ros-overlay:

https://github.com/search?q=repo%3Alopsided98%2Fnix-ros-overlay+patchVendorUrl&type=code

This has worked for us thus far as we only sync our distribution yaml every few months, so it's not a huge burden to coordinate that with updating the patching calls. Obviously, though, it's still not ideal— the main idea I'd had for improving the situation would be a colcon plugin that would scan for ament_cmake_vendor_package calls and stash the repo/ref/hash tuples in the colcon PackageDescriptor metadata for colcon-distro. Having it there would allow the generator to generate the patching calls automatically.

Regarding the GNUInstallDirs work that would enable split outputs to be done more sanely, I'm still very interested in this but my efforts thus far are trapped in the valley of "adequate enough to work internally, hacky enough to not really be presentable externally", and I haven't been able to muster the interest in moving it beyond that, unfortunately.