What Other Distros Can Learn from NixOS

As a warning, this blog post contains the ramblings of an Arch user who barely knows what a flatpak is and has like, three of them on their system. Their background is in web deployment pipelines and they're trying to apply their personal beliefs to Linux distributions, like an idiot, without even making a proof-of-concept. Reader discretion is advised.

One debate that often comes up is static versus dynamic linking. Most Linux distributions try to manage dependencies by dynamically linking them. This can be good, because if a previous version of a library contains a bug, then it's easy to distribute the bug fix and have it apply to all applications simultaneously. The downside is that if a newer version of a library contains a bug, or a backwards-incompatible change, then that will apply to all installed applications. Dynamic linking is a double-edged sword that makes it both easier to fix bugs and easier to create new ones.

But NixOS offers strict control over both the release and use of libraries, which allows applications to get both the ease of rollout that comes with dynamic linking, and the control that comes from static linking.

Background

When an application is installed on NixOS, it is not put in the /bin folder. In fact, this folder is almost entirely empty. Instead, it goes to /nix/store, in a directory with an unreadable hash. Then, when you boot the system, the files for your system applications are symlinked into /run/current-system/sw, which is in the PATH, so these applications can run in your terminal.

But that's only system applications. For libraries, the application sets its own LD_LIBRARY_PATH variable. This tells the dynamic linker where to fetch the required libraries from. This isn't exactly what the file looks like, but you can imagine it looking something like this:

#!/path/to/bash
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/nix/store/naioghuraopjfi4738-libgit2-2.23"
export LD_LIBRARY_PATH
exec -a git "$@"

The benefit of this system is that you can multiple versions of the same library installed on your system, and different applications can depend on different versions, depending on what they need.

I want to note that there's nothing special about NixOS that would make this impossible on other systems. Flatpak already allows applications to bundle their own versions of libraries, and Snap has a similar system. So I think other distros can still benefit from this system.

Testing

Before I go into the benefits of the system described above, I'll touch on something that distros can do already, but that specifying dependency versions might make easier.

Before you can even update the repository, you'll have to test the library. Hopefully, the maintainer of the library has already done this for you, by creating a suite of unit tests that they ran before creating the release. I suppose it wouldn't hurt to double check.

The more interesting test you'll need to run is similar to what crater does for the Rust ecosystem. Whenever a change is made to the language, Rust runs crater to determine if the change would cause a breaking change for any packages on crates.io. It's a very brute-force way to test for new bugs.

In order for this to work, not only does the library need a unit test suite, but so does every application that depends on that library. You'll build every single dependent against the newer version of the library, and then test all of them to see if anything breaks.

Inevitably, some applications will fail under a new version of a library. This doesn't necessarily need to be because of a problem in the library itself, but it could be due to a transient failure in the application. In such a scenario, the application's dependency would be updated to say 3.13 instead of 3, so it doesn't block other applications from using the newer version.

It would also be beneficial to store somewhere in the package which version of each dependency an application was last tested with. Then, if you want to be extra safe, you can delay upgrading the dependency until you know that a test has been run on the latest dependencies.

Beta Channel

The delayed release of packages is what separates Arch Linux from Manajaro. Some users are more interested in being on the bleeding edge than others. I don't see any reason why two separate package repositories are needed for this. We can allow users to opt-in to the bleeding edge for specific packages faster. These people can submit bug reports before the new features make their way to regular users.

NixOS has this in the form of the unstable channel. Users can opt-in to using the unstable channel to get an experience more similar to a rolling release distro. And judging from some Reddit threads, there are people using the unstable channel regularly. This makes sense to me, since not everyone who uses Nix is there for stability. Some people just like having a central config file. NixOS also lets people go one step further and use unstable for specific packages, but leave the rest of their entire system stable.

Other distros do have this. Most commonly this comes in the form of an LTS variant of the distro, which most users are expected to stay on. The non-LTS variant is often treated as a beta.

Rollout

The rest of the ideas presented in this post are ideas that can be done as long as you have a way to pin dependency versions, but they're not done by NixOS. How important these are is debatable, since NixOS already tests their stable channels before releasing them.

Not every package is of equal importance. A change to a library affecting pacman is far more critical to a library that only affects GIMP. But thanks to the system we have here, it doesn't need to affect every package at the same time. We can rollout a library change to less important packages first before we apply it to every library.

To give an example of how this would work, let's say a new library version releases, and it affects both pacman and GIMP. For this to work, we'd need to assign a priority level to every package. Pacman would probably be a level 2 package, meaning it's very important. There should only be a few level 1 packages. GIMP would be level 4, so not very important. As a level 4 package, GIMP would be one of the first packages to use the new version of the library. After a couple days, pacman would also start using the new library.

Rollback

So with all that in mind, what happens when a library actually is bad? Well, marking the library version as bad is luckily pretty simple. We just update the version resolver to say that if a package asks for version 3, don't use 3.14. Prefer 3.13 instead. Pacman already caches older versions of packages, so you might not even need to re-install 3.13.

Of course, rolling backwards isn't always safe. Some newer packaages might already be using features of 3.14 that don't exist in 3.13. To partially resolve this, we can run the test suite against each package before downgrading to 3.13. Hopefully everything will pass, but if a particular package does not, then we can set that package specifically to require a minimum of 3.14. We can also mark this downstream package as bad, and prevent new installations of it until a patch is made.

Distribution

There's one thing here that we can't control: the distribution. Meaning, we don't know when the user is going to run an upgrade. Unlike in Windows land or web land, we don't run the upgrade automatically. Some systems will at least automatically check for updates and notify the user when there is one, but there's usually no distinction between critical bug fixes and feature updates.

I would recommend having a daemon that at least automatically downgrades in the case of bad library versions. Since we can cache older versions of the library, this wouldn't require too much network activity. And we can always give the user an option to disable it.

NixOS actually does support full automatic upgrades in the background. Some NixOS users report that because NixOS is so stable, they feel comfortable to just letting everything update in the background without intervention. The NixOS wiki provides this example on how to do so.

system.autoUpgrade = {
  enable = true;
  flags = [
    "--print-build-logs"
  ];
  dates = "02:00";
  randomizedDelaySec = "45min";
  allowReboot = false;  # Set to true if you want automatic reboots
};

Conclusion

I don't want to say none of these solutions come with caveats. NixOS has trouble running arbitrary executables because of how it goes about accomplishing its goals. The way things are set up on NixOS can sometimes be confusing to newcomers, and I understand why a Linux distro wouldn't want to copy everything from NixOS. But I think even a best-effort attempt at some of the ideas that NixOS provides could provide major benefits for stability. It's just about choosing the right compromises.