There was a sweet spot in like 2024 when it seemed like everything was possible with Nix, but now it seems like everything “experimental” is permanently so (like flakes), packages I care about are not as fresh as I want, I have no mental recall for Nix commands…
Meanwhile, my company is using Nix for everything heavily internally. Everyone gets their dependencies via Nix unless you are like using PAM or something.
Reminds me of Bazel, in the way that it gets adopted by companies with developer support teams (because it solves real problems) but feels frustrating for us ordinary folk. Nixpkgs is kind of a critical part of “ordinary folk” and with the core team disbanding, I feel like my personal moves away from Nix (for projects) are proving correct.
I was lead on a Nix adoption effort for a few years that was kind of like that: it solved real problems, unlocked far faster, smaller, and cheaper builds than would have been possible any other way, and let us ship delta updates over crappy wifi connections to Linux computers on robots. Flakes were a perfect fit for our model, and we were just in time for stuff like up to date versions of cuda and tensorflow to be delivered via nixpkgs.
In my mind, Nix was unstoppable, and I particularly loved how empowered I imagined developers would feel. No mystery-meat CI scripts pushing packages to distant infrastructure that no one understands or even has the permissions to interact with, just the entire build system in one repo, trivially cloneable and hackable... add patches or build steps to anything and it's the same build as always, build it locally or send it to Hydra, it doesn't matter.
A devs "got" it and did leverage those things, used PRs to do safe evaluation of bumps to core dependencies, but I think on the whole it was regarded as cool but not tractable, and a few years after leaving, it sounds like plans are being laid to replace it all with something containers or whatever.
Very frustrating, particularly in a world where it should be trivial to identify the 5-10 typical tasks that people want to do with the nix code, and set up Claude skills to handle those. Given how easy and self-contained the build-test loop is, it feels like an almost perfect fit for agent-led development.
We're using it in the same way - running NixOS on robots.
Overall I think it is succeeding, though it definitely doesn't 'just work' out of the box. We've invested a fair bit of effort into things like:
- containerised services[1]
- UI for branch deployment/service management
- delta patching (at the byte level, to save bandwidth)
The main problem was always that nixlang/nixpkgs are arcane and have a steep learning curve. The majority of your colleagues (in any workplace) don't care about build systems and just want things to work. I think that has been more or less solved by LLMs, and it's now more viable than ever to adopt Nix.
Meanwhile I’m over here with my 100 line flake.nix that installs system dependencies with direnv across hundreds of dev users and dozens of repositories wondering why everyone is all up in arms about boring and stable technology that solves a specific problem
In my personal life, I also like Nix for small projects, especially where you're crossing multiple ecosystems so that any one system's tooling/lockfiles aren't really enough to "enclose" the whole thing.
For that work project though, the final closure was thousands of store paths, with hundreds of source repos being brought together across a multiple of languages and build systems. A lot of that was/is essential complexity, just the reality that robotics is hard, the tools aren't as mature as in other domains, and so you have to be able to develop features and bugfixes all over a huge software stack simultaneously. Doing that with conventional tooling basically gets you a rigid world where you have to tag/release intermediate packages all the time just to get changes into testing, or you have to build the world on every push (lol docker).
With input addressing and hermetic building of the intermediate stages, Nix and Bazel are systems that don't make you make that choice.
My gut feeling is that Nix is just not quite the right abstraction level. I can’t quite articulate this. And hell maybe I’m wrong.
I don’t want to configure a global environment. I just want a build system that works reliably in any environment and can cross-compile from any platform to any platform.
Nix does too damn much. All I want is a build system that doesn’t suck. And I want to run it on windows + Mac + Linux as a first class citizen. And I want to arbitrarily target any platform.
What this really comes down to is that Linux C/C++ toolchains are badly designed. Nix is a huge massive convoluted architecture to try and twist itself around that unfortunate reality.
At least that’s my spicy unpopular opinion that is probably wrong but is at least has elements of truth.
Shared Dynamic Libraries. Not to say they aren't useful. Keeping applications small, keeping security updates simple, sub linear ram scaling, yada-yada-yada, big wins cross the board, not debating that.
The pervasive link detection structure of effectively everything on GNU/Linux assumes is (more-or-less) objectively incorrect behavior in any sane security minded context. Binaries should be able to declare the interface/contract they expect (args/types/abi/exceptions/cryptographic signatures/digests). Then the runtime linker "match" against the local system. The current system is basically 2 levels of string equality checking.
Nix goes into the right direction by getting your runtime linker/elf-runtime & package manager "integrated". But this sort of just feels like putting 'lipstick on a pig' and dancing around the core issue that `foo.exe` cannot ever realize `foobar-v1.8` and `foobar-v1.7` are both installed on the same computer. Nix just manages the environment/symlinks such that `foo.exe` doesn't realize this fact.
> dancing around the core issue that `foo.exe` cannot ever realize `foobar-v1.8` and `foobar-v1.7` are both installed on the same computer
I’m curious if you have an opinion on how you think this should be handled?
At this point I’m just on team static linking or Windows DLL black box style. The Linux approach of dynamic libraries which function same as static is just the worst of every world.
I also blame C/C++ toolchains for being very very bad on Linux.
> I’m curious if you have an opinion on how you think this should be handled?
To "totally overcomplicate things" but do it correctly
- Binary states a list of constraints (namespace:name [<|>|>=|<=|!=] semver).
- ldconf/ld.so either integrate into your package manager and/or are easier to update (I'm not writing conf files by hand and/or flakes). I should simply be able to recursively scan.
- Give the runtime linker an SMT constraint solver (when <1000 this is nearly instant)
- Cache known states to avoid solving NP hard problems every time you launch `cat`.
> At this point I’m just on team static linking or Windows DLL black box style. The Linux approach of dynamic libraries which function same as static is just the worst of every world.
Honestly same.
Having the package manager <-> elf runtime <-> shared libraries more-or-less be a blackbox is probably for the best (which is sort of what windows does with the install-shield/install-wizard stuff). But nobody in Linux Land really wants to "improve" userland, other then change to flavor-of-the-month display managers.
> I also blame C/C++ toolchains for being very very bad on Linux.
They honestly aren't, it is more your pacakge manager *is* your library manager. Because of the absolute bullshit of shared libraries.
If you pretend it is the 1970/80s everyone at your company has the architecture, unix version, etc. It is pretty nice. No cross compiling, multiple OSs, everything just sort of works pretty well. But like I said, "pretend".
> They honestly aren't, it is more your pacakge manager is your library manager. Because of the absolute bullshit of shared libraries.
Hrm. It’s trivial for Linux to cross-compile for Windows because Windows is sane. It requires moving mountains for Windows to cross-compile to Linux. Hell, the only way Linux can reasonably compile to target an older version of glibc is to create a full container image containing and using the old glibc. It’s absurd. I suppose this isn’t the root evil. But it feels related to me. Maybe not.
I mean... a big part of that with conventional Linux distros is the idea that you can security-update a low level library and all the stuff linking to it gets the fixes "for free", a dream abandoned both by Nix and also the movement toward distribution via containers/flatpaks/VMs/whatever.
> The truth is like the sun. The light from the sun touches everything, it's bright and it's obvious... yet no one can stare directly at it.
bad analogy. It can be very painful to accept the truth, but doing so is always a good thing. Staring directly into the sun can cause permanent damage to your eyes -- not a good thing.
No this is not true. There is an evolutionary reason why people lie to themselves. Look at the world.
Almost everyone believes in religions. The world lies to themselves on an unprecedented scale. If accepting the truth had an evolutionary benefit then most of the world would not be so delusional. Delusion dominates the population because lying to oneself conveys a survival benefit such that natural selection selects for this trait.
This short infographic illustrates and proves it. But you know what else proves it? Me getting voted down as I throw the truth down in front of everyone’s faces. I’m a target for selection.
Maybe nix will one day be a tool that’s actually good. It has elements of it but complexity and usability hinder adoption. So overall nix is just shit imo. It’s shit wrapped in a dream, a dream of an ideal way to do things. How do we make that dream a reality?
By lying to ourselves. By claiming nix is already there and that other people can’t see the light such that over the years after enough improvements nix becomes an actual good tool because enough delusional people pushed the dream far enough that it actually happened. But in order to do that… people like me must be culled from the herd.
At the time I was there we hadn't made it all the way to NixOS on the targets, so it was an Ubuntu "base" + Nix managing the app workspace as a kind of pseudo container, though able to bring more of its own system configuration with it (https://github.com/numtide/system-manager) than the more typical nuisance setup of container + separate outer config managed by ansible or a deb or whatever.
As a result of that, in "production" we still actually delivered full rootfs images just to be absolutely certain, and the delta upgrades were for smaller test fleets that could updated hourly just via simple push tooling.
If I was building it from scratch though, I'd probably just do NixOS + colmena, and do a push model forever. It's not worth the saved SSH connection to not have those logs and status messages coming back to the central coordinator immediately.
I’ve been running a NixOS based homelab for awhile now with 5 physical hosts and about 30 NixOS containers/VMs in an Incus cluster and I can’t imagine moving away from my central Nix repo and ability to rebuild/upgrade the entire fleet in one command and feel confident that things will work.
While I’m concerned about the disbanding, it would take quite a lot to make me look elsewhere, and there’s enough critical mass that I feel confident others will step up.
YMMV, I don’t run a home lab, I just have a NAS and run a personal website. I definitely don’t have a “fleet”. I have two servers running Debian and I can bring them back up from zero in <30 minutes.
“Rebuild everything in one command and have it work” is not something that I’m chasing after and I am skeptical that it would work anyway. Maybe there is something I’m missing, but when I update the software, the updates come with changes and it’s possible that things break. My goals are to keep reasonably up to date and to be able to fix things quickly if they do break.
Regarding Nix complaints:
Package maintenance is kind of a crapshoot. Maybe your package is in nixpkgs, maybe there is a flake for it, maybe you find something that is actively updated, maybe not. Maybe there is a package but half the features are turned off because the maintainer didn’t bother. Maybe there is a package but half the features are turned off on macOS for unknown reasons. What I want is just to know what level of distro-level maintenance the package has, including its transitive dependencies.
Docs are just kinda bad. Fragmented across different sites. Docs teaching you Nix forwards or backwards, or teaching old Nix, or teaching Nix with flakes, or teaching you Nix for end-users or developers or package maintainers, or Nix on Mac or NixOS. A surprising number of broken links. What I want is one site, with a little drop-down menu to select the version I am using. What I have is hours spent on the NixOS Discord trying to figure out how to do basic stuff.
Yeah. I fixed AI/GPU stack consisting of 6 existing packages to run CLI on Mac. Fixed all comments during several weeks. And then silence. No approve no nothing for few months. Closed PR and will never come back.
> “Rebuild everything in one command and have it work” is not something that I’m chasing after and I am skeptical that it would work anyway.
I'm not using "rebuild" to describe recreating an instance from scratch, but in the "nixos-rebuild" sense, i.e. I can deploy a fleet-wide configuration shared by all hosts, or apply package updates across the fleet, etc.
As for the skepticism, all I can offer is my experience which is that it does indeed work, and I've been running this way for awhile.
> Maybe there is something I’m missing, but when I update the software, the updates come with changes and it’s possible that things break.
The only time I've had breaking changes that required manual intervention was on a major release update. The issue amounted to "You're using abc configuration option but should be using xyz instead". This message was clearly logged, and fixing it was a matter of a few minutes of reading why a particular property name had changed.
Those updates happen twice/year, and in the last ~3 years I think two packages have forced me to make a change. Unless you're running unstable, you won't encounter this for ongoing package updates within a release.
On the topic of rebuilding from scratch, that's not a single command, but it's only a few. VMs/containers mount incus volumes (zfs-backed) for data storage, and a fresh rebuild is a matter of spawning a new instance from my homelab base image (I use OpenTofu to orchestrate this), running a nixos-rebuild targeting the new instance from my workstation, and things are back up and running. But I'm less focused on full rebuilds since I'm already doing Incus backups and can restore on any of the nodes in my Incus cluster.
> My goals are to keep reasonably up to date and to be able to fix things quickly if they do break.
My goals are similar. I can completely understand sticking with Debian for two hosts. I had experimented with NixOS for years but never really went all-in until I started expanding my homelab environment. At that point, NixOS just clicked and I'm far more productive with 30+ hosts than I used to be with 1-2.
> Package maintenance is kind of a crapshoot. Maybe your package is in nixpkgs, maybe there is a flake for it, maybe...
Which channel were you running on? And do you have any examples of specific packages? What you're describing just sounds foreign to me. I currently default to the latest stable release but if there's something that isn't in stable I'll override that specific package to use unstable. Much of the community just runs everything on unstable, but I prefer a slightly slower pace of updates.
Nixpkgs is the largest package repository across distros by volume, and as much as I love Debian, it's far more common to find things missing there.
> Maybe there is a package but half the features are turned off on macOS for unknown reasons
This sounds like you're branching into territory that can no longer be reasonably framed as NixOS vs. Debian (or other distro of choice).
> Docs are just kinda bad.
This is by far the project's greatest weakness. LLMs have been the saving grace. The frontier models are excellent at NixOS and I've switched to mostly having a conversation in my NixOS Claude project when I need info. This is in no way a defense of the docs, but for anyone motivated to use NixOS, there is at least a good option beyond the docs themselves.
I moved my homelab to Kubernetes. I just don’t see the use of single VPS’es. But I use the cluster as a training platform, so if I didn’t need Kubernetes in other parts of my life, I probably would have stuck with my NixOS VPS’es that all share config.
I use kubenix to manage the cluster, of course. ;-)
The two aren't mutually exclusive! I just finished porting my personal kubernetes cluster from running on Debian to running on NixOS. Now whenever I want to update the host OS on my kubernetes nodes, I just build a new bootable ISO from my Nix configs and upload it to my servers, swapping out the whole OS atomically. If anything goes wrong, I can swap back to my previous build. No more anxiety running `sudo apt-get upgrade` and hoping for the best.
But as the host OS I use Talos Linux, which itself is declarative and minimal. For small Kubernetes version upgrades, I upgrade the kubelet with an API command. When that isn't possible, I upload a new Talos image and swap them.
Since the nodes are cloud VMs, I will use a spare VM to add as a new node, and remove an old node. So I never run with fewer nodes. For my on-prem cluster, I will do the same with a spare hypervisor VM. It really helps ensure my nodes are ephemeral.
So NixOS, for now, has become a desktop/laptop operating system for me, and whenever I need an execution environment, e.g. a CI runner or a remote shell, I choose "something Nix-like" which is `nix` in a container without systemd.
> Meanwhile, my company is using Nix for everything heavily internally. Everyone gets their dependencies via Nix unless you are like using PAM or something.
Why doesn't your company fund development of nix and its associated ecosystem?
That's the social contract with open-source software. You are not owed a cent for your work. But they're not owed support either.
On the other hand, building an entire company atop experimental software that might explode or cease to exist tomorrow seems so monumentally stupid that the inevitable business failure is the outcome they don't need, but so badly deserve.
>building an entire company atop experimental software that might explode or cease to exist tomorrow seems so monumentally stupid
All things come to an end. Why not ride the wave of a FOSS project, eg by offering support as a business. As long as you are prepared to pivot or close the business when it's no longer needed, why not?
Is it really that complicated? I personally use direnv with nix to have per-project dependency versions installed automatically, and I’ve never had too much trouble. I know that the flake.nix files can become somewhat complicated, but mine have been pretty simple thus far.
Not OP, I’m not really sure what your specific complaints are based on the original comment. They seemed more like general/nonspecific concerns, which left the comment pretty open to interpretation.
I’m here in the thread and I can answer questions or elaborate on things, so if you want to know what I meant you can just ask me and there’s a good chance I’ll respond.
It gets worse when you are not a solo dev, and there's sufficient variety on people's setups. Oops, someone updated a version, and then built things for just their processor, and now I am stuck in a 20 minute compilation loop because some bad pin. Debugging Nix problems like those makes me think that old Gentoo Linux back in 2005 was easy and user friendly.
Updating causing a rebuild isn't really a Nix or even a build tool problem though, is it? cache.nixos.org can't provide everything for every architecture all the time.
With multiple types of machines involved, you want some sort of binary cache setup, ideally automated with CI. There's niks3 [0], celler [1] (a more active fork of attic [2]), hydra [3].
The way I set it up at my shop was that everything would build on your PR, and so by the time it merged, everything was already cached and no one should see a rebuild... at most a download.
Im thinking through this for my team, any previous writeups on it? Or do you mind brain dumping the high level of how you had it setup from a design perspective : D
The first was the "toolkit" repo that had all the manual maintained dependencies and functions, while the second was a managed repo which would have tags pushed to it by the pipeline.
So basically all the repos had a push hook attached to them that would run this centralized dispatch workflow (on Jenkins, but it could be anything). The dispatch job would sanitize the branch name like joey-b/fancy-feature and attach it to the most recent "released version" + devel timestamp, so you'd end up with like 2.25+20260808-12345+joey-b-fancy-feature, and that would be pushed as a floating tag to the nix-ros repo with all the sources from the hundreds of participating repos either locked to the versions set in the top level 2.25 devel branches or to the specified feature branch, so that a given build could pull together multiple same-named branches from across repos. That would be sent off to Hydra, and nix2container outputs were also built that went to simulator based validation.
But the end UI was pretty nice, basically a nix configuration mapped the names to the those repos so after the build was done you could just pull it and "enter the workspace" environment like:
And if you were hacking on the "base" repo itself, it was very easy to pass flags like --override-input base=ros-base#some-ref or path/to/local for quick iteration.
Anyway, as I say, I'm obviously proud of what was achieved, particularly in a pre-LLM world, and the experience of building this has given me many of pg's blub experiences over the years, where I look at the ways that other build and packaging systems solve these problems and think yes, yes I can see how that works, but also, I have experienced a world where in exchange for some relatively minor strictness tradeoffs, the set of problems that that is solving never had to exist in the first place.
Awesome, thanks for taking the time to put this together, that really is impressive pre-LLM. We wont be at this level for a while but its great to have resources to look at for examples of others have solved things.
That's already the case in determinate-nix. Flakes do have big problems though - try cloning a big data project with GBs of data and see how much you love flakes when it copies it to the store every time you run `nix run .#analysis`.
I don’t have all of the details, sorry, but PAM is a bunch of .so that are dlopen'd and if you mess them up you lose access to your system pretty damn quick. I would tell people to just use the system libc but I don’t know the specific failure modes for using libc via Nix. Using system libc = building your project outside of Nix.
There could be similar issues with NSS but I think more people are ready to bypass NSS altogether.
It has to do with installing packages that use PAM to e.g. validate user passwords (e.g. display managers, screen lockers). So there is a system PAM with its configuration, but making a program packaged with nix respect that system PAM config is not as easy as just linking with libpam from nixpkgs.
Meanwhile, my company is using Nix for everything heavily internally. Everyone gets their dependencies via Nix unless you are like using PAM or something.
Reminds me of Bazel, in the way that it gets adopted by companies with developer support teams (because it solves real problems) but feels frustrating for us ordinary folk. Nixpkgs is kind of a critical part of “ordinary folk” and with the core team disbanding, I feel like my personal moves away from Nix (for projects) are proving correct.