The Nix daemon is a very powerful piece of software. It can reproducibly build tens of thousands of packages from Nixpkgs, securely limited in a small sandbox, triggered by requests from users. It can detect what packages are currently in use and delete unused ones. And it can even build packages remotely, connecting via SSH to another machine.

Unfortunately this flexibility comes with a downside: A large attack surface. The Nix daemon includes over 100 000 lines of C++ code, parsing untrusted user input while running as root, only one incorrect check away from disaster. In the past there have been multiple privilege escalation vulnerabilities in the Nix daemon. Understandably, some security-conscious administrators are hesitant to run Nix on their systems.

Obsidian Systems undertook a project funded by the Sovereign Tech Agency via the Nix Foundation to allow the Nix daemon to run as an unprivileged user. Although users may envision this being most useful on non-NixOS Linux, it can also be useful on NixOS: an administrator could create a system image to configure a server, then only grant users access to an unprivileged Nix daemon, without the permissions to modify the boot closure.

Deletion failure

The simplest part of this project was answering "what happens when the daemon doesn't own the entire store?"

There are only two reasons the Nix daemon should ever need permission to modify a path: fixing a corrupted path, and deleting no-longer-used paths. Corruption can be prevented in the first place with modern filesystems, so it can be left entirely out of scope.

Deleting paths is not strictly necessary for the functioning of Nix, but it can be useful. Without support for garbage collection, the size of the Nix store will grow unbounded as updates are installed and old versions are left around.

While performing initial testing to determine what changes were needed, we discovered that the Nix daemon would fail when presented with a file that should be deleted, but could not be.

The solution was a flag to ignore failures deleting paths while garbage collecting.

Roots daemon

However, the actual deletion is not the challenging part of unprivileged garbage collection.

One of the less obvious ways the Nix daemon needs to interact with the operating system is when discovering garbage collector roots. The Nix daemon can discover most paths to keep around with so-called "garbage collector roots" — explicit entries in the filesystem informing it of programs that are still needed. When asked to delete unused paths, the Nix daemon will skip all garbage collector roots and their transitive dependencies.

These roots can be made either as "temporary" and linked to the life of a process, or as non-temporary, kept around until manually deleted. Temporary roots are created by nix-internal programs like nix-shell and nix-build while they are running, while non-temporary roots are created while installing programs with tools like nix-env or nixos-rebuild.

That method is simple and OS-independent, but unfortunately does not catch every use of packages from Nix. Consider this: A user installs a copy of Firefox on their computer, opens a documentation file directly from /nix/store, updates their system (including Firefox), then asks the daemon to garbage collect. At the time of garbage collection, Firefox is still running and has a file open.

Neither Firefox nor the open documentation file would be kept around, since the Nix daemon would not have received instructions not to delete it.

Thus, Firefox and the file would both be deleted while running, possibly causing a crash and data corruption.

Therefore, the Nix daemon includes support for another type of root: runtime roots. Before deleting paths, the Nix daemon scans all the running programs, libraries, open files, and environment variables to discover what must be kept around. The approach isn't entirely foolproof — it leaves a period of time between the scan and the deletion when what is needed might change — but it does substantially decrease the number of files deleted while they are in use.

Understandably, Linux does not permit unprivileged users to list every open file, environment variable, and running program from other users. This information could reveal secret keys or usage information that are meant only for the system administrator.

The solution was to create a small daemon running as root that performed scanning and sent the results to the ordinary Nix daemon upon request, with no extraneous information that could contain secrets.

Thus, the first substantial change in the rootless Nix daemon is a new daemon that runs as root. Although this sounds counterintuitive ("why start a new project to reduce privileges by adding another privileged component?") it does serve a purpose. Our new "roots daemon" is much smaller than the complete Nix daemon, and its interface does not take in any untrusted user data. It contains many fewer opportunities for exploitation than the full Nix daemon interface.

We took the existing scanning code and moved it out into a library, then wrote the daemon using existing Nix infrastructure. The protocol is about as simple as possible: the Nix daemon connects to a special socket, then the roots daemon returns a list of roots, separated by null bytes. The end of the list is signaled by closing the connection.

Although not expandable, there is a limited amount of information the daemon must convey, and the simpler the design the fewer opportunities there are for vulnerabilities in the implementation.

Sandboxing

By far the most complex part of this project was the sandboxing. Although the final commit is deceptively simple — under 100 lines of C++ — the research and failed attempts took weeks of work.

The question we needed to answer was thus: How can one drop privileges without being root in the first place?

This sounds like an easy question: A process should be able to remove its own privileges at any time. Unfortunately, Linux doesn't work that way. The Linux sandboxing method, so-called "namespaces", work by creating a new view of some part of the system, e.g. mounted filesystems (CLONE_NEWNS), network interfaces (CLONE_NEWNET), or hostname (CLONE_NEWUTS). These can be used to drop privileges, but they could also be used to escalate, so Linux only lets root use them.

There is, however, one way of bypassing the root requirement: user namespaces. By creating a process with the CLONE_NEWUSER flag (or doing it in-process by calling unshare(CLONE_NEWUSER)), an unprivileged program can become a king of its own castle, something that looks like root for all intents and purposes, but has no privileges over the remaining system. Its CAP_SYS_ADMIN (effectively "root permissions") only apply within its user namespace and other namespaces created at the same time. To all other programs, the program appears to be running as an ordinary unprivileged user.

Unfortunately, there is still a catch. The "user namespace" contains a map from internal UIDs (what the sandboxed programs see) and external UIDs (what other programs see). This defaults to empty, so the sandboxed process will only ever see itself running as "nobody". It will be unable to further drop its privileges by becoming a different user id.

Only root in the parent user namespace (i.e. the external system) can set this mapping. Luckily, this is a known problem, and there are already several solutions.

systemd-nsresourced

Our first attempt was with systemd-nsresourced. As the name suggests, it is a daemon that comes as part of the systemd project. It runs as root, and allows allocating 2¹⁶ fresh, random UIDs to a user namespace.

This sounds incredibly convenient. There is no need for manual configuration — nsresourced will automatically choose which IDs to use — and nsresourced handles UID locking and allocation. We thought so too, so we started by writing a patch for the Nix daemon. Instead of using the traditional build user logic, where users are chosen out of a build user group, we configured the daemon to create a new user namespace and request a set of UIDs from nsresourced to fill it. Before running the builder, Nix entered the namespace with nsresourced users and changed to a user with a common builder UID.

We went so far as to post our pull request on GitHub before we realised nsresourced did not meet our requirements.

As it turns out, nsresourced is designed to be used alongside systemd-mountfsd to make containers from VM system images, like one might download from a Linux distribution website. Users allocated through nsresourced may only be used to write to filesystems when specifically authorised, generally from mountfsd. It is possible to bypass the checks by "stealing" the UIDs and using them outside the expected namespace, but it would require the Nix daemon to have root, defeating the purpose of this project. Limiting write permissions is an understandable security feature: if a program could create setuid files for the temporary user then they could gain persistence, controlling the user even after nsresourced granted it to another container. It is, however, not usable for our use case.

Although that problem is possible to work around — e.g. by creating temporary disk images — it was not a clean solution. Worse, nsresourced will not place both the permanent user and temporary users in the same user namespace; thus it is impossible to change ownership of files from the daemon user to the builder, or vice versa. Although workarounds are possible, they add significant complexity. At this point we started looking for other solutions.

newuidmap/newgidmap

We eventually settled on writing a wrapper for the entire Nix daemon which sets up its namespace and executes it. This comes with some downsides — most notably the daemon cannot recognise users other than itself — but the upsides — no daemon code changes, no ownership issues, and no dependencies outside of default NixOS — greatly outweigh them.

We used some old setuid tools, called newuidmap and newgidmap, that were designed for traditional lxc containers with hardcoded UIDs. The system administrator creates a list of UIDs and GIDs each user is allowed to use — in /etc/subuid and /etc/subgid respectively — then unprivileged users call the setuid newuidmap and newgidmap programs with a PID and list of IDs to populate the namespace. Importantly, these IDs are not required to be contiguous. We can map the Nix daemon's external IDs as 0, allowing the nix daemon to own the store and making the daemon's isRoot checks succeed without code changes. We can then map a number of external UIDs to a high number and configure the daemon to use the existing auto-allocate-uids to use those for builds.

Unlike systemd-nsresourced, this is already supported in NixOS. The wrappers are installed by default, and users can be granted UIDs with the users.users.<name>.subUidRanges and users.users.<name>.subGidRanges options.

At first we tried running newuidmap and newgidmap from within the nix daemon when it was given a certain flag. Unfortunately, it is not possible to run unshare on the current process when it has multiple threads. Even though we ran unshare at the very beginning of the main function, Nix had already created threads in static initialisers.

Instead of reorganising the Nix code in order to support a hack for a specific feature, we decided to create an external wrapper for the nix daemon that would set up users as needed.

This new program, called nix-nswrapper, is not called while running an ordinary privileged nix daemon; only from the system service running the unprivileged nix daemon.

The source code is relatively simple:

The wrapper validates its inputs, forks off a helper that will remain in the parent namespace, then moves itself into a user namespace.

Once it has moved into a namespace, the parent signals the helper to map users into the parent's namespace with newuidmap and newgidmap (they cannot be run from within an empty namespace), then waits for completion.

Finally, the parent sets itself to UID and GID 0 within the namespace, fixes permissions on /dev/pts, and executes the Nix daemon.

By the time the Nix daemon has started, the process is, as far as it can tell, running as root.

NixOS

While we had tested all of the components in isolation, they would not be useful without a way to configure them all together; and they would quickly rot without testing.

Thus, we patched the NixOS module in the Nixpkgs repository to support an unprivileged daemon. With the latest version of Nix, running an unprivileged daemon is as simple as adding a user/group, setting the required features, and configuring Nix to use the users:

users.groups.nix-daemon = { };
users.users.nix-daemon = {
    isSystemUser = true;
    group = "nix-daemon";
};

nix = {
    daemonUser = "nix-daemon";
    daemonGroup = "nix-daemon";
    settings.experimental-features = [
        "local-overlay-store"
        "auto-allocate-uids"
    ];
};

The module will warn on common configuration errors and sets up the necessary UIDs, nix daemon service settings, nix store mount options, nix roots daemon, and directory permissions.

It is already fully upstream and included in NixOS 26.05.

Future Work

Just a few weeks after our pull requests were merged, systemd version 260 was released. With it came new features, including the ability to map both temporary and permanent users into the same namespace with nsresourced. Along with support for making tmpfs mounts within the namespace that we initially missed, it would be possible to use nsresourced to allocate UIDs for namespaces after all.

That method does have some downsides compared to the method we eventually chose — namely that using nsresourced would require changes to the Nix daemon and that builds would necessarily happen on a tmpfs without the option of using the host filesystem. However, it would do away with the need to manually configure users.

One notable missing feature of our work was mounting the Nix store read-only. Ordinarily, the Nix store is made read-only for all users; when the Nix daemon starts it creates a new mount namespace and remounts the store as writable for only itself. To most users this is not an issue — only root and the owner can write to the store — but Nix takes a belt-and-suspenders approach to prevent erroneous programs running as root corrupting the store.

Due to permission issues, this would have been challenging with an unprivileged daemon. The read/write remount would need to happen as root, making the entire nix-nswrapper executable require root.

An express design goal in this project was requiring no programs running as root for basic functionality, so this was untenable.

Another issue is trusted users. All users other than the Nix daemon's appear as nobody to the daemon, since their UIDs are not mapped into its namespace. Mapping them in would present a security issue (the Nix daemon would be able to gain control of every user), so another approach is needed.

We have considered creating separate "trusted" and "untrusted" nix sockets, with Linux ACLs controlling who is allowed to connect to each socket. While that would remove some daemon complexity and work with unprivileged nix, it was determined to be out of scope and not implemented in this project.