Compiler bootstrapping in Nixpkgs

Nixpkgs is a very large pure functional program, a recipe for building 10,000s of open source projects — building just about every bit of software we can get our hands on. For a program this large, with this many authors, it is very important to think about how it should be structured in order for everyone to best coordinate and to limit accidental complexity. A key idea here is that packages should be legible in isolation — as much as possible, we should be writing separate recipes for small packages, think about them and maintain them relatively independently, and not have to think much about the package-agnostic machinery in the background that stitches them together.

An Example — Cross Compilation

To make this point clearer, it will help to look at cases where we deviated from this ideal, and the boundaries between individual packages and machinery to glue everything together became a bit more blurred. A good example of this is the early history of cross compilation in Nixpkgs. Let's look at part of one commit message from the some of the earlier work on cross compilation, way back in 2009, 2aba922d:

This new idea on cross compiling is not similar to that of the nixpkgs/branches/cross-compilation, which mostly added bare new expressions for anything to be cross compiled, if I understood it correctly.

This ancient branch is deleted upstream, but it happens lives on in old Nixpkgs forks ^[1] such as ours. Here's a stable link from Software Heritage that won't go away. If you look at pkgs/top-level/all-packages.nix on that branch, you will see separate packages at the top level for different platforms, like binutilsMips and helloCross:

  binutilsMips = import ../development/tools/misc/binutils-cross {
    inherit fetchurl stdenv noSysDirs;
    cross = "mipsel-linux";
  };

  # [...]

  gcc41mipsboot = import ../development/compilers/gcc-4.1-cross {
    inherit fetchurl stdenv noSysDirs;
    langF77 = false;
    langCC = false;
    binutilsCross = binutilsMips;
    kernelHeadersCross = kernelHeadersMips;
    cross = "mipsel-linux";
  };

  # [...]

  helloCross = import ../applications/misc/hello/ex-1 {
    inherit fetchurl perl;
    stdenv = overrideGCC stdenv gcc41mips;
  };

This is what @viric is referring to in that commit message. Now, having handwritten separate copies of packages for each platform like this clearly doesn't scale — between the different OSes, different CPUs, different libc choices, there are far to many combinations of valid platforms we might want to build for.

What @viric did in 2aba922d instead was get rid of these per-platform packages, and just have *Cross with a global cross parameter. Also, the underlying native and cross package functions (with the exception of gcc-cross-wrapper) were deduplicated, into a single package function. The last part was a major improvement with respect to the idea we started this piece with — if one wants to write some package, let's call it foo, but that entails packaging it twice one for native, and one for all cross platforms, that would a stark example of global concerns (cross, bootstrapping) creeping inside individual packages.

The combined packages were much better, but still there were some things left to be desired. For example, check out the modification to glibc for adding cross compilation support:

--- a/pkgs/development/libraries/glibc-2.9/default.nix
+++ b/pkgs/development/libraries/glibc-2.9/default.nix
@@
 { stdenv, fetchurl, kernelHeaders
 , installLocales ? true
 , profilingLibraries ? false
+, cross ? null
+, binutilsCross ? null
+, gccCross ? null
 }:
@@
+  buildInputs = stdenv.lib.optionals (cross != null) [ binutilsCross gccCross ];
+

While a single binutilsCross is less clutter than many separate binutilsMips, binutilsSparc, etc., it still puts downstream packages in a bind — do I want binutils or binutilsCross? Moreover, the regular binutils is provided by default as part of stdenv, why isn't binutilsCross? What platform should the stdenv-provided binutils target anyways? If every package had to repeat this boilerplate in order to build for the proper platform, cross support would be doomed to always be bit-rotting very quickly.

In 2017, when I first joined Obsidian, I had the great privilege of being able to finish at work my project of overhauling how Nixpkgs does cross compilation, because we needed it for creating mobile apps from web apps written in Haskell using (what would become) our app framework Obelisk. Among other changes, there was no longer any need for any binutilsCross or gccCross, at the top-level or in the parameters of specific packages, because of the insight that in general we need multiple variants (build ≟ host ≟ target) of all packages. And indeed, in PR #25897, one can see gccCross finally go away in the glibc package. The resolution was simple enough: the stdenv.cc that is provided automatically by default should always be a toolchain that targets the platform we're supposed to be building the package for, whether or not we're doing cross compilation.

This is very good for cross in particular: replacing pesky conditional code with better defaults (and more a finer categorization of dependencies) means that far fewer packages had to think about cross — it was much more likely to "just work" with no extra effort on the part of the package maintainer. But it also is, cross aside, better fidelity to the original principle. If we anthropomorphize a package function's parameters as a "I need a ...!" request, the package shouldn't be saying "I need a native C compiler" or "I need a cross C compiler", it should just say "Tell me what platform we're building for" and "I need a C compiler for that platform". And indeed, in most cases the latter alone is fine — "just give me the compiler, I don't even care what platform we're building for". All this fits the principle of packages being legible in isolation and (back to anthropomorphizing), only "caring" about local decisions that are immediately relevant to them, not the global big picture.

libc

Still, not every stone was turned. And for many years later, the libcCross package lived on despite the general intent of getting rid of all such *Cross packages. I don't remember the exact reason I left libcCross intact, but back in 2017 there was still much fragility around compilers and dreaded "infinite recursion" errors, and I think they were making it difficult to remove. (More on this topic below...). Most packages got their libc from the default C/C++ compiler (in particular stdenv.cc.libc), which automatically was the correct one. But there were still some instances of the likes of if stdenv.hostPlatform != stdenv.buildPlatform then libcCross else stdenv.cc.libc littered around the Nixpkgs — no good. "libc" is already not a single package, but a platform-specific alias (since there are many libc implementations, and different ones work on different platforms). There is no reason to make users jump though extra cross-or-native hoops, when the alias should just work in all cases.

I'm happy to say that this is finally fixed, almost a decade later. Enough other issues had been fixed ^[2] over the intervening years that I could just "do the thing" without running into any major roadblocks. That fix just happened in PRs #414321 and #414573, and is the first occasion for writing this blog post. The new top level libc attribute again will not be needed too often — stdenv.cc.libc is fine — but when it is needed will always work, in cross builds, native builds, and weird native bootstrapping builds all alike.

Compilers and standard libraries

Above, I mentioned "fragility around compilers and dreaded 'infinite recursion' errors". Let's discuss this more now.

Compilers and standard libraries are often maintained in the same repo, e.g. GCC and libstdc++, Clang and libc++ (and now the LLVM libc), GHC and base, and rustc and core and std. Most distros likewise build the compiler and standard library together, one package, for basically Conway's law reasons. This means that newly-built compilers in the first part of the build are used to build the libraries in the second part of the build.

The problem with this is that it makes a huge mess, especially for cross compilation. First of all there might be other libraries needed in the bootstrap graph not from the combined compiler and standard library repo.

Here's one example: when building a C/C++ compiler, some of the compiler-provided libraries (C++ standard library, santizers, etc.) depend on libc, and libc is usually maintained separately. This means that one needs to sandwich the libc build between the builtins/intrinsics (the main part of compiler-rt or libgcc), the first compiler-provided library built after the compiler, and the other compiler-provided libraries. ^[3] For example with LLVM, we have something like this:

flowchart TD
%% Nodes

    subgraph LLVMX [LLVM]
    subgraph LLVMTools [Tools]
    LLD>"LLD"]
    LLVM("LLVM")
    Clang>"Clang"]
    end

    subgraph LLVMLibs [Runtime Libs]
    CompilerRT0("Compiler-RT (for builtins)")
    CompilerRT1("Compiler-RT (for sanitizers)")
    Libcxx("Libc++")
    end
    end

    subgraph Platform [Platform-specific]
    Libc
    end

%% Edge connections between nodes

    LLVM -.-> Clang & LLD
    LLVMTools  --> LLVMLibs
    LLVMTools --> Libc

    CompilerRT0 -.-> Libc

    Libc -.-> Libcxx & CompilerRT1

Legend:

• Solid edges mean "built with"
• Dashed edges mean "linked with"
• Edges to/from containers ("Tools" and "Runtime Libs") are shorthands for edges to/from all the nodes inside those containers.

In these situations, a combined package now no longer makes sense even from a Conway's law perspective, because it would be gobbling up bits of other projects' sources. (Continuing with the graphed LLVM example, we'd have to add the platform libc's source to the monolithic LLVM package, even though it is not part of LLVM, because those edges.)

Cross-compilation inherits these general bootstrapping issues, but also introduces additional ones. When the compiler itself is being cross compiled (build ≠ host ≟ target), we have a compiler we cannot run to build the standard library. In this case, packages need to build an entire second build = host version of the compiler, just to build the standard library. And then, since this extra compiler wasn't asked to be installed, it is typically just... thrown away. Likewise, when the compiler is multi-target (which is increasingly common, as it should be), and is doing a combined build for multiple targets, the same multi-target compiler will end up being pointlessly rebuilt again and again.

If we zoom out a bit from all of these issues, there is a common thread, which is that the issues arise when packages make "bootstrapping decisions", like how many compilers do we need to build, what tools should be used to build one library etc. I submit the right design is instead that packages should never make these decisions, and indeed, newly-built artifacts to be installed (like the compiler) should never be run within the same build to build anything else.

I'm not the first to rethink compiler bootstrapping along these lines. musl-cross-make, an alternative GCC bootstap, is careful to preserve a single GCC build across slightly different targets. Exherbo's multiarch infrastructure was made with something similar. LLVM has for a while suppported independent builds of its components. But for a while, Nixpkgs always built compilers the traditional way, which made my initial cross work a lot harder --- certainly more waiting for redundant builds!

Thankfully, that's started to change. First to get reworked was LLVM, a few years ago. If one looks at the curent state of pkgs/development/compilers/llvm/common/default.nix, it might a bit hard to read, but one will see:

• An attribute set of libraries
• An attribute set of tools containing
  • LLVM, clang itself (unwrapped), and other unwrapped compilers
  • Many invocations of cc-wrapper, assembling toolchains with increasingly many components.

The libraries are both used by and built using those toolchains, so dependencies criss-cross the libraries and tools attribute sets, but packages themselves are strictly acyclic (as Nix requires) giving us a pure bootstrap. The general cross infrastructure likewise ensures that the build host and target platforms line up properly (there is an off-by-one aspect to this: your toolchain's target platform is your host platform) meaning cross compilation support in all the myriad combination falls out for free.

Ignoring the wrapper scripts which tie everything together, the package and dependency structure quite closesly matches the graph above, when we were just describing the components and their dependencies more conceptually.

I had long wanted to do the same thing for GCC too, so both compilers used at the heart of Nixpkgs were cleaned up, and a lot of other tech debt (in the native bootstraps, and cc-wrapper, for example) could finally be cleaned up. But I personally was low on time. After working on this over the years with many people --- in chronological order, @alexfmpe, @philiptaron, @cloudripper, and finally @RossComputerGuy --- I'm happy to report we have finally created a new "GCC NG" package set with PR #414299. This is the second occasion for writing this blog post. Fingers crossed 🤞, but the goal is now to migrate Nixpkgs to use this instead over the next release cycle or two, and get the patches upstream. @RossComputerGuy had the good idea of looking at gfortran as a test-case on staging, since that is far less widely used than GCC's C/C++ compiler --- stay tuned for that! I am focusing more on the patch-upstreaming side of things, and spent a little time chatting with the GCC devs on IRC resulting in https://gcc.gnu.org/pipermail/gcc-patches/2025-July/689429.html which is a new version of a patch I originally wrote 4 years ago.

GCC, as important as it is, is still only one compiler. Other compilers (GHC, rustc, Go, ...) will need the same treatment in Nixpkgs, but I am confident with the momentum from GCC we could soon do them also. For example, see these positive developments unrelated to Nixpkgs:

• This PR to GHC, in addition to making a simpler wrapper build system analogous to musl-cross-make, also improves the ability to build components separately; the latter we would reuse for a repackaging in the style of LLVM and "GCC NG".

• The Rust community is also looking at stabilizing the build-std feature in some fashion, which might possibly allow simplifying their python bootstrap glue code as the gap between compiler bootstrapping and users doing arbitrary cross compilation narrows.

Ultimately, I hope in the next few years, it could become common knowledge that building an (ideally multi-target) compiler separately from any runtime libraries is always best practice, with or without Nix.