cargo-dyndrv: A Beginning
Building Rust within Nix has always been somewhat of an annoying problem. Nixpkgs includes tools like buildRustCrate to build a single crate and buildRustPackage to build an entire project with all its dependencies, but both have substantial issues: buildRustCrate requires manually specifying every dependency in the entire tree in Nix; while buildRustPackage compiles the entire dependency tree every time, making every code change take hours.
Existing programs can help. For example, crate2nix can convert an entire dependency tree into buildRustCrate calls, but it either requires slow import-from-derivation calls in Nix or a manual generation every time the crate graph is changed. Furthermore, it relies on Nixpkgs for its crate builds, which adds a small but noticeable overhead on every evaluation and crate build.
Obsidian Systems, supported by Saronic, decided to bridge this gap with a new program we call cargo-dyndrv, using the new "Dynamic Derivations" feature of Nix.
If you would like to try out cargo-dyndrv right now, the code is available on GitHub and we have prepared an example. You will need to enable the ca-derivations and dynamic-derivations experimental features and, until Nix 2.36 is released, run a recent git version of the Nix daemon.
artemis@koali ~> nix build github:obsidiansystems/cargo-dyndrv/ffmpeg-example
artemis@koali ~> result-ffmpeg-example/ffmpeg_example
libavcodec version 63.1.101, configured with --disable-static <clipped>
Dynamic Derivations
Traditionally, if one wanted to programmatically generate derivations, they had two options: generate Nix code before running the Nix interpreter, or generate Nix code within a derivation builder and import it from the result (so-called "Import from Derivation" or IFD).
Manual building is easy to forget and requires checked-in build artefacts for every program or library.
IFD requires a bit more context. Nix builds are divided into two stages: first the Nix language code is evaluated, and while doing so it calls the builtins.derivation code, often via a wrapper like stdenv.mkDerivation. Each of these calls adds the low-level derivation, including a name, list of inputs and outputs, environment variables, and a command to run. These derivations rely on each other, until at last the Nix code returns a single final derivation to build.
After the evaluation is done, the derivations are built (in a sandbox), one by one. These stages are designed to be entirely separate, so evaluation and building could happen on entirely separate machines. That is a core part of the design of remote builders in Nix and large CI systems like Hydra.
IFD breaks this assumption by interleaving evaluation and realisation stages, slowing down the process. It is also complex to implement in CI systems like Hydra, and requires the called program to use string interpolation to generate Nix code.
Dynamic Derivations flip all this on its head by removing the requirement for Nix language at all. Instead of outputting a regular file or directory that is passed to the Nix interpreter or the user, a derivation builder can output another derivation. Within the initial Nix invocation, it is possible to use the builtins.outputOf function to take the output of a generated derivation, then use that in another build. In the internal derivation file, this is expressed as multiple levels of "output" on a derivation in the input set. Since all these relations can be expressed at initial evaluation time, there is no need to ping-pong back and forth between evaluation and realisation.
For example, a derivation that works on the output of a generated derivation could look like:
let
inherit (pkgs) runCommand;
generated = runCommand "generated.drv" { } "<command that generates derivation with one output: out>";
in
runCommand "wrapping" {} ''
contentPath="${builtins.outputOf generated.outPath "out"}"
cat "$contentPath"
cp "$contentPath" "$out"
'';
Outputting Multiple Store Objects
Dynamic Derivations on their own have one big problem: It is only possible to create one derivation at a time. The generating derivation must declare its outputs up front, and what it declares is all it gets to create. If building an entire tree of dependencies, it would have to stuff them all into one derivation, losing all of the caching and distributed build benefits of Nix.
Technically, there was a workaround for this: recursive-nix. Derivation builders are granted access to the Nix daemon socket, and can use standard Nix commands or their own libraries to connect to the socket and add items to the store. This socket connects to a "restricted store" and will only acknowledge the existence of the builder's dependencies and objects it has added, somewhat reducing impurity.
Unfortunately, recursive-nix has no hope of stabilisation. It exposes a massive frequently-changing protocol to builders that includes many unnecessary features. It would be easy to accidentally implement derivations that only work with certain versions of the Nix daemon, and it's possible that some of the functions could present security vulnerabilities.
In order to fix this, Obsidian developed another Nix feature: builder-rpc-v0. Conceptually, it consists of two parts: a minimal Nix daemon interface to create store objects and the ability to "submit outputs", registering the path of one of the recently-created store objects to an output.
The minimal Nix daemon interface reuses the standard Nix socket protocol, but disables most features. The daemon rejects all commands over the socket except the few required to add items to the store. It also comes with two new commands: the convenience feature AddToStoreScanning which automatically scans for dependencies in the added object, as Nix would do if it were put in the output directory; and SubmitOutput, which we will talk more about in the next blog post, where it plays a special role.
When using either builder-rpc-v0 or recursive-nix, a derivation builder may create as many store objects as it likes by calling into the Nix daemon, and allow those to rely on each other and on the builder's dependencies. The final output may rely on any of these created store objects, so a derivation is allowed to return a tree of objects instead of a single object.
For example, in a traditional derivation the "out" variable is declared for the path of the output, and one could create a file with the following bash:
echo "hello, world" > $out
A derivation using builder-rpc-v0 could use the nix command to submit multiple store objects and link one to the output as follows
echo "hello, world" > dep
dep=$(nix store add --scan ./dep)
echo "this depends on $dep" > out
out="$(nix store add --scan ./out)
nix store submit-output "$out" out
These store objects need not be simple files. They can be entire directory trees, like from subsetting source code, or (importantly for us) derivations.
Unit Graph
Using this, cargo-dyndrv is able to create a derivation for every crate in the graph, then attach the final "root crates" to its outputs. However, that begs the question: How does cargo-dyndrv discover what and how to build in the first place?
Cargo build graphs can rely on thousands of crates, each with their own special build flags, environment variables, targets, and dependencies. Discovering the tree is not trivial: crates can have optional dependencies, and whether each one is enabled is an arbitrarily complex function of crate features, target architectures, and crate build types. It may be possible to reimplement the tree discovery, but it could quickly go out of date as Cargo introduces new features, requiring constant maintenance.
Luckily, Cargo includes an unstable feature called the unit graph to provide all the information we need. When run with the --unit-graph option, any Cargo command that builds (e.g. build, test, doc) will skip the build stage, and instead dump a graph of every build it would perform, along with its build settings and dependencies. This graph, combined with crate metadata from the cargo metadata command, provides all information required to create rustc calls.
What's Inside?
Consider a simple Rust crate with one dependency and a build script, being built for a different platform in debug mode. The unit graph, generated with the command:
cargo build --unit-graph --target aarch64-unknown-linux-gnu | jq
looks like this:
{
"version": 1,
"units": [
{
"pkg_id": "path+file:///tmp/hello2#0.1.0",
"target": {
"kind": [
"bin"
],
"crate_types": [
"bin"
],
"name": "hello2",
"src_path": "/tmp/hello2/src/main.rs",
"edition": "2024",
"doc": true,
"doctest": false,
"test": true
},
"profile": {
"name": "dev",
"opt_level": "0",
"lto": "false",
"codegen_backend": null,
"codegen_units": null,
"debuginfo": 2,
"split_debuginfo": null,
"debug_assertions": true,
"overflow_checks": true,
"rpath": false,
"incremental": true,
"panic": "unwind",
"strip": {
"deferred": "None"
}
},
"platform": "aarch64-unknown-linux-gnu",
"mode": "build",
"features": [],
"dependencies": [
{
"index": 1,
"extern_crate_name": "build_script_build",
"public": false,
"noprelude": false,
"nounused": false
},
{
"index": 3,
"extern_crate_name": "json",
"public": false,
"noprelude": false,
"nounused": false
}
]
},
{
"pkg_id": "path+file:///tmp/hello2#0.1.0",
"target": {
"kind": [
"custom-build"
],
"crate_types": [
"bin"
],
"name": "build-script-build",
"src_path": "/tmp/hello2/build.rs",
"edition": "2024",
"doc": false,
"doctest": false,
"test": false
},
"profile": {
"name": "dev",
"opt_level": "0",
"lto": "false",
"codegen_backend": null,
"codegen_units": null,
"debuginfo": 2,
"split_debuginfo": null,
"debug_assertions": true,
"overflow_checks": false,
"rpath": false,
"incremental": false,
"panic": "unwind",
"strip": {
"deferred": "None"
}
},
"platform": "aarch64-unknown-linux-gnu",
"mode": "run-custom-build",
"features": [],
"dependencies": [
{
"index": 2,
"extern_crate_name": "build_script_build",
"public": false,
"noprelude": false,
"nounused": false
}
]
},
{
"pkg_id": "path+file:///tmp/hello2#0.1.0",
"target": {
"kind": [
"custom-build"
],
"crate_types": [
"bin"
],
"name": "build-script-build",
"src_path": "/tmp/hello2/build.rs",
"edition": "2024",
"doc": false,
"doctest": false,
"test": false
},
"profile": {
"name": "dev",
"opt_level": "0",
"lto": "false",
"codegen_backend": null,
"codegen_units": null,
"debuginfo": 2,
"split_debuginfo": null,
"debug_assertions": true,
"overflow_checks": true,
"rpath": false,
"incremental": true,
"panic": "unwind",
"strip": {
"deferred": "None"
}
},
"platform": null,
"mode": "build",
"features": [],
"dependencies": []
},
{
"pkg_id": "registry+https://github.com/rust-lang/crates.io-index#json@0.12.4",
"target": {
"kind": [
"lib"
],
"crate_types": [
"lib"
],
"name": "json",
"src_path": "/home/artemis/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/json-0.12.4/src/lib.rs",
"edition": "2018",
"doc": true,
"doctest": true,
"test": true
},
"profile": {
"name": "dev",
"opt_level": "0",
"lto": "false",
"codegen_backend": null,
"codegen_units": null,
"debuginfo": 2,
"split_debuginfo": null,
"debug_assertions": true,
"overflow_checks": true,
"rpath": false,
"incremental": false,
"panic": "unwind",
"strip": {
"deferred": "None"
}
},
"platform": "aarch64-unknown-linux-gnu",
"mode": "build",
"features": [],
"dependencies": []
}
],
"roots": [
0
]
}
Although the overall text is long, it is very clear. Each build (the dependency, building the build.rs, running the build.rs, and the main crate) is given separate items describing the main file and many of the flags that must be passed.
While the unit graph feature is unstable and may change, there is no indication it is going away. It displaced the now-removed --build-plan option to provide a higher-level access that is less likely to require changes as Cargo's architecture changes. While ordinarily using the flag would require an unstable version of Cargo, it is also possible to use the feature in the ordinary cargo package in Nixpkgs due to its configuration options.
The rustc command-line arguments and Cargo environment variables we require are all stable and documented, since both are regularly used by external programs: the former in non-Cargo build systems including Bazel, and the latter in build scripts and crates accessing them.
One of the most convenient parts of the unit graph is how easy it makes cross compiling. Procedural macros (also called "proc macros"), used to perform complex code generation in crates like serde; and build scripts (i.e. build.rs files), used to link to system libraries and compile/generate custom code, and all their dependencies, must be compiled for the building system even when the final binary will run on another architecture. The unit graph explicitly sets the target of every item, meaning cargo-dyndrv does not even have to reimplement the algorithms cargo uses to determine target platforms.
Build Scripts
Build scripts must also be built for the building system, but they then must be executed with complex sets of environment variables describing the enabled features of the crate, a description of the target system, and flags set by previous build scripts. Then, the build script will generate files and output new flags and environment variables to add to future invocations of rustc.
Since cargo-dyndrv doesn't have access to the output of build scripts (the derivation generation stage finishes well before build scripts are executed), we wrote several small, extremely simple wrapper scripts. None require any dependencies, meaning they can be built with Nixpkgs buildRustCrate, are easy to maintain, and are extremely fast.
First, we have target-env. It runs the Rust commands rustc --print=cfg and rustc --print=host-tuple to discover information about the building and target systems, then converts them into environment variables and saves them into a file. This file is then passed into each build script and read. Although these could be run in the initial cargo-dyndrv derivation generation stage, we decided to put them in a separate derivation to maximise the amount of work that could be cached by Nix.
Next, we have build-wrap. It executes build scripts after they have been built, listening for all the build script flags they output and saving them into files for future derivations.
Finally, we have env-wrap. It reads environment variables from a set of files, applies them, then executes the process — generally rustc or build-wrap. We searched for existing programs that could do that, but surprisingly none are built-in on Linux systems: only the env program that reads environment variables from arguments.
The only other utility binary cargo-dyndrv requires is the standard Unix tool ln, in order to create symlinks. Due to strict requirements on output derivation names in dynamic derivations, cargo-dyndrv creates an intermediate derivation with the correct name, then symlinks the output of the incorrectly-named root derivation in.
You may notice that there is no binary to add arguments to cargo. This is actually not required, since rustc, along with many other compilers, supports loading arguments from a file with an @ sign prefixed before a filename. cargo-dyndrv uses this to load all arguments created by build scripts from the outputs of the derivations that run them. In fact, it would be possible to entirely remove env-wrap if the proposed --env-set option in rustc had been stabilised, but it was eventually removed.
External Dependencies
While the derivation generation we have discussed so far is sufficient for crates with complex build processes, it leaves out one major feature: external dependencies.
Many projects require reading from databases, compiling C code, or linking to non-Rust libraries like OpenSSL or FFmpeg. We decided to handle this in cargo-dyndrv by passing in a file named extern.json with a mapping from crate ID to environment variables, path entries, and extra inputs to the build script derivation.
For example, when building ffmpeg-next our extern.json looked like this:
{
"registry+https://github.com/rust-lang/crates.io-index#clang-sys@1.8.1": {
"inputs": [
"/nix/store/yd8la9pq2znpq6wxi5hfg18i90d8dlzi-clang-21.1.8-lib"
],
"env": {
"LIBCLANG_PATH": "/nix/store/yd8la9pq2znpq6wxi5hfg18i90d8dlzi-clang-21.1.8-lib/lib"
}
},
"registry+https://github.com/rust-lang/crates.io-index#ffmpeg-sys-next@9.0.0": {
"inputs": [
"/nix/store/zi7hs9vvxgx6ld1lyyk6szswdy39xgmp-ffmpeg-9.0-dev",
"/nix/store/whfndkdx65fm92p1xj438zmpgrhk5khq-pkg-config-wrapper-0.29.2",
"/nix/store/20rbjvhw43h4p0z8db4iimbvv1h5bh2s-glibc-2.42-67-dev",
"/nix/store/yd8la9pq2znpq6wxi5hfg18i90d8dlzi-clang-21.1.8-lib"
],
"env": {
"PKG_CONFIG_PATH": "/nix/store/zi7hs9vvxgx6ld1lyyk6szswdy39xgmp-ffmpeg-9.0-dev/lib/pkgconfig",
"PKG_CONFIG": "pkg-config",
"BINDGEN_EXTRA_CLANG_ARGS": "-I/nix/store/20rbjvhw43h4p0z8db4iimbvv1h5bh2s-glibc-2.42-67-dev/include",
"LIBCLANG_PATH": "/nix/store/yd8la9pq2znpq6wxi5hfg18i90d8dlzi-clang-21.1.8-lib/lib"
},
"path": [
"/nix/store/whfndkdx65fm92p1xj438zmpgrhk5khq-pkg-config-wrapper-0.29.2/bin"
]
}
}
This is an incredibly low-level primitive. Most concerning is the inputs field, that directly lists all store paths specified in env and path. It is required because derivations are so low-level: while the Nix evaluator passes around all of the derivations a string refers to as the so-called "context" and can automatically include them as input when creating a derivation, cargo-dyndrv has no such luxury. The program must manually list all inputs itself.
However, this does not mean users of cargo-dyndrv must list all inputs by themselves. Nix provides two features to determine the context of a string: One may use builtins.getContext from within Nix code and immediately receive a list of derivations and outputs as an attribute set, or pass the string in as an input to a derivation with the exportReferencesGraph function and receive the full tree of dependencies as JSON within the derivation.
We decided to use the latter option because it was simpler to implement and moved more work into the parallelizable builders, but either could work for our purposes.
Users of cargo-dyndrv can use our writeExtern function directly, or via the buildDynamicCrate function designed to work similarly to the existing Nixpkgs buildRustPackage function. In both cases, users provide an attrset with the extra path and environment, but leave out the inputs. For example, one can generate the above extern.json with this:
pkgs.writeExtern {
"registry+https://github.com/rust-lang/crates.io-index#clang-sys@1.8.1" = {
env.LIBCLANG_PATH = "${lib.getLib pkgs.buildPackages.libclang}/lib";
};
"registry+https://github.com/rust-lang/crates.io-index#ffmpeg-sys-next@9.0.0" = {
path = [ "${lib.getBin pkgs.buildPackages.pkg-config}/bin" ];
env = {
PKG_CONFIG_PATH = "${lib.getDev pkgs.ffmpeg-headless}/lib/pkgconfig";
PKG_CONFIG = lib.getExe pkgs.buildPackages.pkg-config;
BINDGEN_EXTRA_CLANG_ARGS = "-I${pkgs.stdenv.cc.libc.dev}/include";
LIBCLANG_PATH = "${lib.getLib pkgs.buildPackages.libclang}/lib";
};
};
};
The writeExtern function splits the attribute set by package ID, and for each creates a JSON file in the store and passes it to a derivation. That derivation uses exportReferencesGraph and jq to create a new JSON file with the generated inputs field in addition to the existing env and path fields. Finally, an output derivation takes all of the per-extern derivations and merges them with jq, in order to pass them into cargo-dyndrv.
Future Work
Cargo still has one major advantage over cargo-dyndrv: It can start building libraries after only the metadata of their dependencies are built.
When rustc builds a crate, it outputs two files: a .rmeta metadata file containing a list of functions and values and a .rlib binary containing the generated machine code. The former is produced before the time-consuming code generation stage, and contains all the information required to build dependent libraries.
Cargo uses this to start building libraries before all their dependencies are finished. This sounds like it would be impossible in Nix: derivations have always been atomic build steps that either succeed or fail, while Nix carefully ensures that nothing can observe them while they are in an incomplete state. However, we are working on a solution that achieves this without violating the strict purity of Nix, and we hope to share it soon.
Saronic is a heavy user of Rust and Nix. Like all heavy users of both, they faced a tradeoff: fast eval times, or build reuse from building crates in their own derivations.
Saronic understood this early, when dynamic derivations were still in development. They saw how the feature could resolve that tradeoff and allow fine-grained build plans without an eval-time penalty, revolutionizing software development with Nix.
Now, as dynamic derivations reach maturity, they are joining us in building out the integrations to prove it.