Working with nix-shell
Managing runtime dependencies using Nix
We’re setting up a new project at work, and wanted to apply some lessons learned from previous projects. One of the things we ran into was that when onboarding new people, or setting up a new development environment on a different machine, the dependencies were sometimes out of sync. Usually this was quite easy to fix, but keeping all the versions in sync, and dealing with deprecated or not working features (e.g terraform or kustomize) quickly becomes very annoying.
Recently we ran into another issues where a newer version of java caused issues, which led to a new version of gradle, which meant updating our build files etc. While not an issue in itself, and something that we had to do eventually, but we didn’t really planned to do this at that moment.
So looking around a bit we decided to look deeper into nix, and more specifically into nix-shell. This will allow us to create (and commit to git) a basic shell that will install all the dependencies we need, so everybody has the same starting point. Onboarding a new developer will then be just as easy as running:
➜ nix-shell dev-env.nix
➜ { dev } nix
Note that in the previous shell, we’ve used the zsh nix shell extension, which make sure nix and zsh play nicely together. If we’re in a nix-shell I add the { dev } prefix to indicate this
The biggest part of this setup is inspired on the repo from Gabriel Volpe where he explains how to use it for sbt. So for more ideas and some more advanced usages check out that one as well.
Basic setup
So what do you need to do to get this working. The first thing to do is install nix itself. This can be easily done by following the instructions from here: https://nixos.org/download.html. Once installed you can use it using the nix command:
$ nix --version
nix (Nix) 2.3.12
There is a whole lot learning material out there on how to use nix, also for setup and managing of your own environment, but that’s a bit out of scope for this article on nix-shell. Just a quick couple of commands to show you how interesting the approach of nix is. Whenever you install a package (or set of packages) a new state (a generation in nix terms) is created. So when we start we just have a pretty empty profile:
$ nix-env --list-generations | cat
1 2021-06-16 15:49:02
2 2021-06-30 15:56:56
3 2021-06-30 15:56:56 (current)
So we’ve got a couple of generations, when you install it you should see something similar:
$ nix-env --query "*" | cat
nix-2.3.13
nss-cacert-3.66
If we install a package, we’ll get a new generation, and the commands will become available. You can also remove generations, important note here, don’t remove the generations created by the install of nix. If you do this, you’ll run into strange behavior.
For instance lets say we want to install a specific version of java. Currently I’ve got a system wide one installed (outside of nix):
$ java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
So let’s see what kind of versions are provided by nix, and install one. The following command will search (using a regex) through all the package descriptions that match jdk.
$ nix-env -qaP --description '.*jdk.*' | cat
nixpkgs.adoptopenjdk-bin adoptopenjdk-hotspot-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-hotspot-bin-13 adoptopenjdk-hotspot-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-hotspot-bin-14 adoptopenjdk-hotspot-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-hotspot-bin-15 adoptopenjdk-hotspot-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-hotspot-bin-16 adoptopenjdk-hotspot-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-hotspot-bin-8 adoptopenjdk-hotspot-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-icedtea-web adoptopenjdk-icedtea-web-1.8.6 Java web browser plugin and an implementation of Java Web Start
nixpkgs.icedtea8_web adoptopenjdk-icedtea-web-1.8.6 Java web browser plugin and an implementation of Java Web Start
nixpkgs.icedtea_web adoptopenjdk-icedtea-web-1.8.6 Java web browser plugin and an implementation of Java Web Start
nixpkgs.adoptopenjdk-jre-bin adoptopenjdk-jre-hotspot-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-hotspot-bin-13 adoptopenjdk-jre-hotspot-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-hotspot-bin-14 adoptopenjdk-jre-hotspot-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-hotspot-bin-15 adoptopenjdk-jre-hotspot-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-hotspot-bin-16 adoptopenjdk-jre-hotspot-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-hotspot-bin-8 adoptopenjdk-jre-hotspot-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-11 adoptopenjdk-jre-openj9-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-13 adoptopenjdk-jre-openj9-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-14 adoptopenjdk-jre-openj9-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-15 adoptopenjdk-jre-openj9-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-16 adoptopenjdk-jre-openj9-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-8 adoptopenjdk-jre-openj9-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-11 adoptopenjdk-openj9-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-13 adoptopenjdk-openj9-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-14 adoptopenjdk-openj9-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-15 adoptopenjdk-openj9-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-16 adoptopenjdk-openj9-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-8 adoptopenjdk-openj9-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.jetbrains.jdk jetbrains-jdk-11.0.10-b1427 An OpenJDK fork to better support Jetbrains's products.
nixpkgs.oraclejdk11 oraclejdk-11.0.10
nixpkgs.oraclejdk14 oraclejdk-14.0.2
nixpkgs.oraclejdk8 oraclejdk-8u281
nixpkgs.oraclejdk oraclejdk-8u281
nixpkgs.jdk11 zulu11.48.21-ca-jdk-11.0.11 The open-source Java Development Kit
nixpkgs.jdk zulu16.30.15-ca-jdk-16.0.1 The open-source Java Development Kit
nixpkgs.jre_minimal zulu16.30.15-ca-jdk-16.0.1-minimal-jre
nixpkgs.jre8 zulu8.54.0.21-ca-jdk-8.0.292 The open-source Java Development Kit
nixpkgs.jdk8 zulu8.54.0.21-ca-jdk-8.0.292 The open-source Java Development Kit
Here we see the package name, which you use to install the package, a human readable name, and some description. You can also search using the nix command tool. This provides pretty much the same results, but adds a bit more human readable output.
$ nix search jdk | cat
* nixpkgs.adoptopenjdk-bin (adoptopenjdk-hotspot-bin)
AdoptOpenJDK, prebuilt OpenJDK binary
* nixpkgs.adoptopenjdk-hotspot-bin-11 (adoptopenjdk-hotspot-bin)
AdoptOpenJDK, prebuilt OpenJDK binary
* nixpkgs.adoptopenjdk-hotspot-bin-13 (adoptopenjdk-hotspot-bin)
AdoptOpenJDK, prebuilt OpenJDK binary
And finally you can also search online (https://search.nixos.org/packages), but it is always good to at least know some of the command line tools you can use. So from the list above lets say that currently we’re interested in installing jdk-15, just because we can.
$ nix-env -i adoptopenjdk-hotspot-bin-15.0.2
This will kick off the installation of this package, and any dependencies it might have. Once done we’ll have an additional generation:
$ nix-env --list-generations
1 2021-06-16 15:49:02
2 2021-06-30 15:56:56
3 2021-06-30 15:56:56
4 2021-06-30 16:01:57 (current)
We can see what we installed:
$ nix-env --query "*" | cat
adoptopenjdk-hotspot-bin-15.0.2
nix-2.3.13
nss-cacert-3.66
And we’ve got the expected version of java:
$ java -version
openjdk version "15.0.2" 2021-01-19
But, wait, maybe you made an error and didn’t really want to use openjdk-15. Then you can just do a rollback, and you go back to the exact state your system was in before you installed the new libraries and tools that led to this version:
$ nix-env --rollback
switching from generation 4 to 3
$ java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
And if you want to switch back to generation 4, you can also do that:
$ nix-env --switch-generation 4
$ java -version
openjdk version "15.0.2" 2021-01-19
Easy right? So you can very easily setup your environment, experiment with new packages and tools, without having to worry that you might break your system.
nix-shell
In the previous setup we just showed you how to setup your own environment, your own shell. Often, though, when you start a new project, or switch to some legacy repository you need to work on, you require specific tools and versions to be able to work with that setup. It might need an older version of gradle, or hasn’t been updated to the latest java version, or any of the other hundreds of reasons why specific versions are needed. For this nix comes with nix-shell. With nix-shell you can parse a configuration file written in nix (https://nixos.org/guides/nix-pills/basics-of-language.html), and get a new shell that has installed exactly what was defined in the script. So when you start something new, or someone needs to be onboarded, all they have to do to get started (without messing up their laptops and running into conflicts) is install nix, and run nix-shell env.nix, where env-nix is the configuration of your environment.
To set this up, we define a script written in nix, that defines the packages (and other stuff) that we need to have in our environment (once again, look at the stuff from https://github.com/gvolpe/sbt-nix.g8 where most of this is based on). We’ll first look at what happens when we run this script, and then what’s in it.
$ nix-shell dev-env.nix
unpacking 'https://github.com/NixOS/nixpkgs/archive/189a13688782.tar.gz'...
these paths will be fetched (462.10 MiB download, 1343.75 MiB unpacked):
/nix/store/0w0hm71xblin82k00mlb7b8dsnf9qxl6-hook
/nix/store/2s9yf506ma88n1flxnv10swv6n5klfzf-zulu16.30.15-ca-jdk-16.0.1
/nix/store/34r722s1g3wx73v6prmx0djaq0gw95p3-jq-1.6-dev
/nix/store/3q5c3nvgmd5h0s7h1gks30b009g68ki1-terraform-1.0.0
/nix/store/4gkpfv7x4qdchwfalsb8hh2q62x9l6nl-kustomize-4.1.3
/nix/store/5im8ywgl0cilxlysvrwdnhik954nivbm-clang-7.1.0
/nix/store/64d69jqbz4s8ziqbpam41sd70w338ars-libcxx-7.1.0
...
What happens here is that all the dependencies we’ve defined in the dev-env.nix script are downloaded, and installed for this local shell. They won’t interfere with anything else you’ve got running or if you’re using nix standalone as well. In our case we installed specific versions (which are installed the first time you run this) of java, jq, terraform, gradlew and kustomize.
Once this is done running, we’ve got a shell, where these specific versions are available, without interfering with the rest of the OS and any packages installed globally. To set this up we use a number of different files. The first one we’ll look at is the dev-env.nix file:
{ jdk ? "jdk16" }:
let
# get a normalized set of packages, from which
# we will install all the needed dependencies
pkgs = import ./pkgs.nix { inherit jdk; };
in
pkgs.mkShell {
buildInputs = [
pkgs.${jdk}
pkgs.gradle
pkgs.jq
pkgs.kubectl
pkgs.kustomize
pkgs.terraform_1_0
];
shellHook = ''
export NIX_ENV=dev
'';
}
Here we specify with the mkShell function which packages we want to have available. At the top is an argument to this shell, so we can override the JDK version should we want to do that. If we don’t want this, then we’ll get jdk16. The goal is to have a reproducible environment, so we need a way to fix the packages. In nix we can do this by pointing to a specific git commit of where we want to get the package list from. For this we need to look at the pkgs.nix file next:
{ jdk }:
let
pinned = import ./pinned.nix;
config = import ./config.nix { inherit jdk; };
# if we want to configure a new version of terraform which isn't available
# yet, we could use an overlay. eg.
# overlays = [
# (import ./terraform.nix)
#];
# pkgs = import pinned.nixpkgs { inherit config; inherit overlays;};
pkgs = import pinned.nixpkgs { inherit config;};
in
pkgs
Here we configure the pkgs variable, which will contain all the packages and settings which are available to install from. As you can see from this short script, we define a pkgs variable which is based on the information from pinned. Which in itself is read from pinned.nix. Before we look at the specific config.nix we’ll first look at the pinned.nix file:
{
# We fix to a specific nixos version.
nixpkgs = fetchTarball {
name = "nixos-unstable-2021-06-172";
url = "https://github.com/NixOS/nixpkgs/archive/189a13688782.tar.gz";
sha256 = "09a027w36x05c8m8rwa7lr2g4sc12hx502xbkxhrpa3vmcryrc51";
};
}
What this means is that we define the nixpkgs variable to a specific archive, which can be downloaded from that location. This is just the total list of packages available, including version information, that was available at that time. Since everyone will use this same version, we can assure that everyone gets the same dependencies. There are different initiatives that aim in making working with specific versions easier:
- niv: Niv stores the specific versions in a separate json file, and seems like a good approach.
- nix-dev: Provides additional information and resources on how to use nix for development.
Now we have a look at the config.nix (where we pass in the jdk as a parameter):
{ jdk }:
{
packageOverrides = p: {
gradle = (p.gradleGen.override {
java = p.${jdk};
}).gradle_latest;
};
}
Here we define the packageOverride, where we fix the java version of gradle to the version that we want. So when we install pkgs.gradle, this configuration will be used. If we got more packageOverrides we can add them here for our specific configurations. Note here that packageOverrides can only be set once, and don’t compose like overlays do. But for this example, it’s an easy way to configure gradle.
As an example of how this would work in an overlay, we can define one for a custom version of terraform like this:
# Example of an overlay, where we replace the default from nix, with
# a new description.
final: prev: {
terraform = prev.terraform_1_0.overrideAttrs (old: rec {
version = "1.0.0";
name = "terraform-${version}";
src = prev.fetchFromGitHub {
owner = "hashicorp";
repo = "terraform";
rev = "v${version}";
sha256 = "sha256-ddcT/I2Qn1pKFyhXgh+CcD3fSv2steSNmjyyiS2SE/o=";
};
});
}
And in the pkgs.nix you can see how we can also add overlays. And as you can see, overlays is provided as an array, while the packageOverrides is a single argument to nixpkgs.
In the end, with this configuration we’ve got a nice basic setup to fix development (or build) tools to specific versions, which can easily be shared in git. Using this will allow new people to be quickly onboarded, without having to find and install specific versions, which might conflict with already installed tools.
wrap up
What was shown in this article is a very small set of what nix is capable of. You can easily create specific environments for docker, for building, or for different development environments. And while in this example we used nix-shell to setup a simple development environment, you can do much more. You can for instance use it to create a default home shell on your system, which you can quickly provision whenever you switch machines, or need to setup a new system: Nix Home Manager
A couple of other nice articles on this:
- https://ghedam.at/15978/an-introduction-to-nix-shell
- https://myme.no/posts/2020-01-26-nixos-for-development.html