Posted on August 24, 2017  
by Dmitry Bushev

# Stackage to Nix  
Generate Nix build instructions from a Stack file

[stackage2nix](https://github.com/4e6/stackage2nix) is a tool that generates Nix build instructions from Stack file. Just like `cabal2nix` but for Stack.

Here I would like to tell you a story of its creation, show some usage examples, and finally talk about problems solved.

### The story of one build  
In Typeable we use Stack for development. It solves a bunch of problems you face during a development of the large Haskell projects. But the main benefit you get is Stackage integration.

For the production builds we use Nix. It has some nice properties like declarative build configuration, reproducible builds, etc. If you, for some reason, was not aware of it, I encourage you to give it a try. Nixpkgs differs from other build tools in a variety of ways. I’d call its approach developers friendly. It may take some time to get into, but it’s worth it.

So, we got Stack on one side and Nix on the other. The question is, how would you build the Stack project with Nix?

### Nix buildStackProject  
Our first approach was defining Nix derivation using Stack as a builder. Nixpkgs already has [haskell.lib.buildStackProject](https://nixos.org/nixpkgs/manual/#how-to-build-a-haskell-project-using-stack) helper function. Unfortunately, this method has several downsides. The main one is that Nix does not have control over the Stack cache. In fact, we end up with the builds that failed quite frequently and required manual interventions. Usually, all the problems could be resolved by dropping the Stack cache, followed by the painfully slow compilation from scratch.

### Manual Stackage LTS  
The more _native_ way to build Haskell in Nix is to describe each Haskell package as a separate Nix derivation. This way we get the caching property and all Nix benefits out of the box. Nixpkgs repository already maintains some version of the Hackage snapshot predefined, but we need packages of particular versions from Stackage. So the logical outcome would be to create Stackage snapshot for Nixpkgs.

At the time when I got this idea, I did a quick search and there already was a discussion of such thing on cabal2nix issue [#212](https://github.com/NixOS/cabal2nix/issues/212), where Benno @bennofs mentioned his WIP implementation.

The second approach was to create an override for Nixpkgs manually using existing tool. The new Nixpkgs contained Stackage packages set instead of default Hackage snapshot, with additional overrides from `stack.yaml`. This solution worked better than the first one with bare Stack. The downside was that we got some extra Nix code to maintain in our project.

### stackage2nix  
After the creation of manual Nixpkgs override, it became apparent that the procedure could be automated. We just need to parse `stack.yaml` build definition, generate Nix Stackage packages set for particular LTS snapshot, and then apply the overrides from Stack file on top of it. Sounds manageable.

So, the third approach was to build the tool that would generate Nixpkgs override given the Stack build definition. That’s how we get to `stackage2nix`.

## Example Usage  
In the current implementation, `stackage2nix` has three required arguments.

```bash
stackage2nix \
  --lts-haskell "$LTS_HASKELL_REPO" \
  --all-cabal-hashes "$ALL_CABAL_HASHES_REPO" \
  ./stack.yaml
```

- `--lts-haskell` path to [fpco/lts-haskell](https://github.com/fpco/lts-haskell) repository  
- `--all-cabal-hashes` path to [commercialhaskell/all-cabal-hashes](https://github.com/commercialhaskell/all-cabal-hashes) repository checked out to `hackage` branch  
- path to `stack.yaml` file or directory containing it

Produced Nix derivation split into the following files:

- packages.nix - Base Stackage packages set  
- configuration-packages.nix - Compiler configuration  
- default.nix - Final Haskell packages set with all overrides applied

The result Haskell packages set defined the same way as in Nixpkgs:

```nix
callPackage <nixpkgs/pkgs/development/haskell-modules> {
  ghc = pkgs.haskell.compiler.ghc7103;
  compilerConfig = self: extends pkgOverrides (extends stackageConfig (stackagePackages self));
}
```

That means you can apply the same overrides as for default Haskell packages in Nixpkgs. As an example, the following snippet `release.nix` prepares project for release. It compiles all packages with `-O2` GHC flag and enables static linking for `stackage2nix` executable.

```nix
with import <nixpkgs> {};
with pkgs.haskell.lib;

let haskellPackages = import ./. {};
in haskellPackages.override {
  overrides = self: super: {
    mkDerivation = args: super.mkDerivation (args // {
      configureFlags = (args.configureFlags or []) ++ ["--ghc-option=-O2"];
    });

stackage2nix = disableSharedExecutables super.stackage2nix;
  };
}
```

The build is straightforward:

```bash
nix-build -A stackage2nix release.nix
```

For other examples you can check [4e6/stackage2nix-examples](https://github.com/4e6/stackage2nix-examples) repository. I created it during development, as a sandbox to verify `stackage2nix` by running it on different OSS projects.

## How it works  
Apparently, assembling things from parts is hard. And it’s not an exception in Haskell. In this final section, I’ll explain what `stackage2nix` does to produce the correct Nix build.

As a small step aside, I like to think about `stackage2nix` as a function that translates Stack build definition to Nix in an idempotent way. Once again, the inputs are:

- [fpco/lts-haskell](https://github.com/fpco/lts-haskell) repository with Stackage LTS snapshots.  
- [commercialhaskell/all-cabal-hashes](https://github.com/commercialhaskell/all-cabal-hashes) repository containing information about packages from Hackage.  
- Stack build definition `stack.yaml.`

Now, we got the inputs. First things first, parse `stack.yaml` file to obtain the configuration of the current build. And load appropriate LTS Stackage packages set from `fpco/lts-haskell`.

And here’s the first challenge. Every package on Hackage for a single version can have several revisions. Like here, [mtl-2.2.1](https://hackage.haskell.org/package/mtl-2.2.1/revisions/) has two variants with different constraints on the dependencies. That said, we would like to get the exact revision of the package that was used in Stackage LTS because otherwise, in the worst case we might not be able to resolve the correct dependencies, and the final build may not work. Luckily, LTS metadata contains the SHA1 hash of the package in `commercialhaskell/all-cabal-hashes` repo.

So far so good. First we try to load package by hash. But in reality, that might be the case that SHA1 hash is missing, or repository doesn’t contain an object with this hash. Then we fall back and try to load the latest revision of the package from `commercialhaskell/all-cabal-hashes` repo. But this could also fail because apparently, some files can be incomplete and missing its accompanying metadata. The real world is a rough place. Finally, we try to load the package from local Cabal database. Either the default one in `~/.cabal` directory, or can be overridden by `--hackage-db` flag database will be used.

Okay cool, we’ve loaded the packages. But then we got another problem. Stackage LTS packages set `fpco/lts-haskell` is a list of packages with their dependencies. It forms a graph with packages as vertices and dependencies as edges. The problem is this that this graph might have cycles, and when it does, Nix fails when tries to resolve target dependencies. Usually, cycles are caused by test dependencies, and we can break them by removing test dependencies from problematic packages. As a result in `configuration-packages.nix` you can see something like:

```nix
# break cycle: statistics monad-par mwc-random vector-algorithms
"mwc-random" = dontCheck super.mwc-random;
```

Okay, now we’ve got Stackage LTS packages for Nix. The final step is to apply package overrides from `stack.yaml` file. Remember the _revisions_ thing? Right, the new packages were never tested with the LTS snapshot. They add new constraints to the play that may break the integrity of Stackage LTS packages. The best thing we can do here is to bump revisions for their dependencies and rely on the fact that Stack solver checked them when the project was compiled with Stack tool.

So apparently, building Haskell is not quite trivial as it first seems. And `stackage2nix` makes its best attempt to construct something buildable.

Regarding the further development plans, I would like to focus on usability first. The project made its first release to the [Hackage](https://hackage.haskell.org/package/stackage2nix), have fun with it.

## Update  
A small follow-up to address the questions about the differences with [input-output-hk/stack2nix](https://github.com/input-output-hk/stack2nix). These two tools were started independently with the same intention - to translate Stack build configuration to Nix. But they followed different paths.

`stackage2nix` is focused on the creation of Stackage LTS packages set from [lts-x.y.yaml](https://github.com/fpco/lts-haskell) config, that can be used as a replacement for the default Haskell packages in Nixpkgs. Finally, it can override Stackage LTS with the extra-deps from `stack.yaml` Stack config file.

`stack2nix` produces only final derivation. Also, it relies on external tools in runtime. It utilizes _stack_ to obtain the build plan from `stack.yaml` Stack config, and _cabal2nix_ to generate derivation from it.

So, to conclude, both tools are used to produce the build derivation from the Stack file, but they use different approaches.