Component-based build support for cabal2nix

[TeofilC]

I would love to have component-based build support for the nixpkgs haskell infra.
I understand that adding support for this is quite a big change. And I'm hoping to invest some time into implementing it.

Since this is quite a big addition I'm hoping to split it up into some smaller tasks.
The things I can think of are:

• Add support to cabal2nix to just emit one component
• Add support to the nixpkgs builder to build just one component. (I think some of this already exists but haven't tested it)
• Add support for the nixpkgs builder to put tests into passthru.tests rather than have them run as part of a library's checkPhase
• Add support to cabal2nix to emit all components as different .nix files.
• Allow splitting out the building of Setup.hs into a separate shared derivation.

I'm thinking of keeping all of these things opt-in to avoid breakage especially while support is only partial.

How does this sound as a vague plan?

[maralorn]

This sounds like a pretty big idea. I am generally very much in favor of improvements to cabal2nix, but the constraints of living in nixpkgs are tough.

Before you take on such a big undertaking, I wonder what use cases you are envisioning and if this approach actually helps.

[TeofilC]

That's a really good point!

Currently I'm switching my work CI over from using haskell.nix to the nixpkgs Haskell infra. The main thing I'm missing is component based builds.

While this isn't blocking me from switching, I think it would be good to try to implement this to make things work better (faster builds, failing builds/tests blocking less of the graph).

Since this is quite a big thing, I hope to whittle away at it when I get some time to improve CI.

I'm also keeping in mind that the maintainers have limited time, so I want to approach this in whatever way y'all think is best, including dropping this if y'all think it's too much complexity.

[maralorn]

If we can find a good design for the features in question, I think it would be worth the effort. At least from my personal maintainer perspective.

I wonder if it’d make sense to have a call about this, because it is really not clear where we want to go with this.

• The technically hardest part is probably using the outputs of one component as input to another component.
• One question I have: Can we trim inputs for certain component builds in a way so that we can get at least sometimes recompilation avoidance. (Mainly thinking about fixing tests without recompiling the lib.) This is not a must, but would be super cool.
• The biggest question is: Do we generally want to make a package usable without depending on its test. The general design in nixpkgs is to not do it, I think? I am not aware of any language builder which separates tests into an extra derivation. The danger is very large, that we simply don’t run tests if we can get away with not doing it.
  On the other hand, making the hot path for building shorter sounds really intriguing.

[TeofilC]

A call definitely sounds like a good shout.

One question I have: Can we trim inputs for certain component builds in a way so that we can get at least sometimes recompilation avoidance. (Mainly thinking about fixing tests without recompiling the lib.) This is not a must, but would be super cool.

I would definitely like to have this as well!

In terms of the other points. I think you are right that these are going to be tricky. I think it should be doable to ensure that tests get run. One possibility is that the derivation that we make available to users is a package level summary of the components and so includes checks that the test suite and the test suites of dependencies have run.

[sternenseemann]

I think solving #539 will give us a better sense of the problem space of components, since we'd have to deal with them more explicitly in haskellPackages.mkDerivation and cabal2nix in order to solve it—without splitting anything up at that point.

Then, for me, the logical next step would be to work on having haskellPackages.mkDerivation or rather a mkDerivation-ng support per-component builds in some way. This would be a relatively easy first step and would give us a better sense of what would need to change in cabal2nix and hackage2nix.

One difficulty seems to indeed be how to depend on specific components of another package without breaking splicing, i.e. support for cross-compilation.

The biggest question is: Do we generally want to make a package usable without depending on its test. The general design in nixpkgs is to not do it, I think? I am not aware of any language builder which separates tests into an extra derivation. The danger is very large, that we simply don’t run tests if we can get away with not doing it.

It seems to me that this is a big concern. If test suites can't fail the derivation, they become pointless. It seems to me that we need wrapping derivations

• around each component, that depends on the tests,
• around all components, that depends on the tests and re-instates the package unity (e.g. a pandoc wrapping derivation that unites pandoc:lib and pandoc:exe:* again).

These wrapping derivations would be fairly trivial to build, ideally just symlinking a bunch of stuff and ensuring tests succeeded.

There'd be an explosion in the number of derivations definitely. Another question is how to go about tests: Should the test component derivation directly run it? Probably, to avoid it having to be executed multiple times…