Nix shell

[alecandido]

These are just a bunch of decoupled files to allow me to have a nix shell defined for PineAPPL.

My current goal with Nix is to have a fully reproducible environment for development (and it is fully reproducible because I'm committing it), and at the same time one I can tune for different needs (e.g. it is an easy way to test multiple Python versions, and maybe even Rust ones).

Eventually, I might even propose to package PineAPPL for Nix, within the same files. But this is not what I'm currently proposing, since the current files only support the dev shell.

FYI:

• flake.nix is the main file, describing the Nix properties (it is a way to declare some attributes, including the shell)
• flake.lock is the corresponding lock file
• .envrc is there to automatically enable the environment shell
• .gitignore is updated to avoid storing the local environment
• (everything else should not be updated, most likely I just have to rebase)

[cschwan]

I'm a bit worried about adding more files to the top directory, but nevermind that. Before merging I'd like to test Nix myself.

Are the build failures in commit 04fc920 unrelated? I've seen them before locally, but I don't know exactly what happens.

[alecandido]

I believe it should be because you updated something in master, and I'm based on an old version of master (I will rebase as soon as I have a stable connection).

The workflows should be completely unaffected by these files.

Btw, Nix is an optimal tool for workflows as well, because we can reproduce perfectly what is happening in the remote machine (but I never wrote yet my first Nix workflow, and the current ones are working, so it is not at all a priority).
Up to action usage, of course, but with Nix you can limit the actions used (e.g. no need for actions to install software, since it can all be installed by Nix packages, and cached as well, since they are all marked by hashes).

[alecandido]

Ok, I just rebased on master (it was pretty much outdated), and I believe at this point there is no reason to fail any longer.

Nix is coming as a whole Linux distribution (NixOS, managing even your system configurations), but it is a build tool in the first place (just Nix) and a distribution of packages built with it (Nixpkgs), that guarantee reproducibility.

On top of Nix, I'm also using devenv, that is kind of a library to write Nix development shells + a CLI to manage them https://devenv.sh/ (and in particular I'm using it with flakes, https://devenv.sh/guides/using-with-flakes/, that is the new "flavor" of Nix, which adoption is debated, and surveys pointed slightly more than half the people answering using flakes).
Development shells are already a Nix feature on their own. Devenv is just for extra convenience.
(this is for @cschwan, since you said you want to try yourself - just trying to ease your way in)

[alecandido]

I'm stupid, and I forgot to switch to master when merging #247. I will fix it locally and push.

[alecandido]

Ok, solving the aftermath #247 has been effectively rebased and merged in master (in the sense that I did it locally), and this one is minimal again, and based on current master.

Sorry for the mess.

[alecandido]

@cschwan I managed to have a consistent configuration, where the Python package is working as well.

I thought I would have had to patch the LHAPDF distribution, but people contributing to Nix are making a great job at providing valuable and updated packages (they were up-to-date even with unmerged patches, so the Nix package was less broken than the official one).
Interestingly enough, there is also a reproducible (hashed) distribution of the PDF sets themselves.
https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/physics/lhapdf/pdf_sets.nix

I'm quite convinced this could be useful, since managing a multilanguage environment could be non-trivial. And it is also handling locally and consistently the LHAPDF dependency.
Are you still worried by top-level files? Otherwise, could we merge?

Instructions to test

You don't need a whole NixOS to test, Nix is enough. It is available as a package on major distributions:
https://repology.org/project/nix/versions

Once installed, test it with nix-info.
Old-style commands begin with nix-, while there is an experimental (but very much used) interface with nix <subcommand>. Flakes themselves are marked as experimental (they have widespread adoption, half the community loves and use them apparently, but the other half hates - most likely that's the reason why they are still experimental).

To test, I would use:

nix develop . --impure

Most likely[*], you should add --extra-experimental-features nix-command --extra-experimental-features nix-command flakes (the old-style equivalent would be nix-shell, but I believe it requires a shell.nix file, and I didn't bother making it compatible).

(If you use direnv, it is integrated with Nix, and the shell will be enabled by changing the current directory within the pineappl repository. Not required.)

[*]: i.e. unless you or your distribution already configured to use experimental features

[alecandido]

I wanted to add target/prefix/bin to the shell path. But I need to extend the one defined by the shell, to keep access to all the other binaries provided by the Nix packages.

However, it is definitely possible, since the Python environment generated is working that way. But I'm not handling it manually, so I'm trying to understand how it works in
https://github.com/cachix/devenv/blob/main/src/modules/languages/python.nix

The drawback is that, for the time being, the installed pineappl CLI won't be on your PATH by default.

[cschwan]

Are you still worried by top-level files? Otherwise, could we merge?

No, especially now that we document what all the files do in this repositories. Can you document them in https://github.com/NNPDF/pineappl/blob/master/docs/maintainers-guide.md?

It will take some time to review this, but after Turkey I'm confident I can.

[alecandido]

I fixed the $PATH issue, there was a simple solution.

No, especially now that we document what all the files do in this repositories. Can you document them in master/docs/maintainers-guide.md?

I didn't notice it, it's definitely a good idea. It takes me nothing to write a few lines to document them (I will link the relevant references to avoid verbose descriptions).

It will take some time to review this, but after Turkey I'm confident I can.

That would be great!

There is no special hurry for this, I made it in the first place because it was useful for myself. But now that is there, I believe it could be useful even for other people, so it would be a (tiny) pity to forget it forever.

[alecandido]

The description is slightly beyond what is currently provided, since I have no output.packages in the flake (yet).

However, that is perfectly possible, and I'm also going to do it (sooner or later).
In principle, we could use Nix to build packages even in the CI (those for which we're distributing binaries). But dynamically-linked binaries won't be portable outside Nix (not sure if it's possible to make them such).

In any case, even spelling out the build is already a source distribution on its own, that will make it possible to compile PineAPPL with just:

nix build github:NNPDF/pineappl

and no extra requirement (not regarding build tools installation, nor dependencies).

[alecandido]

For my future self:

• compiling fully dependency-less packages could be hard (since they could be statically linked at most, or not linked at all), and there is still LHAPDF around (and if we compile with Nix's LHAPDF, the user will have to use Nix)
• however, nix bundle should make possible to package at least the CLI, carrying its own copy of LHAPDF within the bundle

[alecandido]

Any chance for this PR to be merged?

[alecandido]

@cschwan I wanted to rebase this branch on master, but GitHub is complaining...

❯ git push --force-with-lease
Enumerating objects: 41, done.
Counting objects: 100% (41/41), done.
Delta compression using up to 20 threads
Compressing objects: 100% (26/26), done.
Writing objects: 100% (37/37), 8.03 KiB | 2.68 MiB/s, done.
Total 37 (delta 29), reused 15 (delta 10), pack-reused 0
remote: Resolving deltas: 100% (29/29), completed with 3 local objects.
remote: error: GH013: Repository rule violations found for refs/heads/nix.
remote: Review all repository rules at http://github.com/NNPDF/pineappl/rules?ref=refs%2Fheads%2Fnix
remote:
remote: - Cannot force-push to this branch
remote:
To github.com:NNPDF/pineappl.git
 ! [remote rejected] nix -> nix (push declined due to repository rule violations)
error: failed to push some refs to 'github.com:NNPDF/pineappl.git'

It is weird, because I rebased even one week ago, and it said nothing...

[alecandido]

I temporarily bypassed the limitation, but it was happening because of a new ruleset

https://github.com/NNPDF/pineappl/settings/rules/660032