Don't download any artifacts if JULIA_MPI_BINARY=system

[carstenbauer]

MPI.jl seems to always download the MPICH and OpenMPI artifacts despite having set JULIA_MPI_BINARY=system. It is using the correct system MPI eventually but ideally it shouldn’t download the unused artifacts in the first place.

I guess / fear that this is a limitation of Pkg and the lack of optional dependencies. But I wanted to voice and archive this point somewhere and thought here would be the right place.

[simonbyrne]

I guess we could make them lazy, and download the appropriate ones at package build time?

[simonbyrne]

There are some new features in 1.6 for handling lazy artifacts, i haven't looked into it though. If someone wants to try it, i'd certainly be interested.

[carstenbauer]

MKL_jll seems to be lazy, see discussion over at JuliaLang/Pkg.jl#2664. However, don't know what that laziness means exactly. Does it also allow optional download without an Override.toml?

[simonbyrne]

We don't support Overrrides.toml, as it isn't flexible enough for our needs (e.g. Overrides.toml requires the library version number to be the same, which means we can't support different MPI library versions, let alone different implementations).

We should also look into using Preferences.jl instead of our current homebrew approach. I don't have time to tackle this at the moment, but am happy to review PRs.

[sloede]

However, don't know what that laziness means exactly. Does it also allow optional download without an Override.toml?

lazy means that the artifact is downloaded at first use. That is, the actual artifact is not downloaded until you use something like artifact"XYZ" to get the path, at which point all lazy artifacts will be downloaded.

From https://pkgdocs.julialang.org/v1/artifacts/#Using-Artifacts

For the specific use case of using artifacts that were previously bound, we have the shorthand notation artifact"name" which will automatically search for the Artifacts.toml file contained within the current package, look up the given artifact by name, install it if it is not yet installed, then return the path to that given artifact.

If you want to make use of laziness to prevent downloading artifacts greedily, e.g., for the MPICH and OpenMPI packages, you have to go to the package that defines the artifacts for them, e.g., MPICH_jll, and check their Artifacts.toml (spoiler: the artifacts are not marked as lazy there). Since the Artifacts.toml of JLL packages is auto-generated by BinaryBuilder, you have to go look there if you want to mark their artifacts as lazy. And indeed (from the docs for BinaryBuilder.build_tarballs):

[...] The kwargs are passed on to autobuild, see there for a list of supported ones. A few additional keyword arguments are accept: [...]

• lazy_artifacts sets whether the artifacts should be lazy.

Thus going back to the OP,

But I wanted to voice and archive this point somewhere and thought here would be the right place

I think this here is the right place to discuss whether it's useful and necessary to make the MPI artifacts all lazy, but the correct place to actually effect these changes are the build_tarballs(...) calls in the build_tarballs.jl for the respective package on Yggdrasil, e.g., for MPICH:
https://github.com/JuliaPackaging/Yggdrasil/blob/572f34dad652e4fb49ad030bee8250ba2c81ed70/M/MPICH/build_tarballs.jl#L85

[vchuravy]

I am sympathetic with not wanting to download unnecessary binaries, but making them lazy has its own problems, I shuder thinking of a thousand processes hitting the network and racing to download an artifact.

[carstenbauer]

I am sympathetic with not wanting to download unnecessary binaries, but making them lazy has its own problems, I shuder thinking of a thousand processes hitting the network and racing to download an artifact.

We should still be able to avoid this by forcing the eager download of the artifact at build time if necessary, right? We don't really want to use the laziness "dynamically" (i.e. at runtime after MPI.jl is built). Instead, can't we just explicitly make sure at build time (in e.g. build.jl) that if JULIA_MPI_BINARY!=system the artifact is downloaded (by calling artifact"XYZ" or whatever necessary) and if JULIA_MPI_BINARY=system it is not? By a quick look, it already seems like there are no "runtime references" to the jll packages / the artifacts anywhere outside build.jl (and Project.toml).

[sloede]

Yes, I think that should work (from my limited experience, though).

[carstenbauer]

BTW, if no one disagrees with the described approach, I'm willing to test-implement this in a PR (and a corresponding PR to Yggdrasil). Actually, I might try the same strategy over at MKL.jl first. There the artifact is much bigger (~1.5 GB) and, fortunately, is already marked as lazy. (see JuliaLinearAlgebra/MKL.jl#82)

[simonbyrne]

Certainly, please feel free.

We could try to trigger the downloading at Pkg.build time (though i guess upgrade may not trigger that).

[carstenbauer]

Works nicely for MKL.jl, see my PR here. Though, MKL.jl is simpler and I could avoid build.jl altogether.

[simonbyrne]

Nice. Once you're happy with that, any chance you can work your magic here as well?

[carstenbauer]

Yes, I will give it a shot next week.

[giordano]

I agree with Valentin that making these artifacts lazy isn't a good idea. Also because loading time of JLL packages with lazy artifacts is larger than for non-lazy packages.

I think the solution should be resolving JuliaLang/Pkg.jl#2664, coming up with custom solutions for each package is not useful, nor scalable. BTW, my experience with Pkg is that if you really want something to be implemented, you should do it yourself.

[simonbyrne]

But we can't use Overrides.toml, because it assumes the same library ABI

[vchuravy]

JLLWrappers switching to Preferences fixed that. Now you can provide the path directly without having to specify an directory.

https://github.com/maleadt/LLVM.jl/blob/54cbc92e4f39621d3570849363c19e7238c2b0cf/deps/build_local.jl#L46-L51

[giordano]

But we can't use Overrides.toml, because it assumes the same library ABI

I think that with JLLWrappers.jl v1.3.0 you can choose a different name via Preferences.jl, but I never played much with that.

[vchuravy]

Jinx.

[carstenbauer]

I think the solution should be resolving JuliaLang/Pkg.jl#2664, coming up with custom solutions for each package is not useful, nor scalable. BTW, my experience with Pkg is that if you really want something to be implemented, you should do it yourself.

@giordano: I agree with the scalable part but it's certainly useful and, in fact, used in practice, see packages like MPI.jl and CUDA.jl. Of course, a general solution is much desired but implementing/changing something in Pkg can be a much more time consuming and complex than implementing a "custom" solution for, say, the top 3 HPC packages (MKL.jl, MPI.jl, CUDA.jl).

JLLWrappers switching to Preferences fixed that. Now you can provide the path directly without having to specify an directory.

I think that with JLLWrappers.jl v1.3.0 you can choose a different name via Preferences.jl, but I never played much with that.

I'm not sure I understand what the overall strategy / flow of information would be with this new preference feature. In particular, how would it interact with the Overrides.toml (if I specify the path to the MPI lib by setting a "libmpi_path" preference in MPI.jl, why would I still need Overrides.toml)? Can you elaborate a bit more?

Side comment: IMHO, from a users perspective, specifying a JULIA_MPI_PATH=/path/to/libmpi.so env variable to make MPI.jl use the "system" MPI is somewhat nicer than to create an Overrides.toml file somewhere (even more so if I need to first create "custom artifacts" mimicking the folder structure of the respective artifact).

[vchuravy]

In particular, how would it interact with the Overrides.toml

It is meant as a successor/replacements to Overrides.toml. But @staticfloat might have some more thoughts on that.

env variable to make MPI.jl

Up to the moment where they forget to set the environment variable. Environment variables are also problematic due the question of caching. E.g. the .ji file goes stale without the user noticing it. Setting it through preferences is strictly nicer for that.

[sloede]

Up to the moment where they forget to set the environment variable.

On the other hand, this would integrate nicely with the ubiquitous modules system widely used on HPC systems: You type in, e.g., module load intel-mpich and voilà, also Julia picks up the correct MPI version automatically from the environment variables set by the module command.

[carstenbauer]

It is meant as a successor/replacements to Overrides.toml

Ah, that makes sense (and explains my confusion) 😄

[staticfloat]

I think for maximum compatibility, we're just going to have to listen to both environment variables and Preferences;

• Preferences, because these pieces of information are often used at compile-time, so if they change, we need to retrigger precompilation.
• Environment variables, for the reasons you have listed here.

There's a PR to MKL.jl open right now that implements this, but it has one big caveat: if the value of the environment variable is read at compile time, it won't be re-read in the future; so you won't be able to force recompilation by just changing the environment variable. You must change the value through Preferences.jl, and then the next time you load Julia, it will re-precompile.

There are alternative things we can do (such as at __init__() time detect a mismatch between the environment variable and the preference setting, automatically set the preference, then tell the user that they need to restart Julia) but I feel that's a pretty bad user experience.

[carstenbauer]

I see the issue but don't think it's such a big caveat. The fact that environment variables might only effect the ] build MyPackage (or precompilation) phase of a package has been around for quite some time (the PYTHON env variable for PyCall.jl is one example) and I don't think it has caused much confusion.