Throw error if CUDA-aware MPI is not found for distributed GPU architecture

[glwagner]

cc @simone-silvestri

[simone-silvestri]

nice

[glwagner]

Hmm, we need to take care because apparently only openmpi allows us to check this:

help?> MPI.has_cuda
  MPI.has_cuda()

  Check if the MPI implementation is known to have CUDA support. Currently only Open MPI provides a mechanism to check, so it will return false with other implementations (unless overridden). For "IBMSpectrumMPI" it will
  return true.

  This can be overridden by setting the JULIA_MPI_HAS_CUDA environment variable to true or false.

  │ Note
  │
  │  For OpenMPI or OpenMPI-based implementations you first need to call Init().

  See also MPI.has_rocm for ROCm support.

[glwagner]

Revisiting this briefly --- we have found that has_cuda only tests Open MPI. So we probably don't want this PR as-is.

However, there are other possible solutions. For example we can write a test like

using MPI
using CUDA
MPI.Init()

function sendrecv_works(grid)
    arch = architecture(grid)
    comm = arch.communicator
    rank = arch.local_rank
    size = MPI.Comm_size(comm)
    dst = mod(rank+1, size)
    src = mod(rank-1, size)
    N = 4
    FT = eltype(grid)
    send_mesg = CuArray{FT}(undef, N)
    recv_mesg = CuArray{FT}(undef, N)
    fill!(send_mesg, FT(rank))
    CUDA.synchronize()
    try
        MPI.Sendrecv!(send_mesg, dst, 0, recv_mesg, src, 0, comm)
        return true
    catch err
        @warn "MPI.Sendrecv test failed." exception=(err, catch_backtrace())
        return false
    end
end

adapted from https://gist.github.com/luraess/0063e90cb08eb2208b7fe204bbd90ed2

[glwagner]

We're also discussing whether such a helper could be added to MPI here: JuliaParallel/MPI.jl#886

[glwagner]

We may want a similar and independent test for Allreduce based on the discussion at

CliMA/ClimaOcean.jl#225

[glwagner]

Also because of the intricacies of configuring CUDA-aware MPI (this can fail due to issues out of your control, for example due to incorrect installation of libraries on a cluster which I experienced recently) we should give more information on failure --- MPI configuration, architecture details, etc.

[simone-silvestri]

I wonder if they have solved this problem in https://github.com/CliMA/ClimaComms.jl

We could think of supporting also non-CUDA-aware MPI by switching the buffers to the CPU and performing a copy to the CPU before the send/recv operations, but that would be a big step because then we would have to support MPI routines everywhere all the time, not only for fill_halo_regions! (for example also in the distributed Fourier transforms or in the all reduce operations)

[glwagner]

I wonder if they have solved this problem in https://github.com/CliMA/ClimaComms.jl

Why do you say that?

We could think of supporting also non-CUDA-aware MPI by switching the buffers to the CPU and performing a copy to the CPU before the send/recv operations, but that would be a big step because then we would have to support MPI routines everywhere all the time, not only for fill_halo_regions! (for example also in the distributed Fourier transforms or in the all reduce operations)

Is there a point? Would such models be useful / run efficiently?

[simone-silvestri]

Why do you say that?

I think they have non-gpu aware support.

Is there a point? Would such models be useful / run efficiently?

There would be some overhead, and hiding communication latency would be more difficult.
Probably not the way forward.

[glwagner]

Why do you say that?

I think they have non-gpu aware support.

Why would non-GPU aware support mean that ClimaComms throws useful errors when CUDA-aware MPI is not available? I don't totally grasp. Anyways if they do have nice error messages let's use those utilities.

There would be some overhead

The question is really about how much overhead there would be. Would it be a small thing, or would it be so large that there would be no point in even trying to run such a simulation.

The other question is, are there systems where CUDA-aware MPI cannot be installed?