[mpich-discuss] MPI_Comm_spawn crosses node boundaries

[Raffenetti, Ken]

On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall at nasa.gov> wrote:

    Ken,

    I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option.

Dang, I thought that would work :(.

    Looking at the Slurm documentation https://slurm.schedmd.com/mpi_guide.html#intel_mpiexec_hydra
    it states  "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface."   The full quote is below for reference.

    1) how do I build MPICH to support hydra's PMI 1.1 interface?

That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes?

    2) Can you offer any guesses on how to build Slurm to do the same?  (I realize this isn't a Slurm forum  😊)

Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes.

Ken