[mpich-devel] mpi process wireup and apache yarn

Kenneth Raffenetti raffenet at mcs.anl.gov
Thu Jul 28 09:42:40 CDT 2016

On 07/27/2016 12:31 AM, Ryan Lewis wrote:
> However, assuming that this is all I need, it seems that essentially
> each YARN container needs to execute these command lines:
>  /usr/lib64/mpich/bin/hydra_pmi_proxy --control-port skynet03:58584
> --rmk user --launcher manual --demux poll --pgid 0 --retries 10 --usize
> -2 --proxy-id 0
> Which they can get from starting the mpi control process on the machine
> which runs the YARN Application Master.
> And then they will all just work. Is this accurate? Is this a
> "supported" mode of operation? this certainly is an extremely easy way
> to get MPI to run on top of YARN, with zero code change necessary to
> the MPICH codebase. I'm not sure how portable (across MPI
> implementations) this is though, but, for now I don't care.

This is correct. In a typical launch mode, the PMI server (mpiexec) 
takes care of starting the proxies for you, but the proxy startup 
commands are the same as the manual ones you discovered.


More information about the devel mailing list