[mpich-devel] mpi process wireup and apache yarn

[Kenneth Raffenetti]

On 07/27/2016 12:31 AM, Ryan Lewis wrote:
> However, assuming that this is all I need, it seems that essentially
> each YARN container needs to execute these command lines:
>
>  /usr/lib64/mpich/bin/hydra_pmi_proxy --control-port skynet03:58584
> --rmk user --launcher manual --demux poll --pgid 0 --retries 10 --usize
> -2 --proxy-id 0
>
> Which they can get from starting the mpi control process on the machine
> which runs the YARN Application Master.
>
> And then they will all just work. Is this accurate? Is this a
> "supported" mode of operation? this certainly is an extremely easy way
> to get MPI to run on top of YARN, with zero code change necessary to
> the MPICH codebase. I'm not sure how portable (across MPI
> implementations) this is though, but, for now I don't care.

This is correct. In a typical launch mode, the PMI server (mpiexec) 
takes care of starting the proxies for you, but the proxy startup 
commands are the same as the manual ones you discovered.

Ken