<div class="gmail_extra"><div class="gmail_quote">On Mon, Nov 5, 2012 at 9:37 PM, Pavan Balaji <span dir="ltr"><<a href="mailto:balaji@mcs.anl.gov" target="_blank">balaji@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="im"><br>
On 11/05/12 13:12, John Fettig wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I believe I have a working build, I'll append my cross file to the end<br>
of this email if anybody else wants to try it.<br>
</blockquote>
<br></div>
Thanks!<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I have a followup question: is there any support for launching jobs<br>
that use both the MIC and the host CPU?<br>
</blockquote>
<br></div>
Yes. Once you have setup MPICH on both the host and MIC, you can launch jobs across them.<br>
<br>
If you didn't pass any configure option, it'll use TCP/IP, which is very slow. If you configure with --with-device=ch3:nemesis:scif, it'll use the SCIF protocol, which is much faster.<br></blockquote><div><br>
I compiled examples/hellow.c for both the MIC and the host CPU, and copied it to the card. This seems to work:<br><br>$ mpiexec -hosts <a href="http://172.31.1.1:1">172.31.1.1:1</a>,<a href="http://172.31.1.254:1">172.31.1.254:1</a> -n 1 ./hellow.mic : -n 1 ./hellow<br>
Hello world from process 1 of 2<br>Hello world from process 0 of 2<br><br>However, if I try to run more processes it crashes:<br><br>$ mpiexec -hosts <a href="http://172.31.1.1:3">172.31.1.1:3</a>,<a href="http://172.31.1.254:3">172.31.1.254:3</a> -n 3 ./hellow.mic : -n 3 ./hellow<br>
Hello world from process 4 of 6<br>Hello world from process 0 of 6<br>Hello world from process 3 of 6<br>Hello world from process 1 of 6<br> 0: 3: 00000033: 00000042: readv err 0<br>Fatal error in MPI_Finalize: Other MPI error, error stack:<br>
MPI_Finalize(293).................: MPI_Finalize failed<br>MPI_Finalize(213).................: <br>MPID_Finalize(117)................: <br>MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close<br>
MPIDI_CH3I_Progress(367)..........: <br>MPID_nem_mpich2_blocking_recv(904): <br>state_commrdy_handler(175)........: <br>state_commrdy_handler(138)........: <br>MPID_nem_scif_recv_handler(115)...: Communication error with rank 3<br>
MPID_nem_scif_recv_handler(35)....: scif_scif_read failed (scif_scif_read failed with error 'Success')<br> 1: 3: 00000033: 00000042: readv err 0<br>Fatal error in MPI_Finalize: Other MPI error, error stack:<br>MPI_Finalize(293).................: MPI_Finalize failed<br>
MPI_Finalize(213).................: <br>MPID_Finalize(117)................: <br>MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close<br>MPIDI_CH3I_Progress(367)..........: <br>
MPID_nem_mpich2_blocking_recv(904): <br>state_commrdy_handler(175)........: <br>state_commrdy_handler(138)........: <br>MPID_nem_scif_recv_handler(115)...: Communication error with rank 3<br>MPID_nem_scif_recv_handler(35)....: scif_scif_read failed (scif_scif_read failed with error 'Success')<br>
Hello world from process 5 of 6<br>Fatal error in MPI_Finalize: Other MPI error, error stack:<br>MPI_Finalize(293).................: MPI_Finalize failed<br>MPI_Finalize(213).................: <br>MPID_Finalize(117)................: <br>
MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close<br>MPIDI_CH3I_Progress(367)..........: <br>MPID_nem_mpich2_blocking_recv(904): <br>state_commrdy_handler(184)........: poll of socket fds failed<br>
Fatal error in MPI_Finalize: Other MPI error, error stack:<br>MPI_Finalize(293).................: MPI_Finalize failed<br>MPI_Finalize(213).................: <br>MPID_Finalize(117)................: <br>MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close<br>
MPIDI_CH3I_Progress(367)..........: <br>MPID_nem_mpich2_blocking_recv(904): <br>state_commrdy_handler(184)........: poll of socket fds failed<br><br>===================================================================================<br>
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>= EXIT CODE: 1<br>= CLEANING UP REMAINING PROCESSES<br>= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>===================================================================================<br>
[proxy:0:0@mic0.local] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:883): assert (!closed) failed<br>[proxy:0:0@mic0.local] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status<br>
[proxy:0:0@mic0.local] main (./pm/pmiserv/pmip.c:210): demux engine error waiting for event<br>[mpiexec@host] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting<br>
[mpiexec@host] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion<br>[mpiexec@host] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:216): launcher returned error waiting for completion<br>
[mpiexec@host] main (./ui/mpich/mpiexec.c:325): process manager error waiting for completion<br><br>Any ideas?<br><br>John<br></div></div></div>