<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">Hello, <span style="font-family:arial,sans-serif;font-size:13px">Neil,</span><div><span style="font-family:arial,sans-serif;font-size:13px"> Could you send us a copy of the code? Thanks. </span></div></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr">--Junchao Zhang</div></div>
<br><div class="gmail_quote">On Mon, Sep 22, 2014 at 4:53 PM, Neil Spruit <span dir="ltr"><<a href="mailto:nrspruit@gmail.com" target="_blank">nrspruit@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p class="MsoNormal">Hello,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I have been experimenting with mpich’s implementation of
one-sided communication with no remote synchronization that was introduced in
the mpi3.0 spec and I have run into a hang issue during connect/accept MPI scenarios.</p><p class="MsoNormal"></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Here is the situation, I have two applications that connect to each
other after they have been launched separately with mpiexec on two different
hosts. </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">After connecting, both hosts allocate memory and run
MPI_Win_Create as the following:</p>
<p class="MsoNormal">Host1_program:</p>
<p class="MsoNormal"> void* buffer =
mmap(0, buf_length, PROT_READ | PROT_WRITE, MAP_SHARED|MAP_POPULATE|MAP_ANON,
-1, 0);</p>
<p class="MsoNormal">
MPI_Intercomm_merge(connected_comm, 1, &comm);</p>
<p class="MsoNormal">
MPI_Comm_rank(comm, &myrank);</p>
<p class="MsoNormal"> MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL,
comm, &win);</p>
<p class="MsoNormal">Host2_program:</p>
<p class="MsoNormal"> void* buffer =
mmap(0, buf_length, PROT_READ | PROT_WRITE, MAP_SHARED|MAP_POPULATE|MAP_ANON,
-1, 0);</p>
<p class="MsoNormal">
MPI_Intercomm_merge(connected_comm, 1, &comm);</p>
<p class="MsoNormal">
MPI_Comm_rank(comm, &myrank);</p>
<p class="MsoNormal"> MPI_Win_create(buffer, buf_length, 1,
MPI_INFO_NULL, comm, &win); </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Once the MPI_Window has been created I execute the following
on host1:</p>
<p class="MsoNormal"> Host1_program(continued):</p>
<p class="MsoNormal"> err =
MPI_Win_lock(MPI_LOCK_SHARED, target_rank, 0, win);</p>
<p class="MsoNormal"> if(err !=
MPI_SUCCESS)</p>
<p class="MsoNormal"> {</p>
<p class="MsoNormal"> return 1;</p>
<p class="MsoNormal"> }</p>
<p class="MsoNormal"> err =
MPI_Put(buffer, buf_length, MPI_BYTE, target_rank, 0, buf_length, MPI_BYTE,
win);</p>
<p class="MsoNormal"> if(err !=
MPI_SUCCESS)</p>
<p class="MsoNormal"> {</p>
<p class="MsoNormal"> return 1;</p>
<p class="MsoNormal"> }</p>
<p class="MsoNormal"> err =
MPI_Win_flush(target_rank, win); </p>
<p class="MsoNormal"> if(err !=
MPI_SUCCESS)</p>
<p class="MsoNormal"> {</p>
<p class="MsoNormal"> return 1;</p>
<p class="MsoNormal"> }</p>
<p class="MsoNormal"> err =
MPI_Win_unlock(target_rank, win);</p>
<p class="MsoNormal"> if(err !=
MPI_SUCCESS)</p>
<p class="MsoNormal"> {</p>
<p class="MsoNormal"> return 1;</p>
<p class="MsoNormal"> }</p>
<p class="MsoNormal">While this is running, host2’s program is simply sleeping in
a getchar().</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">This should run and complete, however, if the other host has
not called another mpi function such as MPI_Recv and instead is in a sleep,
getchar(), etc. the MPI_Win_flush above will simply hang.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Why is the one-sided operation hanging until another MPI
operation is called on the target host? My understanding is that utilizing
MPI_Win_Flush does not require any action by the target host to complete the
operation.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">This is a stack trace of where the flush gets stuck:</p>
<p class="MsoNormal">#0 0x0000003b456dea28
in poll () from /lib64/libc.so.6</p>
<p class="MsoNormal">#1 0x00007fe1c08004ee
in MPID_nem_tcp_connpoll ()</p>
<p class="MsoNormal"> from
/usr/local/lib/libmpich.so.12</p>
<p class="MsoNormal">#2 0x00007fe1c07efab7
in MPIDI_CH3I_Progress ()</p>
<p class="MsoNormal"> from
/usr/local/lib/libmpich.so.12</p>
<p class="MsoNormal">#3 0x00007fe1c07d35ff
in MPIDI_CH3I_Wait_for_lock_granted ()</p>
<p class="MsoNormal"> from
/usr/local/lib/libmpich.so.12</p>
<p class="MsoNormal">#4 0x00007fe1c07d88bf
in MPIDI_Win_flush () from /usr/local/lib/libmpich.so.12</p>
<p class="MsoNormal">#5 0x00007fe1c08c1b46
in PMPI_Win_flush () from /usr/local/lib/libmpich.so.12</p>
<p class="MsoNormal">#6 0x000000000040115c
in main ()</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">According to this it seems that somehow the remote host is
holding something after MPI_Win_Create that in my scenario gets unlocked if I call
MPI_Recv on the connected_comm from above after the MPI_Win_Create.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thank you for your time and I look forward to your reply.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Respectfully,</p>
<p class="MsoNormal">Neil Spruit</p><p class="MsoNormal"><br></p>
<p class="MsoNormal">P.S. MPICH version info</p><p class="MsoNormal"> Version: 3.1</p><p class="MsoNormal"> Release Date: Thu Feb 20 11:41:13 CST 2014</p>
<p class="MsoNormal"> </p></div>
<br>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div>