<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">Thank you, Kenneth.<div><br></div><div>Here is a simple C++ equivalent of what i am doing:</div><div><br></div><div>server.cpp:</div><div>=============================</div><div><br></div><div><div>#include <iostream></div><div>#include <mpi/mpi.h></div><div>#include <stdlib.h></div><div><br></div><div>using namespace std;</div><div><br></div><div>// The only argument must be the number of processes in communicator we expect to build.</div><div>int main(int argc, char** argv)</div><div>{</div><div><br></div><div> int np = atoi(argv[1]);</div><div><br></div><div> int ac = 0;</div><div> MPI_Init(&ac, &argv);</div><div><br></div><div> char portName[MPI_MAX_PORT_NAME];</div><div><br></div><div> MPI_Open_port(MPI_INFO_NULL, portName);</div><div><br></div><div> cout << portName << "\n";</div><div><br></div><div> MPI_Comm intercomm, intracomm = MPI_COMM_SELF;</div><div><br></div><div> // Build an intracom dynamically until n processes are reached.</div><div> for (int i = 1; i < np; i++) {</div><div><br></div><div> MPI_Comm_accept(portName, MPI_INFO_NULL, 0, intracomm, &intercomm);</div><div><br></div><div> cout << "Accepted.\n";</div><div><br></div><div> MPI_Intercomm_merge(intercomm, false, &intracomm);</div><div><br></div><div> cout << "Merged to an intracom.\n";</div><div><br></div><div> MPI_Comm_free(&intercomm);</div><div> }</div><div><br></div><div> // Intracomm contains the one we can now use with n-grid.</div><div> MPI_Comm_free(&intracomm);</div><div><br></div><div> MPI_Close_port(portName);</div><div><br></div><div> MPI_Finalize();</div><div>}</div></div><div><br></div><div><br></div><div>client.cpp:</div><div>===============================</div><div><div>#include <iostream></div><div>#include <mpi/mpi.h></div><div>#include <stdlib.h></div><div><br></div><div>using namespace std;</div><div><br></div><div>// This expects intracom size and the port name to connect to.</div><div>// When using with shell, use single quotas to avoid shell substitution.</div><div>int main(int argc, char** argv)</div><div>{</div><div><br></div><div> int ac = 0;</div><div> MPI_Init(&ac, &argv);</div><div><br></div><div> int np = atoi(argv[1]);</div><div> char* portName = argv[2];</div><div><br></div><div> cout << "Connecting to " << portName << "\n";</div><div><br></div><div> MPI_Comm intercomm, intracomm;</div><div><br></div><div> MPI_Comm_connect(portName, MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm);</div><div><br></div><div> cout << "Connected.\n";</div><div><br></div><div> MPI_Intercomm_merge(intercomm, true, &intracomm);</div><div><br></div><div> cout << "Merged.\n";</div><div><br></div><div> MPI_Comm_free(&intercomm);</div><div><br></div><div> int i;</div><div><br></div><div> MPI_Comm_size(intracomm, &i);</div><div><br></div><div> // Build an intracom dynamically until n processes are reached.</div><div> for (; i < np; i++) {</div><div><br></div><div> MPI_Comm_accept(portName, MPI_INFO_NULL, 0, intracomm, &intercomm);</div><div><br></div><div> cout << "Accepted.\n";</div><div><br></div><div> MPI_Intercomm_merge(intercomm, false, &intracomm);</div><div><br></div><div> cout << "Merged to an intracom.\n";</div><div><br></div><div> MPI_Comm_free(&intercomm);</div><div> }</div><div><br></div><div> // Intracomm contains the one we can now use with n-grid.</div><div> MPI_Comm_free(&intracomm);</div><div><br></div><div> MPI_Finalize();</div><div>}</div></div><div><br></div><div>============================</div><div>Run example on one machine, intracom size=2 (in this case I have run 3.3a)</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">dmitriy@Intel-Kubu:~/projects/mpitests$ mpic++ server.cpp -o server
</span><br>dmitriy@Intel-Kubu:~/projects/mpitests$ mpic++ client.cpp -o client
<br>dmitriy@Intel-Kubu:~/projects/mpitests$ <br>dmitriy@Intel-Kubu:~/projects/mpitests$ <br>dmitriy@Intel-Kubu:~/projects/mpitests$ mpiexec ./server 2
<br>tag#0$description#Intel-Kubu$port#39210$ifname#127.0.1.1$</span></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">Accepted.
</span><br>Merged to an intracom.
<br>dmitriy@Intel-Kubu:~/projects/mpitests$ <br></span><span style="font-family:monospace">
<br></span></div><div>(in another shell) </div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">dmitriy@Intel-Kubu:~/projects/mpitests$ mpiexec ./client 2 'tag#0$description#Intel-Kubu$port#39210$ifname#127.0.1.1$'
</span><br>Connecting to tag#0$description#Intel-Kubu$port#39210$ifname#127.0.1.1$
<br>Connected.
<br>Merged.
<br>dmitriy@Intel-Kubu:~/projects/mpitests$ <br>
<br></span></div><div><font face="monospace">First parameter is eventual size of intracom we are trying to build dynamically, and client also needs to know the port reported by server process. So there's therfore 1 server and (n-1) clients that connect to form the intracom.</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">Now, if i do that for 192 processes on a 192 core cluster (occasionally slightly overloaded in terms of cpu load), I more often get a lock-up than not. The incidence is more frequent for 3.2 than 3.3a2. This cluster has mellanox infiniband.</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">3.2 usually locks up on merge intercom call; and 3.3a2 locked up at least once on 2 clients connected and waiting on merge intercom at the same time (but my understanding only one client should be connected at a time, even if there massive connects pending from other clients).</font></div><div><font face="monospace"><br></font></div><div>Hope this gives a little more material.</div><div>-Dmitriy</div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 3, 2017 at 8:40 AM, Kenneth Raffenetti <span dir="ltr"><<a href="mailto:raffenet@mcs.anl.gov" target="_blank">raffenet@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Dmitriy,<br>
<br>
MPICH does appear to be reported a process exit/crash in this case. A simple reproducer would be useful to test if that is indeed the cause or if there's something else going on.<br>
<br>
I see below that you are using a non-standard MPI binding. If the test case is simple enough, we can try to port it and investigate further.<br>
<br>
Ken<span class="gmail-"><br>
<br>
On 01/19/2017 06:58 PM, Dmitriy Lyubimov wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">
These lock-ups seem to be gone in 3.3a2.<br>
<br>
I do occasionally get the following though:<br>
<br>
Unknown error class, error stack:<br>
PMPI_Comm_accept(129).........<wbr>........:<br>
MPI_Comm_accept(port="tag#0$de<wbr>scription#<a href="http://aaa.com" rel="noreferrer" target="_blank">aaa.com</a><br></span>
<<a href="http://aaa.com" rel="noreferrer" target="_blank">http://aaa.com</a>>$port#36230$if<wbr>name#192.168.42.99$", MPI_INFO_NULL, ro<span class="gmail-"><br>
ot=180, comm=0x84000003, newcomm=0x7f3cf681842c) failed<br>
MPID_Comm_accept(153).........<wbr>........:<br>
MPIDI_Comm_accept(1244).......<wbr>........:<br>
MPIR_Get_contextid_sparse_grou<wbr>p(499)..:<br>
MPIR_Allreduce_impl(755)......<wbr>........:<br>
MPIR_Allreduce_intra(414).....<wbr>........:<br>
MPIDU_Complete_posted_with_err<wbr>or(1137): Process failed<br>
<br>
What does this message mean? some process just exited/died (like with<br>
seg fault?)<br>
<br>
Thank you.<br>
-Dmitriy<br>
<br>
On Thu, Jan 12, 2017 at 11:55 AM, Dmitriy Lyubimov <<a href="mailto:dlieu.7@gmail.com" target="_blank">dlieu.7@gmail.com</a><br></span><span class="gmail-">
<mailto:<a href="mailto:dlieu.7@gmail.com" target="_blank">dlieu.7@gmail.com</a>>> wrote:<br>
<br>
further debugging shows that it's not actually mergeIntercom that<br>
locks up but a pair of send/recv that two nodes decide to execute<br>
before MPI_intercom_merge.<br>
<br>
so the total snapshot of the situation is that everyone waits on<br>
mergeIntercom except for two processes that wait in send/recv<br>
respectively, while majority of others already have entered<br>
collective barrier.<br>
<br>
it would seem that this sort of assymetric logic would be<br>
acceptable, since the send/recv pair is balanced before the merge is<br>
to occur, but in practice it seems to lock up -- increasingly so as<br>
the number of participating processes increases. It almost like<br>
once collective barrier of certain cardinality is formed,<br>
point-to-point messages are not going thru any longer.<br>
<br>
If this scenario begets any ideas, please let me know.<br>
<br>
thank you!<br>
-Dmitriy<br>
<br>
<br>
<br>
On Wed, Jan 11, 2017 at 9:38 AM, Dmitriy Lyubimov <<a href="mailto:dlieu.7@gmail.com" target="_blank">dlieu.7@gmail.com</a><br></span><span class="gmail-">
<mailto:<a href="mailto:dlieu.7@gmail.com" target="_blank">dlieu.7@gmail.com</a>>> wrote:<br>
<br>
Maybe it has something to do with the fact that it is stepping<br>
thru JVM JNI and that somehow screws threading model of MPI,<br>
although it is a single threaded JVM process, and MPI mappings<br>
are known to have been done before (e.g., openmpi had an effort<br>
towards that).<br>
<br>
Strange thing is that i never had lock up with # of processes<br>
under 120 but something changes after that, the spurious<br>
condition becomes much more common after that. By the time I am<br>
at 150 processes in the intercom, I am almost certain to have a<br>
merge lock-up.<br>
<br>
<br>
On Wed, Jan 11, 2017 at 9:34 AM, Dmitriy Lyubimov<br></span><span class="gmail-">
<<a href="mailto:dlieu.7@gmail.com" target="_blank">dlieu.7@gmail.com</a> <mailto:<a href="mailto:dlieu.7@gmail.com" target="_blank">dlieu.7@gmail.com</a>>> wrote:<br>
<br>
Thanks.<br>
it would not be easy for me to do immediately as i am using<br>
proprietary scala binding api for MPI.<br>
<br>
it would help me to know if there's a known problem like<br>
that in the past, or generally mergeIntercomm api is known<br>
to work on hundreds of processes. Sounds like there are no<br>
known issues with that.<br>
<br>
<br>
<br>
On Tue, Jan 10, 2017 at 11:53 PM, Oden, Lena <<a href="mailto:loden@anl.gov" target="_blank">loden@anl.gov</a><br></span><span class="gmail-">
<mailto:<a href="mailto:loden@anl.gov" target="_blank">loden@anl.gov</a>>> wrote:<br>
<br>
Hello Dmittiy,<br>
<br>
can you maybe create a simple example-program to<br>
reproduce this failure?<br>
It is also often easier also to look at a code example<br>
to identify a problem.<br>
<br>
Thanks,<br>
Lena<br>
> On Jan 11, 2017, at 2:45 AM, Dmitriy Lyubimov<br></span><span class="gmail-">
<<a href="mailto:dlieu.7@gmail.com" target="_blank">dlieu.7@gmail.com</a> <mailto:<a href="mailto:dlieu.7@gmail.com" target="_blank">dlieu.7@gmail.com</a>>> wrote:<br>
><br>
> Hello,<br>
><br>
> (mpich 3.2)<br>
><br>
> I have a scenario when i add a few extra processes do<br>
existing intercom.<br>
><br>
> it works as a simple loop --<br>
> (1) n processes accept on n-intercom<br>
> (2) 1 process connects<br>
> (3) intracom is merged into n+1 intercom, intracom and<br>
n-intercom are closed<br>
> (4) repeat 1-3 as needed.<br>
><br>
> Occasionally, i observe that step 3 spuriously locks<br>
up (once i get in the range of 100+ processes). From<br>
what i can tell, all processes in step 3 are accounted<br>
for, and are waiting on the merge, but nothing happens.<br>
the collective barrier locks up.<br>
><br>
> I really have trouble resolving this issue, any ideas<br>
are appreciated!<br>
><br>
> Thank you very much.<br>
> -Dmitriy<br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br></span>
<mailto:<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><span class="gmail-"><br>
> To manage subscription options or unsubscribe:<br>
> <a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discuss</a><br>
<<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailm<wbr>an/listinfo/discuss</a>><br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br></span>
<mailto:<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><span class="gmail-"><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discuss</a><br>
<<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailm<wbr>an/listinfo/discuss</a>><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discuss</a><br>
<br>
</span></blockquote><div class="gmail-HOEnZb"><div class="gmail-h5">
______________________________<wbr>_________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discuss</a><br>
</div></div></blockquote></div><br></div></div></div>