<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Can you also provide your config.log and any CVARs or other relevant environment variables that you might be setting (for instance, in relation to fault tolerance)?
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Wesley<br class="">
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Nov 25, 2014, at 3:58 PM, Amin Hassani <<a href="mailto:ahassani@cis.uab.edu" class="">ahassani@cis.uab.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
This is the simplest code I have that doesn't run.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br class="">
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br class="">
</div>
<div class="gmail_default" style="">
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">#include <mpi.h></font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">#include <stdio.h></font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">#include <malloc.h></font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">#include <unistd.h></font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">#include <stdlib.h></font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default" style=""><span style="font-family:tahoma,sans-serif" class="">int main(int argc, char** argv)</span><br class="">
</div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">{</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> int rank, size;</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> int i, j, k;</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> double t1, t2;</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> int rc;</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> MPI_Init(&argc, &argv);</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> MPI_Comm world = MPI_COMM_WORLD, newworld, newworld2;</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> MPI_Comm_rank(world, &rank);</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> MPI_Comm_size(world, &size);</font></div>
<div class="gmail_default" style=""><span style="font-family:tahoma,sans-serif" class=""><br class="">
</span></div>
<div class="gmail_default" style=""><span style="font-family:tahoma,sans-serif" class=""> t2 = 1;</span></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> MPI_Allreduce(&t2, &t_avg, 1, MPI_DOUBLE, MPI_SUM, world);</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> t_avg = t_avg / size;</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default" style=""><span style="font-family:tahoma,sans-serif" class=""> MPI_Finalize();</span><br class="">
</div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""> return 0;</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">}</font></div>
</div>
</div>
<div class="gmail_extra"><br clear="all" class="">
<div class="">
<div class="gmail_signature">
<div dir="ltr" class="">Amin Hassani,<br class="">
CIS department at UAB,<br class="">
Birmingham, AL, USA.</div>
</div>
</div>
<br class="">
<div class="gmail_quote">On Tue, Nov 25, 2014 at 2:46 PM, "Antonio J. Peña" <span dir="ltr" class="">
<<a href="mailto:apenya@mcs.anl.gov" target="_blank" class="">apenya@mcs.anl.gov</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class=""><br class="">
Hi Amin,<br class="">
<br class="">
Can you share with us a minimal piece of code with which you can reproduce this issue?<br class="">
<br class="">
Thanks,<br class="">
Antonio
<div class="">
<div class="h5"><br class="">
<br class="">
<br class="">
On 11/25/2014 12:52 PM, Amin Hassani wrote:<br class="">
</div>
</div>
</div>
<blockquote type="cite" class="">
<div class="">
<div class="h5">
<div dir="ltr" class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Hi,</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br class="">
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
I am having problem running MPICH, on multiple nodes. When I run an multiple MPI processes on one node, it totally works, but when I try to run on multiple nodes, it fails with the error below.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
My machines have Debian OS, Both infiniband and TCP interconnects. I'm guessing it has something do to with the TCP network, but I can run openmpi on these machines with no problem. But for some reason I cannot run MPICH on multiple nodes. Please let me know
if more info is needed from my side. I'm guessing there are some configuration that I am missing. I used MPICH 3.1.3 for this test. I googled this problem but couldn't find any solution.</div>
<div class=""><br class="">
</div>
<div class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
In my MPI program, I am doing a simple allreduce over MPI_COMM_WORLD.</div>
<br class="">
</div>
<div class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
my host file (hosts-hydra) is something like this:</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">oakmnt-0-a:1</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
oakmnt-0-b:1 </div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
I get this error:</div>
<br class="">
</div>
<div class="">
<div class="gmail_default">
<div class="gmail_default"><font face="tahoma, sans-serif" class="">$ mpirun -hostfile hosts-hydra -np 2 test_dup</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: status->MPI_TAG == recvtag</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: status->MPI_TAG == recvtag</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">internal ABORT - process 1</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">internal ABORT - process 0</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">===================================================================================</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">= PID 30744 RUNNING AT oakmnt-0-b</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">= EXIT CODE: 1</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">= CLEANING UP REMAINING PROCESSES</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">===================================================================================</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">[mpiexec@vulcan13] HYDU_sock_read (../../../../src/pm/hydra/utils/sock/sock.c:239): read error (Bad file descriptor)</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">[mpiexec@vulcan13] control_cb (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:199): unable to read command from proxy</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">[mpiexec@vulcan13] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">[mpiexec@vulcan13] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">[mpiexec@vulcan13] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">Thanks.</font></div>
</div>
</div>
<div class="">
<div class="">
<div dir="ltr" class="">Amin Hassani,<br class="">
CIS department at UAB,<br class="">
Birmingham, AL, USA.</div>
</div>
</div>
</div>
<br class="">
<fieldset class=""></fieldset> <br class="">
</div>
</div>
<pre class="">_______________________________________________
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank" class="">discuss@mpich.org</a>
To manage subscription options or unsubscribe:
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></pre>
<span class="HOEnZb"><font color="#888888" class=""></font></span></blockquote>
<span class="HOEnZb"><font color="#888888" class=""><br class="">
<br class="">
<pre cols="72" class="">--
Antonio J. Peña
Postdoctoral Appointee
Mathematics and Computer Science Division
Argonne National Laboratory
9700 South Cass Avenue, Bldg. 240, Of. 3148
Argonne, IL 60439-4847
<a href="mailto:apenya@mcs.anl.gov" target="_blank" class="">apenya@mcs.anl.gov</a>
<a href="http://www.mcs.anl.gov/~apenya" target="_blank" class="">www.mcs.anl.gov/~apenya</a></pre>
</font></span></div>
<br class="">
_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank" class="">https://lists.mpich.org/mailman/listinfo/discuss</a><br class="">
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>