<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
It’s hard to tell then. Other than some problems compiling (not declaring all of your variables), everything seems ok. Can you try running with the most recent alpha. I have no idea what bug we could have fixed here to make things work, but it’d be good to
eliminate the possibility.
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Wesley<br class="">
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Nov 25, 2014, at 10:11 PM, Amin Hassani <<a href="mailto:ahassani@cis.uab.edu" class="">ahassani@cis.uab.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Here I attached config.log exits in the root folder where it is compiled. I'm not too familiar with MPICH but, there are other config.logs in other directories also but not sure if you needed them too. </div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
I don't have any specific environment variable that can relate to MPICH. Also tried with</div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">export HYDRA_HOST_FILE=<address to host file>,</font><br class="">
</div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">but have the same problem.</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">I don't do anything FT related in MPICH, I don't think this version of MPICH has anything related to FT in it.</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default" style=""><font face="tahoma, sans-serif" class="">Thanks.</font></div>
</div>
<div class="gmail_extra"><br clear="all" class="">
<div class="">
<div class="gmail_signature">
<div dir="ltr" class="">Amin Hassani,<br class="">
CIS department at UAB,<br class="">
Birmingham, AL, USA.</div>
</div>
</div>
<br class="">
<div class="gmail_quote">On Tue, Nov 25, 2014 at 9:02 PM, Bland, Wesley B. <span dir="ltr" class="">
<<a href="mailto:wbland@anl.gov" target="_blank" class="">wbland@anl.gov</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Can you also provide your config.log and any CVARs or other relevant environment variables that you might be setting (for instance, in relation to fault tolerance)?
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Wesley
<div class="">
<div class="h5"><br class="">
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Nov 25, 2014, at 3:58 PM, Amin Hassani <<a href="mailto:ahassani@cis.uab.edu" target="_blank" class="">ahassani@cis.uab.edu</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
This is the simplest code I have that doesn't run.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br class="">
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br class="">
</div>
<div class="gmail_default">
<div class="gmail_default"><font face="tahoma, sans-serif" class="">#include <mpi.h></font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">#include <stdio.h></font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">#include <malloc.h></font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">#include <unistd.h></font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">#include <stdlib.h></font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default"><span style="font-family:tahoma,sans-serif" class="">int main(int argc, char** argv)</span><br class="">
</div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">{</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> int rank, size;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> int i, j, k;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> double t1, t2;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> int rc;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> MPI_Init(&argc, &argv);</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> MPI_Comm world = MPI_COMM_WORLD, newworld, newworld2;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> MPI_Comm_rank(world, &rank);</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> MPI_Comm_size(world, &size);</font></div>
<div class="gmail_default"><span style="font-family:tahoma,sans-serif" class=""><br class="">
</span></div>
<div class="gmail_default"><span style="font-family:tahoma,sans-serif" class=""> t2 = 1;</span></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> MPI_Allreduce(&t2, &t_avg, 1, MPI_DOUBLE, MPI_SUM, world);</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> t_avg = t_avg / size;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default"><span style="font-family:tahoma,sans-serif" class=""> MPI_Finalize();</span><br class="">
</div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""> return 0;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">}</font></div>
</div>
</div>
<div class="gmail_extra"><br clear="all" class="">
<div class="">
<div class="">
<div dir="ltr" class="">Amin Hassani,<br class="">
CIS department at UAB,<br class="">
Birmingham, AL, USA.</div>
</div>
</div>
<br class="">
<div class="gmail_quote">On Tue, Nov 25, 2014 at 2:46 PM, "Antonio J. Peña" <span dir="ltr" class="">
<<a href="mailto:apenya@mcs.anl.gov" target="_blank" class="">apenya@mcs.anl.gov</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class=""><br class="">
Hi Amin,<br class="">
<br class="">
Can you share with us a minimal piece of code with which you can reproduce this issue?<br class="">
<br class="">
Thanks,<br class="">
Antonio
<div class="">
<div class=""><br class="">
<br class="">
<br class="">
On 11/25/2014 12:52 PM, Amin Hassani wrote:<br class="">
</div>
</div>
</div>
<blockquote type="cite" class="">
<div class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Hi,</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br class="">
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
I am having problem running MPICH, on multiple nodes. When I run an multiple MPI processes on one node, it totally works, but when I try to run on multiple nodes, it fails with the error below.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
My machines have Debian OS, Both infiniband and TCP interconnects. I'm guessing it has something do to with the TCP network, but I can run openmpi on these machines with no problem. But for some reason I cannot run MPICH on multiple nodes. Please let me know
if more info is needed from my side. I'm guessing there are some configuration that I am missing. I used MPICH 3.1.3 for this test. I googled this problem but couldn't find any solution.</div>
<div class=""><br class="">
</div>
<div class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
In my MPI program, I am doing a simple allreduce over MPI_COMM_WORLD.</div>
<br class="">
</div>
<div class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
my host file (hosts-hydra) is something like this:</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">oakmnt-0-a:1</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
oakmnt-0-b:1 </div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
I get this error:</div>
<br class="">
</div>
<div class="">
<div class="gmail_default">
<div class="gmail_default"><font face="tahoma, sans-serif" class="">$ mpirun -hostfile hosts-hydra -np 2 test_dup</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: status->MPI_TAG == recvtag</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: status->MPI_TAG == recvtag</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">internal ABORT - process 1</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">internal ABORT - process 0</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">===================================================================================</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">= PID 30744 RUNNING AT oakmnt-0-b</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">= EXIT CODE: 1</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">= CLEANING UP REMAINING PROCESSES</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">===================================================================================</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">[mpiexec@vulcan13] HYDU_sock_read (../../../../src/pm/hydra/utils/sock/sock.c:239): read error (Bad file descriptor)</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">[mpiexec@vulcan13] control_cb (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:199): unable to read command from proxy</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">[mpiexec@vulcan13] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">[mpiexec@vulcan13] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">[mpiexec@vulcan13] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class=""><br class="">
</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif" class="">Thanks.</font></div>
</div>
</div>
<div class="">
<div class="">
<div dir="ltr" class="">Amin Hassani,<br class="">
CIS department at UAB,<br class="">
Birmingham, AL, USA.</div>
</div>
</div>
</div>
<br class="">
<fieldset class=""></fieldset> <br class="">
</div>
</div>
<pre class="">_______________________________________________
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank" class="">discuss@mpich.org</a>
To manage subscription options or unsubscribe:
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></pre>
<span class=""><font color="#888888" class=""></font></span></blockquote>
<span class=""><font color="#888888" class=""><br class="">
<br class="">
<pre cols="72" class="">--
Antonio J. Peña
Postdoctoral Appointee
Mathematics and Computer Science Division
Argonne National Laboratory
9700 South Cass Avenue, Bldg. 240, Of. 3148
Argonne, IL 60439-4847
<a href="mailto:apenya@mcs.anl.gov" target="_blank" class="">apenya@mcs.anl.gov</a>
<a href="http://www.mcs.anl.gov/~apenya" target="_blank" class="">www.mcs.anl.gov/~apenya</a></pre>
</font></span></div>
<br class="">
_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank" class="">https://lists.mpich.org/mailman/listinfo/discuss</a><br class="">
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</div>
<br class="">
_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank" class="">https://lists.mpich.org/mailman/listinfo/discuss</a><br class="">
</blockquote>
</div>
<br class="">
</div>
<span id="cid:E22E4B73-2CF4-4527-91F3-118A3796897F@tds.net"><config.log></span>_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>