<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
Hi Abhishek,
<div><br>
</div>
<div>Can you try with the recent MPICH release to see if the same error happens? You can download the recent release, 3.1.2, from <a href="http://www.mpich.org/downloads/">http://www.mpich.org/downloads/</a>.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Sangmin</div>
<div><br>
</div>
<div><br>
<div>
<div>On Sep 12, 2014, at 12:59 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com">abhat@trinityconsultants.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div class="WordSection1" style="page: WordSection1;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">I am running a photochemical modeling on Linux cluster (CentOS_64 bit) with 1 master and 8 slave nodes with quad core (intel i7) on each node. I have two scenarios, in first scenario, I am running less data intensive
run on all 8 nodes (NUMPROCS = 9) and the run will go fine. When running same configuration for a more intense run, I am getting following error.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">Fatal error in MPI_Recv: Other MPI error, error stack:<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">MPI_Recv(187).....................: MPI_Recv(buf=0x7fff989d53b0, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff995d96a0) failed<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">MPIDI_CH3I_Progress(150)..........:<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">MPID_nem_mpich2_blocking_recv(948):<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">MPID_nem_tcp_connpoll(1720).......:<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">state_commrdy_handler(1556).......:<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">MPID_nem_tcp_recv_handler(1446)...: socket closed<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">rank 1 in job 1 dfw-camx_55000 caused collective abort of all ranks<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> exit status of rank 1: killed by signal 9<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">If I run the program with smaller nodes (smaller than 7 NUMPROCS) the run goes fine.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">It appears that the rank 1 (my first node) is collectively causing all the ranks, but I could identify why. I tried following solutions <o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;">
<span style="font-family: Cambria, serif;"><span>1.<span style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-family: 'Times New Roman';"> <span class="Apple-converted-space"> </span></span></span></span><span style="font-family: Cambria, serif;">Increased
master memory to 32 gb<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;">
<span style="font-family: Cambria, serif;"><span>2.<span style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-family: 'Times New Roman';"> <span class="Apple-converted-space"> </span></span></span></span><span style="font-family: Cambria, serif;">Increased
all nodes memory to 32 gb<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;">
<span style="font-family: Cambria, serif;"><span>3.<span style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-family: 'Times New Roman';"> <span class="Apple-converted-space"> </span></span></span></span><span style="font-family: Cambria, serif;">Exchanged
the rank 1 to different node in the parallel.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">In all situations, I am getting this error. Surprisingly, when I am running smaller (less data intensive runs), I am not getting this error even if I increase the NUMPROCS to 32 processes.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">Any help will be highly appreciated.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">I am running mpich 1.4<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">Thank You<br>
Abhishek<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif; color: rgb(0, 64, 128);">
.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<b><span style="font-family: Cambria, serif; color: rgb(0, 64, 128);">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-family: Cambria, serif; color: rgb(0, 64, 128);">Senior Consultant<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif; color: rgb(0, 64, 128);"> </span></div>
<div style="margin: 0in 0in 0.0001pt 0.7pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<b><span style="font-family: Cambria, serif; color: rgb(0, 64, 128);">Trinity Consultants</span></b><span style="font-family: Cambria, serif;"><o:p></o:p></span></div>
<p class="MsoNormal" style="margin: 0in 0in 6pt 0.7pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">12770 Merit Drive, Suite 900 | Dallas, Texas 75251<o:p></o:p></span></p>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">Office: <span class="Apple-converted-space"> </span></span><b><span style="font-family: Cambria, serif; color: rgb(194, 0, 0);">972-661-8100</span></b><span style="font-family: Cambria, serif;">| Mobile: 806-281-7617<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">Email: <span class="Apple-converted-space"> </span></span><span style="font-family: Cambria, serif;"><a href="mailto:abhat@trinityconsultants.com" style="color: rgb(149, 79, 114); text-decoration: underline;"><span style="color: rgb(5, 99, 193);">abhat@trinityconsultants.com</span></a></span><u><span style="font-family: Cambria, serif; color: rgb(0, 64, 128);"> </span></u><span style="font-family: Cambria, serif;"> |
LinkedIn: </span><span style="font-family: Cambria, serif;"><a href="http://www.linkedin.com/in/abhattrinityconsultants" style="color: rgb(149, 79, 114); text-decoration: underline;"><span style="color: rgb(5, 99, 193);">www.linkedin.com/in/abhattrinityconsultants</span></a><span style=""><o:p></o:p></span></span></div>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">Stay current on environmental issues. <span class="Apple-converted-space"> </span></span><span style="font-family: Cambria, serif;"><a href="http://www.trinityconsultants.com/Subscribe/" style="color: rgb(149, 79, 114); text-decoration: underline;"><span style="color: rgb(0, 64, 128);">Subscribe</span></a></span><span style="font-family: Cambria, serif;"><span class="Apple-converted-space"> </span>today
to receive Trinity's free<span class="Apple-converted-space"> </span></span><span style="font-family: Cambria, serif;"><a href="http://www.trinityconsultants.com/EnvironmentalQuarterly/" style="color: rgb(149, 79, 114); text-decoration: underline;"><i><span style="color: rgb(0, 64, 128);">Environmental
Quarterly</span></i></a></span><span style="font-family: Cambria, serif;">.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;">Learn about Trinitys<span class="Apple-converted-space"> </span></span><span style="font-family: Cambria, serif;"><a href="http://www.trinityconsultants.com/Training/" style="color: rgb(149, 79, 114); text-decoration: underline;"><span style="color: rgb(0, 64, 128);">courses</span></a></span><span style="font-family: Cambria, serif;"><span class="Apple-converted-space"> </span>for
environmental professionals.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<a href="http://www.linkedin.com/company/trinity-consultants" style="color: rgb(149, 79, 114); text-decoration: underline;"><span style="font-family: Cambria, serif; color: rgb(5, 99, 193); text-decoration: none;"><image001.gif></span></a><span style="font-family: Cambria, serif;"> <span class="Apple-converted-space"> </span></span><a href="http://www.facebook.com/TrinityConsults" style="color: rgb(149, 79, 114); text-decoration: underline;"><span style="font-family: Cambria, serif; color: rgb(5, 99, 193); text-decoration: none;"><image002.gif></span></a><span style="font-family: Cambria, serif;"> </span><a href="http://twitter.com/trinityconsults" style="color: rgb(149, 79, 114); text-decoration: underline;"><span style="font-family: Cambria, serif; color: rgb(5, 99, 193); text-decoration: none;"><image003.gif></span></a><span style="font-family: Cambria, serif;"> </span><a href="http://www.youtube.com/trinityconsultants" style="color: rgb(149, 79, 114); text-decoration: underline;"><span style="font-family: Cambria, serif; color: rgb(5, 99, 193); text-decoration: none;"><image004.gif></span></a><span style="font-family: Cambria, serif;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: Cambria, serif;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="color: rgb(31, 73, 125);"><image005.jpg></span><span style="font-family: Cambria, serif;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p> </o:p></div>
</div>
<br>
______________________________<wbr>______________________________<wbr>_____________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
______________________________<wbr>______________________________<wbr>_____________<br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" style="color: rgb(149, 79, 114); text-decoration: underline;">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" style="color: rgb(149, 79, 114); text-decoration: underline;">https://lists.mpich.org/mailman/listinfo/discuss</a></div>
</blockquote>
</div>
<br>
</div>
</body>
</html>