<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helv;
panose-1:2 11 6 4 2 2 2 3 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Cambria","serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1513304433;
mso-list-type:hybrid;
mso-list-template-ids:-42972234 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">I am running a photochemical modeling on Linux cluster (CentOS_64 bit) with 1 master and 8 slave nodes with quad core (intel i7) on each node. I have two scenarios, in first scenario, I am running
less data intensive run on all 8 nodes (NUMPROCS = 9) and the run will go fine. When running same configuration for a more intense run, I am getting following error.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">Fatal error in MPI_Recv: Other MPI error, error stack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">MPI_Recv(187).....................: MPI_Recv(buf=0x7fff989d53b0, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff995d96a0) failed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">MPIDI_CH3I_Progress(150)..........:
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">MPID_nem_mpich2_blocking_recv(948):
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">MPID_nem_tcp_connpoll(1720).......:
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">state_commrdy_handler(1556).......:
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">MPID_nem_tcp_recv_handler(1446)...: socket closed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">rank 1 in job 1 dfw-camx_55000 caused collective abort of all ranks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif""> exit status of rank 1: killed by signal 9<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">If I run the program with smaller nodes (smaller than 7 NUMPROCS) the run goes fine.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">It appears that the rank 1 (my first node) is collectively causing all the ranks, but I could identify why. I tried following solutions –
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:"Cambria","serif""><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-family:"Cambria","serif"">Increased master memory to 32 gb<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:"Cambria","serif""><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-family:"Cambria","serif"">Increased all nodes memory to 32 gb<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:"Cambria","serif""><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-family:"Cambria","serif"">Exchanged the rank 1 to different node in the parallel.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">In all situations, I am getting this error. Surprisingly, when I am running smaller (less data intensive runs), I am not getting this error even if I increase the NUMPROCS to 32 processes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">Any help will be highly appreciated.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">I am running mpich 1.4
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif"">Thank You<br>
Abhishek<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-family:"Cambria","serif";color:#004080">………………………………………………………………………………………………….<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><b><span style="font-family:"Cambria","serif";color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-family:"Cambria","serif";color:#004080">Senior Consultant<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-family:"Cambria","serif";color:#004080"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.7pt;text-autospace:none"><b><span style="font-family:"Cambria","serif";color:#004080">Trinity Consultants</span></b><span style="font-family:"Cambria","serif";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:6.0pt;margin-left:.7pt;text-autospace:none">
<span style="font-family:"Cambria","serif";color:black">12770 Merit Drive, Suite 900 | Dallas, Texas 75251<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-family:"Cambria","serif";color:black">Office:
</span><b><span style="font-family:"Cambria","serif";color:#C20000">972-661-8100</span></b><span style="font-family:"Cambria","serif";color:black">| Mobile: 806-281-7617<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-family:"Cambria","serif";color:black">Email:
</span><span style="font-family:"Cambria","serif""><a href="mailto:abhat@trinityconsultants.com"><span style="color:#0563C1">abhat@trinityconsultants.com</span></a></span><u><span style="font-family:"Cambria","serif";color:#004080">
</span></u><span style="font-family:"Cambria","serif";color:black"> | LinkedIn:
</span><span style="font-family:"Cambria","serif""><a href="http://www.linkedin.com/in/abhattrinityconsultants"><span style="color:#0563C1">www.linkedin.com/in/abhattrinityconsultants</span></a><span style="color:black"><o:p></o:p></span></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-family:"Cambria","serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-family:"Cambria","serif";color:black">Stay current on environmental issues.
</span><span style="font-family:"Cambria","serif""><a href="http://www.trinityconsultants.com/Subscribe/"><span style="color:#004080">Subscribe</span></a></span><span style="font-family:"Cambria","serif";color:black"> today to receive Trinity's free
</span><span style="font-family:"Cambria","serif""><a href="http://www.trinityconsultants.com/EnvironmentalQuarterly/"><i><span style="color:#004080">Environmental Quarterly</span></i></a></span><span style="font-family:"Cambria","serif";color:black">.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-family:"Cambria","serif";color:black">Learn about Trinity’s
</span><span style="font-family:"Cambria","serif""><a href="http://www.trinityconsultants.com/Training/"><span style="color:#004080">courses</span></a></span><span style="font-family:"Cambria","serif";color:black"> for environmental professionals.
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-family:"Cambria","serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><a href="http://www.linkedin.com/company/trinity-consultants"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><img border="0" width="23" height="23" id="_x0000_i1029" src="cid:image001.gif@01CFCE89.5805A4C0" alt="LinkedIn icon_23p"></span></a><span style="font-family:"Cambria","serif"">
</span><a href="http://www.facebook.com/TrinityConsults"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><img border="0" width="23" height="23" id="Picture_x0020_4" src="cid:image002.gif@01CFCE89.5805A4C0" alt="Facebook icon_23p"></span></a><span style="font-family:"Cambria","serif""> </span><a href="http://twitter.com/trinityconsults"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><img border="0" width="23" height="23" id="Picture_x0020_3" src="cid:image003.gif@01CFCE89.5805A4C0" alt="Twitter icon_23p"></span></a><span style="font-family:"Cambria","serif""> </span><a href="http://www.youtube.com/trinityconsultants"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><img border="0" width="23" height="23" id="Picture_x0020_2" src="cid:image004.gif@01CFCE89.5805A4C0" alt="YouTube icon_23p"></span></a><span style="font-family:"Cambria","serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><img border="0" width="238" height="59" id="Picture_x0020_5" src="cid:image005.jpg@01CFCE89.5805A4C0" alt="https://corporate.trinityconsultants.com/Departments/Marketing/Community%20Shared%20Library/Logos/TCI_40%20Yr%20Logo.jpg"></span><span style="font-family:"Cambria","serif""><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>
<br>
______________________________<WBR>______________________________<WBR>_____________<br/><br/>The information transmitted is intended only for the person or entity to<br/>which it is addressed and may contain confidential and/or privileged<br/>material. Any review, retransmission, dissemination or other use of, or<br/>taking of any action in reliance upon, this information by persons or<br/>entities other than the intended recipient is prohibited. If you received<br/>this in error, please contact the sender and delete the material from any<br/>computer.<br/>______________________________<WBR>______________________________<WBR>_____________<br/>