<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helv;
panose-1:2 11 6 4 2 2 2 3 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Cambria","serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Cambria","serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Any luck on finding solution on the error in the log file provided?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Thank You<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Abhishek<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">………………………………………………………………………………………………….<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Senior Consultant<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#004080"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Abhishek Bhat
<br>
<b>Sent:</b> Tuesday, September 16, 2014 11:36 AM<br>
<b>To:</b> <discuss@mpich.org><br>
<b>Cc:</b> 'sseo@anl.gov'<br>
<b>Subject:</b> RE: [mpich-discuss] Error Running MPICH for Photochemical Modeling<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Sangmin,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Sorry for the confusion, I was looking in the wrong location. Please find attached the splice of the debug file 0 where the Error is occurring.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Thank You<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Abhishek<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">………………………………………………………………………………………………….<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Senior Consultant<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#004080"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Seo, Sangmin [<a href="mailto:sseo@anl.gov">mailto:sseo@anl.gov</a>]
<br>
<b>Sent:</b> Tuesday, September 16, 2014 11:12 AM<br>
<b>To:</b> Abhishek Bhat<br>
<b>Cc:</b> <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Subject:</b> Re: [mpich-discuss] Error Running MPICH for Photochemical Modeling<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">The log files should be found in the directory where you execute the application. Did you run the application with debug options, -mpich-dbg=file -mpich-dbg-class=all -mpich-dbg-level=verbose?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">— Sangmin<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Sep 16, 2014, at 11:05 AM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com">abhat@trinityconsultants.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Sangmin,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">I configured the MPICH with revised code and included the additional options as described. Where should I be looking for the log files?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">I cannot see any files generated.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Thank You<br>
Abhishek</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
<div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">………………………………………………………………………………………………….</span><o:p></o:p></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Senior Consultant</span><o:p></o:p></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#004080"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Seo,
Sangmin [<a href="mailto:sseo@anl.gov"><span style="color:purple">mailto:sseo@anl.gov</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Monday, September 15, 2014 12:43 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Abhishek Bhat<br>
<b>Cc:</b><span class="apple-converted-space"> </span><<a href="mailto:discuss@mpich.org"><span style="color:purple">discuss@mpich.org</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [mpich-discuss] Error Running MPICH for Photochemical Modeling</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Sorry, “./make” should have been “make”<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">— Sangmin<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">On Sep 15, 2014, at 12:40 PM, Seo, Sangmin <<a href="mailto:sseo@anl.gov"><span style="color:purple">sseo@anl.gov</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal">Did you run dmesg on the node which executed the rank failed? I don't see any error message from your log file.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Anyway, let's try with MPICH DEBUG messages. As described in<a href="https://wiki.mpich.org/mpich/index.php/Debug_Event_Logging"><span style="color:purple">https://wiki.mpich.org/mpich/index.php/Debug_Event_Logging</span></a>, first configure
MPICH with "--enable-g=dbg,log" to enable event logging and built it. For example,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">$ ./configure --prefix=$INSTALL_DIR --enable-g=dbg,log<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">$ ./make -j8<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">$ ./make install<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Then, execute your application with additional options, "-mpich-dbg=file -mpich-dbg-class=all -mpich-dbg-level=verbose." With the recent MPICH, you don't need to use mpdboot. Just execute your application with mpiexec like:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">mpiexec -machinefile nodes -np $NUMPROCS $EXEC -mpich-dbg=file -mpich-dbg-class=all -mpich-dbg-level=verbose<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">This will generate log files, such as dbg0-xxxx.log, dbg1-xxxx.log, etc. Since the log files may be large, it would be difficult to send them to us. If so, please take a look at each file, find error messages, and send the error message
to us.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">And, if you can reproduce the same error with small code, it will be really helpful to figure out your problem. Could you write a simplified code and send us the code?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Sangmin<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Sep 15, 2014, at 10:58 AM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com"><span style="color:purple">abhat@trinityconsultants.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Sangmin,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Please see attached the logs from demsg. I apologies but I am not a computer expert so it’s all Greek to me.. Can you please see if you can find any error or reason for the failure?</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Thank You</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Abhishek</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">………………………………………………………………………………………………….</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Senior Consultant</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#004080"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Seo,
Sangmin [<a href="mailto:sseo@anl.gov"><span style="color:purple">mailto:sseo@anl.gov</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Monday, September 15, 2014 9:21 AM<br>
<b>To:</b><span class="apple-converted-space"> </span>Abhishek Bhat<br>
<b>Cc:</b><span class="apple-converted-space"> </span><<a href="mailto:discuss@mpich.org"><span style="color:purple">discuss@mpich.org</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [mpich-discuss] Error Running MPICH for Photochemical Modeling</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Can you run dmesg on the node of rank 1, which is killed by signal 9, after you execute your application? You can find the reason that the process is killed at the end of dmesg output, e.g., out of memory.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">— Sangmin<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Sep 14, 2014, at 12:37 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com"><span style="color:purple">abhat@trinityconsultants.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Because the application works when less intensive runs and fails for more intensive runs, it is likely that the application is requesting too many resources. When\where should
I run ulimit –a and dmesg, after I get the error? If that is true, is there any way to change the environment in MPI to increase the capacity so that the increased resources can be accommodated?</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">If I run it in new terminal – here is what I get</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">core file size (blocks, -c) 0</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">data seg size (kbytes, -d) unlimited</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">scheduling priority (-e) 0</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">file size (blocks, -f) unlimited</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">pending signals (-i) 250598</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">max locked memory (kbytes, -l) 64</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">max memory size (kbytes, -m) unlimited</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">open files (-n) 1024</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">pipe size (512 bytes, -p) 8</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">POSIX message queues (bytes, -q) 819200</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">real-time priority (-r) 0</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">stack size (kbytes, -s) 10240</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">cpu time (seconds, -t) unlimited</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">max user processes (-u) 1024</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">virtual memory (kbytes, -v) unlimited</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">file locks (-x) unlimited</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">in my job, I try to set the stack size to – unlimited but I guess it is not working.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Let me know. Thank you for all the help.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Abhishek</span><o:p></o:p></p>
</div>
</div>
<div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">………………………………………………………………………………………………….</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Senior Consultant</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#004080"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Seo,
Sangmin [<a href="mailto:sseo@anl.gov"><span style="color:purple">mailto:sseo@anl.gov</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Sunday, September 14, 2014 11:16 AM<br>
<b>To:</b><span class="apple-converted-space"> </span><<a href="mailto:discuss@mpich.org"><span style="color:purple">discuss@mpich.org</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [mpich-discuss] Error Running MPICH for Photochemical Modeling</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Abhishek,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Signal 9 is caused by many reasons, e.g., CPU time, out of memory, etc., but it is mostly because the application requests too many resources. You can check the environment settings with ulimit -a. And, you may find some information about
your error from dmesg.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Sangmin<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Sep 12, 2014, at 5:51 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com"><span style="color:purple">abhat@trinityconsultants.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Sangmin.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">I updated to mpich3 and getting the following error</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Fatal error in MPI_Recv: A process has failed, error stack:</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPI_Recv(187).............: MPI_Recv(buf=0x7fff93840c30, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff94444f20) failed</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dequeue_and_set_error(865): Communication error with rank 1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">rank 1 in job 1 dfw-camx_55000 caused collective abort of all ranks</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> exit status of rank 1: killed by signal 9</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Same situation, successful runs for smaller resource runs and for up to 7 processes. Error at more than 7. Here is the mpich command I am using to run from my job file…</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">cat << ieof > nodes</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx:1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n1:1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n2:1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n3:1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n4:1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n5:1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n6:1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n7:1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">ieof</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">set NUMPROCS =<span class="apple-converted-space"> </span><span style="color:red">8</span></span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">set RING = `wc -l nodes | awk '{print $1}'`</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">mpdboot -n $RING -f nodes –verbose</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">if( ! { mpiexec -machinefile nodes -np $NUMPROCS $EXEC } ) then</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> mpdallexit</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> exit</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">endif</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">For a successful run the NUMPROCS has to be < = 7.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Any help is much appreciated.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Thank You</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Abhishek</span><o:p></o:p></p>
</div>
</div>
<div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">………………………………………………………………………………………………….</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Senior Consultant</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#004080"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Seo,
Sangmin [<a href="mailto:sseo@anl.gov"><span style="color:purple">mailto:sseo@anl.gov</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Friday, September 12, 2014 1:11 PM<br>
<b>To:</b><span class="apple-converted-space"> </span><<a href="mailto:discuss@mpich.org"><span style="color:purple">discuss@mpich.org</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [mpich-discuss] Error Running MPICH for Photochemical Modeling</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Hi Abhishek,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Can you try with the recent MPICH release to see if the same error happens? You can download the recent release, 3.1.2, from <a href="http://www.mpich.org/downloads/"><span style="color:purple">http://www.mpich.org/downloads/</span></a>.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Sangmin<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Sep 12, 2014, at 12:59 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com"><span style="color:purple">abhat@trinityconsultants.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">I am running a photochemical modeling on Linux cluster (CentOS_64 bit) with 1 master and 8 slave nodes with quad core (intel i7) on each node. I have two scenarios, in first
scenario, I am running less data intensive run on all 8 nodes (NUMPROCS = 9) and the run will go fine. When running same configuration for a more intense run, I am getting following error.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Fatal error in MPI_Recv: Other MPI error, error stack:</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPI_Recv(187).....................: MPI_Recv(buf=0x7fff989d53b0, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff995d96a0) failed</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPIDI_CH3I_Progress(150)..........:</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPID_nem_mpich2_blocking_recv(948):</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPID_nem_tcp_connpoll(1720).......:</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">state_commrdy_handler(1556).......:</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPID_nem_tcp_recv_handler(1446)...: socket closed</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">rank 1 in job 1 dfw-camx_55000 caused collective abort of all ranks</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> exit status of rank 1: killed by signal 9</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">If I run the program with smaller nodes (smaller than 7 NUMPROCS) the run goes fine.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">It appears that the rank 1 (my first node) is collectively causing all the ranks, but I could identify why. I tried following solutions –</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.5in">
<div>
<p class="MsoNormal" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Cambria","serif"">1.</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Cambria","serif"">Increased
master memory to 32 gb</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.5in">
<div>
<p class="MsoNormal" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Cambria","serif"">2.</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Cambria","serif"">Increased
all nodes memory to 32 gb</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.5in">
<div>
<p class="MsoNormal" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Cambria","serif"">3.</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Cambria","serif"">Exchanged
the rank 1 to different node in the parallel.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">In all situations, I am getting this error. Surprisingly, when I am running smaller (less data intensive runs), I am not getting this error even if I increase the NUMPROCS to
32 processes.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Any help will be highly appreciated.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">I am running mpich 1.4</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Thank You<br>
Abhishek</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">………………………………………………………………………………………………….</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Senior Consultant</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080"> </span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.7pt">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Trinity Consultants</span></b><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:6.0pt;margin-left:.7pt">
<span style="font-size:11.0pt;font-family:"Cambria","serif"">12770 Merit Drive, Suite 900 | Dallas, Texas 75251</span><o:p></o:p></p>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Office: <span class="apple-converted-space"> </span><b><span style="color:#C20000">972-661-8100</span></b>| Mobile: 806-281-7617</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Email: <span class="apple-converted-space"> </span><a href="mailto:abhat@trinityconsultants.com"><span style="color:#0563C1">abhat@trinityconsultants.com</span></a><u><span style="color:#004080"> </span></u> |
LinkedIn: <a href="http://www.linkedin.com/in/abhattrinityconsultants"><span style="color:#0563C1">www.linkedin.com/in/abhattrinityconsultants</span></a></span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Stay current on environmental issues. <span class="apple-converted-space"> </span><a href="http://www.trinityconsultants.com/Subscribe/"><span style="color:#004080">Subscribe</span></a><span class="apple-converted-space"> </span>today
to receive Trinity's free<span class="apple-converted-space"> </span><a href="http://www.trinityconsultants.com/EnvironmentalQuarterly/"><i><span style="color:#004080">Environmental Quarterly</span></i></a>.</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Learn about Trinity’s<span class="apple-converted-space"> </span><a href="http://www.trinityconsultants.com/Training/"><span style="color:#004080">courses</span></a><span class="apple-converted-space"> </span>for
environmental professionals.</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><a href="http://www.linkedin.com/company/trinity-consultants"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><image001.gif></span></a></span><span style="font-size:11.0pt;font-family:"Cambria","serif""> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><a href="http://www.facebook.com/TrinityConsults"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><image002.gif></span></a></span><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><a href="http://twitter.com/trinityconsults"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><image003.gif></span></a></span><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><a href="http://www.youtube.com/trinityconsultants"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><image004.gif></span></a></span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><image005.jpg></span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org"><span style="color:#954F72">discuss@mpich.org</span></a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss"><span style="color:#954F72">https://lists.mpich.org/mailman/listinfo/discuss</span></a></span><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org"><span style="color:purple">discuss@mpich.org</span></a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss"><span style="color:purple">https://lists.mpich.org/mailman/listinfo/discuss</span></a></span><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________</span><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<br>
<dmesg3.txt><messages.txt></span><o:p></o:p></p>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org"><span style="color:purple">discuss@mpich.org</span></a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss"><span style="color:purple">https://lists.mpich.org/mailman/listinfo/discuss</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>
<br>
______________________________<wbr>______________________________<wbr>_____________<br><br>The information transmitted is intended only for the person or entity to<br>which it is addressed and may contain confidential and/or privileged<br>material. Any review, retransmission, dissemination or other use of, or<br>taking of any action in reliance upon, this information by persons or<br>entities other than the intended recipient is prohibited. If you received<br>this in error, please contact the sender and delete the material from any<br>computer.<br>______________________________<wbr>______________________________<wbr>_____________<br>