<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helv;
panose-1:2 11 6 4 2 2 2 3 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Cambria","serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Sangmin.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">I updated to mpich3 and getting the following error<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Fatal error in MPI_Recv: A process has failed, error stack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPI_Recv(187).............: MPI_Recv(buf=0x7fff93840c30, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff94444f20) failed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dequeue_and_set_error(865): Communication error with rank 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">rank 1 in job 1 dfw-camx_55000 caused collective abort of all ranks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> exit status of rank 1: killed by signal 9<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Same situation, successful runs for smaller resource runs and for up to 7 processes. Error at more than 7. Here is the mpich command I am using to run from my job file…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">cat << ieof > nodes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx:1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n1:1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n2:1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n3:1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n4:1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n5:1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n6:1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">dfw-camx-n7:1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">ieof<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">set NUMPROCS =
<span style="color:red">8</span><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">set RING = `wc -l nodes | awk '{print $1}'`<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">mpdboot -n $RING -f nodes –verbose<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">if( ! { mpiexec -machinefile nodes -np $NUMPROCS $EXEC } ) then<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> mpdallexit<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> exit<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">endif<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">For a successful run the NUMPROCS has to be < = 7.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Any help is much appreciated.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Thank You<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Abhishek<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">………………………………………………………………………………………………….<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Senior Consultant<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#004080"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Seo, Sangmin [mailto:sseo@anl.gov]
<br>
<b>Sent:</b> Friday, September 12, 2014 1:11 PM<br>
<b>To:</b> <discuss@mpich.org><br>
<b>Subject:</b> Re: [mpich-discuss] Error Running MPICH for Photochemical Modeling<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Abhishek, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Can you try with the recent MPICH release to see if the same error happens? You can download the recent release, 3.1.2, from <a href="http://www.mpich.org/downloads/">http://www.mpich.org/downloads/</a>.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Sangmin<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Sep 12, 2014, at 12:59 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com">abhat@trinityconsultants.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">I am running a photochemical modeling on Linux cluster (CentOS_64 bit) with 1 master and 8 slave nodes with quad core (intel i7) on each node. I have two scenarios, in first
scenario, I am running less data intensive run on all 8 nodes (NUMPROCS = 9) and the run will go fine. When running same configuration for a more intense run, I am getting following error.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Fatal error in MPI_Recv: Other MPI error, error stack:</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPI_Recv(187).....................: MPI_Recv(buf=0x7fff989d53b0, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff995d96a0) failed</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPIDI_CH3I_Progress(150)..........:</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPID_nem_mpich2_blocking_recv(948):</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPID_nem_tcp_connpoll(1720).......:</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">state_commrdy_handler(1556).......:</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">MPID_nem_tcp_recv_handler(1446)...: socket closed</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">rank 1 in job 1 dfw-camx_55000 caused collective abort of all ranks</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> exit status of rank 1: killed by signal 9</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">If I run the program with smaller nodes (smaller than 7 NUMPROCS) the run goes fine.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">It appears that the rank 1 (my first node) is collectively causing all the ranks, but I could identify why. I tried following solutions –</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Cambria","serif"">1.</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Cambria","serif"">Increased
master memory to 32 gb</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Cambria","serif"">2.</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Cambria","serif"">Increased
all nodes memory to 32 gb</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Cambria","serif"">3.</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Cambria","serif"">Exchanged
the rank 1 to different node in the parallel.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">In all situations, I am getting this error. Surprisingly, when I am running smaller (less data intensive runs), I am not getting this error even if I increase the NUMPROCS to
32 processes.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Any help will be highly appreciated.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">I am running mpich 1.4</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Thank You<br>
Abhishek</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">………………………………………………………………………………………………….</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Senior Consultant</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.7pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#004080">Trinity Consultants</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:6.0pt;margin-left:.7pt">
<span style="font-size:11.0pt;font-family:"Cambria","serif"">12770 Merit Drive, Suite 900 | Dallas, Texas 75251</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Office: <span class="apple-converted-space"> </span><b><span style="color:#C20000">972-661-8100</span></b>| Mobile: 806-281-7617</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Email: <span class="apple-converted-space"> </span><a href="mailto:abhat@trinityconsultants.com"><span style="color:#0563C1">abhat@trinityconsultants.com</span></a><u><span style="color:#004080"> </span></u> |
LinkedIn: <a href="http://www.linkedin.com/in/abhattrinityconsultants"><span style="color:#0563C1">www.linkedin.com/in/abhattrinityconsultants</span></a></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Stay current on environmental issues. <span class="apple-converted-space"> </span><a href="http://www.trinityconsultants.com/Subscribe/"><span style="color:#004080">Subscribe</span></a><span class="apple-converted-space"> </span>today
to receive Trinity's free<span class="apple-converted-space"> </span><a href="http://www.trinityconsultants.com/EnvironmentalQuarterly/"><i><span style="color:#004080">Environmental Quarterly</span></i></a>.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif"">Learn about Trinity’s<span class="apple-converted-space"> </span><a href="http://www.trinityconsultants.com/Training/"><span style="color:#004080">courses</span></a><span class="apple-converted-space"> </span>for
environmental professionals.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><a href="http://www.linkedin.com/company/trinity-consultants"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><image001.gif></span></a></span><span style="font-size:11.0pt;font-family:"Cambria","serif""> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><a href="http://www.facebook.com/TrinityConsults"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><image002.gif></span></a></span><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><a href="http://twitter.com/trinityconsults"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><image003.gif></span></a></span><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><a href="http://www.youtube.com/trinityconsultants"><span style="font-family:"Cambria","serif";color:#0563C1;text-decoration:none"><image004.gif></span></a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><image005.jpg></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org"><span style="color:#954F72">discuss@mpich.org</span></a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss"><span style="color:#954F72">https://lists.mpich.org/mailman/listinfo/discuss</span></a><o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>
<br>
______________________________<WBR>______________________________<WBR>_____________<br/><br/>The information transmitted is intended only for the person or entity to<br/>which it is addressed and may contain confidential and/or privileged<br/>material. Any review, retransmission, dissemination or other use of, or<br/>taking of any action in reliance upon, this information by persons or<br/>entities other than the intended recipient is prohibited. If you received<br/>this in error, please contact the sender and delete the material from any<br/>computer.<br/>______________________________<WBR>______________________________<WBR>_____________<br/>