<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helv;
panose-1:2 11 6 4 2 2 2 3 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Cambria",serif;
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Sangmin,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">All the firewalls are off. This might be a very basic question but do I need to install MPICH libraries on all nodes even if I have installed MPICH (including libraries) on the
master in a shared file between all nodes?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Thank You<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Abhishek<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">………………………………………………………………………………………………….<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><b><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">Senior Consultant<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75pt;text-autospace:none"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#004080"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Seo, Sangmin [mailto:sseo@anl.gov]
<br>
<b>Sent:</b> Wednesday, February 04, 2015 1:41 PM<br>
<b>To:</b> <discuss@mpich.org><br>
<b>Subject:</b> Re: [mpich-discuss] MPICH3 Problem<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Can you also check whether the firewalls are turned off on all nodes, as described in <a href="https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_program_aborts_with_an_error_saying_it_cannot_communicate_with_other_processes">https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_program_aborts_with_an_error_saying_it_cannot_communicate_with_other_processes</a>?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">— Sangmin<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Feb 4, 2015, at 12:55 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com">abhat@trinityconsultants.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Sangmin,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">All nodes can communicate with each other with ssh. I ran the MPI example program and here is the error I got</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[12:52:46] [clusteruser@Earth examples]$ mpiexec -n 5 -f machinefile ./cpi</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Process 0 of 5 is on Earth</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Process 1 of 5 is on Earth</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Process 4 of 5 is on Earth</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Process 2 of 5 is on node1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Process 3 of 5 is on node1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Fatal error in PMPI_Reduce: Unknown error class, error stack:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">PMPI_Reduce(1263)...............: MPI_Reduce(sbuf=0x7fff3bea9148, rbuf=0x7fff3bea9140, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPIR_Reduce_impl(1075)..........:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPIR_Reduce_intra(826)..........:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPIR_Reduce_impl(1075)..........:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPIR_Reduce_intra(881)..........:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPIR_Reduce_binomial(188).......:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPIDI_CH3U_Recvq_FDU_or_AEP(636): Communication error with rank 2</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPIR_Reduce_intra(846)..........:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPIR_Reduce_impl(1075)..........:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPIR_Reduce_intra(881)..........:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPIR_Reduce_binomial(250).......: Failure during collective</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">===================================================================================</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">= PID 11705 RUNNING AT Earth</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">= EXIT CODE: 1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">= CLEANING UP REMAINING PROCESSES</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">===================================================================================</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[proxy:0:1@node1] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[proxy:0:1@node1] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[proxy:0:1@node1] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Thank You</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Abhishek</span><o:p></o:p></p>
</div>
<div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">………………………………………………………………………………………………….</span><o:p></o:p></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">Senior Consultant</span><o:p></o:p></p>
</div>
<div style="margin-left:.75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#004080"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Seo,
Sangmin [<a href="mailto:sseo@anl.gov">mailto:sseo@anl.gov</a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Tuesday, February 03, 2015 6:26 PM<br>
<b>To:</b><span class="apple-converted-space"> </span><<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [mpich-discuss] MPICH3 Problem</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Can all nodes that you are using do ssh each other? And, can you also try to run an MPICH example, cpi, located in <mpich_top_dir>/examples with the same nodes to see if you run into the same error?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">— Sangmin<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">On Feb 3, 2015, at 6:06 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com"><span style="color:purple">abhat@trinityconsultants.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">So, after installing the PGI fortran on the shared drive, I re-ran and now I am getting communication error –</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Model startup ......Fatal error in MPI_Send: Unknown error class, error stack:</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPI_Send(174)..............: MPI_Send(buf=0x1983bef8, count=1, MPI_INTEGER, dest=2, tag=1, MPI_COMM_WORLD) failed</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">MPID_nem_tcp_connpoll(1832): Communication error with rank 2: Connection timed out</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">………………………………………………………………………………………………….</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">Senior Consultant</span><o:p></o:p></p>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#004080"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Seo,
Sangmin [<a href="mailto:sseo@anl.gov"><span style="color:purple">mailto:sseo@anl.gov</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Tuesday, February 03, 2015 5:57 PM<br>
<b>To:</b><span class="apple-converted-space"> </span><a href="mailto:discuss@mpich.org"><span style="color:purple">discuss@mpich.org</span></a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [mpich-discuss] MPICH3 Problem</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Correct. libpgmp.so should be from your PGI compiler installation.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">— Sangmin<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">On Feb 3, 2015, at 5:53 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com"><span style="color:purple">abhat@trinityconsultants.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Sangmin,</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">I have installed MPICH on shared drive but PGI fortran is installed on /opt/pgi which node do not have access. I am assuming that is the issue currently. I am trying to re-install
PGI on the shared drive to see if that will fix the problem.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Just to confirm libpgmp.so is not mpich file correct?</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Thank You</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Abhishek</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div style="margin-left:.75pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">………………………………………………………………………………………………….</span><o:p></o:p></p>
</div>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">Senior Consultant</span><o:p></o:p></p>
</div>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#004080"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Seo,
Sangmin [<a href="mailto:sseo@anl.gov"><span style="color:purple">mailto:sseo@anl.gov</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Tuesday, February 03, 2015 5:50 PM<br>
<b>To:</b><span class="apple-converted-space"> </span><<a href="mailto:discuss@mpich.org"><span style="color:purple">discuss@mpich.org</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [mpich-discuss] MPICH3 Problem</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hi Abhishek,<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">As the error message says, it looks that the node running the application doesn’t have libpgmp.so. Can you confirm whether the node has libpgmp.so and whether LD_LIBRARY_PATH is correctly set on the node in case that it has libpgmp.so?<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Best regards,<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Sangmin<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal">On Feb 3, 2015, at 5:31 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com"><span style="color:purple">abhat@trinityconsultants.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">HI All,</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">I installed MPICH3 on a master terminal along with PGI Fortran. Then used MPICH3 and PGI to compile my software. When I run the program on the master terminal only I do not have
any error messages but when I am trying to run them on one of the nodes, I am getting following message error –</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">/home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">/home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">===================================================================================</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">= PID 5200 RUNNING AT node3</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">= EXIT CODE: 127</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">= CLEANING UP REMAINING PROCESSES</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">===================================================================================</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[proxy:0:0@Earth] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[proxy:0:0@Earth] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[proxy:0:0@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">[mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">The /home/Earth is shared and mapped on all nodes</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Any help is much appreciated.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Thank You</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif">Abhishek</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">………………………………………………………………………………………………….</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div style="margin-left:.75pt">
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">Abhishek Bhat, PhD, EPI,<br>
</span></b><span style="font-size:11.0pt;font-family:"Cambria",serif;color:#004080">Senior Consultant</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<br>
_______________________________________________<br>
discuss mailing list </span><a href="mailto:discuss@mpich.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#954F72">discuss@mpich.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
To manage subscription options or unsubscribe:<br>
</span><a href="https://lists.mpich.org/mailman/listinfo/discuss"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#954F72">https://lists.mpich.org/mailman/listinfo/discuss</span></a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<br>
_______________________________________________<br>
discuss mailing list </span><a href="mailto:discuss@mpich.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:purple">discuss@mpich.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
To manage subscription options or unsubscribe:<br>
</span><a href="https://lists.mpich.org/mailman/listinfo/discuss"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:purple">https://lists.mpich.org/mailman/listinfo/discuss</span></a><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<br>
_______________________________________________<br>
discuss mailing list </span><a href="mailto:discuss@mpich.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:purple">discuss@mpich.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
To manage subscription options or unsubscribe:<br>
</span><a href="https://lists.mpich.org/mailman/listinfo/discuss"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:purple">https://lists.mpich.org/mailman/listinfo/discuss</span></a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<br>
_______________________________________________<br>
discuss mailing list </span><a href="mailto:discuss@mpich.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:purple">discuss@mpich.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
To manage subscription options or unsubscribe:<br>
</span><a href="https://lists.mpich.org/mailman/listinfo/discuss"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:purple">https://lists.mpich.org/mailman/listinfo/discuss</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>
<br>
______________________________<wbr>______________________________<wbr>_____________<br><br>The information transmitted is intended only for the person or entity to<br>which it is addressed and may contain confidential and/or privileged<br>material. Any review, retransmission, dissemination or other use of, or<br>taking of any action in reliance upon, this information by persons or<br>entities other than the intended recipient is prohibited. If you received<br>this in error, please contact the sender and delete the material from any<br>computer.<br>______________________________<wbr>______________________________<wbr>_____________<br>