<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">No, you dont need to install MPICH on all nodes if it is shared and LD_LIBRARY_PATH on all nodes is set correctly for the MPICH installation.</div>
<div class=""><br class="">
</div>
<div class=""> Sangmin</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Feb 4, 2015, at 3:32 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com" class="">abhat@trinityconsultants.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Sangmin,<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">All the firewalls are off. This might be a very basic question but do I need to install MPICH libraries on all nodes even if I have installed MPICH (including libraries) on the master in
a shared file between all nodes?<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Thank You<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Abhishek<o:p class=""></o:p></span></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">
.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">Abhishek Bhat, PhD, EPI,<br class="">
</span></b><span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">Senior Consultant<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.75pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(0, 64, 128);" class=""> </span></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span></div>
<div class="">
<div style="border-style: solid none none; border-top-color: rgb(225, 225, 225); border-top-width: 1pt; padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span>Seo, Sangmin [<a href="mailto:sseo@anl.gov" class="">mailto:sseo@anl.gov</a>]<span class="Apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, February 04, 2015 1:41 PM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span><<a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [mpich-discuss] MPICH3 Problem<o:p class=""></o:p></span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Can you also check whether the firewalls are turned off on all nodes, as described in <a href="https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_program_aborts_with_an_error_saying_it_cannot_communicate_with_other_processes" style="color: purple; text-decoration: underline;" class="">https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_program_aborts_with_an_error_saying_it_cannot_communicate_with_other_processes</a>?<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Sangmin<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
On Feb 4, 2015, at 12:55 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com" style="color: purple; text-decoration: underline;" class="">abhat@trinityconsultants.com</a>> wrote:<o:p class=""></o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Sangmin,</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">All nodes can communicate with each other with ssh. I ran the MPI example program and here is the error I got</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[12:52:46] [clusteruser@Earth examples]$ mpiexec -n 5 -f machinefile ./cpi</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Process 0 of 5 is on Earth</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Process 1 of 5 is on Earth</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Process 4 of 5 is on Earth</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Process 2 of 5 is on node1</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Process 3 of 5 is on node1</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Fatal error in PMPI_Reduce: Unknown error class, error stack:</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">PMPI_Reduce(1263)...............: MPI_Reduce(sbuf=0x7fff3bea9148, rbuf=0x7fff3bea9140, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPIR_Reduce_impl(1075)..........:</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPIR_Reduce_intra(826)..........:</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPIR_Reduce_impl(1075)..........:</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPIR_Reduce_intra(881)..........:</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPIR_Reduce_binomial(188).......:</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPIDI_CH3U_Recvq_FDU_or_AEP(636): Communication error with rank 2</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPIR_Reduce_intra(846)..........:</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPIR_Reduce_impl(1075)..........:</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPIR_Reduce_intra(881)..........:</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPIR_Reduce_binomial(250).......: Failure during collective</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">===================================================================================</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">= PID 11705 RUNNING AT Earth</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">= EXIT CODE: 1</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">= CLEANING UP REMAINING PROCESSES</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">===================================================================================</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[proxy:0:1@node1] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[proxy:0:1@node1] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[proxy:0:1@node1] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Thank You</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Abhishek</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin-left: 0.75pt;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">
.</span><o:p class=""></o:p></div>
</div>
<div style="margin-left: 0.75pt;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">Abhishek Bhat, PhD, EPI,<br class="">
</span></b><span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">Senior Consultant</span><o:p class=""></o:p></div>
</div>
<div style="margin-left: 0.75pt;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(0, 64, 128);" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="border-style: solid none none; border-top-color: rgb(225, 225, 225); border-top-width: 1pt; padding: 3pt 0in 0in;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span class="apple-converted-space"><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Seo,
Sangmin [<a href="mailto:sseo@anl.gov" style="color: purple; text-decoration: underline;" class="">mailto:sseo@anl.gov</a>]<span class="apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="apple-converted-space"> </span>Tuesday, February 03, 2015 6:26 PM<br class="">
<b class="">To:</b><span class="apple-converted-space"> </span><<a href="mailto:discuss@mpich.org" style="color: purple; text-decoration: underline;" class="">discuss@mpich.org</a>><br class="">
<b class="">Subject:</b><span class="apple-converted-space"> </span>Re: [mpich-discuss] MPICH3 Problem</span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Can all nodes that you are using do ssh each other? And, can you also try to run an MPICH example, cpi, located in <mpich_top_dir>/examples with the same nodes to see if you run into the same error?<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Sangmin<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
On Feb 3, 2015, at 6:06 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">abhat@trinityconsultants.com</span></a>> wrote:<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">So, after installing the PGI fortran on the shared drive, I re-ran and now I am getting communication error </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Model startup ......Fatal error in MPI_Send: Unknown error class, error stack:</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPI_Send(174)..............: MPI_Send(buf=0x1983bef8, count=1, MPI_INTEGER, dest=2, tag=1, MPI_COMM_WORLD) failed</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">MPID_nem_tcp_connpoll(1832): Communication error with rank 2: Connection timed out</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin-left: 0.75pt;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">
.</span><o:p class=""></o:p></div>
</div>
</div>
<div style="margin-left: 0.75pt;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">Abhishek Bhat, PhD, EPI,<br class="">
</span></b><span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">Senior Consultant</span><o:p class=""></o:p></div>
</div>
</div>
<div style="margin-left: 0.75pt;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(0, 64, 128);" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="border-style: solid none none; border-top-color: rgb(225, 225, 225); border-top-width: 1pt; padding: 3pt 0in 0in;" class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span class="apple-converted-space"><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Seo,
Sangmin [<a href="mailto:sseo@anl.gov" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">mailto:sseo@anl.gov</span></a>]<span class="apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="apple-converted-space"> </span>Tuesday, February 03, 2015 5:57 PM<br class="">
<b class="">To:</b><span class="apple-converted-space"> </span><a href="mailto:discuss@mpich.org" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">discuss@mpich.org</span></a><br class="">
<b class="">Subject:</b><span class="apple-converted-space"> </span>Re: [mpich-discuss] MPICH3 Problem</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Correct. libpgmp.so should be from your PGI compiler installation.<o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Sangmin<o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
On Feb 3, 2015, at 5:53 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">abhat@trinityconsultants.com</span></a>> wrote:<o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Sangmin,</span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">I have installed MPICH on shared drive but PGI fortran is installed on /opt/pgi which node do not have access. I am assuming that is the issue currently. I am trying to re-install PGI on
the shared drive to see if that will fix the problem.</span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Just to confirm libpgmp.so is not mpich file correct?</span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Thank You</span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Abhishek</span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div style="margin-left: 0.75pt;" class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">
.</span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div style="margin-left: 0.75pt;" class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">Abhishek Bhat, PhD, EPI,<br class="">
</span></b><span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">Senior Consultant</span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div style="margin-left: 0.75pt;" class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(0, 64, 128);" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div style="border-style: solid none none; border-top-color: rgb(225, 225, 225); border-top-width: 1pt; padding: 3pt 0in 0in;" class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span class="apple-converted-space"><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Seo,
Sangmin [<a href="mailto:sseo@anl.gov" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">mailto:sseo@anl.gov</span></a>]<span class="apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="apple-converted-space"> </span>Tuesday, February 03, 2015 5:50 PM<br class="">
<b class="">To:</b><span class="apple-converted-space"> </span><<a href="mailto:discuss@mpich.org" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">discuss@mpich.org</span></a>><br class="">
<b class="">Subject:</b><span class="apple-converted-space"> </span>Re: [mpich-discuss] MPICH3 Problem</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Hi Abhishek,<o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
As the error message says, it looks that the node running the application doesnt have libpgmp.so. Can you confirm whether the node has libpgmp.so and whether LD_LIBRARY_PATH is correctly set on the node in case that it has libpgmp.so?<o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Best regards,<o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Sangmin<o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
On Feb 3, 2015, at 5:31 PM, Abhishek Bhat <<a href="mailto:abhat@trinityconsultants.com" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">abhat@trinityconsultants.com</span></a>> wrote:<o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">HI All,</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">I installed MPICH3 on a master terminal along with PGI Fortran. Then used MPICH3 and PGI to compile my software. When I run the program on the master terminal only I do not have any error
messages but when I am trying to run them on one of the nodes, I am getting following message error </span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">/home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">/home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">===================================================================================</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">= PID 5200 RUNNING AT node3</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">= EXIT CODE: 127</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">= CLEANING UP REMAINING PROCESSES</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">===================================================================================</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[proxy:0:0@Earth] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[proxy:0:0@Earth] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[proxy:0:0@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">[mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">The /home/Earth is shared and mapped on all nodes</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Any help is much appreciated.</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Thank You</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif;" class="">Abhishek</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div style="margin-left: 0.75pt;" class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">
.</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div style="margin-left: 0.75pt;" class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">Abhishek Bhat, PhD, EPI,<br class="">
</span></b><span style="font-size: 11pt; font-family: Cambria, serif; color: rgb(0, 64, 128);" class="">Senior Consultant</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">
_________________________________________________________________________<br class="">
<br class="">
The information transmitted is intended only for the person or entity to<br class="">
which it is addressed and may contain confidential and/or privileged<br class="">
material. Any review, retransmission, dissemination or other use of, or<br class="">
taking of any action in reliance upon, this information by persons or<br class="">
entities other than the intended recipient is prohibited. If you received<br class="">
this in error, please contact the sender and delete the material from any<br class="">
computer.<br class="">
_________________________________________________________________________<br class="">
_______________________________________________<br class="">
discuss mailing list </span><a href="mailto:discuss@mpich.org" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: rgb(149, 79, 114);" class="">discuss@mpich.org</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">
To manage subscription options or unsubscribe:<br class="">
</span><a href="https://lists.mpich.org/mailman/listinfo/discuss" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: rgb(149, 79, 114);" class="">https://lists.mpich.org/mailman/listinfo/discuss</span></a><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">
_________________________________________________________________________<br class="">
<br class="">
The information transmitted is intended only for the person or entity to<br class="">
which it is addressed and may contain confidential and/or privileged<br class="">
material. Any review, retransmission, dissemination or other use of, or<br class="">
taking of any action in reliance upon, this information by persons or<br class="">
entities other than the intended recipient is prohibited. If you received<br class="">
this in error, please contact the sender and delete the material from any<br class="">
computer.<br class="">
_________________________________________________________________________<br class="">
_______________________________________________<br class="">
discuss mailing list </span><a href="mailto:discuss@mpich.org" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;" class="">discuss@mpich.org</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">
To manage subscription options or unsubscribe:<br class="">
</span><a href="https://lists.mpich.org/mailman/listinfo/discuss" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;" class="">https://lists.mpich.org/mailman/listinfo/discuss</span></a><o:p class=""></o:p></div>
</div>
</div>
</div>
</blockquote>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">
_________________________________________________________________________<br class="">
<br class="">
The information transmitted is intended only for the person or entity to<br class="">
which it is addressed and may contain confidential and/or privileged<br class="">
material. Any review, retransmission, dissemination or other use of, or<br class="">
taking of any action in reliance upon, this information by persons or<br class="">
entities other than the intended recipient is prohibited. If you received<br class="">
this in error, please contact the sender and delete the material from any<br class="">
computer.<br class="">
_________________________________________________________________________<br class="">
_______________________________________________<br class="">
discuss mailing list </span><a href="mailto:discuss@mpich.org" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;" class="">discuss@mpich.org</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">
To manage subscription options or unsubscribe:<br class="">
</span><a href="https://lists.mpich.org/mailman/listinfo/discuss" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;" class="">https://lists.mpich.org/mailman/listinfo/discuss</span></a><o:p class=""></o:p></div>
</div>
</div>
</blockquote>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""></o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">
_________________________________________________________________________<br class="">
<br class="">
The information transmitted is intended only for the person or entity to<br class="">
which it is addressed and may contain confidential and/or privileged<br class="">
material. Any review, retransmission, dissemination or other use of, or<br class="">
taking of any action in reliance upon, this information by persons or<br class="">
entities other than the intended recipient is prohibited. If you received<br class="">
this in error, please contact the sender and delete the material from any<br class="">
computer.<br class="">
_________________________________________________________________________<br class="">
_______________________________________________<br class="">
discuss mailing list </span><a href="mailto:discuss@mpich.org" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;" class="">discuss@mpich.org</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">
To manage subscription options or unsubscribe:<br class="">
</span><a href="https://lists.mpich.org/mailman/listinfo/discuss" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;" class="">https://lists.mpich.org/mailman/listinfo/discuss</span></a><o:p class=""></o:p></div>
</div>
</blockquote>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">______________________________</span><wbr style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">______________________________</span><wbr style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_____________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">The
information transmitted is intended only for the person or entity to</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">which
it is addressed and may contain confidential and/or privileged</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">material.
Any review, retransmission, dissemination or other use of, or</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">taking
of any action in reliance upon, this information by persons or</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">entities
other than the intended recipient is prohibited. If you received</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">this
in error, please contact the sender and delete the material from any</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">computer.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">______________________________</span><wbr style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">______________________________</span><wbr style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_____________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">discuss
mailing list </span><a href="mailto:discuss@mpich.org" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">discuss@mpich.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">To
manage subscription options or unsubscribe:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></div>
</blockquote>
</div>
<br class="">
</body>
</html>