<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:537159213;
mso-list-type:hybrid;
mso-list-template-ids:-1442815430 252478578 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Bruce, <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I noticed you are mixing up things a bit, so let’s clear it up first:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><span style="font-size:11.0pt">If you compile with slurm then you should only launch with srun, not mpirun. Otherwise it won’t work.<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><span style="font-size:11.0pt">If you compile without slurm, then you should launch with mpirun, or more precisely, mpiexec.hydra.
<o:p></o:p></span></li></ul>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:11.0pt"><br>
Depend on which you are doing, each may have issues in your envioronment, but whatever issues are probably unrelated, and should not be discussed in a same context.
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:11.0pt">What is working with your with 3.3.1? With slurm? If that’s what’s working for you, then let’s focus on compile and run 3.3.2 with slurm as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <br>
Hui Zhou<o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Date: </b>Thursday, April 23, 2020 at 11:08 AM<br>
<b>To: </b>"Zhou, Hui" <zhouh@anl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Hui,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I was launching the jobs with mpirun. I tried launching the jobs with srun and they no longer hang, but it looks like they are returning an MPI_COMM_WORLD with only one process, although I didn’t investigate
this extensively. I also tried Pavan’s suggestion and rebuilt 3.3.2 without slurm and ran it with mpiexec. This also hangs and the error I was seeing previously reappears</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[proxy:0:1@node168.local] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node168.local" to "node100.local" (Connection refused)</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[proxy:0:1@node168.local] main (pm/pmiserv/pmip.c:183): unable to connect to server node100.local at port 54762 (check for firewalls!)</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">srun: error: node168: task 1: Exited with exit code 5</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">srun: Job step aborted: Waiting up to 2 seconds for job step to finish.</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: *** STEP 13201325.0 CANCELLED AT 2020-04-23T08:58:02 *** on node100</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: *** JOB 13201325 CANCELLED AT 2020-04-23T08:58:02 *** on node100</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor)</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I suppose this could be a firewall issue, but there must also be some changes between 3.3.1 and 3.3.2, otherwise it should be a problem for all versions of mpich.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bruce</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Zhou, Hui" <zhouh@anl.gov><br>
<b>Date: </b>Wednesday, April 22, 2020 at 10:22 AM<br>
<b>To: </b>"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Bruce,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">How did you launch your jobs? Since you configured with slurm, you should launch your job with `srun`, right?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <br>
Hui Zhou</span><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Date: </b>Wednesday, April 22, 2020 at 11:55 AM<br>
<b>To: </b>"Zhou, Hui" <zhouh@anl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Hui,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Its currently 14.03.8</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bruce</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Zhou, Hui" <zhouh@anl.gov><br>
<b>Date: </b>Wednesday, April 22, 2020 at 9:36 AM<br>
<b>To: </b>"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Bruce,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for the effort checking these versions. What is the slurm versions that you have on cluster?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <br>
Hui Zhou</span><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Date: </b>Wednesday, April 22, 2020 at 11:02 AM<br>
<b>To: </b>"Zhou, Hui" <zhouh@anl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Hui,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I rebuilt everything from scratch and tried running several versions of mpich. Release 3.3.1 seems to work okay but 3.3.2 hangs. Here is a complete summary of the versions I ran</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">3.3rc1: Works</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">3.3: Works</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">3.3.1: Works</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">3.3.2: Hangs</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">3.4a2: Hangs</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m not seeing the error message from hydra anymore (I have no idea why not), but I logged into one of the hung processes when running with 3.3.2 and got the following listing from gdb</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">(gdb) where</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#0 0x0000003d7ce0e810 in __read_nocancel () from /lib64/libpthread.so.0</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#1 0x00002aaaac6e936e in PMIU_readline () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#2 0x00002aaaac6e985b in GetResponse.part.0 () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#3 0x00002aaaac6e4e36 in MPIDU_shm_seg_commit () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#4 0x00002aaaabc541dc in MPIR_Init_thread () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#5 0x00002aaaabc3db7e in PMPI_Init () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#6 0x0000000000408d0e in main ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bruce</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Zhou, Hui" <zhouh@anl.gov><br>
<b>Date: </b>Monday, April 20, 2020 at 10:25 AM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <br>
Hui Zhou</span><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Palmer, Bruce J via discuss" <discuss@mpich.org><br>
<b>Reply-To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Date: </b>Thursday, April 16, 2020 at 5:48 PM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Subject: </b>[mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’ve been building MPICH on are aging Infiniband cluster using the following formula</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code
seems to hang on MPI_Init and it looks like it is producing the error message</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">[proxy:0:1@node013.local] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l</span><o:p></o:p></p>
<p class="p1"><span class="s1">ocal" to "node012.local" (Connection refused)</span><o:p></o:p></p>
<p class="p1"><span class="s1">[proxy:0:1@node013.local] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p</span><o:p></o:p></p>
<p class="p1"><span class="s1">ort 37769 (check for firewalls!)</span><o:p></o:p></p>
<p class="p1"><span class="s1">srun: error: node013: task 1: Exited with exit code 5</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration
formula is no good, or is this more likely to be due to some system modification?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bruce Palmer</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Senior Research Scientist</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Pacific Northwest National Laboratory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Richland, WA 99352</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">(509) 375-3899</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</body>
</html>