<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:537159213;
mso-list-type:hybrid;
mso-list-template-ids:-1442815430 252478578 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:1466968777;
mso-list-template-ids:959621284;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Bruce,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Before give you solution (which I have no clue yet), let’s understand what is needed. Do you have a firewall rules between nodes?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <br>
Hui Zhou<o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Date: </b>Friday, April 24, 2020 at 2:54 PM<br>
<b>To: </b>"Zhou, Hui" <zhouh@anl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hui,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">mpirun is pointing to mpiexec.hydra. I’m using “mpirun -n 6 executable.x” to launch jobs. The system guys have given me some information about linking to the pmi libraries that I’m going to try to see if I
can get srun to work properly. I will give that a try and see if it enables me to use srun. They also suggested trying to configure MPICH with --add-pmi, although that doesn’t look like an MPICH configuration option. I did see a –enable-pmiport option. Is
that something I should try?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bruce</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Zhou, Hui" <zhouh@anl.gov><br>
<b>Date: </b>Friday, April 24, 2020 at 10:59 AM<br>
<b>To: </b>"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Bruce,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Mpirun, mpiexec, and mpiexec.hydra, are all the same binary, with previous two symbolic links to the last. Please verify. If not, then you have installation issue. Hydra will detect environment and utilize
information gathered from, e.g. slurm. For example, it will gather that you are launching with a given number of processes. When in doubt, use explicit command line option, such as `-n <numprocs>`. By the way, what is your complete command line that you used
to launch jobs in all the cases?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <br>
Hui Zhou</span><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Date: </b>Friday, April 24, 2020 at 11:49 AM<br>
<b>To: </b>"Zhou, Hui" <zhouh@anl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Hui,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m not sure what is going on with srun, but it doesn’t seem to work the way you are describing, at least on our system. I’ve got an inquiry into our system administrators asking about it but generally, when
I launch with srun it looks like MPI_Comm_size is returns 1 for the size of MPI_COMM_WORLD no matter how many processors I’m actually running on.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’ve tried the following combinations:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">mpich-3.3.2 built without slurm</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Launch with mpiexec.hydra: hangs in MPI_Init</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Launch with srun: only get 1 processor</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Launch with mpirun: hangs in MPI_Init</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Launch with mpiexec: hangs in MPI_Init</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">mpich-3.3.2 built with slurm</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Launch with mpiexec: hangs in MPI_Init</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Launch with srun: only get 1 processor</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Launch with mpirun: hangs in MPI_Init</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">mpich-3.3.1 built with slurm</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Launch with mpiexec: runs okay</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Launch with srun: only get 1 processor</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Launch with mpirun: runs okay</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I was under the impression that MPICH created its own version of mpirun/mpiexec depending on what it found out about the scheduling system during configuration and then built mpirun or mpiexec accordingly.
Is this not correct?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bruce</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Zhou, Hui" <zhouh@anl.gov><br>
<b>Date: </b>Thursday, April 23, 2020 at 1:31 PM<br>
<b>To: </b>"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Bruce, </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I noticed you are mixing up things a bit, so let’s clear it up first:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3"><span style="font-size:11.0pt">If you compile with slurm then you should only launch with srun, not mpirun. Otherwise it won’t work.</span><o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3"><span style="font-size:11.0pt">If you compile without slurm, then you should launch with mpirun, or more precisely, mpiexec.hydra.
</span><o:p></o:p></li></ul>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:11.0pt"><br>
Depend on which you are doing, each may have issues in your envioronment, but whatever issues are probably unrelated, and should not be discussed in a same context.
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:11.0pt">What is working with your with 3.3.1? With slurm? If that’s what’s working for you, then let’s focus on compile and run 3.3.2 with slurm as well.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <br>
Hui Zhou</span><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Date: </b>Thursday, April 23, 2020 at 11:08 AM<br>
<b>To: </b>"Zhou, Hui" <zhouh@anl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Hui,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I was launching the jobs with mpirun. I tried launching the jobs with srun and they no longer hang, but it looks like they are returning an MPI_COMM_WORLD with only one process, although I didn’t investigate
this extensively. I also tried Pavan’s suggestion and rebuilt 3.3.2 without slurm and ran it with mpiexec. This also hangs and the error I was seeing previously reappears</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[proxy:0:1@node168.local] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node168.local" to "node100.local" (Connection refused)</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[proxy:0:1@node168.local] main (pm/pmiserv/pmip.c:183): unable to connect to server node100.local at port 54762 (check for firewalls!)</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">srun: error: node168: task 1: Exited with exit code 5</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">srun: Job step aborted: Waiting up to 2 seconds for job step to finish.</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: *** STEP 13201325.0 CANCELLED AT 2020-04-23T08:58:02 *** on node100</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: *** JOB 13201325 CANCELLED AT 2020-04-23T08:58:02 *** on node100</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor)</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">[mpiexec@node100.local] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I suppose this could be a firewall issue, but there must also be some changes between 3.3.1 and 3.3.2, otherwise it should be a problem for all versions of mpich.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bruce</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Zhou, Hui" <zhouh@anl.gov><br>
<b>Date: </b>Wednesday, April 22, 2020 at 10:22 AM<br>
<b>To: </b>"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Bruce,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">How did you launch your jobs? Since you configured with slurm, you should launch your job with `srun`, right?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <br>
Hui Zhou</span><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Date: </b>Wednesday, April 22, 2020 at 11:55 AM<br>
<b>To: </b>"Zhou, Hui" <zhouh@anl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Hui,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Its currently 14.03.8</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bruce</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Zhou, Hui" <zhouh@anl.gov><br>
<b>Date: </b>Wednesday, April 22, 2020 at 9:36 AM<br>
<b>To: </b>"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Bruce,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for the effort checking these versions. What is the slurm versions that you have on cluster?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <br>
Hui Zhou</span><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Date: </b>Wednesday, April 22, 2020 at 11:02 AM<br>
<b>To: </b>"Zhou, Hui" <zhouh@anl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Hui,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I rebuilt everything from scratch and tried running several versions of mpich. Release 3.3.1 seems to work okay but 3.3.2 hangs. Here is a complete summary of the versions I ran</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">3.3rc1: Works</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">3.3: Works</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">3.3.1: Works</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">3.3.2: Hangs</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">3.4a2: Hangs</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m not seeing the error message from hydra anymore (I have no idea why not), but I logged into one of the hung processes when running with 3.3.2 and got the following listing from gdb</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">(gdb) where</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#0 0x0000003d7ce0e810 in __read_nocancel () from /lib64/libpthread.so.0</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#1 0x00002aaaac6e936e in PMIU_readline () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#2 0x00002aaaac6e985b in GetResponse.part.0 () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#3 0x00002aaaac6e4e36 in MPIDU_shm_seg_commit () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#4 0x00002aaaabc541dc in MPIR_Init_thread () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#5 0x00002aaaabc3db7e in PMPI_Init () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12</span><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black">#6 0x0000000000408d0e in main ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bruce</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Zhou, Hui" <zhouh@anl.gov><br>
<b>Date: </b>Monday, April 20, 2020 at 10:25 AM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <br>
Hui Zhou</span><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Palmer, Bruce J via discuss" <discuss@mpich.org><br>
<b>Reply-To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Date: </b>Thursday, April 16, 2020 at 5:48 PM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>"Palmer, Bruce J" <Bruce.Palmer@pnnl.gov><br>
<b>Subject: </b>[mpich-discuss] MPICH configure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’ve been building MPICH on are aging Infiniband cluster using the following formula</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code
seems to hang on MPI_Init and it looks like it is producing the error message</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">[proxy:0:1@node013.local] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l</span><o:p></o:p></p>
<p class="p1"><span class="s1">ocal" to "node012.local" (Connection refused)</span><o:p></o:p></p>
<p class="p1"><span class="s1">[proxy:0:1@node013.local] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p</span><o:p></o:p></p>
<p class="p1"><span class="s1">ort 37769 (check for firewalls!)</span><o:p></o:p></p>
<p class="p1"><span class="s1">srun: error: node013: task 1: Exited with exit code 5</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration
formula is no good, or is this more likely to be due to some system modification?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bruce Palmer</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Senior Research Scientist</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Pacific Northwest National Laboratory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Richland, WA 99352</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">(509) 375-3899</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</body>
</html>