<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hui,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">“Failing” == spinning in place, deep in the MPI library inside the MPI_Comm_spawn call. I can provide the stack trace if you need it. Slurm says this, but not sure why it mentions srun since I am calling that nowhere.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">srun: error: Unable to create step for job 34378: Requested node configuration is not available<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I tried ntasks = 4, 20 and 48 and the results were always the same.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Kurt<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Zhou, Hui <zhouh@anl.gov> <br>
<b>Sent:</b> Friday, January 7, 2022 4:13 PM<br>
<b>To:</b> Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov>; discuss@mpich.org<br>
<b>Subject:</b> [EXTERNAL] Re: Slurm and MPI_Comm_spawn<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hi Kurt,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Thanks for the details. When you say the job is failing, is it process hanging or abort? Are there any error messages?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">My suspicion is that slurm is preventing extra process to be launched since you have assigned all the resources to the first two MPI processes. Could you try increase the ntasks in the batch command?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">-- <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hui Zhou<o:p></o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov">kurt.e.mccall@nasa.gov</a>><br>
<b>Sent:</b> Friday, January 7, 2022 3:33 PM<br>
<b>To:</b> Zhou, Hui <<a href="mailto:zhouh@anl.gov">zhouh@anl.gov</a>>; <a href="mailto:discuss@mpich.org">
discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Cc:</b> Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov">kurt.e.mccall@nasa.gov</a>><br>
<b>Subject:</b> Re: Slurm and MPI_Comm_spawn</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal">Thanks for the reply, Hui.<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">configure --prefix=/home/kmccall/mpich-install-4.0b1 --with-device=ch3:nemesis --disable-fortran -enable-debuginfo --enable-g=debug
<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">The program is run via sbatch, which is given a bash script as an argument.
<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">sbatch --nodes=2 --ntasks=2 --cpus-per-task=24 <bash_script><o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">The bash script calls mpiexec:<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">mpiexec -print-all-exitcodes -enable-x -np 2 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <cmd><o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xmsonormal"><b>From:</b> Zhou, Hui <<a href="mailto:zhouh@anl.gov">zhouh@anl.gov</a>>
<br>
<b>Sent:</b> Friday, January 7, 2022 2:39 PM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
<b>Cc:</b> Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov">kurt.e.mccall@nasa.gov</a>><br>
<b>Subject:</b> [EXTERNAL] Re: Slurm and MPI_Comm_spawn<o:p></o:p></p>
</div>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">MPICH uses PMI 1 by default.<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">How is your MPICH configured? And how do you run your program, is it via srun?<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<div>
<div>
<p class="xmsonormal">-- <br>
Hui Zhou<o:p></o:p></p>
</div>
</div>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xmsonormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Mccall, Kurt E. (MSFC-EV41) via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Date: </b>Friday, January 7, 2022 at 2:21 PM<br>
<b>To: </b><a href="mailto:discuss@mpich.org">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Cc: </b>Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov">kurt.e.mccall@nasa.gov</a>><br>
<b>Subject: </b>[mpich-discuss] Slurm and MPI_Comm_spawn</span><o:p></o:p></p>
</div>
<p class="xmsonormal" style="margin-left:.5in">My MPICH/Slurm job is failing when the call to MPI_Comm_spawn is made. The Slurm MPI guide
<a href="https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fmpi_guide.html%23mpich2&data=04%7C01%7Ckurt.e.mccall%40nasa.gov%7Cfceae7104eb74518db6a08d9d22af26d%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637771904164104729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5t1oJdGqmfsWlEqBOGR401n6wGvAev3mgucRCR%2FnSLk%3D&reserved=0">
https://slurm.schedmd.com/mpi_guide.html#mpich2</a> specifically states that MPI_Comm_spawn will work going through Hydra’s PMI 1.1 interface.<o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in">How do I ensure that it goes through that interface?
<o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in">Maybe we’ll have to rebuild Slurm to support PMI 1.1. This Slurm command yields the following and PMI 1.1 is not mentioned, although PMI 2 is.<o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in">$ srun –mpi=list<o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in">srun: MPI types are...<o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in">srun: cray_shasta<o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in">srun: pmi2<o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in">srun: none<o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="xmsonormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>