<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoPlainText">Ken,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Looking at the Slurm documentation
<a href="https://slurm.schedmd.com/mpi_guide.html#intel_mpiexec_hydra">https://slurm.schedmd.com/mpi_guide.html#intel_mpiexec_hydra</a><o:p></o:p></p>
<p class="MsoPlainText">it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">1) how do I build MPICH to support hydra's PMI 1.1 interface?<o:p></o:p></p>
<p class="MsoPlainText">2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum
<span style="font-family:"Segoe UI Emoji",sans-serif">😊</span>)<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Thanks,<o:p></o:p></p>
<p class="MsoPlainText">Kurt<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">=========== Full quote ===============<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">MPICH2 with mpiexec<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Do not add any flags to mpich and build the default (e.g. "./configure -prefix ... ". Do NOT pass the --with-slurm, --with-pmi, --enable-pmiport options).<o:p></o:p></p>
<p class="MsoPlainText">Do not add -lpmi to your application (it will force slurm's pmi 1 interface which doesn't support PMI_Spawn_multiple).<o:p></o:p></p>
<p class="MsoPlainText">Launch the application using salloc to create the job allocation and mpiexec to launch the tasks. A simple example is shown below.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">salloc -N 2 mpiexec my_application<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<br>
From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss@mpich.org> <br>
Sent: Friday, January 28, 2022 1:43 PM<br>
To: Raffenetti, Ken <raffenet@anl.gov>; discuss@mpich.org<br>
Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov><br>
Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I have built in the past without the pmi2 option, and MPI_Comm_spawn doesn't work at all. My memory is a little fuzzy, so I'll try that to make sure. In the meantime, can you recommend an option with which to build Slurm that MPICH
also supports and has tested? Here is what we have now:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">$ srun --mpi=list<o:p></o:p></p>
<p class="MsoPlainText">srun: MPI types are...<o:p></o:p></p>
<p class="MsoPlainText">srun: cray_shasta<o:p></o:p></p>
<p class="MsoPlainText">srun: pmi2<o:p></o:p></p>
<p class="MsoPlainText">srun: none<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<o:p></o:p></p>
<p class="MsoPlainText">From: Raffenetti, Ken <<a href="mailto:raffenet@anl.gov"><span style="color:windowtext;text-decoration:none">raffenet@anl.gov</span></a>>
<o:p></o:p></p>
<p class="MsoPlainText">Sent: Friday, January 28, 2022 1:35 PM<o:p></o:p></p>
<p class="MsoPlainText">To: Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov"><span style="color:windowtext;text-decoration:none">kurt.e.mccall@nasa.gov</span></a>>;
<a href="mailto:discuss@mpich.org"><span style="color:windowtext;text-decoration:none">discuss@mpich.org</span></a><o:p></o:p></p>
<p class="MsoPlainText">Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I'm just now realizing that we don't test spawn functionality with our pmi2 implementation. Can you try rebuilding without that option and see if it works as expected?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Ken<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">On 1/28/22, 1:18 PM, "Mccall, Kurt E. (MSFC-EV41)" <<a href="mailto:kurt.e.mccall@nasa.gov"><span style="color:windowtext;text-decoration:none">kurt.e.mccall@nasa.gov</span></a>> wrote:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> ../mpich-4.0rc3/configure --prefix=/opt/mpich --with-pmi=pmi2/simple --with-device=ch3:nemesis --disable-fortran -enable-debuginfo --enable-g=debug<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> -----Original Message-----<o:p></o:p></p>
<p class="MsoPlainText"> From: Raffenetti, Ken <<a href="mailto:raffenet@anl.gov"><span style="color:windowtext;text-decoration:none">raffenet@anl.gov</span></a>>
<o:p></o:p></p>
<p class="MsoPlainText"> Sent: Friday, January 28, 2022 1:16 PM<o:p></o:p></p>
<p class="MsoPlainText"> To: Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov"><span style="color:windowtext;text-decoration:none">kurt.e.mccall@nasa.gov</span></a>>;
<a href="mailto:discuss@mpich.org"><span style="color:windowtext;text-decoration:none">discuss@mpich.org</span></a><o:p></o:p></p>
<p class="MsoPlainText"> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> From what I can tell, the info keyvals are not being sent to the process manager, which would explain what you are seeing. Need to investigate why that is happening next. What is your MPICH ./configure line? That'll help narrow where
to look.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Ken<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> On 1/28/22, 12:49 PM, "Mccall, Kurt E. (MSFC-EV41)" <<a href="mailto:kurt.e.mccall@nasa.gov"><span style="color:windowtext;text-decoration:none">kurt.e.mccall@nasa.gov</span></a>> wrote:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Ken, <o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> There is a lot of my own output mixed up in the "mpiexec -v" output. I hope you can make sense of this.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Thanks,<o:p></o:p></p>
<p class="MsoPlainText"> Kurt<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> -----Original Message-----<o:p></o:p></p>
<p class="MsoPlainText"> From: Raffenetti, Ken <<a href="mailto:raffenet@anl.gov"><span style="color:windowtext;text-decoration:none">raffenet@anl.gov</span></a>>
<o:p></o:p></p>
<p class="MsoPlainText"> Sent: Friday, January 28, 2022 12:36 PM<o:p></o:p></p>
<p class="MsoPlainText"> To: Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov"><span style="color:windowtext;text-decoration:none">kurt.e.mccall@nasa.gov</span></a>>;
<a href="mailto:discuss@mpich.org"><span style="color:windowtext;text-decoration:none">discuss@mpich.org</span></a><o:p></o:p></p>
<p class="MsoPlainText"> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> "ip_address" won't be recognized, only "host", "hosts", or "hostfile". Could you run an example using "mpiexec -v" and capture/share the output? That should help tell us if the hostname information is being fed correctly to the
process manager by the spawn command.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Ken<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> On 1/28/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <<a href="mailto:kurt.e.mccall@nasa.gov"><span style="color:windowtext;text-decoration:none">kurt.e.mccall@nasa.gov</span></a>> wrote:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Ken,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> I'm using sbatch, which calls a bash script that calls mpiexec (4.0rc3). Which host name convention is correct, the short or the long host name? Would the "ip_address" info key work?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Kurt<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> -----Original Message-----<o:p></o:p></p>
<p class="MsoPlainText"> From: Raffenetti, Ken <<a href="mailto:raffenet@anl.gov"><span style="color:windowtext;text-decoration:none">raffenet@anl.gov</span></a>>
<o:p></o:p></p>
<p class="MsoPlainText"> Sent: Friday, January 28, 2022 10:49 AM<o:p></o:p></p>
<p class="MsoPlainText"> To: <a href="mailto:discuss@mpich.org"><span style="color:windowtext;text-decoration:none">discuss@mpich.org</span></a><o:p></o:p></p>
<p class="MsoPlainText"> Cc: Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov"><span style="color:windowtext;text-decoration:none">kurt.e.mccall@nasa.gov</span></a>><o:p></o:p></p>
<p class="MsoPlainText"> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Are you using mpiexec or srun when initially launching your job? Hydra (mpiexec) should support the "host" info key, but I'm not sure if srun will.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Ken<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> On 1/28/22, 10:41 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <<a href="mailto:discuss@mpich.org"><span style="color:windowtext;text-decoration:none">discuss@mpich.org</span></a>> wrote:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Hi,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called
MPI_Comm_spawn.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> I’ve tried passing MPI_Comm_spawn an info struct created like this:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> MPI_Info info;<o:p></o:p></p>
<p class="MsoPlainText"> MPI_Info_create(&info);<o:p></o:p></p>
<p class="MsoPlainText"> MPI_Info_set(info, "host", host_name);<o:p></o:p></p>
<p class="MsoPlainText"> MPI_Info_set(info, "bind_to", "core");<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> where hostname = “n001” or even the full name “n001.cluster.pssclabs.com”<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> but that doesn’t prevent the problem. Any suggestions?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Thanks,<o:p></o:p></p>
<p class="MsoPlainText"> Kurt<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">_______________________________________________<o:p></o:p></p>
<p class="MsoPlainText">discuss mailing list <a href="mailto:discuss@mpich.org">
<span style="color:windowtext;text-decoration:none">discuss@mpich.org</span></a><o:p></o:p></p>
<p class="MsoPlainText">To manage subscription options or unsubscribe:<o:p></o:p></p>
<p class="MsoPlainText"><a href="https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mpich.org%2Fmailman%2Flistinfo%2Fdiscuss&data=04%7C01%7Ckurt.e.mccall%40nasa.gov%7C132072f5f8994f8cb31308d9e2966ce7%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637789957980739463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5q%2FzYyNnO9HMxQ%2Bq1xDDaCLC06vCuCDolIuS1Oe%2FLNo%3D&reserved=0"><span style="color:windowtext;text-decoration:none">https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mpich.org%2Fmailman%2Flistinfo%2Fdiscuss&data=04%7C01%7Ckurt.e.mccall%40nasa.gov%7C132072f5f8994f8cb31308d9e2966ce7%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637789957980739463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5q%2FzYyNnO9HMxQ%2Bq1xDDaCLC06vCuCDolIuS1Oe%2FLNo%3D&reserved=0</span></a><o:p></o:p></p>
</div>
</body>
</html>