<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Things were working fine when I was launching 1 node jobs under Slurm 20.11.8, but when I launched a 20 node job, MPICH hangs in MPI_Init. The output of “mpiexec -verbose” is attached, and the stack trace at the point where it hangs is
below.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In the “mpiexec -verbose” output, I wonder why variables such as PATH_modshare point to our Intel MPI implementation, which I am no using. I am using MPICH 4.0 with a patch that Ken Raffenetti provided (which makes MPICH recognize the
“host” info key). My $PATH and $LD_LIBRARY_PATH variables definitely point to the correct MPICH installation.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I appreciate any help you can give.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is the Slurm sbatch command:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">sbatch --nodes=20 --ntasks=20 --job-name $job_name --exclusive –verbose
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is the mpiexec command:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <many more args…><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Stack trace at MPI_Init:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#0 0x00007f6d85f499b2 in read () from /lib64/libpthread.so.0<o:p></o:p></p>
<p class="MsoNormal">#1 0x00007f6d87a5753a in PMIU_readline (fd=5, buf=buf@entry=0x7ffd6fb596e0 "", maxlen=maxlen@entry=1024)<o:p></o:p></p>
<p class="MsoNormal"> at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmiutil.c:134<o:p></o:p></p>
<p class="MsoNormal">#2 0x00007f6d87a57a56 in GetResponse (request=0x7f6d87b48351 "cmd=barrier_in\n",<o:p></o:p></p>
<p class="MsoNormal"> expectedCmd=0x7f6d87b48345 "barrier_out", checkRc=0) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmi.c:818<o:p></o:p></p>
<p class="MsoNormal">#3 0x00007f6d87a29915 in MPIDI_PG_SetConnInfo (rank=rank@entry=0,<o:p></o:p></p>
<p class="MsoNormal"> connString=connString@entry=0x1bbf5a0 "description#n001$port#33403$ifname#172.16.56.1$")<o:p></o:p></p>
<p class="MsoNormal"> at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpidi_pg.c:559<o:p></o:p></p>
<p class="MsoNormal">#4 0x00007f6d87a38611 in MPID_nem_init (pg_rank=pg_rank@entry=0, pg_p=pg_p@entry=0x1bbf850, has_parent=<optimized out>)<o:p></o:p></p>
<p class="MsoNormal"> at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c:393<o:p></o:p></p>
<p class="MsoNormal">#5 0x00007f6d87a2ad93 in MPIDI_CH3_Init (has_parent=<optimized out>, pg_p=0x1bbf850, pg_rank=0)<o:p></o:p></p>
<p class="MsoNormal"> at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/ch3_init.c:83<o:p></o:p></p>
<p class="MsoNormal">#6 0x00007f6d87a1b3b7 in init_world () at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:190<o:p></o:p></p>
<p class="MsoNormal">#7 MPID_Init (requested=<optimized out>, provided=provided@entry=0x7f6d87e03540 <MPIR_ThreadInfo>)<o:p></o:p></p>
<p class="MsoNormal"> at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:76<o:p></o:p></p>
<p class="MsoNormal">#8 0x00007f6d879828eb in MPII_Init_thread (argc=argc@entry=0x7ffd6fb5a5cc, argv=argv@entry=0x7ffd6fb5a5c0,<o:p></o:p></p>
<p class="MsoNormal"> user_required=0, provided=provided@entry=0x7ffd6fb5a574, p_session_ptr=p_session_ptr@entry=0x0)<o:p></o:p></p>
<p class="MsoNormal"> at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:208<o:p></o:p></p>
<p class="MsoNormal">#9 0x00007f6d879832a5 in MPIR_Init_impl (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0)<o:p></o:p></p>
<p class="MsoNormal"> at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:93<o:p></o:p></p>
<p class="MsoNormal">#10 0x00007f6d8786388e in PMPI_Init (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0)<o:p></o:p></p>
<p class="MsoNormal"> at ../mpich-slurm-patch-4.0/src/binding/c/init/init.c:46<o:p></o:p></p>
<p class="MsoNormal">#11 0x000000000040640d in main (argc=23, argv=0x7ffd6fb5ad68) at src/NeedlesMpiManagerMain.cpp:53<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>