<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:"Lucida Grande";
panose-1:2 11 6 0 4 5 2 2 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Lucida Grande",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande",sans-serif">Hi Howard,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande",sans-serif"> What was the latency with the 4.3.x branch?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande",sans-serif">Rajeev<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande",sans-serif"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri",sans-serif;color:black">From:
</span></b><span style="font-family:"Calibri",sans-serif;color:black">Howard Pritchard via discuss <discuss@mpich.org><br>
<b>Reply-To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Date: </b>Wednesday, July 30, 2025 at 1:43 PM<br>
<b>To: </b>"Zhou, Hui" <zhouh@anl.gov><br>
<b>Cc: </b>Howard Pritchard <hppritcha@gmail.com>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">3.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">x
branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%">
<tbody>
<tr>
<td style="padding:12.0pt 0in 12.0pt 0in">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt;border-radius:4px">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0in 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Hi Hui<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Howard<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Mon, Jul 28, 2025 at 4:15<span style="font-family:"Arial",sans-serif"> </span>PM Zhou, Hui <<a href="mailto:zhouh@anl.gov">zhouh@anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="color:black">Hi Howard,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Hui<o:p></o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="m_6369988255225199108divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Howard Pritchard via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Sent:</b> Monday, July 28, 2025 9:41 AM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Cc:</b> Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>><br>
<b>Subject:</b> [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric
1.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">22.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">0
slurm - 24.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">11.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">5
Here's my mpichversion output: MPICH Version:</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white"> 5.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">0.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">0a1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<div style="border:none;border-top:solid #90A4AE 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#d0d8dc!important;border-top:#90a4ae!important" id="m_6369988255225199108x_pfptBanner53g6uvq">
<div id="m_6369988255225199108x_pfptBanner53g6uvq">
<div id="m_6369988255225199108x_pfptBanner53g6uvq">
<p class="MsoNormal" style="line-height:13.5pt;background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</div>
<div id="m_6369988255225199108x_pfptBanner53g6uvq">
<p class="MsoNormal" style="line-height:13.5pt;background:#D0D8DC"><span style="font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="background:#D0D8DC"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Hi Folks, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">First the info:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">system using libfabric 1.22.0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">slurm - 24.11.5<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Here's my mpichversion output:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH Version: 5.0.0a1</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH Release date: unreleased development copy</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH ABI: 0:0:0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH Device: ch4:ofi</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci
--with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH CC: gcc -O3</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH CXX: g++ -O3</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH F77: gfortran -O3</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH FC: gfortran -O3</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH features: threadcomm<br>
<br>
<br>
<br>
And here's the OSU latency results:<br>
<br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective.</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get()</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]:</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: -</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]:</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: -</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"># OSU MPI Latency Test v5.8</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"># Size Latency (us)</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">0 1.66</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">1 9.29</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">2 9.57</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">4 9.69</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">8 9.76</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">16 9.77</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">32 9.76</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">64 9.77</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">128 10.32</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">256 7.54</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">512 7.45</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">1024 7.38</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">2048 7.37</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">4096 7.45</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">8192 9.21</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">16384 9.70</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">32768 10.63</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">65536 13.15</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">131072 16.96</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">262144 23.84</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">524288 36.16</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">1048576 60.36</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">2097152 108.43</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">4194304 228.31</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs.</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like
1.6 or so.</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">I am confident the slurm issue is unrelated to the latency issue.<br>
<br>
Thanks for any suggestions on how to address either issue however.</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
<p style="margin:0in;background:#FEF48B;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>