<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerdirxo1q { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtondirxo1q:hover, .pfptPrimaryButtondirxo1q:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtondirxo1q:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Rajeev, Here are the results for 4. 3. x branch: hpp@ nid001293: /usr/projects/artab/users/hpp/osu-micro-benchmarks-5. 8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency # OSU MPI Latency Test v5. 8 # Size Latency (us) 0 1. 92 1 1. 98 2 1. 98</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerdirxo1q" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerdirxo1q" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerdirxo1q" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerdirxo1q" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr"><div dir="ltr">Hi Rajeev,<div><br></div><div>Here are the results for 4.3.x branch:</div><div><br></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hpp@nid001293:/usr/projects/artab/users/hpp/osu-micro-benchmarks-5.8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># OSU MPI Latency Test v5.8</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># Size<span class="gmail-Apple-converted-space"> </span>Latency (us)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">0 <span class="gmail-Apple-converted-space"> </span>1.92</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">1 <span class="gmail-Apple-converted-space"> </span>1.98</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2 <span class="gmail-Apple-converted-space"> </span>1.98</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">4 <span class="gmail-Apple-converted-space"> </span>1.98</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">8 <span class="gmail-Apple-converted-space"> </span>1.98</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">16<span class="gmail-Apple-converted-space"> </span>1.98</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">32<span class="gmail-Apple-converted-space"> </span>1.99</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">64<span class="gmail-Apple-converted-space"> </span>1.99</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">128 <span class="gmail-Apple-converted-space"> </span>2.47</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">256 <span class="gmail-Apple-converted-space"> </span>2.59</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">512 <span class="gmail-Apple-converted-space"> </span>2.65</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">1024<span class="gmail-Apple-converted-space"> </span>2.76</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2048<span class="gmail-Apple-converted-space"> </span>2.95</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">4096<span class="gmail-Apple-converted-space"> </span>3.00</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">8192<span class="gmail-Apple-converted-space"> </span>5.96</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">16384 <span class="gmail-Apple-converted-space"> </span>6.64</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">32768 <span class="gmail-Apple-converted-space"> </span>7.44</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">65536 <span class="gmail-Apple-converted-space"> </span>8.75</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">131072 <span class="gmail-Apple-converted-space"> </span>11.52</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">262144 <span class="gmail-Apple-converted-space"> </span>17.08</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">524288 <span class="gmail-Apple-converted-space"> </span>27.96</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">1048576<span class="gmail-Apple-converted-space"> </span>49.38</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2097152<span class="gmail-Apple-converted-space"> </span>92.96</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">4194304 <span class="gmail-Apple-converted-space"> </span>179.74</span></p></div></div><br><div class="gmail_quote gmail_quote_container"><div class="gmail_attr">These are more like i would expect for SS11/OFI CXI provider.</div><div class="gmail_attr"><br></div><div class="gmail_attr">Howard</div><div class="gmail_attr"><br></div><div dir="ltr" class="gmail_attr">Am Mi., 30. Juli 2025 um 12:48 Uhr schrieb Thakur, Rajeev <<a href="mailto:thakur@anl.gov">thakur@anl.gov</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-2350182920575357441">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_-2350182920575357441WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif">Hi Howard,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif"> What was the latency with the 4.3.x branch?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif">Rajeev<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif"><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri,sans-serif;color:black">From:
</span></b><span style="font-family:Calibri,sans-serif;color:black">Howard Pritchard via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Reply-To: </b>"<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>" <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Date: </b>Wednesday, July 30, 2025 at 1:43 PM<br>
<b>To: </b>"Zhou, Hui" <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>><br>
<b>Cc: </b>Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>>, "<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>" <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1pt;color:white">Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">3.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">x
branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1pt;color:white">ZjQcmQRYFpfptBannerStart<u></u><u></u></span></p>
</div>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100%">
<tbody>
<tr>
<td style="padding:12pt 0in">
<table border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100%;background:rgb(208,216,220);border-right:none;border-bottom:none;border-left:none;border-top:3pt solid rgb(144,164,174);border-radius:4px">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0in 7.5pt 3.75pt 4.5pt">
<table border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3pt 6pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:Arial,sans-serif;color:black">This Message Is From an External Sender
<u></u><u></u></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3pt 6pt">
<p class="MsoNormal"><span style="font-size:9pt;font-family:Arial,sans-serif;color:black">This message came from outside your organization.
<u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal"><span style="font-size:1pt;color:white">ZjQcmQRYFpfptBannerEnd<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal">Hi Hui<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Howard<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Mon, Jul 28, 2025 at 4:15<span style="font-family:Arial,sans-serif"> </span>PM Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="color:black">Hi Howard,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Hui<u></u><u></u></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="m_-2350182920575357441m_6369988255225199108divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif;color:black">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif;color:black"> Howard Pritchard via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Sent:</b> Monday, July 28, 2025 9:41 AM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Cc:</b> Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>><br>
<b>Subject:</b> [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem</span>
<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:1pt;color:white">Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric
1.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">22.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">0
slurm - 24.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">11.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">5
Here's my mpichversion output: MPICH Version:</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white"> 5.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">0.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">0a1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1pt;color:white">ZjQcmQRYFpfptBannerStart<u></u><u></u></span></p>
</div>
<div style="border-width:initial;border-right-style:none;border-bottom-style:none;border-left-style:none;border-right-color:initial;border-bottom-color:initial;border-left-color:initial;display:block;text-align:left;margin:0px;padding:16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top-style:initial;border-top-color:rgb(144,164,174)" id="m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq">
<div id="m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq">
<div id="m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq">
<p class="MsoNormal" style="line-height:13.5pt;background:rgb(208,216,220)"><b><span style="font-family:Arial,sans-serif;color:black">This Message Is From an External Sender
<u></u><u></u></span></b></p>
</div>
<div id="m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq">
<p class="MsoNormal" style="line-height:13.5pt;background:rgb(208,216,220)"><span style="font-family:Arial,sans-serif;color:black">This message came from outside your organization.
<u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="background:rgb(208,216,220)"><span style="color:black"> </span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1pt;color:white">ZjQcmQRYFpfptBannerEnd<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal">Hi Folks, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">First the info:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">system using libfabric 1.22.0<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">slurm - 24.11.5<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Here's my mpichversion output:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH Version: 5.0.0a1</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH Release date: unreleased development copy</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH ABI: 0:0:0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH Device: ch4:ofi</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci
--with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH CC: gcc -O3</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH CXX: g++ -O3</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH F77: gfortran -O3</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH FC: gfortran -O3</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH features: threadcomm<br>
<br>
<br>
<br>
And here's the OSU latency results:<br>
<br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective.</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get()</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]:</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: -</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]:</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: -</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"># OSU MPI Latency Test v5.8</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"># Size Latency (us)</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">0 1.66</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">1 9.29</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">2 9.57</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">4 9.69</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">8 9.76</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">16 9.77</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">32 9.76</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">64 9.77</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">128 10.32</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">256 7.54</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">512 7.45</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">1024 7.38</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">2048 7.37</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">4096 7.45</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">8192 9.21</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">16384 9.70</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">32768 10.63</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">65536 13.15</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">131072 16.96</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">262144 23.84</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">524288 36.16</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">1048576 60.36</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">2097152 108.43</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">4194304 228.31</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs.</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like
1.6 or so.</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">I am confident the slurm issue is unrelated to the latency issue.<br>
<br>
Thanks for any suggestions on how to address either issue however.</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo"><u></u> <u></u></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div></blockquote></div></div>