<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Howard,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Could you run with `MPIR_CVAR_DEBUG_SUMMARY=1`? It should print some debug messages. I want to confirm it is running the `cxi` provider.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Howard Pritchard <hppritcha@gmail.com><br>
<b>Sent:</b> Wednesday, July 30, 2025 4:37 PM<br>
<b>To:</b> Thakur, Rajeev <thakur@anl.gov><br>
<b>Cc:</b> discuss@mpich.org <discuss@mpich.org>; Zhou, Hui <zhouh@anl.gov><br>
<b>Subject:</b> Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem</font>
<div> </div>
</div>
<style>
<!--
#x_pfptBannerbiv7aor
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#D0D8DC!important;
max-width:none!important;
max-height:none!important}
-->
</style>
<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; border:0!important; display:table!important; width:100%!important; table-layout:fixed!important; border-collapse:seperate!important; float:none!important; border-spacing:0px 0px!important">
<tbody style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; display:block!important">
<tr style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important">
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; padding:7px 2px 7px 2px!important; background-color:#A6A6A6!important; width:0px!important">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; width:100%!important; background-color:#EAEAEA!important; padding:7px 5px 7px 15px!important; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif!important; font-size:12px!important; font-weight:normal!important; color:#212121!important; text-align:left!important; word-wrap:break-word!important">
<div style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important">
You don't often get email from hppritcha@gmail.com. <a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!auMZ5HDttmXdzic7Af_6iAv9fkjrSrKuj5_hA-q8bi10GjJl8M9BVmTrW57ZlwjcCQEiJ-quEYeU$" style="background:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; opacity:revert!important; visibility:revert!important">
Learn why this is important</a> </div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; width:75px!important; background-color:#EAEAEA!important; padding:7px 5px 7px 5px!important; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif!important; font-size:12px!important; font-weight:normal!important; color:#212121!important; text-align:left!important; word-wrap:break-word!important">
</td>
</tr>
</tbody>
</table>
<div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
Hi Rajeev, Here are the results for 4. 3. x branch: hpp@ nid001293: /usr/projects/artab/users/hpp/osu-micro-benchmarks-5. 8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency # OSU MPI Latency Test v5. 8 # Size Latency (us) 0 1. 92 1 1. 98 2 1. 98</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="x_pfptBannerbiv7aor" style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#D0D8DC!important; background-color:#D0D8DC; border-top:4px solid #90a4ae!important; border-top:4px solid #90a4ae">
<div id="x_pfptBannerbiv7aor" style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBannerbiv7aor" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBannerbiv7aor" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">
<div dir="ltr">Hi Rajeev,
<div><br>
</div>
<div>Here are the results for 4.3.x branch:</div>
<div><br>
</div>
<div>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">hpp@nid001293:/usr/projects/artab/users/hpp/osu-micro-benchmarks-5.8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"># OSU MPI Latency Test v5.8</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"># Size<span class="x_gmail-Apple-converted-space">
</span>Latency (us)</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">0 <span class="x_gmail-Apple-converted-space">
</span>1.92</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">1 <span class="x_gmail-Apple-converted-space">
</span>1.98</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">2 <span class="x_gmail-Apple-converted-space">
</span>1.98</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">4 <span class="x_gmail-Apple-converted-space">
</span>1.98</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">8 <span class="x_gmail-Apple-converted-space">
</span>1.98</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">16<span class="x_gmail-Apple-converted-space">
</span>1.98</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">32<span class="x_gmail-Apple-converted-space">
</span>1.99</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">64<span class="x_gmail-Apple-converted-space">
</span>1.99</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">128 <span class="x_gmail-Apple-converted-space">
</span>2.47</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">256 <span class="x_gmail-Apple-converted-space">
</span>2.59</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">512 <span class="x_gmail-Apple-converted-space">
</span>2.65</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">1024<span class="x_gmail-Apple-converted-space">
</span>2.76</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">2048<span class="x_gmail-Apple-converted-space">
</span>2.95</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">4096<span class="x_gmail-Apple-converted-space">
</span>3.00</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">8192<span class="x_gmail-Apple-converted-space">
</span>5.96</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">16384
<span class="x_gmail-Apple-converted-space"> </span>6.64</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">32768
<span class="x_gmail-Apple-converted-space"> </span>7.44</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">65536
<span class="x_gmail-Apple-converted-space"> </span>8.75</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">131072
<span class="x_gmail-Apple-converted-space"> </span>11.52</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">262144
<span class="x_gmail-Apple-converted-space"> </span>17.08</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">524288
<span class="x_gmail-Apple-converted-space"> </span>27.96</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">1048576<span class="x_gmail-Apple-converted-space">
</span>49.38</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">2097152<span class="x_gmail-Apple-converted-space">
</span>92.96</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">4194304
<span class="x_gmail-Apple-converted-space"> </span>179.74</span></p>
</div>
</div>
<br>
<div class="x_gmail_quote x_gmail_quote_container">
<div class="x_gmail_attr">These are more like i would expect for SS11/OFI CXI provider.</div>
<div class="x_gmail_attr"><br>
</div>
<div class="x_gmail_attr">Howard</div>
<div class="x_gmail_attr"><br>
</div>
<div dir="ltr" class="x_gmail_attr">Am Mi., 30. Juli 2025 um 12:48 Uhr schrieb Thakur, Rajeev <<a href="mailto:thakur@anl.gov">thakur@anl.gov</a>>:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg-2350182920575357441">
<div lang="EN-US" style="">
<div class="x_m_-2350182920575357441WordSection1">
<p class="x_MsoNormal"><span style="font-size:11pt; font-family:"Lucida Grande",sans-serif">Hi Howard,<u></u><u></u></span></p>
<p class="x_MsoNormal"><span style="font-size:11pt; font-family:"Lucida Grande",sans-serif"> What was the latency with the 4.3.x branch?<u></u><u></u></span></p>
<p class="x_MsoNormal"><span style="font-size:11pt; font-family:"Lucida Grande",sans-serif"><u></u> <u></u></span></p>
<p class="x_MsoNormal"><span style="font-size:11pt; font-family:"Lucida Grande",sans-serif">Rajeev<u></u><u></u></span></p>
<p class="x_MsoNormal"><span style="font-size:11pt; font-family:"Lucida Grande",sans-serif"><u></u> <u></u></span></p>
<p class="x_MsoNormal"><span style="font-size:11pt; font-family:"Lucida Grande",sans-serif"><u></u> <u></u></span></p>
<div style="border-right:none; border-bottom:none; border-left:none; border-top:1pt solid rgb(181,196,223); padding:3pt 0in 0in">
<p class="x_MsoNormal"><b><span style="font-family:Calibri,sans-serif; color:black">From:
</span></b><span style="font-family:Calibri,sans-serif; color:black">Howard Pritchard via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Reply-To: </b>"<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>" <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Date: </b>Wednesday, July 30, 2025 at 1:43 PM<br>
<b>To: </b>"Zhou, Hui" <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>><br>
<b>Cc: </b>Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>>, "<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>" <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem<u></u><u></u></span></p>
</div>
<div>
<p class="x_MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:1pt; color:white">Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.</span><span style="font-size:1pt; font-family:Arial,sans-serif; color:white"> </span><span style="font-size:1pt; color:white">3.</span><span style="font-size:1pt; font-family:Arial,sans-serif; color:white"> </span><span style="font-size:1pt; color:white">x
branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch.</span><span style="font-size:1pt; font-family:Arial,sans-serif; color:white"> </span><span style="font-size:1pt; color:white"><u></u><u></u></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:1pt; color:white">ZjQcmQRYFpfptBannerStart<u></u><u></u></span></p>
</div>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100%">
<tbody>
<tr>
<td style="padding:12pt 0in">
<table border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100%; background:rgb(208,216,220); border-right:none; border-bottom:none; border-left:none; border-top:3pt solid rgb(144,164,174); border-radius:4px">
<tbody>
<tr>
<td valign="top" style="border:none; padding:0in 7.5pt 3.75pt 4.5pt">
<table border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3pt 6pt">
<p class="x_MsoNormal"><b><span style="font-size:10.5pt; font-family:Arial,sans-serif; color:black">This Message Is From an External Sender
<u></u><u></u></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3pt 6pt">
<p class="x_MsoNormal"><span style="font-size:9pt; font-family:Arial,sans-serif; color:black">This message came from outside your organization.
<u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="x_MsoNormal"><span style="font-size:1pt; color:white">ZjQcmQRYFpfptBannerEnd<u></u><u></u></span></p>
</div>
<div>
<p class="x_MsoNormal">Hi Hui<u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="x_MsoNormal">That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that
branch.<u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="x_MsoNormal">Howard<u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="x_MsoNormal">On Mon, Jul 28, 2025 at 4:15<span style="font-family:Arial,sans-serif"> </span>PM Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none; border-right:none; border-bottom:none; border-left:1pt solid rgb(204,204,204); padding:0in 0in 0in 6pt; margin-left:4.8pt; margin-right:0in">
<div>
<div>
<p class="x_MsoNormal"><span style="color:black">Hi Howard,<u></u><u></u></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"> I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves?<u></u><u></u></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">Hui<u></u><u></u></span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_m_-2350182920575357441m_6369988255225199108divRplyFwdMsg">
<p class="x_MsoNormal"><b><span style="font-size:11pt; font-family:Calibri,sans-serif; color:black">From:</span></b><span style="font-size:11pt; font-family:Calibri,sans-serif; color:black"> Howard Pritchard via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Sent:</b> Monday, July 28, 2025 9:41 AM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Cc:</b> Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>><br>
<b>Subject:</b> [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem</span>
<u></u><u></u></p>
<div>
<p class="x_MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="x_MsoNormal"><span style="font-size:1pt; color:white">Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1.</span><span style="font-size:1pt; font-family:Arial,sans-serif; color:white"> </span><span style="font-size:1pt; color:white">22.</span><span style="font-size:1pt; font-family:Arial,sans-serif; color:white"> </span><span style="font-size:1pt; color:white">0
slurm - 24.</span><span style="font-size:1pt; font-family:Arial,sans-serif; color:white"> </span><span style="font-size:1pt; color:white">11.</span><span style="font-size:1pt; font-family:Arial,sans-serif; color:white"> </span><span style="font-size:1pt; color:white">5
Here's my mpichversion output: MPICH Version:</span><span style="font-size:1pt; font-family:Arial,sans-serif; color:white"> </span><span style="font-size:1pt; color:white"> 5.</span><span style="font-size:1pt; font-family:Arial,sans-serif; color:white"> </span><span style="font-size:1pt; color:white">0.</span><span style="font-size:1pt; font-family:Arial,sans-serif; color:white"> </span><span style="font-size:1pt; color:white">0a1<u></u><u></u></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:1pt; color:white">ZjQcmQRYFpfptBannerStart<u></u><u></u></span></p>
</div>
<div id="x_m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq" style="border-width:initial; border-right-style:none; border-bottom-style:none; border-left-style:none; border-right-color:initial; border-bottom-color:initial; border-left-color:initial; display:block; text-align:left; margin:0px; padding:16px; border-radius:4px; min-width:200px; background-color:rgb(208,216,220); border-top-style:initial; border-top-color:rgb(144,164,174)">
<div id="x_m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq">
<div id="x_m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq">
<p class="x_MsoNormal" style="line-height:13.5pt; background:rgb(208,216,220)"><b><span style="font-family:Arial,sans-serif; color:black">This Message Is From an External Sender
<u></u><u></u></span></b></p>
</div>
<div id="x_m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq">
<p class="x_MsoNormal" style="line-height:13.5pt; background:rgb(208,216,220)"><span style="font-family:Arial,sans-serif; color:black">This message came from outside your organization.
<u></u><u></u></span></p>
</div>
</div>
<div>
<p class="x_MsoNormal" style="background:rgb(208,216,220)"><span style="color:black"> </span><u></u><u></u></p>
</div>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:1pt; color:white">ZjQcmQRYFpfptBannerEnd<u></u><u></u></span></p>
</div>
<div>
<p class="x_MsoNormal">Hi Folks, <u></u><u></u></p>
<div>
<p class="x_MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="x_MsoNormal">We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH.<u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="x_MsoNormal">First the info:<u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal">system using libfabric 1.22.0<u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal">slurm - 24.11.5<u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="x_MsoNormal">Here's my mpichversion output:<u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">MPICH Version: 5.0.0a1</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">MPICH Release date: unreleased development copy</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">MPICH ABI: 0:0:0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">MPICH Device: ch4:ofi</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci
--with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">MPICH CC: gcc -O3</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">MPICH CXX: g++ -O3</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">MPICH F77: gfortran -O3</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">MPICH FC: gfortran -O3</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">MPICH features: threadcomm<br>
<br>
<br>
<br>
And here's the OSU latency results:<br>
<br>
<br>
</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective.</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get()</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]:</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: -</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]:</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: -</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black"># OSU MPI Latency Test v5.8</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black"># Size Latency (us)</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">0 1.66</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">1 9.29</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">2 9.57</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">4 9.69</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">8 9.76</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">16 9.77</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">32 9.76</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">64 9.77</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">128 10.32</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">256 7.54</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">512 7.45</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">1024 7.38</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">2048 7.37</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">4096 7.45</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">8192 9.21</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">16384 9.70</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">32768 10.63</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">65536 13.15</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">131072 16.96</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">262144 23.84</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">524288 36.16</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">1048576 60.36</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">2097152 108.43</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">4194304 228.31</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black"><br>
<br>
</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs.</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black"><br>
<br>
</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like
1.6 or so.</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black"><br>
<br>
</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black">I am confident the slurm issue is unrelated to the latency issue.<br>
<br>
Thanks for any suggestions on how to address either issue however.</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo; color:black"><br>
<br>
</span><span style="font-size:8.5pt; font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in; background:rgb(254,244,139); font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal">
<span style="font-size:8.5pt; font-family:Menlo"><u></u> <u></u></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>