<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanner7jf65b4 { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButton7jf65b4:hover, .pfptPrimaryButton7jf65b4:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButton7jf65b4:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Here you go Hui! MPICH debug output and slurm steps output to boot. Again no such slurmy errors with the 4. 3. 1 release. Something must have changed in the way MPICH is using the PMIX group constructor ops or something like that. Required minimum</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanner7jf65b4" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanner7jf65b4" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanner7jf65b4" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanner7jf65b4" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr"><div dir="ltr"><div dir="ltr">Here you go Hui!<div><br></div><div>MPICH debug output and slurm steps output to boot. Again no such slurmy errors with the 4.3.1 release.</div><div>Something must have changed in the way MPICH is using the PMIX group constructor ops or something like that.</div><div><br></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Required minimum FI_VERSION: 0, current version: 10016</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Required minimum FI_VERSION: 10005, current version: 10016</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">==== Capability set configuration ====</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">libfabric provider: cxi - cxi</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_DATA: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_AV_TABLE: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_MR_ALLOCATED: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_MR_PROV_KEY: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_TAGGED: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_AM: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_RMA: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_ATOMICS: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_TRIGGERED: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_ENABLE_HMEM: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_NUM_AM_BUFFERS: 8</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_CONTEXT_BITS: 20</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_SOURCE_BITS: 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_TAG_BITS: 20</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_VNI_USE_DOMAIN: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MAXIMUM SUPPORTED RANKS: 4294967296</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MAXIMUM TAG: 1048576</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">==== Provider global thresholds ====</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">max_buffered_send: 192</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">max_buffered_write: 192</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">max_msg_size: 4294967295</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">max_order_raw: -1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">max_order_war: -1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">max_order_waw: -1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">tx_iov_limit: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">rx_iov_limit: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">rma_iov_limit: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">max_mr_key_size: 4</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">==== Various sizes and limits ====</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_AM_MSG_HEADER_SIZE: 24</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_MAX_AM_HDR_SIZE: 255</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">sizeof(MPIDI_OFI_am_request_header_t): 416</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">sizeof(MPIDI_OFI_per_vci_t): 52480</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">======================================</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_belong_chk: nid001406 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective.</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: _process_server_request: nid001406 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get()</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_check: nid001405 [0]: pmixp_coll_ring.c:614: 0x14b448006e10: unexpected contrib from nid001406:1, expected is 0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: _process_server_request: nid001405 [0]: pmixp_server.c:937: 0x14b448006e10: unexpected contrib from nid001406:1, coll->seq=0, seq=0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001405 [0]: pmixp_coll_ring.c:738: 0x14b454052fc0: collective timeout seq=0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_log: nid001405 [0]: pmixp_coll.c:286: Dumping collective state</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:756: 0x14b454052fc0: COLL_FENCE_RING state seq=0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001405</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001406, prev 1:nid001406</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:775: Context ptr=0x14b454053038, #0, in-use=0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:775: Context ptr=0x14b454053070, #1, in-use=0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:775: Context ptr=0x14b4540530a8, #2, in-use=1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:786:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>seq=0 contribs: loc=1/prev=0/fwd=1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:788:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>neighbor contribs [2]:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:821:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>done contrib: -</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:823:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>wait contrib: nid001406</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:825:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>status=PMIXP_COLL_RING_PROGRESS</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:829:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>buf (offset/size): 36/16384</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001406 [1]: pmixp_coll_ring.c:738: 0x14aa28053100: collective timeout seq=0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_log: nid001406 [1]: pmixp_coll.c:286: Dumping collective state</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:756: 0x14aa28053100: COLL_FENCE_RING state seq=0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001406</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001405, prev 0:nid001405</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:775: Context ptr=0x14aa28053178, #0, in-use=0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:775: Context ptr=0x14aa280531b0, #1, in-use=0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:775: Context ptr=0x14aa280531e8, #2, in-use=1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:786:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>seq=0 contribs: loc=1/prev=0/fwd=1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:788:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>neighbor contribs [2]:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:821:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>done contrib: -</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:823:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>wait contrib: nid001405</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:825:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>status=PMIXP_COLL_RING_PROGRESS</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:829:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>buf (offset/size): 36/16384</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">==== Various sizes and limits ====</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">sizeof(MPIDI_per_vci_t): 128</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">==== collective selection ====</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIR_CVAR_DEVICE_COLLECTIVES: percoll</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPIR: MPII_coll_generic_json</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPID: MPIDI_coll_generic_json</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPID/shm: MPIDI_POSIX_coll_generic_json</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">==== OFI dynamic settings ====</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">num_vcis: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">num_nics: 1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">======================================</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">error checking<span class="gmail-Apple-converted-space"> </span>: disabled</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">QMPI<span class="gmail-Apple-converted-space"> </span>: disabled</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">debugger support<span class="gmail-Apple-converted-space"> </span>: disabled</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">thread level<span class="gmail-Apple-converted-space"> </span>: MPI_THREAD_SINGLE</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">thread CS <span class="gmail-Apple-converted-space"> </span>: per-vci</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">threadcomm<span class="gmail-Apple-converted-space"> </span>: enabled</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">==== data structure summary ====</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">sizeof(MPIR_Comm): 1832</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">sizeof(MPIR_Request): 520</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">sizeof(MPIR_Datatype): 280</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">================================</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># OSU MPI Latency Test v5.8</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># Size<span class="gmail-Apple-converted-space"> </span>Latency (us)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">0 <span class="gmail-Apple-converted-space"> </span>2.04</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">1<span class="gmail-Apple-converted-space"> </span>10.08</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2<span class="gmail-Apple-converted-space"> </span>10.10</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">4<span class="gmail-Apple-converted-space"> </span>10.11</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">8<span class="gmail-Apple-converted-space"> </span>10.12</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">16 <span class="gmail-Apple-converted-space"> </span>10.12</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">32 <span class="gmail-Apple-converted-space"> </span>10.13</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">64 <span class="gmail-Apple-converted-space"> </span>10.12</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">128<span class="gmail-Apple-converted-space"> </span>10.67</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">256 <span class="gmail-Apple-converted-space"> </span>8.10</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">512 <span class="gmail-Apple-converted-space"> </span>8.18</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">1024<span class="gmail-Apple-converted-space"> </span>8.11</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2048<span class="gmail-Apple-converted-space"> </span>7.86</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">4096<span class="gmail-Apple-converted-space"> </span>7.80</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">8192 <span class="gmail-Apple-converted-space"> </span>10.25</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">16384<span class="gmail-Apple-converted-space"> </span>11.04</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">32768<span class="gmail-Apple-converted-space"> </span>12.04</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">65536<span class="gmail-Apple-converted-space"> </span>14.05</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">131072 <span class="gmail-Apple-converted-space"> </span>17.89</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">262144 <span class="gmail-Apple-converted-space"> </span>24.61</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">524288 <span class="gmail-Apple-converted-space"> </span>37.51</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">1048576<span class="gmail-Apple-converted-space"> </span>61.48</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2097152 <span class="gmail-Apple-converted-space"> </span>110.06</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">4194304 <span class="gmail-Apple-converted-space"> </span>228.67</span></p></div><div><br></div></div></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">Am Mi., 13. Aug. 2025 um 13:10 Uhr schrieb Zhou, Hui <<a href="mailto:zhouh@anl.gov">zhouh@anl.gov</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-7763817599493531614">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Howard,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Could you run with `MPIR_CVAR_DEBUG_SUMMARY=1`? It should print some debug messages. I want to confirm it is running the `cxi` provider.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hui</div>
<div id="m_-7763817599493531614appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-7763817599493531614divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>><br>
<b>Sent:</b> Wednesday, July 30, 2025 4:37 PM<br>
<b>To:</b> Thakur, Rajeev <<a href="mailto:thakur@anl.gov" target="_blank">thakur@anl.gov</a>><br>
<b>Cc:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>>; Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>><br>
<b>Subject:</b> Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem</font>
<div> </div>
</div>
<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="background:revert;color:revert;direction:revert;font-size:revert;height:revert;letter-spacing:revert;line-height:revert;margin:revert;opacity:revert;outline:revert;overflow:revert;padding:revert;text-align:revert;text-indent:revert;text-orientation:revert;text-overflow:revert;text-transform:revert;vertical-align:revert;white-space:revert;word-break:revert;word-spacing:revert;writing-mode:revert;zoom:revert;border:0px;display:table;width:100%;table-layout:fixed;float:none;border-spacing:0px">
<tbody style="background:revert;border:revert;color:revert;direction:revert;font-size:revert;height:revert;letter-spacing:revert;line-height:revert;margin:revert;opacity:revert;outline:revert;overflow:revert;padding:revert;table-layout:revert;text-align:revert;text-indent:revert;text-orientation:revert;text-overflow:revert;text-transform:revert;vertical-align:revert;white-space:revert;width:revert;word-break:revert;word-spacing:revert;writing-mode:revert;zoom:revert;display:block">
<tr style="background:revert;border:revert;color:revert;direction:revert;display:revert;font-size:revert;height:revert;letter-spacing:revert;line-height:revert;margin:revert;opacity:revert;outline:revert;overflow:revert;padding:revert;table-layout:revert;text-align:revert;text-indent:revert;text-orientation:revert;text-overflow:revert;text-transform:revert;vertical-align:revert;white-space:revert;width:revert;word-break:revert;word-spacing:revert;writing-mode:revert;zoom:revert">
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="background-image:revert;background-position:revert;background-size:revert;background-repeat:revert;background-origin:revert;background-clip:revert;border:revert;color:revert;direction:revert;display:revert;font-size:revert;height:revert;letter-spacing:revert;line-height:revert;margin:revert;opacity:revert;outline:revert;overflow:revert;table-layout:revert;text-align:revert;text-indent:revert;text-orientation:revert;text-overflow:revert;text-transform:revert;vertical-align:revert;white-space:revert;word-break:revert;word-spacing:revert;writing-mode:revert;zoom:revert;padding:7px 2px;background-color:rgb(166,166,166);width:0px">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="background-image:revert;background-position:revert;background-size:revert;background-repeat:revert;background-origin:revert;background-clip:revert;border:revert;direction:revert;display:revert;height:revert;letter-spacing:revert;line-height:revert;margin:revert;opacity:revert;outline:revert;overflow:revert;table-layout:revert;text-indent:revert;text-orientation:revert;text-overflow:revert;text-transform:revert;vertical-align:revert;white-space:revert;word-break:revert;word-spacing:revert;writing-mode:revert;zoom:revert;width:100%;background-color:rgb(234,234,234);padding:7px 5px 7px 15px;font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;font-size:12px;font-weight:normal;color:rgb(33,33,33);text-align:left">
<div style="background:revert;border:revert;color:revert;direction:revert;display:revert;font-size:revert;height:revert;letter-spacing:revert;line-height:revert;margin:revert;opacity:revert;outline:revert;overflow:revert;padding:revert;table-layout:revert;text-align:revert;text-indent:revert;text-orientation:revert;text-overflow:revert;text-transform:revert;vertical-align:revert;white-space:revert;width:revert;word-break:revert;word-spacing:revert;writing-mode:revert;zoom:revert">
You don't often get email from <a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>. <a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!aeEBmF_DTUp_lE5ETFEZupObYvUZ6i54jdGlfV3tG05FKqEKN1UmnaLgx1W6epKD1rrcWzppMp6MXXLu$" style="background:revert;color:revert;direction:revert;display:revert;font-size:revert;opacity:revert" target="_blank">
Learn why this is important</a> </div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="background-image:revert;background-position:revert;background-size:revert;background-repeat:revert;background-origin:revert;background-clip:revert;border:revert;direction:revert;display:revert;height:revert;letter-spacing:revert;line-height:revert;margin:revert;opacity:revert;outline:revert;overflow:revert;table-layout:revert;text-indent:revert;text-orientation:revert;text-overflow:revert;text-transform:revert;vertical-align:revert;white-space:revert;word-break:revert;word-spacing:revert;writing-mode:revert;zoom:revert;width:75px;background-color:rgb(234,234,234);padding:7px 5px;font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;font-size:12px;font-weight:normal;color:rgb(33,33,33);text-align:left">
</td>
</tr>
</tbody>
</table>
<div>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">
Hi Rajeev, Here are the results for 4. 3. x branch: hpp@ nid001293: /usr/projects/artab/users/hpp/osu-micro-benchmarks-5. 8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency # OSU MPI Latency Test v5. 8 # Size Latency (us) 0 1. 92 1 1. 98 2 1. 98</div>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;opacity:0;overflow:hidden;display:none">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="m_-7763817599493531614x_pfptBannerbiv7aor" style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)">
<div id="m_-7763817599493531614x_pfptBannerbiv7aor" style="float:left;display:block;margin:0px 0px 1px;max-width:600px">
<div id="m_-7763817599493531614x_pfptBannerbiv7aor" style="display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px">
This Message Is From an External Sender </div>
<div id="m_-7763817599493531614x_pfptBannerbiv7aor" style="font-weight:normal;display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px">
This message came from outside your organization. </div>
</div>
<div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px">
</div>
</div>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;opacity:0;overflow:hidden;display:none">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">
<div dir="ltr">Hi Rajeev,
<div><br>
</div>
<div>Here are the results for 4.3.x branch:</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">hpp@nid001293:/usr/projects/artab/users/hpp/osu-micro-benchmarks-5.8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures"># OSU MPI Latency Test v5.8</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures"># Size<span>
</span>Latency (us)</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">0 <span>
</span>1.92</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">1 <span>
</span>1.98</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">2 <span>
</span>1.98</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">4 <span>
</span>1.98</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">8 <span>
</span>1.98</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">16<span>
</span>1.98</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">32<span>
</span>1.99</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">64<span>
</span>1.99</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">128 <span>
</span>2.47</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">256 <span>
</span>2.59</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">512 <span>
</span>2.65</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">1024<span>
</span>2.76</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">2048<span>
</span>2.95</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">4096<span>
</span>3.00</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">8192<span>
</span>5.96</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">16384
<span> </span>6.64</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">32768
<span> </span>7.44</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">65536
<span> </span>8.75</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">131072
<span> </span>11.52</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">262144
<span> </span>17.08</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">524288
<span> </span>27.96</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">1048576<span>
</span>49.38</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">2097152<span>
</span>92.96</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">4194304
<span> </span>179.74</span></p>
</div>
</div>
<br>
<div>
<div>These are more like i would expect for SS11/OFI CXI provider.</div>
<div><br>
</div>
<div>Howard</div>
<div><br>
</div>
<div dir="ltr">Am Mi., 30. Juli 2025 um 12:48 Uhr schrieb Thakur, Rajeev <<a href="mailto:thakur@anl.gov" target="_blank">thakur@anl.gov</a>>:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="EN-US">
<div>
<p><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif">Hi Howard,<u></u><u></u></span></p>
<p><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif"> What was the latency with the 4.3.x branch?<u></u><u></u></span></p>
<p><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif"><u></u> <u></u></span></p>
<p><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif">Rajeev<u></u><u></u></span></p>
<p><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif"><u></u> <u></u></span></p>
<p><span style="font-size:11pt;font-family:"Lucida Grande",sans-serif"><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p><b><span style="font-family:Calibri,sans-serif;color:black">From:
</span></b><span style="font-family:Calibri,sans-serif;color:black">Howard Pritchard via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Reply-To: </b>"<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>" <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Date: </b>Wednesday, July 30, 2025 at 1:43 PM<br>
<b>To: </b>"Zhou, Hui" <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>><br>
<b>Cc: </b>Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>>, "<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>" <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem<u></u><u></u></span></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p><span style="font-size:1pt;color:white">Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">3.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">x
branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white"><u></u><u></u></span></p>
</div>
<div>
<p><span style="font-size:1pt;color:white">ZjQcmQRYFpfptBannerStart<u></u><u></u></span></p>
</div>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100%">
<tbody>
<tr>
<td style="padding:12pt 0in">
<table border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100%;background:rgb(208,216,220);border-right:none;border-bottom:none;border-left:none;border-top:3pt solid rgb(144,164,174);border-radius:4px">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0in 7.5pt 3.75pt 4.5pt">
<table border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3pt 6pt">
<p><b><span style="font-size:10.5pt;font-family:Arial,sans-serif;color:black">This Message Is From an External Sender
<u></u><u></u></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3pt 6pt">
<p><span style="font-size:9pt;font-family:Arial,sans-serif;color:black">This message came from outside your organization.
<u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p><span style="font-size:1pt;color:white">ZjQcmQRYFpfptBannerEnd<u></u><u></u></span></p>
</div>
<div>
<p>Hi Hui<u></u><u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p>That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that
branch.<u></u><u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p>Howard<u></u><u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
<div>
<div>
<p>On Mon, Jul 28, 2025 at 4:15<span style="font-family:Arial,sans-serif"> </span>PM Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p><span style="color:black">Hi Howard,<u></u><u></u></span></p>
</div>
<div>
<p><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p><span style="color:black"> I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves?<u></u><u></u></span></p>
</div>
<div>
<p><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p><span style="color:black">Hui<u></u><u></u></span></p>
</div>
<div align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="m_-7763817599493531614x_m_-2350182920575357441m_6369988255225199108divRplyFwdMsg">
<p><b><span style="font-size:11pt;font-family:Calibri,sans-serif;color:black">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif;color:black"> Howard Pritchard via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Sent:</b> Monday, July 28, 2025 9:41 AM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Cc:</b> Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>><br>
<b>Subject:</b> [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem</span>
<u></u><u></u></p>
<div>
<p> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p><span style="font-size:1pt;color:white">Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">22.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">0
slurm - 24.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">11.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">5
Here's my mpichversion output: MPICH Version:</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white"> 5.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">0.</span><span style="font-size:1pt;font-family:Arial,sans-serif;color:white"> </span><span style="font-size:1pt;color:white">0a1<u></u><u></u></span></p>
</div>
<div>
<p><span style="font-size:1pt;color:white">ZjQcmQRYFpfptBannerStart<u></u><u></u></span></p>
</div>
<div id="m_-7763817599493531614x_m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq" style="border-width:initial;border-right-style:none;border-bottom-style:none;border-left-style:none;border-right-color:initial;border-bottom-color:initial;border-left-color:initial;display:block;text-align:left;margin:0px;padding:16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top-style:initial;border-top-color:rgb(144,164,174)">
<div id="m_-7763817599493531614x_m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq">
<div id="m_-7763817599493531614x_m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq">
<p style="line-height:13.5pt;background:rgb(208,216,220)"><b><span style="font-family:Arial,sans-serif;color:black">This Message Is From an External Sender
<u></u><u></u></span></b></p>
</div>
<div id="m_-7763817599493531614x_m_-2350182920575357441m_6369988255225199108x_pfptBanner53g6uvq">
<p style="line-height:13.5pt;background:rgb(208,216,220)"><span style="font-family:Arial,sans-serif;color:black">This message came from outside your organization.
<u></u><u></u></span></p>
</div>
</div>
<div>
<p style="background:rgb(208,216,220)"><span style="color:black"> </span><u></u><u></u></p>
</div>
</div>
<div>
<p><span style="font-size:1pt;color:white">ZjQcmQRYFpfptBannerEnd<u></u><u></u></span></p>
</div>
<div>
<p>Hi Folks, <u></u><u></u></p>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p>We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH.<u></u><u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p>First the info:<u></u><u></u></p>
</div>
<div>
<p>system using libfabric 1.22.0<u></u><u></u></p>
</div>
<div>
<p>slurm - 24.11.5<u></u><u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p>Here's my mpichversion output:<u></u><u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH Version: 5.0.0a1</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH Release date: unreleased development copy</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH ABI: 0:0:0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH Device: ch4:ofi</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci
--with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH CC: gcc -O3</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH CXX: g++ -O3</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH F77: gfortran -O3</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH FC: gfortran -O3</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">MPICH features: threadcomm<br>
<br>
<br>
<br>
And here's the OSU latency results:<br>
<br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective.</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get()</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]:</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: -</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]:</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: -</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"># OSU MPI Latency Test v5.8</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"># Size Latency (us)</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">0 1.66</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">1 9.29</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">2 9.57</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">4 9.69</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">8 9.76</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">16 9.77</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">32 9.76</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">64 9.77</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">128 10.32</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">256 7.54</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">512 7.45</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">1024 7.38</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">2048 7.37</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">4096 7.45</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">8192 9.21</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">16384 9.70</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">32768 10.63</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">65536 13.15</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">131072 16.96</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">262144 23.84</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">524288 36.16</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">1048576 60.36</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">2097152 108.43</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">4194304 228.31</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs.</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like
1.6 or so.</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black">I am confident the slurm issue is unrelated to the latency issue.<br>
<br>
Thanks for any suggestions on how to address either issue however.</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo;color:black"><br>
<br>
</span><span style="font-size:8.5pt;font-family:Menlo"><u></u><u></u></span></p>
<p style="margin:0in;background:rgb(254,244,139);font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal">
<span style="font-size:8.5pt;font-family:Menlo"><u></u> <u></u></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div>