<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanner53g6uvq { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButton53g6uvq:hover, .pfptPrimaryButton53g6uvq:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButton53g6uvq:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: MPICH Version: 5. 0. 0a1</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanner53g6uvq" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanner53g6uvq" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanner53g6uvq" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanner53g6uvq" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr">Hi Folks,<div><br></div><div>We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH.</div><div><br></div><div>First the info:</div><div>system using libfabric 1.22.0</div><div>slurm - 24.11.5</div><div><br></div><div>Here's my mpichversion output:</div><div><br></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH Version:<span class="gmail-Apple-converted-space"> </span>5.0.0a1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH Release date: unreleased development copy</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH ABI:<span class="gmail-Apple-converted-space"> </span>0:0:0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH Device: <span class="gmail-Apple-converted-space"> </span>ch4:ofi</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH configure:<span class="gmail-Apple-converted-space"> </span>--prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH CC: <span class="gmail-Apple-converted-space"> </span>gcc <span class="gmail-Apple-converted-space"> </span>-O3</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH CXX:<span class="gmail-Apple-converted-space"> </span>g++ <span class="gmail-Apple-converted-space"> </span>-O3</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH F77:<span class="gmail-Apple-converted-space"> </span>gfortran <span class="gmail-Apple-converted-space"> </span>-O3</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH FC: <span class="gmail-Apple-converted-space"> </span>gfortran <span class="gmail-Apple-converted-space"> </span>-O3</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH features: <span class="gmail-Apple-converted-space"> </span>threadcomm<br><br><br><br>And here's the OSU latency results:<br><br>
</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective.</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get()</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>seq=0 contribs: loc=1/prev=0/fwd=1</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>neighbor contribs [2]:</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>done contrib: -</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>wait contrib: nid001439</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>status=PMIXP_COLL_RING_PROGRESS</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>buf (offset/size): 36/16384</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>seq=0 contribs: loc=1/prev=0/fwd=1</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>neighbor contribs [2]:</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>done contrib: -</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>wait contrib: nid001438</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>status=PMIXP_COLL_RING_PROGRESS</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="gmail-Apple-converted-space"> </span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829:<span class="gmail-Apple-converted-space"> <span class="gmail-Apple-tab-span" style="white-space:pre"> </span> </span>buf (offset/size): 36/16384</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># OSU MPI Latency Test v5.8</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># Size<span class="gmail-Apple-converted-space"> </span>Latency (us)</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">0 <span class="gmail-Apple-converted-space"> </span>1.66</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">1 <span class="gmail-Apple-converted-space"> </span>9.29</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2 <span class="gmail-Apple-converted-space"> </span>9.57</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">4 <span class="gmail-Apple-converted-space"> </span>9.69</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">8 <span class="gmail-Apple-converted-space"> </span>9.76</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">16<span class="gmail-Apple-converted-space"> </span>9.77</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">32<span class="gmail-Apple-converted-space"> </span>9.76</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">64<span class="gmail-Apple-converted-space"> </span>9.77</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">128<span class="gmail-Apple-converted-space"> </span>10.32</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">256 <span class="gmail-Apple-converted-space"> </span>7.54</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">512 <span class="gmail-Apple-converted-space"> </span>7.45</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">1024<span class="gmail-Apple-converted-space"> </span>7.38</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2048<span class="gmail-Apple-converted-space"> </span>7.37</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">4096<span class="gmail-Apple-converted-space"> </span>7.45</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">8192<span class="gmail-Apple-converted-space"> </span>9.21</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">16384 <span class="gmail-Apple-converted-space"> </span>9.70</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">32768<span class="gmail-Apple-converted-space"> </span>10.63</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">65536<span class="gmail-Apple-converted-space"> </span>13.15</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">131072 <span class="gmail-Apple-converted-space"> </span>16.96</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">262144 <span class="gmail-Apple-converted-space"> </span>23.84</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">524288 <span class="gmail-Apple-converted-space"> </span>36.16</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">1048576<span class="gmail-Apple-converted-space"> </span>60.36</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2097152 <span class="gmail-Apple-converted-space"> </span>108.43</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">
</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">4194304 <span class="gmail-Apple-converted-space"> </span>228.31</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs.</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like 1.6 or so.</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">I am confident the slurm issue is unrelated to the latency issue.<br><br>Thanks for any suggestions on how to address either issue however.</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)"><br></p></div></div>