<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannersgvgage { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonsgvgage:hover, .pfptPrimaryButtonsgvgage:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonsgvgage:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4. 3. x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. </div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannersgvgage" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannersgvgage" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannersgvgage" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannersgvgage" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="auto">Hi Hui</div><div dir="auto"><br></div><div dir="auto">That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch.</div><div dir="auto"><br></div><div dir="auto">Howard</div><div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Mon, Jul 28, 2025 at 4:15 PM Zhou, Hui <<a href="mailto:zhouh@anl.gov">zhouh@anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Howard,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves?</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hui</div>
<div id="m_6369988255225199108appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_6369988255225199108divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Howard Pritchard via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Sent:</b> Monday, July 28, 2025 9:41 AM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Cc:</b> Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>><br>
<b>Subject:</b> [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem</font>
<div> </div>
</div>
<div>
<div style="display:none!important;display:none;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden">
Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: MPICH Version: 5. 0. 0a1</div>
<div style="display:none!important;display:none;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="m_6369988255225199108x_pfptBanner53g6uvq" style="display:block!important;text-align:left!important;margin:16px 0px 16px 0px!important;padding:8px 16px 8px 16px!important;border-radius:4px!important;min-width:200px!important;background-color:#d0d8dc!important;background-color:#d0d8dc;border-top:4px solid #90a4ae!important;border-top:4px solid #90a4ae">
<div id="m_6369988255225199108x_pfptBanner53g6uvq" style="float:left!important;display:block!important;margin:0px 0px 1px 0px!important;max-width:600px!important">
<div id="m_6369988255225199108x_pfptBanner53g6uvq" style="display:block!important;background-color:#d0d8dc!important;color:#000000!important;color:#000000;font-family:'Arial',sans-serif!important;font-family:'Arial',sans-serif;font-weight:bold!important;font-weight:bold;font-size:14px!important;line-height:18px!important;line-height:18px">
This Message Is From an External Sender </div>
<div id="m_6369988255225199108x_pfptBanner53g6uvq" style="display:block!important;background-color:#d0d8dc!important;color:#000000!important;color:#000000;font-weight:normal;font-family:'Arial',sans-serif!important;font-family:'Arial',sans-serif;font-size:12px!important;line-height:18px!important;line-height:18px;margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div style="clear:both!important;display:block!important;line-height:0!important;font-size:0.01px!important;height:0px">
</div>
</div>
<div style="display:none!important;display:none;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">Hi Folks,
<div><br>
</div>
<div>We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH.</div>
<div><br>
</div>
<div>First the info:</div>
<div>system using libfabric 1.22.0</div>
<div>slurm - 24.11.5</div>
<div><br>
</div>
<div>Here's my mpichversion output:</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">MPICH Version:<span>
</span>5.0.0a1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">MPICH Release date: unreleased development copy</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">MPICH ABI:<span>
</span>0:0:0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">MPICH Device:
<span> </span>ch4:ofi</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">MPICH configure:<span>
</span>--prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default
--with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">MPICH CC:
<span> </span>gcc <span>
</span>-O3</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">MPICH CXX:<span>
</span>g++ <span> </span>-O3</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">MPICH F77:<span>
</span>gfortran <span> </span>-O3</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">MPICH FC:
<span> </span>gfortran <span>
</span>-O3</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">MPICH features:
<span> </span>threadcomm<br>
<br>
<br>
<br>
And here's the OSU latency results:<br>
<br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective.</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get()</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786:<span> <span style="white-space:pre-wrap">
</span></span>seq=0 contribs: loc=1/prev=0/fwd=1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788:<span> <span style="white-space:pre-wrap">
</span></span>neighbor contribs [2]:</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821:<span> <span style="white-space:pre-wrap">
</span><span style="white-space:pre-wrap"></span></span>done contrib: -</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823:<span> <span style="white-space:pre-wrap">
</span><span style="white-space:pre-wrap"></span></span>wait contrib: nid001439</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825:<span> <span style="white-space:pre-wrap">
</span></span>status=PMIXP_COLL_RING_PROGRESS</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829:<span> <span style="white-space:pre-wrap">
</span></span>buf (offset/size): 36/16384</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786:<span> <span style="white-space:pre-wrap">
</span></span>seq=0 contribs: loc=1/prev=0/fwd=1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788:<span> <span style="white-space:pre-wrap">
</span></span>neighbor contribs [2]:</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821:<span> <span style="white-space:pre-wrap">
</span><span style="white-space:pre-wrap"></span></span>done contrib: -</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823:<span> <span style="white-space:pre-wrap">
</span><span style="white-space:pre-wrap"></span></span>wait contrib: nid001438</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825:<span> <span style="white-space:pre-wrap">
</span></span>status=PMIXP_COLL_RING_PROGRESS</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span>
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829:<span> <span style="white-space:pre-wrap">
</span></span>buf (offset/size): 36/16384</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures"># OSU MPI Latency Test v5.8</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures"># Size<span>
</span>Latency (us)</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">0 <span>
</span>1.66</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">1 <span>
</span>9.29</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">2 <span>
</span>9.57</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">4 <span>
</span>9.69</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">8 <span>
</span>9.76</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">16<span>
</span>9.77</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">32<span>
</span>9.76</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">64<span>
</span>9.77</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">128<span>
</span>10.32</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">256 <span>
</span>7.54</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">512 <span>
</span>7.45</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">1024<span>
</span>7.38</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">2048<span>
</span>7.37</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">4096<span>
</span>7.45</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">8192<span>
</span>9.21</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">16384
<span> </span>9.70</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">32768<span>
</span>10.63</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">65536<span>
</span>13.15</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">131072
<span> </span>16.96</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">262144
<span> </span>23.84</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">524288
<span> </span>36.16</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">1048576<span>
</span>60.36</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">2097152
<span> </span>108.43</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures"></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">4194304
<span> </span>228.31</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs.</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting
more like 1.6 or so.</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
I am confident the slurm issue is unrelated to the latency issue.<br>
<br>
Thanks for any suggestions on how to address either issue however.</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<br>
</p>
</div>
</div>
</div>
</div>
</blockquote></div></div>