<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Howard,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves?</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Howard Pritchard via discuss <discuss@mpich.org><br>
<b>Sent:</b> Monday, July 28, 2025 9:41 AM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Howard Pritchard <hppritcha@gmail.com><br>
<b>Subject:</b> [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem</font>
<div> </div>
</div>
<style>
<!--
#x_pfptBanner53g6uvq
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#D0D8DC!important;
max-width:none!important;
max-height:none!important}
-->
</style>
<div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: MPICH Version: 5. 0. 0a1</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="x_pfptBanner53g6uvq" style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#D0D8DC!important; background-color:#D0D8DC; border-top:4px solid #90a4ae!important; border-top:4px solid #90a4ae">
<div id="x_pfptBanner53g6uvq" style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBanner53g6uvq" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBanner53g6uvq" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">Hi Folks,
<div><br>
</div>
<div>We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH.</div>
<div><br>
</div>
<div>First the info:</div>
<div>system using libfabric 1.22.0</div>
<div>slurm - 24.11.5</div>
<div><br>
</div>
<div>Here's my mpichversion output:</div>
<div><br>
</div>
<div>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH Version:<span class="x_gmail-Apple-converted-space">
</span>5.0.0a1</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH Release date: unreleased development copy</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH ABI:<span class="x_gmail-Apple-converted-space">
</span>0:0:0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH Device:
<span class="x_gmail-Apple-converted-space"> </span>ch4:ofi</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH configure:<span class="x_gmail-Apple-converted-space">
</span>--prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default
--with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH CC:
<span class="x_gmail-Apple-converted-space"> </span>gcc <span class="x_gmail-Apple-converted-space">
</span>-O3</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH CXX:<span class="x_gmail-Apple-converted-space">
</span>g++ <span class="x_gmail-Apple-converted-space"> </span>-O3</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH F77:<span class="x_gmail-Apple-converted-space">
</span>gfortran <span class="x_gmail-Apple-converted-space"> </span>-O3</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH FC:
<span class="x_gmail-Apple-converted-space"> </span>gfortran <span class="x_gmail-Apple-converted-space">
</span>-O3</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">MPICH features:
<span class="x_gmail-Apple-converted-space"> </span>threadcomm<br>
<br>
<br>
<br>
And here's the OSU latency results:<br>
<br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective.</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get()</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span></span>seq=0 contribs: loc=1/prev=0/fwd=1</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span></span>neighbor contribs [2]:</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span><span class="x_gmail-Apple-tab-span" style="white-space:pre"></span></span>done contrib: -</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span><span class="x_gmail-Apple-tab-span" style="white-space:pre"></span></span>wait contrib: nid001439</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span></span>status=PMIXP_COLL_RING_PROGRESS</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span></span>buf (offset/size): 36/16384</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span></span>seq=0 contribs: loc=1/prev=0/fwd=1</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span></span>neighbor contribs [2]:</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span><span class="x_gmail-Apple-tab-span" style="white-space:pre"></span></span>done contrib: -</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span><span class="x_gmail-Apple-tab-span" style="white-space:pre"></span></span>wait contrib: nid001438</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span></span>status=PMIXP_COLL_RING_PROGRESS</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">slurmstepd: error:<span class="x_gmail-Apple-converted-space">
</span>mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829:<span class="x_gmail-Apple-converted-space"> <span class="x_gmail-Apple-tab-span" style="white-space:pre">
</span></span>buf (offset/size): 36/16384</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"># OSU MPI Latency Test v5.8</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"># Size<span class="x_gmail-Apple-converted-space">
</span>Latency (us)</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">0 <span class="x_gmail-Apple-converted-space">
</span>1.66</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">1 <span class="x_gmail-Apple-converted-space">
</span>9.29</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">2 <span class="x_gmail-Apple-converted-space">
</span>9.57</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">4 <span class="x_gmail-Apple-converted-space">
</span>9.69</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">8 <span class="x_gmail-Apple-converted-space">
</span>9.76</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">16<span class="x_gmail-Apple-converted-space">
</span>9.77</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">32<span class="x_gmail-Apple-converted-space">
</span>9.76</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">64<span class="x_gmail-Apple-converted-space">
</span>9.77</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">128<span class="x_gmail-Apple-converted-space">
</span>10.32</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">256 <span class="x_gmail-Apple-converted-space">
</span>7.54</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">512 <span class="x_gmail-Apple-converted-space">
</span>7.45</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">1024<span class="x_gmail-Apple-converted-space">
</span>7.38</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">2048<span class="x_gmail-Apple-converted-space">
</span>7.37</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">4096<span class="x_gmail-Apple-converted-space">
</span>7.45</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">8192<span class="x_gmail-Apple-converted-space">
</span>9.21</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">16384
<span class="x_gmail-Apple-converted-space"> </span>9.70</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">32768<span class="x_gmail-Apple-converted-space">
</span>10.63</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">65536<span class="x_gmail-Apple-converted-space">
</span>13.15</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">131072
<span class="x_gmail-Apple-converted-space"> </span>16.96</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">262144
<span class="x_gmail-Apple-converted-space"> </span>23.84</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">524288
<span class="x_gmail-Apple-converted-space"> </span>36.16</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">1048576<span class="x_gmail-Apple-converted-space">
</span>60.36</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">2097152
<span class="x_gmail-Apple-converted-space"> </span>108.43</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">4194304
<span class="x_gmail-Apple-converted-space"> </span>228.31</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs.</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting
more like 1.6 or so.</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
I am confident the slurm issue is unrelated to the latency issue.<br>
<br>
Thanks for any suggestions on how to address either issue however.</p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; font-kerning:auto; font-feature-settings:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; background-color:rgb(254,244,139)">
<br>
</p>
</div>
</div>
</div>
</body>
</html>