<!DOCTYPE html>
<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Thanks, Hui! I am trying to have one build that works for our cluster and also for AWS, which uses ofi. Is that possible? Thx. . . . John On 6/11/26 10: 06 AM, Zhou, Hui wrote: [External email - use caution] HI John, Try using the UCX instead of</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanner8tkpnce" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanner8tkpnce" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanner8tkpnce" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanner8tkpnce" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html>
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanner8tkpnce { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButton8tkpnce:hover, .pfptPrimaryButton8tkpnce:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButton8tkpnce:active {
background-color: #90a4ae !important; }
html:root, html:root>body { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<font face="monospace">Thanks, Hui!<br>
<br>
I am trying to have one build that works for our cluster and also
for AWS, which uses ofi. Is that possible?<br>
<br>
Thx....John<br>
<br>
<br>
</font><br>
<div class="moz-cite-prefix">On 6/11/26 10:06 AM, Zhou, Hui wrote:<br>
</div>
<blockquote type="cite"
cite="mid:SA0PR09MB7417876DEF2E049F565C5F88A91B2@SA0PR09MB7417.namprd09.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div>[External email - use caution]</div>
<br>
<div>
<div class="elementToProof"
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
HI John,</div>
<div class="elementToProof"
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof"
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Try using the UCX instead of libfabric. You can configure
MPICH with <code>./configure --with-device=ch4:ucx</code> to
use UCX. With libfabric, could you try set environment
variable
<code>FI_PROVIDER=verbs ?</code></div>
<div class="elementToProof"
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof"
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
If you still have issue, try run a dummy MPI program setting
`MPIR_CVAR_DEBUG_SUMMARY=1`. That will provide more logging
details on which libfabric provider is being selected.</div>
<div class="elementToProof"
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof"
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-- </div>
<div class="elementToProof"
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui Zhou</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font
face="Calibri, sans-serif" style="font-size:11pt"
color="#000000"><b>From:</b> John Cary via discuss
<a class="moz-txt-link-rfc2396E" href="mailto:discuss@mpich.org"><discuss@mpich.org></a><br>
<b>Sent:</b> Thursday, June 11, 2026 8:50 AM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:discuss@mpich.org">discuss@mpich.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:discuss@mpich.org"><discuss@mpich.org></a><br>
<b>Cc:</b> John Cary <a class="moz-txt-link-rfc2396E" href="mailto:cary@colorado.edu"><cary@colorado.edu></a><br>
<b>Subject:</b> [mpich-discuss] How to get srun/mpich to use
the right interface?</font>
<div> </div>
</div>
<div>
<div
style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
How to get mpich to use the right interface? mpich
configured and built with libfabric as shown below. It is
run using slurm (srun). The result is
[1781142435. 559314543] ne07: rank64. vorpal: Failed to
modify UD QP to INIT on mlx5_0: Operation</div>
<div
style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="x_pfptBannervvt4hsp"
style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#D0D8DC!important; background-color:#D0D8DC; border-top:4px solid #90a4ae!important; border-top:4px solid #90a4ae">
<div id="x_pfptBannervvt4hsp"
style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBannervvt4hsp"
style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBannervvt4hsp"
style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div
style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div
style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<style>#x_pfptBannervvt4hsp
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#D0D8DC!important;
max-width:none!important;
max-height:none!important}html:root, html:root > div
{display:block!important;
visibility:visible!important;
opacity:1!important}</style>
<pre
style="font-family:sans-serif; font-size:100%; white-space:pre-wrap; word-wrap:break-word">How to get mpich to use the right interface?
mpich configured and built with libfabric as shown below. It is run
using slurm (srun). The result is
[1781142435.559314543] ne07:rank64.vorpal: Failed to modify UD QP to
INIT on mlx5_0: Operation not permitted
[1781142435.563376291] ne07:rank66.vorpal: Failed to modify UD QP to
INIT on mlx5_0: Operation not permitted
Abort(203572367): Fatal error in internal_Init: Other MPI error, error stack
...
which I think means that mpich is txrying to run over the mlx5_0
interface, which does not exist. The interfaces are
eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
and we want to use ib0, infiniband. We tried
export HYDRA_IFACE=ib0
srun ...
and got the same error.
How can srun/mpich be instructed to use the ib0 interface by default?
Also, how can one see which interface mpich is choosing?
Thx...
'/user/builds-linux-centos8-zen2/gvxsimall/mpich-5.0.1/configure' \
--prefix=/home/research/user/installs/linux-centos8-zen2/contrib-gcc1140/mpich-5.0.1-shared
\
--enable-shared \
--disable-static \
CC='/home/common/gcc-11.4.0/bin/gcc' \
CXX='/home/common/gcc-11.4.0/bin/g++' \
FC='/home/common/gcc-11.4.0/bin/gfortran' \
F77='/home/common/gcc-11.4.0/bin/gfortran' \
CFLAGS='-pthread -pipe -fPIC' \
CXXFLAGS='-pthread -Wno-deprecated-declarations -pipe -fPIC' \
FFLAGS='-fallow-argument-mismatch -pipe -fPIC' \
FCFLAGS='-fallow-argument-mismatch -pipe -fPIC' \
LDFLAGS='-L/home/common/gcc-11.4.0/lib64
-Wl,-rpath,/home/common/gcc-11.4.0/lib64' \
LDSHARED='-L/home/common/gcc-11.4.0/lib64
-Wl,-rpath,/home/common/gcc-11.4.0/lib64' \
--with-libfabric=install \
--with-device=ch4:ofi \
--without-cuda \
--disable-gl
</pre>
</div>
</div>
</blockquote>
<br>
</body>
</html>