<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
MPICH doesn't support building with both libfabric and ucx at this point. We understand this use case and we may support it in the future.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-- <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui Zhou<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss@mpich.org><br>
<b>Sent:</b> Friday, January 7, 2022 11:16 AM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson@nasa.gov><br>
<b>Subject:</b> [mpich-discuss] Building for Heterogeneous Clusters</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Consolas}
@font-face
{}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle17
{font-family:Consolas;
color:windowtext;
font-weight:normal;
font-style:normal}
.x_MsoChpDefault
{font-size:12.0pt;
font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="font-family:Consolas">Dear MPICH Gurus,</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">I recently built MPICH 4.0b1 on a cluster I work on that happens to have a couple of different interconnects you can run on. One part is Omnipath and the other is Infiniband.
</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">Now, when I built MPICH, I did so on the Infiniband cluster and a user recently tried my module and:</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">$ mpifort -o helloWorld.mpi3.MPICH.PSM2.exe helloWorld.mpi3.F90</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">/usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0/lib/libmpi.so: undefined reference to `rdma_establish@RDMACM_1.0'</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">collect2: error: ld returned 1 exit status</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">Turns out, you try to use that MPICH on the Omnipath cluster, and boom. My "solution" for him was "Build and run on the Infiniband cluster" and that's fine for now as we don't use MPICH in production.</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">But it got me thinking. Is there any way of building MPICH so that it would "nicely" support both? My first build was a "no extra arguments" type of build, but the configure output did say to maybe try
the ch4:ucx device (I think it chose ch4:ofi first). So, I built with "--with-device=ch4:ucx" (still on Infiniband) and this then does build on the Omnipath system...but throws a message on running:</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">$ mpirun -np 4 ./helloWorld.mpi3.MPICH-UCX.PSM2.exe</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">libibcm: couldn't read ABI version</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">libibcm: couldn't read ABI version</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">libibcm: couldn't read ABI version</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">libibcm: couldn't read ABI version</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">I mean, it still *<b>ran</b>* fine, it just prints out one of those for each process.</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">So I thought I'd ask the experts: is there any way to build an MPICH that is "happy" on both PSM2 and Infiniband? Or should I just tell my users that want to try out MPICH "Stick with Infiniband"</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">Note: There is no way for us to run a job on both clusters simultaneously, so it's not like I need something that will work on both at the same time. Just something that doesn't through warnings/messages
if possible.</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">Thanks,</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">Matt</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">-- </span></p>
<div>
<p class="x_MsoNormal" style="text-autospace:none"><span style="font-family:Consolas">Matt Thompson, SSAI, Ld Scientific Programmer/Analyst</span></p>
<p class="x_MsoNormal" style="text-autospace:none"><span style="font-family:Consolas">NASA GSFC, Global Modeling and Assimilation Office</span></p>
<p class="x_MsoNormal" style="text-autospace:none"><span style="font-family:Consolas">Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771</span></p>
<p class="x_MsoNormal" style="text-autospace:none"><span style="font-family:Consolas">Phone: 301-614-6712 Fax: 301-614-6246</span></p>
</div>
<p class="x_MsoNormal"><u><span style="font-family:Consolas; color:#0000E9">http://science.gsfc.nasa.gov/sed/bio/matthew.thompson</span></u></p>
</div>
</div>
</body>
</html>