<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 2 6 3 5 4 5 2 3 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Consolas;
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:Consolas">Dear MPICH Gurus,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">I recently built MPICH 4.0b1 on a cluster I work on that happens to have a couple of different interconnects you can run on. One part is Omnipath and the other is Infiniband.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Now, when I built MPICH, I did so on the Infiniband cluster and a user recently tried my module and:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">$ mpifort -o helloWorld.mpi3.MPICH.PSM2.exe helloWorld.mpi3.F90<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">/usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0/lib/libmpi.so: undefined reference to `rdma_establish@RDMACM_1.0'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">collect2: error: ld returned 1 exit status<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Turns out, you try to use that MPICH on the Omnipath cluster, and boom. My "solution" for him was "Build and run on the Infiniband cluster" and that's fine for now as we don't use MPICH in production.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">But it got me thinking. Is there any way of building MPICH so that it would "nicely" support both? My first build was a "no extra arguments" type of build, but the configure output did say to maybe try
the ch4:ucx device (I think it chose ch4:ofi first). So, I built with "--with-device=ch4:ucx" (still on Infiniband) and this then does build on the Omnipath system...but throws a message on running:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">$ mpirun -np 4 ./helloWorld.mpi3.MPICH-UCX.PSM2.exe<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">libibcm: couldn't read ABI version<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">libibcm: couldn't read ABI version<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">libibcm: couldn't read ABI version<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">libibcm: couldn't read ABI version<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">I mean, it still *<b>ran</b>* fine, it just prints out one of those for each process.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">So I thought I'd ask the experts: is there any way to build an MPICH that is "happy" on both PSM2 and Infiniband? Or should I just tell my users that want to try out MPICH "Stick with Infiniband"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Note: There is no way for us to run a job on both clusters simultaneously, so it's not like I need something that will work on both at the same time. Just something that doesn't through warnings/messages
if possible.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Matt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">-- <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:Consolas">Matt Thompson, SSAI, Ld Scientific Programmer/Analyst<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:Consolas">NASA GSFC, Global Modeling and Assimilation Office<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:Consolas">Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:Consolas">Phone: 301-614-6712 Fax: 301-614-6246<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><u><span style="font-family:Consolas;color:#0000E9">http://science.gsfc.nasa.gov/sed/bio/matthew.thompson</span></u><o:p></o:p></p>
</div>
</body>
</html>