<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
All, I've been trying to get a code of mine working with MPICH 4. 2. 0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanner61e76b4" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanner61e76b4" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanner61e76b4" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanner61e76b4" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanner61e76b4 { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButton61e76b4:hover, .pfptPrimaryButton61e76b4:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButton61e76b4:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Consolas;
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">I've been trying to get a code of mine working with MPICH 4.2.0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine
on multiple nodes as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">But when I finally try and run our complex model:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">Assertion failed in file ../src/mpid/ch4/src/ch4_impl.h at line 838: map_entry != NULL<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(+0x37d211) [0x14bf4f62c211]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(PMPI_Win_allocate_shared+0x3ba) [0x14bf4f3e452a]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">/discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI3VMK14ssishmAllocateERSt6vectorImSaImEEPNS0_9memhandleEb+0x18b)
[0x14bf6e91481b]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">/discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI5Array6createEPNS_9ArraySpecEPNS_8DistGridEPNS_10InterArrayIiEES7_S7_S7_S7_S7_S7_P14ESMC_IndexFlagP13ESMC_Pin_FlagS7_S7_S7_PiPNS_2VME+0x2707)
[0x14bf6e44a267]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">What I'm mainly wondering is if anyone has any experience with an error like this? My guess (at the moment) is that I built things wrong for an Infiniband cluster maybe?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">I'm using Intel Fortran Classic 2021.11.0 with GCC 11.4.0 as a backing C compiler and I built as:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> mkdir build-ifort-2021.11.0 && cd build-ifort-2021.11.0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> ../configure CC=icx CXX=icpx FC=ifort \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> --with-ucx=embedded --with-hwloc=embedded --with-libfabric=embedded --with-yaksa=embedded \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0 |& tee configure.ifort-2021.11.0.log<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">All those "embedded" flags are mainly because with Open MPI on this system, I have to do something similar with its configure step:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> --with-hwloc=internal --with-libevent=internal --with-pmix=internal<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">so I figured I should do the same with MPICH><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">Now, at the end of the configure step I did see:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">*****************************************************<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">***<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">*** device : ch4:ofi (embedded libfabric)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">*** shm feature : auto<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">*** gpu support : disabled<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">***<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> MPICH is configured with device ch4:ofi, which should work<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> for TCP networks and any high-bandwidth interconnect<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> supported by libfabric. MPICH can also be configured with<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> "--with-device=ch4:ucx", which should work for TCP networks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> and any high-bandwidth interconnect supported by the UCX<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> library. In addition, the legacy device ch3 (--with-device=ch3)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"> is also available.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">*****************************************************<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">And I did try the `--with-device=ch4:ucx` but that didn't seem to help. And the system I am on is an Infiniband network, so I imagine ofi should work.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">Note that this code works fine with Intel MPI and Open MPI (which are our "main" MPI stacks), but some of our external users are asking about MPICH support.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">Matt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;mso-ligatures:none">--
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;mso-ligatures:none">Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;mso-ligatures:none">NASA GSFC, Global Modeling and Assimilation Office<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;mso-ligatures:none">Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;mso-ligatures:none">Phone: 301-614-6712 Fax: 301-614-6246<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;mso-ligatures:none"><a href="https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.thompson__;!!G_uCfscf7eWS!Y6jfxrSalGUYiT8VqK_4OwkY3bftJ-gwM5C6AHyrxvP2BZZvQlHGBeYZnUWmkPQJN7-mWjRBpQg60pHQJKXDetYFzss$">http://science.gsfc.nasa.gov/sed/bio/matthew.thompson</a></span><o:p></o:p></p>
</div>
</body>
</html>