<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Ken, Sorry for the late reply. Things blew up at work. I'll work on getting this into a GitHub issue for you next week! Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610. 1,
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannery87e7yo" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannery87e7yo" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannery87e7yo" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannery87e7yo" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannery87e7yo { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtony87e7yo:hover, .pfptPrimaryButtony87e7yo:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtony87e7yo:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:Consolas;
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">Ken,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><br>
Sorry for the late reply. Things blew up at work. I'll work on getting this into a GitHub issue for you next week!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">Matt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">--<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">Matt Thompson, SSAI, Sr Scientific Programmer/Analyst<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">NASA GSFC, Global Modeling and Assimilation Office<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas">Phone: 301-614-6712 Fax: 301-614-6246<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><a href="https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.thompson__;!!G_uCfscf7eWS!dkcBBGeuCObo8sjHgGIOmCwxAopVeOwBgol4pqtYSPUfWgY7lGnqq-Yj93an-UQV_HWvpN-TwPElQdTI4cBElKLgYD4$"><span style="color:#0563C1">http://science.gsfc.nasa.gov/sed/bio/matthew.thompson</span></a></span><span style="font-size:12.0pt;font-family:Consolas"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Raffenetti, Ken <raffenet@anl.gov><br>
<b>Date: </b>Tuesday, April 2, 2024 at 9:26</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">PM<br>
<b>To: </b>Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson@nasa.gov>, discuss@mpich.org <discuss@mpich.org><br>
<b>Subject: </b>Re: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Can you provide details on how to run the application to reproduce the error? Preferably with as few processes as possible. I think we’ll need to do some more digging to get to the cause on our side. It would
also be good to transfer these details over to Github so we can better track the issue.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ken</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"Calibri",sans-serif;color:black">From:
</span></b><span style="font-family:"Calibri",sans-serif;color:black">"Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson@nasa.gov><br>
<b>Date: </b>Tuesday, March 26, 2024 at 11:11 AM<br>
<b>To: </b>"Raffenetti, Ken" <raffenet@anl.gov>, "discuss@mpich.org" <discuss@mpich.org><br>
<b>Subject: </b>Re: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:1.0pt;color:white">Ken, I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">6.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">0
(https:</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">//github.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">com/esmf-org/esmf/blob/ec5f18667091090df7e7b716d588955ce9aa4bd5/src/Infrastructure/VM/src/ESMCI_VMKernel.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">C#L466-L475)
#if (MPI_VERSION </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;margin-left:.5in;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0in 12.0pt 0in">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0in 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">Ken,</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8.6.0 (<a href="https://urldefense.us/v3/__https:/github.com/esmf-org/esmf/blob/ec5f18667091090df7e7b716d588955ce9aa4bd5/src/Infrastructure/VM/src/ESMCI_VMKernel.C*L466-L475__;Iw!!G_uCfscf7eWS!afLx5kWjU-Q0DepBQPBvQKQ3qCWj5W9vEMDCwhaW37NdsEXg6oFKMOmrmW3rg9nURQgShu7S8wjY7-7mKMT5lx82H_U8$">https://github.com/esmf-org/esmf/blob/ec5f18667091090df7e7b716d588955ce9aa4bd5/src/Infrastructure/VM/src/ESMCI_VMKernel.C#L466-L475</a>)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">#if (MPI_VERSION >= 3)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> // set up communicator across single-system-images SSIs</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> MPI_Comm_split_type(mpi_c, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL,</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> &mpi_c_ssi);</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> // set up communicator across root pets of each SSI</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> int color;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> MPI_Comm_rank(mpi_c_ssi, &color);</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> if (color>0) color = MPI_UNDEFINED; // only root PETs on each SSI</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> MPI_Comm_split(mpi_c, color, 0, &mpi_c_ssi_roots);</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">#endif</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">Indeed, if it's coming from ESMF, that is the *<b>only</b>* MPI_Comm_split_type in the whole code!</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">We also call MPI_Comm_split_type in the overall code (MAPL) and it also only uses that:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"><a href="https://urldefense.us/v3/__https:/github.com/search?q=repo*3AGEOS-ESM*2FMAPL*20mpi_comm_split_type&type=code__;JSUl!!G_uCfscf7eWS!afLx5kWjU-Q0DepBQPBvQKQ3qCWj5W9vEMDCwhaW37NdsEXg6oFKMOmrmW3rg9nURQgShu7S8wjY7-7mKMT5lz5W2yRj$">https://github.com/search?q=repo%3AGEOS-ESM%2FMAPL%20mpi_comm_split_type&type=code</a></span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">--
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">NASA GSFC, Global Modeling and Assimilation Office</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas">Phone: 301-614-6712 Fax: 301-614-6246</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"><a href="https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thompson__;!!G_uCfscf7eWS!afLx5kWjU-Q0DepBQPBvQKQ3qCWj5W9vEMDCwhaW37NdsEXg6oFKMOmrmW3rg9nURQgShu7S8wjY7-7mKMT5lwtzstq4$">http://science.gsfc.nasa.gov/sed/bio/matthew.thompson</a></span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Raffenetti, Ken <raffenet@anl.gov><br>
<b>Date: </b>Tuesday, March 26, 2024 at 10:55</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson@nasa.gov><br>
<b>Subject: </b>[EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">It looks like the crash is happening after shared memory window creation fails. The failure path is getting tripped up removing the window id from the global hash, since it was never
added. We will address this in the code so users get a better error message after the failure.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Can you confirm that the input communicator to the window creation function is one created with MPI_Comm_split_type(…,</span><span style="font-size:9.0pt;font-family:"Verdana",sans-serif;color:black;background:white">MPI_COMM_TYPE_SHARED,…)?</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Verdana",sans-serif;color:black;background:white"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Verdana",sans-serif;color:black;background:white">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Verdana",sans-serif;color:black;background:white">Ken</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:1.0in"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:
</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">"Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss" <discuss@mpich.org><br>
<b>Reply-To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Date: </b>Tuesday, March 26, 2024 at 9:20 AM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>"Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson@nasa.gov><br>
<b>Subject: </b>[mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:1.0pt;color:white">All, I've been trying to get a code of mine working with MPICH 4.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">2.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">0.
I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;margin-left:1.0in;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0in 12.0pt 0in">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0in 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">All,</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">I've been trying to get a code of mine working with MPICH 4.2.0. I can build MPICH just fine and then build our base libraries and then model and all compiles
fine. Hello world runs fine on multiple nodes as well.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">But when I finally try and run our complex model:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">Assertion failed in file ../src/mpid/ch4/src/ch4_impl.h at line 838: map_entry != NULL</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(+0x37d211) [0x14bf4f62c211]</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(PMPI_Win_allocate_shared+0x3ba) [0x14bf4f3e452a]</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">/discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI3VMK14ssishmAllocateERSt6vectorImSaImEEPNS0_9memhandleEb+0x18b)
[0x14bf6e91481b]</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">/discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI5Array6createEPNS_9ArraySpecEPNS_8DistGridEPNS_10InterArrayIiEES7_S7_S7_S7_S7_S7_P14ESMC_IndexFlagP13ESMC_Pin_FlagS7_S7_S7_PiPNS_2VME+0x2707)
[0x14bf6e44a267]</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">What I'm mainly wondering is if anyone has any experience with an error like this? My guess (at the moment) is that I built things wrong for an Infiniband cluster
maybe?</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">I'm using Intel Fortran Classic 2021.11.0 with GCC 11.4.0 as a backing C compiler and I built as:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> mkdir build-ifort-2021.11.0 && cd build-ifort-2021.11.0</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> ../configure CC=icx CXX=icpx FC=ifort \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> --with-ucx=embedded --with-hwloc=embedded --with-libfabric=embedded --with-yaksa=embedded \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0 |& tee configure.ifort-2021.11.0.log</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">All those "embedded" flags are mainly because with Open MPI on this system, I have to do something similar with its configure step:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> --with-hwloc=internal --with-libevent=internal --with-pmix=internal</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">so I figured I should do the same with MPICH></span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">Now, at the end of the configure step I did see:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">*****************************************************</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">***</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">*** device : ch4:ofi (embedded libfabric)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">*** shm feature : auto</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">*** gpu support : disabled</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">***</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> MPICH is configured with device ch4:ofi, which should work</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> for TCP networks and any high-bandwidth interconnect</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> supported by libfabric. MPICH can also be configured with</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> "--with-device=ch4:ucx", which should work for TCP networks</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> and any high-bandwidth interconnect supported by the UCX</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> library. In addition, the legacy device ch3 (--with-device=ch3)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> is also available.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">*****************************************************</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">And I did try the `--with-device=ch4:ucx` but that didn't seem to help. And the system I am on is an Infiniband network, so I imagine ofi should work.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">Note that this code works fine with Intel MPI and Open MPI (which are our "main" MPI stacks), but some of our external users are asking about MPICH support.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">Matt</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">--
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">NASA GSFC, Global Modeling and Assimilation Office</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas">Phone: 301-614-6712 Fax: 301-614-6246</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt;font-family:Consolas"><a href="https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thompson__;!!G_uCfscf7eWS!Y6jfxrSalGUYiT8VqK_4OwkY3bftJ-gwM5C6AHyrxvP2BZZvQlHGBeYZnUWmkPQJN7-mWjRBpQg60pHQJKXDetYFzss$">http://science.gsfc.nasa.gov/sed/bio/matthew.thompson</a></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>