<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Apparently I was having some issues with respect to the libraries that were getting configured when I built MPICH versus the libraries that were available on the compute nodes. I’ve got that sorted out. I configure with --with-device=ch4: ofi
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannervtcmybb" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannervtcmybb" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannervtcmybb" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannervtcmybb" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannervtcmybb { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonvtcmybb:hover, .pfptPrimaryButtonvtcmybb:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonvtcmybb:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.xxxxmsonormal, li.xxxxmsonormal, div.xxxxmsonormal
{mso-style-name:x_xxxmsonormal;
margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
p.xxxxxmsonormal, li.xxxxxmsonormal, div.xxxxxmsonormal
{mso-style-name:x_xxxxmsonormal;
margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;}
p.xxxxxp1, li.xxxxxp1, div.xxxxxp1
{mso-style-name:x_xxxxp1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.xxxxxs2
{mso-style-name:x_xxxxs2;
background:#878A04;}
span.xxxxxs1
{mso-style-name:x_xxxxs1;}
span.xxxxxapple-converted-space
{mso-style-name:x_xxxxapple-converted-space;}
span.EmailStyle34
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Apparently I was having some issues with respect to the libraries that were getting configured when I built MPICH versus the libraries that were available on the compute nodes. I’ve got that sorted out.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I configure with --with-device=ch4:ofi and got rid of the --without-ucx flag. I’m still getting a crash on MPI_Wait. If I don’t set FI_PROVIDER, then I get the error<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="p1"><span class="s1">Bad : modifier in $ (-).</span><o:p></o:p></p>
<p class="p1"><span class="s1">Abort(944950159) on node 1: Fatal error in internal_Wait: Other MPI error, error stack:</span><o:p></o:p></p>
<p class="p1"><span class="s1">internal_Wait(68205)..........: MPI_Wait(request=0x5e88690, status=0x7ffc63581a40) failed</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIR_Wait(780)................:</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIR_Wait_state(737)..........:</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_progress_test(134)......:</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_handle_cq_error(791): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If I set FI_PROVIDER=verbs, I get<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="p1"><span class="s1">Abort(673331087) on node 5 (rank 5 in comm 496): Fatal error in internal_Recv: Other MPI error, error stack:</span><o:p></o:p></p>
<p class="p1"><span class="s1">internal_Recv(61700)..........: MPI_Recv(buf=0x2b4e0fa00000, count=4194304, MPI_CHAR, 0, 27624, comm=0x84000001, status=0x7ffdb2e9bdd0) failed</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIR_Wait(780)................:</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIR_Wait_state(737)..........:</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_progress_test(134)......:</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_handle_cq_error(791): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Bad address)<o:p></o:p></span></p>
<p class="p1"><span class="s1"><o:p> </o:p></span></p>
<p class="p1"><span class="s1">This error appears on ranks 2 and 5 of a 6 processor job on 2 nodes, but the remaining nodes still appear to be failing on an MPI_Wait call.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">Zhou, Hui <zhouh@anl.gov><br>
<b>Date: </b>Friday, June 21, 2024 at 11:03</span><span style="font-family:"Arial",sans-serif;color:black"> </span><span style="color:black">AM<br>
<b>To: </b>Palmer, Bruce J <Bruce.Palmer@pnnl.gov>, discuss@mpich.org <discuss@mpich.org><br>
<b>Subject: </b>Re: Fail on MPI_Wait<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">>../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> --with-device=ch4:ofi:sockets --with-libfabric=embedded \<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> --without-ucx CC=gcc CXX=g++<br>
<br>
You are statically using "sockets" provider. Try </span><code><span style="font-size:10.0pt;color:black">--with-device=ch4:ofi</span></code><span style="font-family:"Arial",sans-serif;color:black"></span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Hui<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="63%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Palmer, Bruce J <Bruce.Palmer@pnnl.gov><br>
<b>Sent:</b> Friday, June 21, 2024 12:27 PM<br>
<b>To:</b> Zhou, Hui <zhouh@anl.gov>; discuss@mpich.org <discuss@mpich.org><br>
<b>Subject:</b> RE: Fail on MPI_Wait</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hui, When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process: #include "mpi.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">h"
int main(int argc, char **argv) { MPI_Init(&argc, &argv); MPI_Finalize(); } I’m running on a
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<div style="border:none;border-top:solid #90A4AE 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#D0D8DC!important;border-top:#90a4ae!important" id="x_pfptBanneraxqgm0d">
<div id="x_pfptBanneraxqgm0d">
<div id="x_pfptBanneraxqgm0d">
<p class="MsoNormal" style="line-height:13.5pt;background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</div>
<div id="x_pfptBanneraxqgm0d">
<p class="MsoNormal" style="line-height:13.5pt;background:#D0D8DC"><span style="font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="background:#D0D8DC"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hui,</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process:</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">#include "mpi.h"</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">int main(int argc, char **argv) {</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> MPI_Init(&argc, &argv);</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> MPI_Finalize();</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">}</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I’m running on a system with the following modules</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[d3g293@deception02 testing]$ module list</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Currently Loaded Modulefiles:</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> 1) gcc/11.2.0 3) python/3.7.0 5) mkl/2019u4</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> 2) cmake/3.21.4 4) git/2.42.0(default) 6) cuda/11.8</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">and a home-built version of mpich-4.2.1 configured with</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> --with-device=ch4:ofi:sockets --with-libfabric=embedded \</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> --without-ucx CC=gcc CXX=g++</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I thought it might have something to do with using a build configuration in my application build that is set up to include Cuda, but it also fails in MPI_Init
with a non-Cuda configuraton if I set the FI_PROVIDER variable.</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
Bruce</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Zhou, Hui <zhouh@anl.gov>
<br>
<b>Sent:</b> Friday, June 14, 2024 9:27 AM<br>
<b>To:</b> Palmer, Bruce J <Bruce.Palmer@pnnl.gov>; discuss@mpich.org<br>
<b>Subject:</b> Re: Fail on MPI_Wait</span><o:p></o:p></p>
</div>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black">Never mind. It is v4.2.1.</span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-family:"Times New Roman",serif">
<hr size="0" width="63%" align="center">
</span></div>
<div id="x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Zhou, Hui <</span><a href="mailto:zhouh@anl.gov"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">zhouh@anl.gov</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">><br>
<b>Sent:</b> Friday, June 14, 2024 11:26 AM<br>
<b>To:</b> Palmer, Bruce J <</span><a href="mailto:Bruce.Palmer@pnnl.gov"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Bruce.Palmer@pnnl.gov</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">>;
</span><a href="mailto:discuss@mpich.org"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">discuss@mpich.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> <</span><a href="mailto:discuss@mpich.org"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">discuss@mpich.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">><br>
<b>Subject:</b> Re: Fail on MPI_Wait</span> <o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black">Bruce,</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black">What is the mpich version, BTW?</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black">--</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black">Hui</span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-family:"Times New Roman",serif">
<hr size="0" width="63%" align="center">
</span></div>
<div id="x_x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Zhou, Hui <</span><a href="mailto:zhouh@anl.gov"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">zhouh@anl.gov</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">><br>
<b>Sent:</b> Friday, June 14, 2024 10:55 AM<br>
<b>To:</b> Palmer, Bruce J <</span><a href="mailto:Bruce.Palmer@pnnl.gov"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Bruce.Palmer@pnnl.gov</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">>;
</span><a href="mailto:discuss@mpich.org"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">discuss@mpich.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> <</span><a href="mailto:discuss@mpich.org"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">discuss@mpich.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">><br>
<b>Subject:</b> Re: Fail on MPI_Wait</span> <o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black">Bruce,</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black">You are using the sockets provider. Could try set
</span><code><span style="font-size:10.0pt;color:black">FI_PROVIDER=tcp</span></code><span style="font-family:"Arial",sans-serif;color:black"></span><span style="font-family:"Aptos",sans-serif;color:black"> to see if it makes a difference?<br>
<br>
Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black">--</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Aptos",sans-serif;color:black">Hui</span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-family:"Times New Roman",serif">
<hr size="0" width="63%" align="center">
</span></div>
<div id="x_x_x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Palmer, Bruce J <</span><a href="mailto:Bruce.Palmer@pnnl.gov"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Bruce.Palmer@pnnl.gov</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">><br>
<b>Sent:</b> Friday, June 14, 2024 10:47 AM<br>
<b>To:</b> Zhou, Hui <</span><a href="mailto:zhouh@anl.gov"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">zhouh@anl.gov</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">>;
</span><a href="mailto:discuss@mpich.org"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">discuss@mpich.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> <</span><a href="mailto:discuss@mpich.org"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">discuss@mpich.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">><br>
<b>Subject:</b> Re: Fail on MPI_Wait</span> <o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-size:1.0pt;color:white">The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org>
Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #90A4AE 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#D0D8DC!important;border-top:#90a4ae!important" id="x_x_x_x_pfptBannerbvkjra5">
<div id="x_x_x_x_pfptBannerbvkjra5">
<div id="x_x_x_x_pfptBannerbvkjra5">
<p class="xmsonormal" style="line-height:13.5pt;background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><o:p></o:p></p>
</div>
<div id="x_x_x_x_pfptBannerbvkjra5">
<p class="xmsonormal" style="line-height:13.5pt;background:#D0D8DC"><span style="font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xmsonormal" style="background:#D0D8DC"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-size:11.0pt">The output to standard out from running on 2 nodes and one process per node is attached.</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div id="x_x_x_x_mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xxxxmsonormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">Zhou, Hui <</span><a href="mailto:zhouh@anl.gov">zhouh@anl.gov</a><span style="color:black">><br>
<b>Date: </b>Tuesday, June 11, 2024 at 5:49</span><span style="font-family:"Arial",sans-serif;color:black"> </span><span style="color:black">PM<br>
<b>To: </b></span><a href="mailto:discuss@mpich.org">discuss@mpich.org</a><span style="color:black"> <</span><a href="mailto:discuss@mpich.org">discuss@mpich.org</a><span style="color:black">><br>
<b>Cc: </b>Palmer, Bruce J <</span><a href="mailto:Bruce.Palmer@pnnl.gov">Bruce.Palmer@pnnl.gov</a><span style="color:black">><br>
<b>Subject: </b>Re: Fail on MPI_Wait</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="color:black">>MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)<br>
<br>
This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable
</span><code><span style="font-size:10.0pt;color:black">MPIR_CVAR_DEBUG_SUMMARY=1</span></code><span style="font-family:"Arial",sans-serif;color:black"></span><span style="color:black"> and run a simple
</span><code><span style="font-size:10.0pt;color:black">MPI_INIT+MPI_Finalize</span></code><span style="font-family:"Arial",sans-serif;color:black"></span><span style="color:black"> test code. Could post its console output?</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="color:black">--</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="color:black">Hui</span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="63%" align="center">
</div>
<div id="x_x_x_x_divRplyFwdMsg">
<p class="xxxxmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Palmer, Bruce J via discuss <</span><a href="mailto:discuss@mpich.org"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">discuss@mpich.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">><br>
<b>Sent:</b> Tuesday, June 11, 2024 3:17 PM<br>
<b>To:</b> </span><a href="mailto:discuss@mpich.org"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">discuss@mpich.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> <</span><a href="mailto:discuss@mpich.org"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">discuss@mpich.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">><br>
<b>Cc:</b> Palmer, Bruce J <</span><a href="mailto:Bruce.Palmer@pnnl.gov"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Bruce.Palmer@pnnl.gov</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">><br>
<b>Subject:</b> [mpich-discuss] Fail on MPI_Wait</span> <o:p></o:p></p>
<div>
<p class="xxxxmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xxxxmsonormal"><span style="font-size:1.0pt;color:white">Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down
why. Currently, we are getting failures in MPI_Wait </span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #90A4AE 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#D0D8DC!important;border-top:#90a4ae!important" id="x_x_x_x_x_pfptBannerbiv18om">
<div id="x_x_x_x_x_pfptBannerbiv18om">
<div id="x_x_x_x_x_pfptBannerbiv18om">
<p class="xxxxmsonormal" style="line-height:13.5pt;background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><o:p></o:p></p>
</div>
<div id="x_x_x_x_x_pfptBannerbiv18om">
<p class="xxxxmsonormal" style="line-height:13.5pt;background:#D0D8DC"><span style="font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xxxxmsonormal" style="background:#D0D8DC"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxxmsonormal">Hi,<o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxmsonormal">I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in
MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is<o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxp1"><span class="xxxxxs1">Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack:</span><o:p></o:p></p>
<p class="xxxxxp1"><span class="xxxxxs1">internal_Wait(68205)..........: </span><span class="xxxxxs2">MPI_Wai</span><span class="xxxxxs1">t(request=0x500847a0, status=0x7ffff9331800) failed</span><o:p></o:p></p>
<p class="xxxxxp1"><span class="xxxxxs1">MPIR_Wait(780)................:</span><o:p></o:p></p>
<p class="xxxxxp1"><span class="xxxxxs1">MPIR_Wait_state(737)..........:</span><o:p></o:p></p>
<p class="xxxxxp1"><span class="xxxxxs1">MPIDI_progress_test(134)......:</span><o:p></o:p></p>
<p class="xxxxxp1"><span class="xxxxxs1">MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)</span><o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxmsonormal">I’ve verified that the handle corresponding to <span class="xxxxxs1">
0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with</span><o:p></o:p></p>
<p class="xxxxxmsonormal"><span class="xxxxxs1"> </span><o:p></o:p></p>
<p class="xxxxxp1"><span class="xxxxxs1">../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \</span><o:p></o:p></p>
<p class="xxxxxp1"><span class="xxxxxapple-converted-space"> </span><span class="xxxxxs1">--with-device=ch4:ofi:sockets --with-libfabric=embedded \</span><o:p></o:p></p>
<p class="xxxxxp1"><span class="xxxxxapple-converted-space"> </span><span class="xxxxxs1">--without-ucx --enable-threads=multiple --with-slurm \</span><o:p></o:p></p>
<p class="xxxxxp1"><span class="xxxxxapple-converted-space"> </span><span class="xxxxxs1">CC=gcc CXX=g+</span><o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxmsonormal">I’ve tried building with UCX and gotten the same results.<o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxmsonormal">Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated.<o:p></o:p></p>
<p class="xxxxxmsonormal"><br>
Thanks,<o:p></o:p></p>
<p class="xxxxxmsonormal">Bruce<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>