<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Dear Brent Morgan,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am glad you resolved your own issue. But I guess you are having the similar after taste that your resolution is less than ideal. Is that the reason for your ranting? It appears the original issue was the MPI_Bcast algorithm, that due
to your particular situation, neither the tree algorithm or ring algorithm are not performing? For the record, this is the first time I get a clue on where the issue is at. I recall the past emails were just complaining Isend and Irecv. There are two key steps
before a good communication or resolution can happen. First, all the key information need be communicated. I am not shifting the blame to users. This can be very difficult especially when we don’t know what information is critical. Very difficult or impossible
before we understand what is the issue – chicken and egg problem. So the best we can do for one, provide as much details as we (means you) think is relevant, and for two, keep finding new details that we omitted, and for three, be understandable. The second
key step is, of course, we (whoever you are asking for help from) need know the solution. Trust me, if we *<b>understand</b>* the issue and *<b>know</b>* the solution, we can’t wait to help you. But when we don’t understand your issue (due to lack of information
or probes) or don’t know the solution (research problems), you may feel that we are completely ignoring your ask for help. What response would you suggest (when we have no clue)?<br>
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <br>
Hui Zhou<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Brent Morgan via devel <devel@mpich.org><br>
<b>Date: </b>Thursday, February 18, 2021 at 1:02 AM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org>, devel@mpich.org <devel@mpich.org>, Robert Katona <robert.katona@hotmail.com><br>
<b>Cc: </b>Brent Morgan <brent.taylormorgan@gmail.com><br>
<b>Subject: </b>Re: [mpich-devel] Fatal error in PMPI_Isend: Unknown error class<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi all,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">As usual, we resolved our own issue. It turns out there was an overloading of MPI_Isend/MPI_Irecv with sending/receiving a vector of ~100 elements. We improved this by playing with the grid configuration of sub-communicators,
but that failed further down the road. We just switched to mpi_send/mpi_recv which we found to be more robust, but our experiments show this is slower (for reasons that differ between MPI_send and MPI_isend). The reason why we are using send/recv and not
BCAST, is because BCAST led to undesired timing behavior. It turns out (that we learned on our own), that broadcast sends data down a tree-like arrangement of ranks (when messages are small and communication time is dominated by network latency) or down a
ring of ranks (with large messages). If a rank on an intermediate tree level is late, it delays the broadcast to all other ranks beneath it in the sub-tree. Similarly, in the ring arrangement, a late rank delays all ranks after it in the ring. This can lead
to delays in subgroups of ranks if the ranks are not aligned in time (like it is for us).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Our team is wondering why there isn't
<b>any</b> support on these email discussion boards- is this the official place to get help for MPICH? Are there any other places like github discussions? I would imagine NASA or Argonne National Labs or other government laboratories are needing to communicate
issues that arise, or need to send/receive much more data than a vector of 100 elements without issues. The more we implement MPICH on our end, the less confident we are about this mpi implementation. Our errors we have observed have no documented error codes
or error traces; we have encountered many issues thus far (and reported them, with no responses)- which led to us scanning mpich source code and trying random things to arrive at solutions, which is far from ideal in any software development environment.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">We've arrived in a position that if we continue to experience more issues, we are being forced to switch to another implementation (openmpi) because of all the questions we've submitted, none of them were answered
or considered at all in neither <a href="mailto:discuss@mpich.org">discuss@mpich.org</a> or devel@mpich.org- and I see others' questions go unanswered as well. We don't understand why there isn't any documentation (at all) on some things we've encountered
(and others as well).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Brent<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On Sun, Feb 14, 2021 at 12:20 AM Brent Morgan <<a href="mailto:brent.taylormorgan@gmail.com" target="_blank">brent.taylormorgan@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal" style="margin-left:.5in">Hello all,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I am seeing this:<br>
<br>
"Abort(105) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Isend: Unknown error class"<br>
<br>
This is after my float vector I am sending to the nodes has increased to a size of ~70 elements. I cannot find any documentation about what this means. I tracked down where this fails and upon sending it to the 300th process (of 600), the MPI_ISend() command
dies and this error shows.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Is there anything I can do to further diagnose the issue? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Brent<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>