<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Alim, <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for the details. That looks like data corruptions, probably due to some of the atomics used are not strong enough on the particular CPU architecture. Ch3 in some place uses weaker atomics that runs fine on the known architecture,
but you may have just provided an example that the weaker atomics is insufficient. If running Ch3 is important for you, please file an issue over
<a href="https://github.com/pmodels/mpich">https://github.com/pmodels/mpich</a> and we will track it down.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <br>
Hui Zhou<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Alim Akhtar <alim.akhtar@gmail.com><br>
<b>Date: </b>Thursday, May 27, 2021 at 9:20 AM<br>
<b>To: </b>Zhou, Hui <zhouh@anl.gov><br>
<b>Cc: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Subject: </b>Re: [mpich-discuss] mpich3 error with ch3<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi Hui<br>
<br>
<br>
On Thu, May 27, 2021 at 7:28 PM Zhou, Hui <zhouh@anl.gov> wrote:<br>
><br>
> Similar issues can be very different in causes. Checking the referenced discussion, I wasn’t sure what was the original issues. We suggested to try ch4 as to get more data points rather than as a solution. Nevertheless, ch4 is the current recommended device
as it is more actively developed.<br>
><br>
><br>
><br>
> Ch3 is not broken as far as we know. Could you describe your issue in more details?<br>
><br>
I get<br>
Assertion failed in file<br>
src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 530:<br>
payload_len >= sizeof (MPIDI_CH3_Pkt_t)<br>
after some loop.<br>
or sometime I see like :<br>
Assertion failed in file<br>
src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 567:<br>
!vc_ch->recv->active<br>
<br>
The number of pass loop does depends on number of CPUs used. (more<br>
number of CPUs more failure).<br>
With One CPU, no failure.<br>
<br>
<br>
><br>
><br>
> --<br>
> Hui Zhou<br>
><br>
><br>
><br>
><br>
><br>
> From: Alim Akhtar via discuss <discuss@mpich.org><br>
> Date: Wednesday, May 26, 2021 at 10:51 PM<br>
> To: discuss@mpich.org <discuss@mpich.org><br>
> Cc: Alim Akhtar <alim.akhtar@gmail.com><br>
> Subject: [mpich-discuss] mpich3 error with ch3<br>
><br>
> Hi mpich dev team,<br>
><br>
> I am facing one issue similar to discussed in below discussion<br>
><br>
> <a href="https://lists.mpich.org/pipermail/devel/2021-January/000826.html">https://lists.mpich.org/pipermail/devel/2021-January/000826.html</a><br>
><br>
> Someone in the mailing list suggested recompiling the mpi bench using<br>
> CH4. as below:<br>
><br>
> <a href="https://lists.mpich.org/pipermail/devel/2021-January/000828.html">https://lists.mpich.org/pipermail/devel/2021-January/000828.html</a><br>
><br>
> "MPICH with ch4, with `--with-device=ch4:ofi`"<br>
><br>
> Actually this fixes the failure on this CPU architecture.<br>
><br>
> Questions:<br>
> 1. Is this a known issue with MPI bench on the recent CPU<br>
> architecture? (I am running on ARM's Cortex -A 75), like ch3 is<br>
> broken?<br>
> 2. With no error after using CH4, does this mean the CPU is all good?<br>
><br>
> Note: using ch3 was working fine on our previous CPU (A-72)<br>
><br>
> Any input will be really appreciated.<br>
><br>
><br>
> --<br>
> Regards,<br>
> Alim<br>
> _______________________________________________<br>
> discuss mailing list discuss@mpich.org<br>
> To manage subscription options or unsubscribe:<br>
> <a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
<br>
<br>
<br>
-- <br>
Regards,<br>
Alim<o:p></o:p></p>
</div>
</div>
</body>
</html>