<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1924727815;
mso-list-type:hybrid;
mso-list-template-ids:1038646140 1709371382 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0D8 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"> “Our MPI implementation is merely finding the sum of the results of the N processes, where N is large. Is MPI_Reduce going to be faster? <o:p></o:p></p>
<p class="MsoNormal">“<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Oh, yeah, if you are doing reduce, you should call `MPI_Reduce`.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">However, I suspect there maybe some usage errors involved. Could you post some sample/pseudo code?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <br>
Hui Zhou<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Brent Morgan <brent.taylormorgan@gmail.com><br>
<b>Date: </b>Thursday, January 14, 2021 at 11:02 PM<br>
<b>To: </b>Robert Katona <robert.katona@hotmail.com><br>
<b>Cc: </b>Zhou, Hui <zhouh@anl.gov>, devel@mpich.org <devel@mpich.org><br>
<b>Subject: </b>Re: [mpich-devel] mpich3 error<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Hi Hui Zhou,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Robert and I managed to incorporate multiple communicators- we use MPI_Gather() of Send/Receives. However the issue remains- for a small (N<50) # of threads, the calculations work and seem fine. For large (N>=50) # of threads, the issue
persists. We will try compiling for ch4 tonight... But I wonder if we're doing something wrong? Should we do Gather of Gathers? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Our MPI implementation is merely finding the sum of the results of the N processes, where N is large. Is MPI_Reduce going to be faster? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Brent<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jan 14, 2021 at 2:15 AM Robert Katona <<a href="mailto:robert.katona@hotmail.com" target="_blank">robert.katona@hotmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hi </span><span style="color:black;background:white">Hui Zhou,</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black;background:white">Thanks for the findings. You are right, we used one communicator. And my idea was also to recompile the MPICH with ch4.</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black;background:white">First I would like to try the MPI_Split() and use multiple communicator. But I stuck on the implementation. In the attachment you can see a toy app where all node want to share a float value with
world rank 0.</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black;background:white">But when I run the app for 20 processes I got the following output:</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black;background:white">0, 1, 2, 3, 4, 5, ,6, 7, 8, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, </span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black;background:white">But the expected is:</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black;background:white">0, 1, 2, 3, 4, 5, ,6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black;background:white">What am I missing?</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black;background:white">Regards,</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black;background:white">Robert </span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="86%" align="center">
</div>
<div id="gmail-m_-1328304056524589182gmail-m_-7390180511783217357divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">Feladó:</span></b><span style="color:black"> Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>><br>
<b>Elküldve:</b> 2021. január 13., szerda 17:39<br>
<b>Címzett:</b> Robert Katona <<a href="mailto:robert.katona@hotmail.com" target="_blank">robert.katona@hotmail.com</a>>;
<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a> <<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>><br>
<b>Másolatot kap:</b> Brent Morgan <<a href="mailto:brent.taylormorgan@gmail.com" target="_blank">brent.taylormorgan@gmail.com</a>><br>
<b>Tárgy:</b> Re: [mpich-devel] mpich3 error</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p style="margin:0in">Hi Robert,<o:p></o:p></p>
<p style="margin:0in"> <o:p></o:p></p>
<p style="margin:0in">Were you running MPI_Gather in multiple threads concurrently on a same communicator?
<o:p></o:p></p>
<p style="margin:0in">That is not allowed. You’ll need at least different communicator for different threads.<br>
<br>
If that was not the issue, could you try compile MPICH with ch4, with `--with-device=ch4:ofi`, assuming you are using the latest release.<o:p></o:p></p>
<p style="margin:0in"> <o:p></o:p></p>
<div>
<div>
<div>
<p style="margin:0in">-- <br>
Hui Zhou<o:p></o:p></p>
</div>
</div>
</div>
<p style="margin:0in"> <o:p></o:p></p>
<p style="margin:0in"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:0in">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Robert Katona <<a href="mailto:robert.katona@hotmail.com" target="_blank">robert.katona@hotmail.com</a>><br>
<b>Date: </b>Wednesday, January 13, 2021 at 10:34 AM<br>
<b>To: </b>Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>>,
<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a> <<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>><br>
<b>Cc: </b>Brent Morgan <<a href="mailto:brent.taylormorgan@gmail.com" target="_blank">brent.taylormorgan@gmail.com</a>><br>
<b>Subject: </b>Re: [mpich-devel] mpich3 error</span><o:p></o:p></p>
</div>
<p style="margin:0in;background:white"><span style="color:#212121">Hello Hui Zhou,</span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121"> </span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121">And can you help us in brainstorming?</span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121"> </span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121">Unfortunately we cannot share the code. But I can give you more details. </span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121"> </span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121">Couple of computers are connected into the same network and we would like to run a distributed calculation. During testing we run the MPI application over 50 job executer processes/threads and
one job evaluter master/root thread. Each job calculate two float value what the master collects. To distribute this data we use the MPI_Gather() function. The execution of the problem runs fine but randomly stops with error message what Brent sent to you.</span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121"> </span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121">Sometimes it stops after 10 iterative calculation, sometimes after 50 or 70. We do the same calculation, with same input, but the event of the error is very random. And it always stops in the
MPI_Gather() function, when the master try to collect the data from all of the jobs.</span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121"> </span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121">If there is a network issue with any of the computer can it produce this error?</span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121"> </span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121">We are using Mpich 3.3.2, in the release note of 3.4 it is written the network communication has changed from ch3 to ch4. With ch4 can we expect better behavior?</span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121"> </span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121">Can you give us any hint where to look and what to check in this topic?</span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121"> </span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121">Regards,</span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121"> </span><o:p></o:p></p>
<p style="margin:0in;background:white"><span style="color:#212121">Robert</span><o:p></o:p></p>
<div id="gmail-m_-1328304056524589182gmail-m_-7390180511783217357x_ms-outlook-mobile-signature">
<div>
<p style="margin:0in"> <o:p></o:p></p>
</div>
<p style="margin:0in">Az <a href="https://aka.ms/ghei36" target="_blank">Android Outlook</a> letöltése<o:p></o:p></p>
</div>
<div id="gmail-m_-1328304056524589182gmail-m_-7390180511783217357x_id-512d1ffc-1bac-4aa0-bcc7-f8897b2efba4">
<div>
<p style="margin:0in"><span style="font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="78%" align="center">
</div>
<div id="gmail-m_-1328304056524589182gmail-m_-7390180511783217357x_divRplyFwdMsg">
<p style="margin:0in"><strong><span style="font-family:"Calibri",sans-serif">Feladó:</span></strong> Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>><br>
<strong><span style="font-family:"Calibri",sans-serif">Elküldve:</span></strong> 2021. január 13., szerda 15:51<br>
<strong><span style="font-family:"Calibri",sans-serif">Címzett:</span></strong> <a href="mailto:devel@mpich.org" target="_blank">
devel@mpich.org</a><br>
<strong><span style="font-family:"Calibri",sans-serif">Másolat:</span></strong> Brent Morgan; Robert Katona<br>
<strong><span style="font-family:"Calibri",sans-serif">Tárgy:</span></strong> Re: [mpich-devel] mpich3 error<o:p></o:p></p>
</div>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:0in">
<o:p> </o:p></p>
<div>
<p style="margin:0in">Hi Brent,<o:p></o:p></p>
<p style="margin:0in"> <o:p></o:p></p>
<p style="margin:0in">Unfortunately, unless you can provide us with a reproducer, there is little we can do to find the issue.<o:p></o:p></p>
<p style="margin:0in"> <o:p></o:p></p>
<div>
<div>
<div>
<p style="margin:0in">-- <br>
Hui Zhou<o:p></o:p></p>
</div>
</div>
</div>
<p style="margin:0in"> <o:p></o:p></p>
<p style="margin:0in"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:0in">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Brent Morgan via devel <<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>><br>
<b>Date: </b>Wednesday, January 13, 2021 at 3:20 AM<br>
<b>To: </b><a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a> <<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>><br>
<b>Cc: </b>Brent Morgan <<a href="mailto:brent.taylormorgan@gmail.com" target="_blank">brent.taylormorgan@gmail.com</a>>, Robert Katona <<a href="mailto:robert.katona@hotmail.com" target="_blank">robert.katona@hotmail.com</a>><br>
<b>Subject: </b>[mpich-devel] mpich3 error</span><o:p></o:p></p>
</div>
<div>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:0in">
Hello mpich dev support,<br>
<br>
I am receiving the following error in my MPI implementation, when I use 110+ threads.<br>
<br>
Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 567: !vc_ch->recv_active<br>
0x7f7f870937 ???<br>
???:0<br>
0x7f7f870957 ???<br>
???:0<br>
0x7f7f8089cf ???<br>
???:0<br>
0x7f7f854983 ???<br>
???:0<br>
0x7f7f75a4f3 ???<br>
???:0<br>
0x7f7f7cd8bf ???<br>
???:0<br>
0x7f7f7cde0f ???<br>
???:0<br>
0x7f7f78d577 ???<br>
???:0<br>
0x7f7f6ea6cb ???<br>
???:0<br>
0x7f7f6ea8f3 ???<br>
???:0<br>
0x7f7f78c7ff ???<br>
???:0<br>
0x7f7f6e9e6f ???<br>
???:0<br>
0x7f7f6e9eb7 ???<br>
???:0<br>
0x7f7f6ea033 ???<br>
???:0<br>
0x55793fe47b ???<br>
???:0<br>
0x55793f96b3 ???<br>
???:0<br>
0x7f7f29908f ???<br>
???:0<br>
0x55793f9a9b ???<br>
???:0<br>
internal ABORT - process 171<o:p></o:p></p>
<div>
<p style="margin:0in">Does this give a clue as to what I may be doing wrong? Thank you,<o:p></o:p></p>
</div>
<div>
<div>
<p style="margin:0in"> <o:p></o:p></p>
</div>
<div>
<p style="margin:0in">Best,<o:p></o:p></p>
</div>
<div>
<p style="margin:0in">Brent<o:p></o:p></p>
</div>
</div>
</div>
</div>
<p style="margin:0in"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>