<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Lucida Grande";
panose-1:2 11 6 0 4 5 2 2 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:-webkit-standard;
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Lucida Grande",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Lucida Grande",sans-serif">There is no limitation. Gather concatenates data at the root so all the data needs to be sent. Reduce calculates intermediate sums in a binomial tree fashion, so if you are summing a
single number, only a single number is communicated. There are many papers on collective communication algorithms.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Lucida Grande",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Lucida Grande",sans-serif">When you run 27 processes, are there 27 physical cores for them to run concurrently? If not, the performance measurements will not show speedups.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Lucida Grande",sans-serif"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Brent Morgan <brent.taylormorgan@gmail.com><br>
<b>Date: </b>Saturday, January 16, 2021 at 7:37 PM<br>
<b>To: </b>"Thakur, Rajeev" <thakur@anl.gov><br>
<b>Cc: </b>Robert Katona <robert.katona@hotmail.com>, "Zhou, Hui" <zhouh@anl.gov>, "devel@mpich.org" <devel@mpich.org><br>
<b>Subject: </b>Re: [mpich-devel] mpich3 error<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Why isn’t the sum all computed after doing a gather to the root- is there a limitation of 27 processes (7 devices) for gather? We thought a collector could handle much more,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Brent<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Sat, Jan 16, 2021 at 2:18 PM Thakur, Rajeev <<a href="mailto:thakur@anl.gov">thakur@anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Lucida Grande",sans-serif">It uses a different algorithm and communicates much less data. The sum is not all computed after doing a gather to the root.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Lucida Grande",sans-serif"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Brent Morgan <<a href="mailto:brent.taylormorgan@gmail.com" target="_blank">brent.taylormorgan@gmail.com</a>><br>
<b>Date: </b>Saturday, January 16, 2021 at 2:58 PM<br>
<b>To: </b>"Thakur, Rajeev" <<a href="mailto:thakur@anl.gov" target="_blank">thakur@anl.gov</a>><br>
<b>Cc: </b>Robert Katona <<a href="mailto:robert.katona@hotmail.com" target="_blank">robert.katona@hotmail.com</a>>, "Zhou, Hui" <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>>, "<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>"
<<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>><br>
<b>Subject: </b>Re: [mpich-devel] mpich3 error</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We will try MPI_Reduce which will improve our code, but it will not solve the underlying problem. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Brent<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Sat, Jan 16, 2021 at 1:31 PM Thakur, Rajeev <<a href="mailto:thakur@anl.gov" target="_blank">thakur@anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Lucida Grande",sans-serif">Your mail all the way below says “</span><span style="font-size:13.5pt;font-family:"-webkit-standard",serif;color:black">We
are using MPI_Gather collector for merely calculating the sum of the result of N processes”. Why don’t you use MPI_Reduce instead then?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:13.5pt;font-family:"-webkit-standard",serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:13.5pt;font-family:"-webkit-standard",serif;color:black">Rajeev</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Lucida Grande",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Lucida Grande",sans-serif"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Brent Morgan via devel <<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>><br>
<b>Reply-To: </b>"<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>" <<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>><br>
<b>Date: </b>Saturday, January 16, 2021 at 1:38 PM<br>
<b>To: </b>"Zhou, Hui" <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>><br>
<b>Cc: </b>Brent Morgan <<a href="mailto:brent.taylormorgan@gmail.com" target="_blank">brent.taylormorgan@gmail.com</a>>, "<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>" <<a href="mailto:devel@mpich.org" target="_blank">devel@mpich.org</a>>,
Robert Katona <<a href="mailto:robert.katona@hotmail.com" target="_blank">robert.katona@hotmail.com</a>><br>
<b>Subject: </b>Re: [mpich-devel] mpich3 error</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi Hui, Mpich community,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks for the response. You're right, I'll provide a toy program that replicates the code structure (and results). The toy program is calculating a sum value from each process-
the value isn't too important for this toy program. The timing, however, is the only thing important in our demonstration. It exactly replicates what we are observing for our actual program. This directly relates to the MPI functionality- we can't find
out what the issue is.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><img border="0" width="1336" height="584" style="width:13.9166in;height:6.0833in" id="_x0000_i1026" src="cid:image001.png@01D6EC3F.F3583F70"><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I have attached the code. Is something wrong with our implementation? It starts with the main() function. Thank you very much for any help,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Brent<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">PS My subscription to
<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> is pending currently.<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Sat, Jan 16, 2021 at 12:24 PM Brent Morgan <<a href="mailto:brent.taylormorgan@gmail.com" target="_blank">brent.taylormorgan@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi Hui, Mpich community,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks for the response. You're right, I'll provide a toy program that replicates the code structure (and results). The toy program is calculating a sum value from each process-
the value isn't too important for this toy program. The timing, however, is the only thing important in our demonstration. It exactly replicates what we are observing for our actual program. This directly relates to the MPI functionality- we can't find
out what the issue is.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><img border="0" width="1336" height="584" style="width:13.9166in;height:6.0833in" id="_x0000_i1025" src="cid:image001.png@01D6EC3F.F3583F70"><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I have attached the code. Is something wrong with our implementation? It starts with the main() function. Thank you very much for any help,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Brent<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, Jan 15, 2021 at 10:43 PM Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Your description only mentions MPI_Gather. If there is indeed problem with MPI_Gather, then you should be able to reproduce the issue with a sample program. Share with us and we
can better assist you. If you can’t reproduce the issue with a simple example, then I suspect there are other problems that you are not able to fully describe. We really can’t help much without able to see the code.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">That said, I am not even sure what is the issue you are describing. 100 process MPI_Gather will be slower than 50 process MPI_Gather. And since it is a collective, if one of your
process is delayed due to some computations or else, the whole collective will take longer to finish just due to waiting for the late process. You really need tell us what your program is doing in order for us to even offer an intelligent guess.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">-- <br>
Hui Zhou<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Brent Morgan <<a href="mailto:brent.taylormorgan@gmail.com" target="_blank">brent.taylormorgan@gmail.com</a>><br>
<b>Date: </b>Friday, January 15, 2021 at 10:42 PM<br>
<b>To: </b>Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>>,
<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Cc: </b>Robert Katona <<a href="mailto:robert.katona@hotmail.com" target="_blank">robert.katona@hotmail.com</a>><br>
<b>Subject: </b>Re: [mpich-devel] mpich3 error</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi MPICH community,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">My team has downloaded mpich 3.3.2 (using ch3 as default) and implemented MPI, and for small # of processes (<50), everything worked fine for our MPI implementation. For >=50 processes,
there was a ch3 error and crashed the program after a random amount of seconds (sometimes 10seconds, sometimes 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) using '--with-device=ch4:ofi` flag and this got rid of the error- but
for >12 processes, the speed would slow down to 2x slower suddenly.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). Everything worked fine until we hit 20 processes;
after >=20 processes, the 2x slowdown is happening again.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We have tried 1 communicator and multiple communicators in an attempt to make the MPI implementation faster, but there's no significant difference in observations. We are using
MPI_Gather collector for merely calculating the sum of the result of N processes, but we can't seem to maintain stability within MPI as we increase N processes. Is there something we are missing that is ultimately causing this error? We are at a loss here,
thank you.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Brent<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>