[mpich-devel] mpich3 error

Sat Jan 16 20:05:07 CST 2021

Hi Rajeev,

For the 27 processes, it consists of 27 physical cores, 7 devices.

I understand Gather may overall be slower, but that should result in a
smaller slope of a linear speedup...  The observed results leads to a
timing 2x of what is expected for 27 processes.
[image: image.png]

One thing I notice is that, in line 126 of my sent code, ranks 20-24 is
processed *after* ranks 1-19, 25-27.  Printing things out above line 126,
ranks 20-24 is printed always after 1-19, 25-27, and a pause; it's as if
the program broadcasts 23 (instead of 27) ranks at a time.  There is
nothing faulty about my devices/cores, as this behavior is consistent when
I switch nodes in the hostname file.

Best,
Brent

On Sat, Jan 16, 2021 at 6:44 PM Thakur, Rajeev <thakur at anl.gov> wrote:

> There is no limitation. Gather concatenates data at the root so all the
> data needs to be sent. Reduce calculates intermediate sums in a binomial
> tree fashion, so if you are summing a single number, only a single number
> is communicated. There are many papers on collective communication
> algorithms.
>
>
>
> When you run 27 processes, are there 27 physical cores for them to run
> concurrently? If not, the performance measurements will not show speedups.
>
>
>
> *From: *Brent Morgan <brent.taylormorgan at gmail.com>
> *Date: *Saturday, January 16, 2021 at 7:37 PM
> *To: *"Thakur, Rajeev" <thakur at anl.gov>
> *Cc: *Robert Katona <robert.katona at hotmail.com>, "Zhou, Hui" <
> zhouh at anl.gov>, "devel at mpich.org" <devel at mpich.org>
> *Subject: *Re: [mpich-devel] mpich3 error
>
>
>
> Why isn’t the sum all computed after doing a gather to the root- is there
> a limitation of 27 processes (7 devices) for gather?  We thought a
> collector could handle much more,
>
>
>
> Best,
>
> Brent
>
>
>
> On Sat, Jan 16, 2021 at 2:18 PM Thakur, Rajeev <thakur at anl.gov> wrote:
>
> It uses a different algorithm and communicates much less data. The sum is
> not all computed after doing a gather to the root.
>
>
>
> *From: *Brent Morgan <brent.taylormorgan at gmail.com>
> *Date: *Saturday, January 16, 2021 at 2:58 PM
> *To: *"Thakur, Rajeev" <thakur at anl.gov>
> *Cc: *Robert Katona <robert.katona at hotmail.com>, "Zhou, Hui" <
> zhouh at anl.gov>, "devel at mpich.org" <devel at mpich.org>
> *Subject: *Re: [mpich-devel] mpich3 error
>
>
>
> We will try MPI_Reduce which will improve our code, but it will not solve
> the underlying problem.
>
>
>
> Best,
>
> Brent
>
>
>
> On Sat, Jan 16, 2021 at 1:31 PM Thakur, Rajeev <thakur at anl.gov> wrote:
>
> Your mail all the way below says “We are using MPI_Gather collector for
> merely calculating the sum of the result of N processes”. Why don’t you use
> MPI_Reduce instead then?
>
>
>
> Rajeev
>
>
>
>
>
> *From: *Brent Morgan via devel <devel at mpich.org>
> *Reply-To: *"devel at mpich.org" <devel at mpich.org>
> *Date: *Saturday, January 16, 2021 at 1:38 PM
> *To: *"Zhou, Hui" <zhouh at anl.gov>
> *Cc: *Brent Morgan <brent.taylormorgan at gmail.com>, "devel at mpich.org" <
> devel at mpich.org>, Robert Katona <robert.katona at hotmail.com>
> *Subject: *Re: [mpich-devel] mpich3 error
>
>
>
> Hi Hui, Mpich community,
>
>
>
> Thanks for the response.  You're right, I'll provide a toy program that
> replicates the code structure (and results).  The toy program is
> calculating a sum value from each process- the value isn't too important
> for this toy program.  The timing, however, is the only thing important in
> our demonstration.  It exactly replicates what we are observing for our
> actual program.  This directly relates to the MPI functionality- we can't
> find out what the issue is.
>
>
>
> I have attached the code.  Is something wrong with our implementation?  It
> starts with the main() function.  Thank you very much for any help,
>
>
>
> Best,
>
> Brent
>
> PS My subscription to discuss at mpich.org is pending currently.
>
>
>
> On Sat, Jan 16, 2021 at 12:24 PM Brent Morgan <
> brent.taylormorgan at gmail.com> wrote:
>
> Hi Hui, Mpich community,
>
>
>
> Thanks for the response.  You're right, I'll provide a toy program that
> replicates the code structure (and results).  The toy program is
> calculating a sum value from each process- the value isn't too important
> for this toy program.  The timing, however, is the only thing important in
> our demonstration.  It exactly replicates what we are observing for our
> actual program.  This directly relates to the MPI functionality- we can't
> find out what the issue is.
>
>
>
> I have attached the code.  Is something wrong with our implementation?  It
> starts with the main() function.  Thank you very much for any help,
>
>
>
> Best,
>
> Brent
>
>
>
> On Fri, Jan 15, 2021 at 10:43 PM Zhou, Hui <zhouh at anl.gov> wrote:
>
> Your description only mentions MPI_Gather. If there is indeed problem with
> MPI_Gather, then you should be able to reproduce the issue with a sample
> program. Share with us and we can better assist you. If you can’t reproduce
> the issue with a simple example, then I suspect there are other problems
> that you are not able to fully describe. We really can’t help much without
> able to see the code.
>
>
>
> That said, I am not even sure what is the issue you are describing. 100
> process MPI_Gather will be slower than 50 process MPI_Gather. And since it
> is a collective, if one of your process is delayed due to some computations
> or else, the whole collective will take longer to finish just due to
> waiting for the late process. You really need tell us what your program is
> doing in order for us to even offer an intelligent guess.
>
>
>
> --
> Hui Zhou
>
>
>
>
>
> *From: *Brent Morgan <brent.taylormorgan at gmail.com>
> *Date: *Friday, January 15, 2021 at 10:42 PM
> *To: *Zhou, Hui <zhouh at anl.gov>, discuss at mpich.org <discuss at mpich.org>
> *Cc: *Robert Katona <robert.katona at hotmail.com>
> *Subject: *Re: [mpich-devel] mpich3 error
>
> Hi MPICH community,
>
>
>
> My team has downloaded mpich 3.3.2 (using ch3 as default) and implemented
> MPI, and for small # of processes (<50), everything worked fine for our MPI
> implementation.  For >=50 processes, there was a ch3 error and crashed the
> program after a random amount of seconds (sometimes 10seconds, sometimes
> 100seconds).  So we compiled mpich 3.3.2 with ch4 (instead of default ch3)
> using '--with-device=ch4:ofi` flag and this got rid of the error- but for
> >12 processes, the speed would slow down to 2x slower suddenly.
>
>
>
> Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with
> '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4).
> Everything worked fine until we hit 20 processes; after >=20 processes, the
> 2x slowdown is happening again.
>
>
>
> We have tried 1 communicator and multiple communicators in an attempt to
> make the MPI implementation faster, but there's no significant difference
> in observations.  We are using MPI_Gather collector for merely calculating
> the sum of the result of N processes, but we can't seem to maintain
> stability within MPI as we increase N processes.  Is there something we are
> missing that is ultimately causing this error?  We are at a loss here,
> thank you.
>
>
>
> Best,
>
> Brent
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/devel/attachments/20210116/d3457d8e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 87386 bytes
Desc: not available
URL: <http://lists.mpich.org/pipermail/devel/attachments/20210116/d3457d8e/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 87383 bytes
Desc: not available
URL: <http://lists.mpich.org/pipermail/devel/attachments/20210116/d3457d8e/attachment-0003.png>