<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">Why are you building MPICH from source on a Cray machine? Cray MPI is based on MPICH and highly optimized for that system.<div><div><br></div><div>Jeff</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 8, 2016 at 2:38 PM, <span dir="ltr"><<a href="mailto:Keith.Bannister@csiro.au" target="_blank">Keith.Bannister@csiro.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">HI Halim,<br>
<br>
Thanks for getting back to me.<br>
<span class=""><br>
On 6 Dec 2016, at 3:48 am, Halim Amer <<a href="mailto:aamer@anl.gov">aamer@anl.gov</a>> wrote:<br>
> > 2) The latency changes by > 10x over 100 iterations. Is that normal?<br>
><br>
> What is the baseline you are comparing against? Do you mean memory latency? If yes, how do you measure it and from where do you fetch the data?<br>
<br>
</span>I mean the (max - min)/min latency for a call to mpi_gather() as reported by the osu_gather [1] benchmark for a 16MB message size with 12 ranks running on the same node:<br>
<span class=""><br>
mpirun -n 12 ./osu_gather -m 33554432 -f -M 1073741842<br>
</span><span class=""># Size Avg Latency(us) Min Latency(us) Max Latency(us) Iterations<br>
</span>...<br>
<span class="">16777216 19515.15 3522.29 49930.88 100<br>
<br>
</span>I haven’t checked the osu_gather benchmark. I imagine it initialises some arrays and just runs mpi_gather.<br>
<br>
I don’t know how to find out whether mpich is using shared memory, or going over some network interface. In either case, the jitter in latency (x13) is much larger on this mpich setup than I’ve seen on another machine (in which it’s a few percent).<br>
<br>
Moreover, that latency should be much smaller if it is using shared memory. If I understand it right, the average throughput = message_size/latency = 6.5 Gbits/sec, which resembles a network throughput to me. I’m sure the memory bus on this machine can sustain much more than that.<br>
<br>
vmstat doesn’t report any swapping, but I’m wondering whether there’s some problem with how the shared memory is working? Some virtual memory setup problem?<br>
<br>
> What is your hardware?<br>
<br>
Cray XC30<br>
Single node:<br>
Memory: 64 GB<br>
CPU<br>
model name : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz<br>
Architecture : x86_64<br>
cpu MHz : 1999.873<br>
cache size : 20480 KB (Last Level)<br>
<br>
Total Number of Sockets : 2<br>
Total Number of Cores : 16 (8 per socket)<br>
Hyperthreading : ON<br>
Total Number of Physical Processors : 16<br>
Total Number of Logical Processors : 32 (2 per Phys Processor)<br>
<span class=""><br>
><br>
> > MPICH configure: --prefix=/group/astronomy856/<wbr>ban115/mpich/build-ingest-<wbr>debug --enable-error-messages=all --enable-timing=all --enable-g=most<br>
><br>
> You are trying to understand if there is a performance anomaly, yet you build MPICH in debugging mode. I suggest building with *--enable-fast=O3,ndebug* and remove the other flags you supplied.<br>
<br>
</span>I’ve compiled as you say, and got essentailly the same results. :-(<br>
<br>
<br>
<br>
[1] <a href="http://mvapich.cse.ohio-state.edu/benchmarks/" rel="noreferrer" target="_blank">http://mvapich.cse.ohio-state.<wbr>edu/benchmarks/</a><br>
<div class="HOEnZb"><div class="h5">--<br>
KEITH BANNISTER<br>
CSIRO Astronomy and Space Science<br>
T +61 2 9372 4295<br>
E keith.bannister@csiro.au<br>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/<wbr>mailman/listinfo/discuss</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div>