<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Abu,<br>
<br>
I think Jeff means that you should run your experiment with more
iterations in order to get a stable results.<br>
- Increase the iteration of for loop in each execution (I think osu
benchmark allows you to set it)<br>
- Run the experiments 10 or 100 times, and take the average and
standard deviation.<br>
<br>
If you see a very small standard deviation (e.g., <=5%), then the
trend is stable and you might not see such gaps.<br>
<br>
Best regards,<br>
Min<br>
<div class="moz-cite-prefix">On 2018/06/20 12:14, Abu Naser wrote:<br>
</div>
<blockquote type="cite" cite="mid:BLUPR0501MB200383DFBCA0721E71DDAC0E97770@BLUPR0501MB2003.namprd05.prod.outlook.com">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hello Jeff,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Yes, I am using a switch
and other machines are also connected with that switch.
<br>
</p>
<p style="margin-top:0;margin-bottom:0">If I remove other
machines and just use my two node with the switch, then will
it improve the performance by 200 ~ 400 iterations?</p>
<p style="margin-top:0;margin-bottom:0">Meanwhile I will give a
try with a single dedicated cable.
<span></span><br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Thank you.<br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0);
font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple
Color Emoji","Segoe UI
Emoji",NotoColorEmoji,"Segoe UI
Symbol","Android Emoji",EmojiSymbols">
<p><br>
</p>
<p align="left"><span style="font-size:10pt;
font-family:Calibri,Helvetica,sans-serif">Best Regards,</span></p>
<span style="font-family:Calibri,Helvetica,sans-serif;
font-size:10pt"></span>
<div align="left"><span style="font-size:11pt;
font-family:Calibri,Helvetica,sans-serif"></span></div>
<span style="font-family:Calibri,Helvetica,sans-serif;
font-size:10pt"></span>
<p align="left"><span style="font-size:10pt;
font-family:Calibri,Helvetica,sans-serif">Abu Naser</span><br>
</p>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Jeff
Hammond <a class="moz-txt-link-rfc2396E" href="mailto:jeff.science@gmail.com"><jeff.science@gmail.com></a><br>
<b>Sent:</b> Wednesday, June 20, 2018 12:52:06 PM<br>
<b>To:</b> MPICH<br>
<b>Subject:</b> Re: [mpich-discuss] osu_latency test: why 8KB
takes less time than 4KB and 2KB takes less time than 1KB?</font>
<div> </div>
</div>
<meta content="text/html; charset=utf-8">
<div>
<div dir="ltr">Is the ethernet connection a single dedicated
cable between the two machines or are you running through a
switch that handles other traffic?
<div><br>
</div>
<div>My best guess is that this is noise and that you may be
able to avoid it by running a very long time, e.g. 10000
iterations.</div>
<div><br>
</div>
<div>Jeff</div>
</div>
<div class="x_gmail_extra"><br>
<div class="x_gmail_quote">On Wed, Jun 20, 2018 at 6:53 AM,
Abu Naser <span dir="ltr">
<<a href="mailto:an16e@my.fsu.edu" target="_blank" moz-do-not-send="true">an16e@my.fsu.edu</a>></span>
wrote:<br>
<blockquote class="x_gmail_quote" style="margin:0 0 0 .8ex;
border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div id="x_m_6077755676379859201divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0);
font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple
Color Emoji","Segoe UI
Emoji",NotoColorEmoji,"Segoe UI
Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">Good day to
all,</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">I had run
point to point osu_latency test in two nodes for 200
times. Followings are the average time in
microsecond for various size of the messages -</p>
<div>1KB 84.8514 us<br>
<span style="color:rgb(255,0,0)">2KB 73.52535</span>
us<br>
4KB 272.55275 us<br>
<span style="color:rgb(255,0,0)">8KB 234.86385</span>
us<br>
16KB 288.88 us<br>
32KB 523.3725 us<br>
64KB 910.4025 us</div>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">From the
above looks like, 2KB message has less latency than
1 KB and 8KB has less latency than 4KB.
<br>
</p>
<p style="margin-top:0; margin-bottom:0">I was looking
for explanation of this behavior but did not get
any.</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<ol style="margin-bottom:0px; margin-top:0px">
<li><span>MPIR_CVAR_CH3_EAGER_MAX_MSG_<wbr>SIZE</span><span>
is set to 128KB. So none of the above message
size is using Rendezvous protocol. Is there any
partition inside eager protocol (e.g. 0 - 512
bytes, 1KB - 8KB, 16KB - 64KB)? If yes then what
are the boundaries for them? Can I log them with
debug-event-logging? </span><br>
</li>
</ol>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">Setup I am
using:</p>
<p style="margin-top:0; margin-bottom:0">- two nodes
has intel core i7, one with 16gb memory another one
8gb</p>
<p style="margin-top:0; margin-bottom:0">- mpich
3.2.1, configured and build to use nemesis tcp</p>
<p style="margin-top:0; margin-bottom:0">- 1gb
Ethernet connection</p>
<p style="margin-top:0; margin-bottom:0">- NFS is
using for sharing<br>
</p>
<p style="margin-top:0; margin-bottom:0">- osu_latency
: uses MPI_Send and MPI_Recv</p>
<p style="margin-top:0; margin-bottom:0">- <span>MPIR_CVAR_CH3_EAGER_MAX_MSG_<wbr>SIZE</span>=
<span>131072</span> (128KB)<br>
</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">Can anyone
help me on that? Thanks in advance.<br>
</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<div id="x_m_6077755676379859201Signature">
<div id="x_m_6077755676379859201divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0);
font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple
Color Emoji","Segoe UI
Emoji",NotoColorEmoji,"Segoe UI
Symbol","Android
Emoji",EmojiSymbols">
<p><br>
</p>
<p align="left"><span style="font-size:10pt;
font-family:Calibri,Helvetica,sans-serif">Best
Regards,</span></p>
<span style="font-family:Calibri,Helvetica,sans-serif;
font-size:10pt"></span>
<div align="left"><span style="font-size:11pt;
font-family:Calibri,Helvetica,sans-serif"></span></div>
<span style="font-family:Calibri,Helvetica,sans-serif;
font-size:10pt"></span>
<p align="left"><span style="font-size:10pt;
font-family:Calibri,Helvetica,sans-serif">Abu
Naser</span><br>
</p>
</div>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" moz-do-not-send="true">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.mpich.org/<wbr>mailman/listinfo/discuss</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="x_gmail_signature">Jeff Hammond<br>
<a href="mailto:jeff.science@gmail.com" target="_blank" moz-do-not-send="true">jeff.science@gmail.com</a><br>
<a href="http://jeffhammond.github.io/" target="_blank" moz-do-not-send="true">http://jeffhammond.github.io/</a></div>
</div>
</div>
<!--'"--><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
discuss mailing list <a class="moz-txt-link-abbreviated" href="mailto:discuss@mpich.org">discuss@mpich.org</a>
To manage subscription options or unsubscribe:
<a class="moz-txt-link-freetext" href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a>
</pre>
</blockquote>
<br>
</body>
</html>