<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">Hi Qiguo,<div><br></div><div>You should try to collect performance statistics, especially regarding your specific nodes and what they are doing at every moment in time.</div><div>If I understand correctly, your algorithm does the following:</div><div>- Master thread: read in data, split it up into pieces, transfer pieces to slaves</div><div>- Slave thread: do calculation, transfer data back to master</div><div>- Master: recombine data, do calculation, split data back up, transfer pieces to slaves</div><div>- etc...</div><div><br></div><div>The reason you do not see linear performance scaling could be due to the following:</div><div>- The master thread recombining and splitting the data set may be responsible for a large part of the work (and therefore is the bottleneck)</div><div>- Work is not divided equally. A significant part of the time is spent waiting on one slave node who has a more difficult problem (takes longer) than the rest.</div><div>- There is a common dataset. I/O takes a larger part of the time when more slave nodes are used.</div><div><br></div><div>The only way to know for sure, is to simply generate a log file that shows the time every process starts and ends a specific process step. Output the time when </div><div>- the slave starts receiving data</div><div>- starts calculation</div><div>- starts sending results back</div><div>- starts waiting for his next piece of data</div><div>This will clearly show you what each node is doing at each moment in time, and should identify the bottleneck.</div><div><br></div><div>/ Ruben</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 23, 2014 at 5:27 PM, Qiguo Jing <span dir="ltr"><<a href="mailto:qjing@trinityconsultants.com" target="_blank">qjing@trinityconsultants.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div><span class="">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Hi Bob,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Thanks for your suggestions. Here are more tests. We actually have three clusters.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Cluster 1 and 2: 8 nodes, (2 Processors, 4 cores/processor, no HT – Total 8 Threads)/node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Cluster 3: 8 nodes, (1 Processors, 4 cores /processor, HT – Total 8 Threads )/node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">We also have a standalone machine: 2 processors, 6 cores/processor, HT – total 24 threads.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">For one particular case:
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Cluster 1 and 2 take 48 min to finish with 8 nodes, 8 threads/node, 60% CPU usage; 53 min to finish with 3 nodes, 8 threads/node, 90% CPU usage;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Cluster 3 takes 227 min to finish with 8 nodes, 8 threads/node, 20% CPU usage; 207 min to finish with 3 nodes, 8 threads/node, 50% CPU usage;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
</span><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Standalone machine takes 82 min to finish with 24 threads, 100% CPU usage.<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">It looks like with 24 threads, they should be pretty busy? Could the above phenomena be a hardware issue more than software?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Qiguo<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Bob Ilgner [<a href="mailto:bobilgner@gmail.com" target="_blank">mailto:bobilgner@gmail.com</a>]
<br>
<b>Sent:</b> Thursday, October 23, 2014 1:11 AM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
<b>Subject:</b> Re: [mpich-discuss] CPU usage versus Nodes, Threads<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Hi Qiguo,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">From the results table it looks as if you are using a computationally sparse algorithm. i.e. the problem may not lie with the inefficiency of your communication between threads but just that your algorithm is not keeping the processor busy
enough for a large number of threads. The only way that you will know for sure whether this is a comms issue or an algorithmic one is to use a profiling tool, such as Vampir or Paraver.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">With the profiling result you will be able to determine whether you need to make algorithmic changes in your bulk processing and enhance comms as per Huiwei's notes.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I would be interested to know what your profiling shows.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Regards, bob<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, Oct 23, 2014 at 12:02 AM, Qiguo Jing <<a href="mailto:qjing@trinityconsultants.com" target="_blank">qjing@trinityconsultants.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">Hi All,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">We have a parallel program running on a cluster. We recently found a case, which decreases the CPU usage and increase the run-time when increases Nodes. Below is the results
table.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">The particular run requires a lot of data communication between nodes.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Any thoughts about this phenomena? Or is there any way we can improve the CPU usage when using higher number of nodes?
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<table border="0" cellspacing="0" cellpadding="0" width="441" style="width:331.0pt;border-collapse:collapse">
<tbody>
<tr style="height:15.0pt">
<td width="153" nowrap="" valign="bottom" style="width:115.0pt;border:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">Average CPU Usage (%)</span><u></u><u></u></p>
</td>
<td width="119" nowrap="" valign="bottom" style="width:89.0pt;border:solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">Number of Nodes</span><u></u><u></u></p>
</td>
<td width="169" nowrap="" valign="bottom" style="width:127.0pt;border:solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">Number of Threads/Node</span><u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="153" nowrap="" valign="bottom" style="width:115.0pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">100</span><u></u><u></u></p>
</td>
<td width="119" nowrap="" valign="bottom" style="width:89.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">1</span><u></u><u></u></p>
</td>
<td width="169" nowrap="" valign="bottom" style="width:127.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">8</span><u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="153" nowrap="" valign="bottom" style="width:115.0pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">92</span><u></u><u></u></p>
</td>
<td width="119" nowrap="" valign="bottom" style="width:89.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">2</span><u></u><u></u></p>
</td>
<td width="169" nowrap="" valign="bottom" style="width:127.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">8</span><u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="153" nowrap="" valign="bottom" style="width:115.0pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">50</span><u></u><u></u></p>
</td>
<td width="119" nowrap="" valign="bottom" style="width:89.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">3</span><u></u><u></u></p>
</td>
<td width="169" nowrap="" valign="bottom" style="width:127.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">8</span><u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="153" nowrap="" valign="bottom" style="width:115.0pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">40</span><u></u><u></u></p>
</td>
<td width="119" nowrap="" valign="bottom" style="width:89.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">4</span><u></u><u></u></p>
</td>
<td width="169" nowrap="" valign="bottom" style="width:127.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">8</span><u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="153" nowrap="" valign="bottom" style="width:115.0pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">35</span><u></u><u></u></p>
</td>
<td width="119" nowrap="" valign="bottom" style="width:89.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">5</span><u></u><u></u></p>
</td>
<td width="169" nowrap="" valign="bottom" style="width:127.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">8</span><u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="153" nowrap="" valign="bottom" style="width:115.0pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">30</span><u></u><u></u></p>
</td>
<td width="119" nowrap="" valign="bottom" style="width:89.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">6</span><u></u><u></u></p>
</td>
<td width="169" nowrap="" valign="bottom" style="width:127.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">8</span><u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="153" nowrap="" valign="bottom" style="width:115.0pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">25</span><u></u><u></u></p>
</td>
<td width="119" nowrap="" valign="bottom" style="width:89.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">7</span><u></u><u></u></p>
</td>
<td width="169" nowrap="" valign="bottom" style="width:127.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">8</span><u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="153" nowrap="" valign="bottom" style="width:115.0pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">20</span><u></u><u></u></p>
</td>
<td width="119" nowrap="" valign="bottom" style="width:89.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">8</span><u></u><u></u></p>
</td>
<td width="169" nowrap="" valign="bottom" style="width:127.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">8</span><u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="153" nowrap="" valign="bottom" style="width:115.0pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">20</span><u></u><u></u></p>
</td>
<td width="119" nowrap="" valign="bottom" style="width:89.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">8</span><u></u><u></u></p>
</td>
<td width="169" nowrap="" valign="bottom" style="width:127.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-color:currentColor windowtext windowtext currentColor">
<p class="MsoNormal" align="center" style="text-align:center">
<span style="color:black">4</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<br>
<br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><br>
_________________________________________________________________________<br>
<br>
The information transmitted is intended only for the person or entity to<br>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<br>
this in error, please contact the sender and delete the material from any<br>
computer.<br>
_________________________________________________________________________<u></u><u></u></p>
</div></div></div>
</div><div class="HOEnZb"><div class="h5">
<br>
______________________________<u></u>______________________________<u></u>_____________<br><br>The information transmitted is intended only for the person or entity to<br>which it is addressed and may contain confidential and/or privileged<br>material. Any review, retransmission, dissemination or other use of, or<br>taking of any action in reliance upon, this information by persons or<br>entities other than the intended recipient is prohibited. If you received<br>this in error, please contact the sender and delete the material from any<br>computer.<br>______________________________<u></u>______________________________<u></u>_____________<br></div></div><br>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br>/ Ruben FAELENS<div>+32 494 06 72 59</div>
</div>