No subject
Tue Jun 18 13:52:11 CDT 2019
sparse algorithm. i.e. the problem may not lie with the inefficiency of
your communication between threads but just that your algorithm is not
keeping the processor busy enough for a large number of threads. The only
way that you will know for sure whether this is a comms issue or an
algorithmic one is to use a profiling tool, such as Vampir or Paraver.
With the profiling result you will be able to determine whether you need to
make algorithmic changes in your bulk processing and enhance comms as per
Huiwei's notes.
I would be interested to know what your profiling shows.
Regards, bob
On Thu, Oct 23, 2014 at 12:02 AM, Qiguo Jing <qjing at trinityconsultants.com>
wrote:
> Hi All,
>
>
>
> We have a parallel program running on a cluster. We recently found a
> case, which decreases the CPU usage and increase the run-time when
> increases Nodes. Below is the results table.
>
>
>
> The particular run requires a lot of data communication between nodes.
>
>
>
> Any thoughts about this phenomena? Or is there any way we can improve the
> CPU usage when using higher number of nodes?
>
>
>
> Average CPU Usage (%)
>
> Number of Nodes
>
> Number of Threads/Node
>
> 100
>
> 1
>
> 8
>
> 92
>
> 2
>
> 8
>
> 50
>
> 3
>
> 8
>
> 40
>
> 4
>
> 8
>
> 35
>
> 5
>
> 8
>
> 30
>
> 6
>
> 8
>
> 25
>
> 7
>
> 8
>
> 20
>
> 8
>
> 8
>
> 20
>
> 8
>
> 4
>
>
>
>
>
> Thanks!
>
> _________________________________________________________________________
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
> _________________________________________________________________________
>
> _______________________________________________
> discuss mailing list discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
--20cf30363855c043ab050610eddd
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8"><d=
iv dir=3D"ltr"><div>Hi Qiguo,</div><div><br></div><div>From the results tab=
le it looks as if you are using a computationally sparse algorithm. i.=
e. the problem may not lie with the inefficiency of your communication betw=
een threads but just that your algorithm is not keeping the processor busy =
enough for a large number of threads. The only way that you will know for s=
ure whether this is a comms issue or an algorithmic one is to use a profili=
ng tool, such as Vampir or Paraver.</div><div><br></div><div>With the profi=
ling result you will be able to determine whether you need to make algorith=
mic changes in your bulk processing and enhance comms as per Huiwei's notes=
.</div><div><br></div><div>I would be interested to know what your profilin=
g shows.</div><div><br></div><div>Regards, bob</div></div><div class=3D"gma=
il_extra"><br><div class=3D"gmail_quote">On Thu, Oct 23, 2014 at 12:02 AM, =
Qiguo Jing <span dir=3D"ltr"><<a href=3D"mailto:qjing at trinityconsultants=
.com" target=3D"_blank">qjing at trinityconsultants.com</a>></span> wrote:<=
br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left=
:1px #ccc solid;padding-left:1ex">
<div lang=3D"EN-US" vlink=3D"#954F72" link=3D"#0563C1">
<div>
<p class=3D"MsoNormal">Hi All,<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<p class=3D"MsoNormal">We have a parallel program running on a cluster.&nbs=
p; We recently found a case, which decreases the CPU usage and increase the=
run-time when increases Nodes. Below is the results table.<u><=
/u><u></u></p>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<p class=3D"MsoNormal">The particular run requires a lot of data communicat=
ion between nodes.
<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<p class=3D"MsoNormal">Any thoughts about this phenomena? Or is there=
any way we can improve the CPU usage when using higher number of nodes?
<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<table width=3D"441" style=3D"width:331pt;border-collapse:collapse" border=
=3D"0" cellspacing=3D"0" cellpadding=3D"0">
<tbody>
<tr style=3D"height:15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"padding:0in 5.4pt;=
border:1pt solid windowtext;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">Average CPU Usage (%)<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width:1pt 1=
pt 1pt medium;border-style:solid solid solid none;border-color:windowtext w=
indowtext windowtext currentColor;padding:0in 5.4pt;width:89pt;height:15pt"=
>
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">Number of Nodes<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width:1pt 1=
pt 1pt medium;border-style:solid solid solid none;border-color:windowtext w=
indowtext windowtext currentColor;padding:0in 5.4pt;width:127pt;height:15pt=
">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">Number of Threads/Node<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height:15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt;border-style:none solid solid;border-color:currentColor windowtex=
t windowtext;padding:0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">100<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">1<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height:15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt;border-style:none solid solid;border-color:currentColor windowtex=
t windowtext;padding:0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">92<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">2<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height:15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt;border-style:none solid solid;border-color:currentColor windowtex=
t windowtext;padding:0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">50<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">3<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height:15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt;border-style:none solid solid;border-color:currentColor windowtex=
t windowtext;padding:0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">40<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">4<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height:15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt;border-style:none solid solid;border-color:currentColor windowtex=
t windowtext;padding:0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">35<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">5<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height:15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt;border-style:none solid solid;border-color:currentColor windowtex=
t windowtext;padding:0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">30<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">6<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height:15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt;border-style:none solid solid;border-color:currentColor windowtex=
t windowtext;padding:0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">25<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">7<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height:15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt;border-style:none solid solid;border-color:currentColor windowtex=
t windowtext;padding:0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">20<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">8<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height:15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt;border-style:none solid solid;border-color:currentColor windowtex=
t windowtext;padding:0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">20<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">8<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width:mediu=
m 1pt 1pt medium;border-style:none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding:0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align:center"><span s=
tyle=3D"color:black">4<u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<p class=3D"MsoNormal">Thanks!<u></u><u></u></p>
</div>
</div>
<br>
______________________________<u></u>______________________________<u></u>_=
____________<br><br>The information transmitted is intended only for the pe=
rson or entity to<br>which it is addressed and may contain confidential and=
/or privileged<br>material. Any review, retransmission, dissemination or o=
ther use of, or<br>taking of any action in reliance upon, this information =
by persons or<br>entities other than the intended recipient is prohibited. =
If you received<br>this in error, please contact the sender and delete th=
e material from any<br>computer.<br>______________________________<u></u>__=
____________________________<u></u>_____________<br><br>___________________=
____________________________<br>
discuss mailing list <a href=3D"mailto:discuss at mpich.org=
">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></=
div><br></div>
--20cf30363855c043ab050610eddd--
--===============0923015504839801834==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss
--===============0923015504839801834==--
More information about the discuss
mailing list