No subject
Tue Jun 18 13:52:11 CDT 2019
e algorithm. i.e. the problem may not lie with the inefficiency of your com=
munication between threads but just that your algorithm is not keeping the =
processor busy enough for a large number of threads. The only way that you =
will know for sure whether this is a comms issue or an algorithmic one is t=
o use a profiling tool, such as Vampir or Paraver.
With the profiling result you will be able to determine whether you need to=
make algorithmic changes in your bulk processing and enhance comms as per =
Huiwei's notes.
I would be interested to know what your profiling shows.
Regards, bob
On Thu, Oct 23, 2014 at 12:02 AM, Qiguo Jing <qjing at trinityconsultants.com<=
mailto:qjing at trinityconsultants.com>> wrote:
Hi All,
We have a parallel program running on a cluster. We recently found a case,=
which decreases the CPU usage and increase the run-time when increases Nod=
es. Below is the results table.
The particular run requires a lot of data communication between nodes.
Any thoughts about this phenomena? Or is there any way we can improve the =
CPU usage when using higher number of nodes?
Average CPU Usage (%)
Number of Nodes
Number of Threads/Node
100
1
8
92
2
8
50
3
8
40
4
8
35
5
8
30
6
8
25
7
8
20
8
8
20
8
4
Thanks!
_________________________________________________________________________
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
_________________________________________________________________________
_______________________________________________
discuss mailing list discuss at mpich.org<mailto:discuss at mpich.org>
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss
--=20
_________________________________________________________________________
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
_________________________________________________________________________
--_000_9D079B25EA4E3149B260846F07AE80754C594461TCIEXCH03ustrin_
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40"><head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:=E5=AE=8B=E4=BD=93;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@=E5=AE=8B=E4=BD=93";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">Hi Bob,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">Thanks for your suggestions. &nb=
sp; Here are more tests. We actually have three clusters.<o:p></o:p><=
/span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">Cluster 1 and 2: 8 nodes, (2 Pr=
ocessors, 4 cores/processor, no HT =E2=80=93 Total 8 Threads)/node<o:p></o:=
p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">Cluster 3: &nb=
sp; 8 nodes, (1 Processors,=
4 cores /processor, HT =E2=80=93 Total 8 Threads )/node<o:p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">We also have a standalone machine:&nb=
sp; 2 processors, 6 cores/processor, HT =E2=80=93 total 24 threads.<o:p></o=
:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">For one particular case:
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">Cluster 1 and 2 take 48 min to finish=
with 8 nodes, 8 threads/node, 60% CPU usage; 53 min to finish =
with 3 nodes, 8 threads/node, 90% CPU usage;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">Cluster 3 takes 227 min to finish wit=
h 8 nodes, 8 threads/node, 20% CPU usage; 207 min to finish with 3 nodes, 8=
threads/node, 50% CPU usage;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">Standalone machine takes 24 min to fi=
nish with 24 threads, 100% CPU usage.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">It looks like with 24 threads, they s=
hould be pretty busy? Could the above phenomena be a hardware i=
ssue more than software?<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D">Qiguo<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size:11.0pt;font-family:"=
;Calibri",sans-serif">From:</span></b><span style=3D"font-size:11.0pt;=
font-family:"Calibri",sans-serif"> Bob Ilgner [mailto:bobilgner at g=
mail.com]
<br>
<b>Sent:</b> Thursday, October 23, 2014 1:11 AM<br>
<b>To:</b> discuss at mpich.org<br>
<b>Subject:</b> Re: [mpich-discuss] CPU usage versus Nodes, Threads<o:p></o=
:p></span></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class=3D"MsoNormal">Hi Qiguo,<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">From the results table it looks as if you are u=
sing a computationally sparse algorithm. i.e. the problem may not lie with =
the inefficiency of your communication between threads but just that your a=
lgorithm is not keeping the processor busy
enough for a large number of threads. The only way that you will know for =
sure whether this is a comms issue or an algorithmic one is to use a profil=
ing tool, such as Vampir or Paraver.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">With the profiling result you will be able to determ=
ine whether you need to make algorithmic changes in your bulk processing an=
d enhance comms as per Huiwei's notes.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">I would be interested to know what your profiling sh=
ows.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">Regards, bob<o:p></o:p></p>
</div>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<div>
<p class=3D"MsoNormal">On Thu, Oct 23, 2014 at 12:02 AM, Qiguo Jing <<a =
href=3D"mailto:qjing at trinityconsultants.com" target=3D"_blank">qjing at trinit=
yconsultants.com</a>> wrote:<o:p></o:p></p>
<blockquote style=3D"border:none;border-left:solid #CCCCCC 1.0pt;padding:0i=
n 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-=
bottom:5.0pt">
<div>
<div>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto">Hi All,<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto">We have a parallel program running on a cluster. We recently=
found a case, which decreases the CPU usage and increase the run-time when=
increases Nodes. Below is the results
table.<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto">The particular run requires a lot of data communication between no=
des.
<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto">Any thoughts about this phenomena? Or is there any way we ca=
n improve the CPU usage when using higher number of nodes?
<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"> <o:p></o:p></p>
<table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding=
=3D"0" width=3D"441" style=3D"width:331.0pt;border-collapse:collapse">
<tbody>
<tr style=3D"height:15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width:115.0pt;bord=
er:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">Average CPU Usage (%)</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width:89.0pt;borde=
r:solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t:15.0pt;border-color:currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">Number of Nodes</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width:127.0pt;bord=
er:solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;heig=
ht:15.0pt;border-color:currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">Number of Threads/Node</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height:15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width:115.0pt;bord=
er:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t:15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">100</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width:89.0pt;borde=
r-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">1</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width:127.0pt;bord=
er-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height:15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width:115.0pt;bord=
er:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t:15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">92</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width:89.0pt;borde=
r-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">2</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width:127.0pt;bord=
er-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height:15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width:115.0pt;bord=
er:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t:15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">50</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width:89.0pt;borde=
r-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">3</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width:127.0pt;bord=
er-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height:15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width:115.0pt;bord=
er:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t:15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">40</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width:89.0pt;borde=
r-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">4</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width:127.0pt;bord=
er-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height:15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width:115.0pt;bord=
er:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t:15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">35</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width:89.0pt;borde=
r-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">5</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width:127.0pt;bord=
er-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height:15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width:115.0pt;bord=
er:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t:15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">30</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width:89.0pt;borde=
r-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">6</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width:127.0pt;bord=
er-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height:15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width:115.0pt;bord=
er:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t:15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">25</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width:89.0pt;borde=
r-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">7</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width:127.0pt;bord=
er-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height:15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width:115.0pt;bord=
er:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t:15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">20</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width:89.0pt;borde=
r-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">8</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width:127.0pt;bord=
er-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height:15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width:115.0pt;bord=
er:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t:15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">20</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width:89.0pt;borde=
r-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">8</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width:127.0pt;bord=
er-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color:currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt:auto;ms=
o-margin-bottom-alt:auto;text-align:center">
<span style=3D"color:black">4</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto">Thanks!<o:p></o:p></p>
</div>
</div>
<p class=3D"MsoNormal"><br>
_________________________________________________________________________<b=
r>
<br>
The information transmitted is intended only for the person or entity to<br=
>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<b=
r>
this in error, please contact the sender and delete the material from any<b=
r>
computer.<br>
_________________________________________________________________________<b=
r>
<br>
_______________________________________________<br>
discuss mailing list <a href=3D"mailto:discuss at mpich.org=
">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk">https://lists.mpich.org/mailman/listinfo/discuss</a><o:p></o:p></p>
</blockquote>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>
<br>
______________________________<wbr>______________________________<wbr>_____=
________<br><br>The information transmitted is intended only for the person=
or entity to<br>which it is addressed and may contain confidential and/or =
privileged<br>material. Any review, retransmission, dissemination or other=
use of, or<br>taking of any action in reliance upon, this information by p=
ersons or<br>entities other than the intended recipient is prohibited. If=
you received<br>this in error, please contact the sender and delete the ma=
terial from any<br>computer.<br>______________________________<wbr>________=
______________________<wbr>_____________<br>=
--_000_9D079B25EA4E3149B260846F07AE80754C594461TCIEXCH03ustrin_--
--===============5521172785385150317==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss
--===============5521172785385150317==--
More information about the discuss
mailing list