<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<div class="_rp_z3 ms-font-weight-regular ms-font-color-neutralDark rpHighlightAllClass rpHighlightBodyClass" id="Item.MessageUniqueBody" role="button" tabindex="0" style="">
<div>
<div dir="ltr" style="background-color:white;"><font size="3" face="Calibri,Arial,Helvetica,sans-serif" color="black"><span style="font-size:12pt;background-color:white;" dir="ltr">
<div>Hi Hui,</div>
<div><br>
</div>
<div>Here is the updated list with your request:<br>
</div>
<ol style="margin-top:14pt;margin-bottom:14pt;">
<li>One mpi job with mpiexec.hydra -n 64 myprog => 57.32s </li><li>One mpi job with mpiexec.hydra -bind-to numa -n 64 myprog => 50.52s</li><li>Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s</li><li>Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 myprog => 72.23s</li><li>Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 myprog => 159.2s</li><li>Four mpi jobs with mpiexec.hydra -n 64 myprog => 159.9s <br>
</li></ol>
<div style="margin-top:0;margin-bottom:0;">Case 6 is the same speed as Case 5. The code is a CFD code based on a subdomain approach. There is communication between neighboring subdomains. The machine is a supermicro server with 512GB of memory. These small
test cases use about 3% of the available memory. I want to run 4 larger MPI jobs concurrently.
</div>
</span></font></div>
</div>
</div>
<p><br>
</p>
<div style="word-wrap:break-word; line-break:after-white-space">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Zhou, Hui <zhouh@anl.gov><br>
<b>Sent:</b> Wednesday, November 13, 2019 8:55 AM<br>
<b>To:</b> discuss@mpich.org<br>
<b>Cc:</b> Douglas Dommermuth<br>
<b>Subject:</b> Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs</font>
<div> </div>
</div>
<div>Hi Doug,
<div class=""><br class="">
</div>
<div class="">What is your number running 4 mpi jobs without -bind-to numa?</div>
<div class=""><br class="">
<div class="">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="word-wrap:break-word; line-break:after-white-space">
<div style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
— </div>
<div style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
Hui Zhou</div>
<div style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<br class="">
</div>
<div class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<br class="">
</div>
<br class="Apple-interchange-newline">
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Nov 13, 2019, at 10:39 AM, Douglas Dommermuth via discuss <<a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="margin-top:0px; margin-bottom:0px; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); text-decoration:none">
I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 processors with hyper threading enabled. My codes are written in MPI and Fortran. The dual AMD processors have 128 cores and
256 threads. I want to optimize the runtime for 4 mpi jobs running concurrently with 64 threads each. Some timings are provided here:</div>
<ol class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); text-decoration:none">
<li class="">One mpi job with mpiexec.hydra -n 64 myprog => 57.32s<span class="Apple-converted-space"> </span></li><li class="">One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s</li><li class="">Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s</li><li class="">Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s</li><li class="">Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s</li></ol>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); text-decoration:none">
The option "-bind-to numa" helps, but even so, running four mpi jobs concurrently with 64 threads each is considerably slower than running one mpi job with 64 threads. I can almost run four mpi jobs sequentially and match the time for running four mpi jobs
concurrently. How can I improve on the result for running 4 mpi jobs concurrently? Thanks, Doug.<br class="">
</div>
<span class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); text-decoration:none; float:none; display:inline!important">_______________________________________________</span><br class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); text-decoration:none">
<span class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); text-decoration:none; float:none; display:inline!important">discuss
mailing list </span><a href="mailto:discuss@mpich.org" class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; background-color:rgb(255,255,255)">discuss@mpich.org</a><br class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); text-decoration:none">
<span class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); text-decoration:none; float:none; display:inline!important">To
manage subscription options or unsubscribe:</span><br class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); text-decoration:none">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; background-color:rgb(255,255,255)">https://lists.mpich.org/mailman/listinfo/discuss</a><br class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); text-decoration:none">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</body>
</html>