<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Doug,
<div class=""><br class="">
</div>
<div class="">When you do -map-by numa, processes should be bound to available numas using a round robin mapping. Thus if you are running only one MPI job with -bind-to numa and you have two numas, you are binding rank 0 to numa0:core0:hwthread0, rank 1 to
numa1:core64:hwthread128, rank 2 to numa0:core0:hwthread1, and so on. Which is what you were expecting. However, if you run multiple MPI jobs with the same bind policy, these are independent from each other and will all map to the same core and hwthreads.
Thus if you have 256 MPI processes running over 4 separate jobs you are still using half of the cores and a forth of the hwthreads. If you want to map to different hwthreads you can use a custom policy different for every job:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">mpiexec -bind-to --help</div>
<div class=""><br class="">
</div>
<div class="">-bind-to: Process-core binding type to use</div>
<div class=""><br class="">
</div>
<div class=""> Binding type options:</div>
<div class=""> Default:</div>
<div class=""> none -- no binding (default)</div>
<div class=""><br class="">
</div>
<div class=""> Architecture unaware options:</div>
<div class=""> rr -- round-robin as OS assigned processor IDs</div>
<div class=""> user:0+2,1+4,3,2 -- user specified binding</div>
</div>
<div class=""><br class="">
</div>
<div class="">The user policy allows you to do exactly that. There you define what hwthreads you want to bind your ranks to (+N) means your MPI process can move across hwthreads skipping every N hwthreads. So if you have 32 hwthreads and one process and request
-bind-to user:0+2 you get your process moving around across hwthreads: 0, 2, 4, 6, 8, …</div>
<div class=""><br class="">
</div>
<div class="">If you want to see what the binding is for your processes you can export HYDRA_TOPO_DEBUG=1 and use /bin/true as simple example code to inspect the binding you get for different options.</div>
<div class=""><br class="">
</div>
<div class="">If you don’t do any binding MPI processes will be re-scheduled over the course of the runtime on different cores, hwthreads (losing cached data) and possibly even different numas. In Linux memory is allocated on first touch policy by default.
Which means that physical memory is allocated on the numa your process is running when it accessed it first. If your process gets moved to another numa later on during the runtime it will be accessing its memory across numa domains which has a bigger impact
on latency. So if your application is latency sensitive you may get a big hit on performance.</div>
<div class=""><br class="">
</div>
<div class="">Hope this helps.</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Giuseppe</div>
<div class=""><br class="">
</div>
<div class=""> <br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Nov 13, 2019, at 10:39 AM, Douglas Dommermuth via discuss <<a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration: none; float: none; display: inline !important;" class="">256</span></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>