<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Sam, First I'm not sure why reaching top performance on a dev system would matter. To understand the production performance, you would need to test on the target system anyways. Especially because a single-node system is no good proxy for</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerbihuehu" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerbihuehu" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerbihuehu" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerbihuehu" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html>
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerbihuehu { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonbihuehu:hover, .pfptPrimaryButtonbihuehu:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonbihuehu:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="auto">Hi Sam,
<div dir="auto"><br>
</div>
<div dir="auto">First I'm not sure why reaching top performance on a dev system would matter. To understand the production performance, you would need to test on the target system anyways. Especially because a single-node system is no good proxy for communication
in a distributed memory system.<br>
<div class="gmail_extra" dir="auto"><br>
<div class="gmail_quote">Am 15.08.2025 02:16 schrieb Sam Austin <sam.austin.p@gmail.com>:
<blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">
<div dir="ltr">
<div><i><font color="#351c75">I would assume, that your MPI bandwidth calculation only accounts for the buffer size (i.e., only read or write).</font></i></div>
<div>This is a good point, although I'm not sure why the bandwidth would be 1.5x higher on the other machine that has very similar memory performance.</div>
<div></div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Depending on the process placement, this might be caused by moving the data from one to the other socket, see below.</div>
<div dir="auto">
<div class="gmail_extra" dir="auto">
<div class="gmail_quote">
<blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>One more piece of information: The single-socket machine (Xeon E5-2650 v4) has four RAM sticks in a quad-channel configuration, all tied to the same socket as you can see in lstopo. On the dual-socket machine (the machine in question), the four RAM sticks
are in a dual-channel configuration, with two sticks on each socket. So, I'm not sure if the dual- vs quad-channel configuration is hurting maximum memory bandwidth per socket on the dual-socket machine, despite the total bandwidths showing approximately the
same in STREAM.</div>
<div></div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">As Peter already pointed out: there are at least two bandwidth limits in a multicore system. One is the bandwidth a core can stream using it's load/store pipelines. The other bandwidth is for the connection between the CPU package and the memory.
Going from Dual-Channel to Quad-Channel you increase the latter bandwidth. Therefore you will need more processes/threads to saturate the bandwidth with Quad-Channel configuration.</div>
<div dir="auto">In Multi-Socket systems you have a third bandwidth limit for accessing memory connected with a different socket, which is typically lower. For the p2p bandwidth test, the bandwidth should not be the limiting factor, but the increased latency
of these accesses might cause a reduced single-core bandwidth.</div>
<div dir="auto">
<div class="gmail_extra" dir="auto">
<div class="gmail_quote">
<blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>I have tried to make the tests more uniform by binding the processes to the same core. But that still calls into question the total memory bandwidth of the dual-channel vs quad-channel memory configuration.</div>
<div></div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Do you mean both processes to the same core, or symmetric proc placement on the two systems?</div>
<div dir="auto">Try binding the processes to cores on the same/different sockets. Also, make sure to initialize the buffers before starting communication, so that they are paged locally. Repeated communication in the same direction might cause the OS to trigger
page migration. So make sure to communicate back and forth.</div>
<div dir="auto">
<div class="gmail_extra" dir="auto">
<div class="gmail_quote">
<blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>Do you have any thoughts on this? The question I'm trying to answer is: for what reasons would the memory bandwidth be significantly lower on the dual-socket Dell machine? I understand that making comparisons across machines is tricky, but I've tried to
provide as much information as possible to isolate the key aspects of the memory configuration.</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">At the end, p2p bandwidth will never be reached in a large-scale program, because filling the node will quickly saturate the overall memory bandwidth.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Best</div>
<div dir="auto">Joachim </div>
<div dir="auto"></div>
</div>
</body>
</html>