<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Sam,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> The main takeaways are that the single- and multi-core memory bandwidth appears to be the same for both machines, although the IPC bandwidth observed using MPI is ><b>1.5x higher on the single-socket machine</b> than the original machine that is showing the
slow-down.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
That seems to say that intra-socket bandwidth is higher than inter-socket bandwidth. That is typical. However, 1.5x seems extreme. Because the single core bandwidth bottleneck should be the CPU pipeline bandwidth rather than the package memory bandwidth. Nevertheless,
3.5GB bandwidth does seem low. It should reach ~10GB/s. Something is slowing it down and I am not sure I have a clue yet.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Just to double check, could you try the following (not at the same time) -</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
1. Set MPIR_CVAR_ODD_EVEN_CLIQUES=1</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2. Set MPIR_CVAR_CH4_CMA_ENABLE=1</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Also I think you made a typo in the last email. Please confirm that you are using MPICH-4.3.1, not MPICH-4.2.1, right?</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Sam Austin via discuss <discuss@mpich.org><br>
<b>Sent:</b> Thursday, August 14, 2025 7:16 PM<br>
<b>To:</b> Boyle, Peter <pboyle@bnl.gov><br>
<b>Cc:</b> Sam Austin <sam.austin.p@gmail.com>; discuss@mpich.org <discuss@mpich.org><br>
<b>Subject:</b> Re: [mpich-discuss] MPICH: SHM bandwidth very low on IPC test</font>
<div> </div>
</div>
<style>
<!--
#x_pfptBanner8k8cr3j
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#D0D8DC!important;
max-width:none!important;
max-height:none!important}
-->
</style>
<div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
Hi all, Thank you for your advice; I really appreciate it. Try repeat the measurement a few times, you should see higher bandwidth number in the later rounds. I did this by running the transfer 100 times and only using the last 20 measurements</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="x_pfptBanner8k8cr3j" style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#D0D8DC!important; background-color:#D0D8DC; border-top:4px solid #90a4ae!important; border-top:4px solid #90a4ae">
<div id="x_pfptBanner8k8cr3j" style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBanner8k8cr3j" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBanner8k8cr3j" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">
<div dir="ltr">Hi all,
<div><br>
</div>
<div>Thank you for your advice; I really appreciate it. </div>
<div><br>
</div>
<div><i><font color="#351c75">Try repeat the measurement a few times, you should see higher bandwidth number in the later rounds.</font></i></div>
<div>I did this by running the transfer 100 times and only using the last 20 measurements (discarding the first 80), and I saw the same performance.<br>
<br>
<i><font color="#351c75">Obviously if there are multiple rank pairs on the node communicating concurrently, then multiple cores will be active copying concurrently increasing aggregate bandwidth to closer to the many core or threaded throughput, which is what
you would get if you compile STREAM with "-fopenmp”. </font></i></div>
<div>Good point -- I ran stream.c with -fopenmp, which returned 48 GB/s.</div>
<div><br>
</div>
<div>So, the discrepancy seems to be when dealing with IPC. Next, I ran these benchmarks, also using MPICH 4.2.1, on a different machine from the same era (Xeon E5-2650 v4). This is a single-socket board with one processor, as opposed to the original Dell Poweredge
in question, which has two sockets, each with a E5-2699A. For both machines, I have attached the following in a zip file to avoid clogging the thread:</div>
<div>
<ul>
<li>Hardware topology (from lstopo)</li><li>lscpu output</li><li>Output from stream.c (single core)</li><li>Output from stream_omp.c (openmp)</li><li>Output from my host-host communication program (shmem_check.cpp):</li></ul>
<div>The main takeaways are that the single- and multi-core memory bandwidth appears to be the same for both machines, although the IPC bandwidth observed using MPI is ><b>1.5x higher on the single-socket machine</b> than the original machine that is showing
the slow-down.</div>
</div>
<div><br>
</div>
<div><i><font color="#351c75">I would assume, that your MPI bandwidth calculation only accounts for the buffer size (i.e., only read or write).</font></i></div>
<div>This is a good point, although I'm not sure why the bandwidth would be 1.5x higher on the other machine that has very similar memory performance.</div>
<div><br>
</div>
<div>One more piece of information: The single-socket machine (Xeon E5-2650 v4) has four RAM sticks in a quad-channel configuration, all tied to the same socket as you can see in lstopo. On the dual-socket machine (the machine in question), the four RAM sticks
are in a dual-channel configuration, with two sticks on each socket. So, I'm not sure if the dual- vs quad-channel configuration is hurting maximum memory bandwidth per socket on the dual-socket machine, despite the total bandwidths showing approximately the
same in STREAM.</div>
<div><br>
</div>
<div>I have tried to make the tests more uniform by binding the processes to the same core. But that still calls into question the total memory bandwidth of the dual-channel vs quad-channel memory configuration.</div>
<div><br>
</div>
<div>Do you have any thoughts on this? The question I'm trying to answer is: for what reasons would the memory bandwidth be significantly lower on the dual-socket Dell machine? I understand that making comparisons across machines is tricky, but I've tried to
provide as much information as possible to isolate the key aspects of the memory configuration.</div>
<div><br>
</div>
<div>Thanks in advance!</div>
<div><br>
</div>
<div>Sam</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Thu, Aug 14, 2025 at 12:41 PM Boyle, Peter <<a href="mailto:pboyle@bnl.gov" target="_blank">pboyle@bnl.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt">
Hi, </div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
1)</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Obviously if there are multiple rank pairs on the node communicating</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
concurrently, then multiple cores will be active copying concurrently increasing</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
aggregate bandwidth to closer to the many core or threaded throughput, which is what you</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
would get if you compile STREAM with "-fopenmp”. </div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The Xeon part you quote has a peak of 76GB/s memory</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
bandwidth per socket and multiple cores will be needed to saturate that.</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
2) It is common for people to use hybrid OpenMP and MPI, to minimize intra-node</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
copy overhead. Often using one rank per NUMA domain.</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
In that context, using MPI_Comm_split(…, MPI_COMM_TYPE_SHARED) will reveal which ranks can </div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
use unix shared memory regions (e.g. shmopen ) and then use ALL their threads concurrently</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
to blast data between sockets, and use OpenMP within a socket.</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Then we get into the topic of careful NUMA binding of ranks to sockets etc...</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Best wishes,</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Peter</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div id="x_m_-582180402405684870m_-7906220285507707488m_3878610124219932801m_-4687774725689945500m_8957142732391415907m_2808622259848803103mail-editor-reference-message-container">
<div></div>
<div style="text-align:left; padding:3pt 0in 0in; border-width:1pt medium medium; border-style:solid none none; border-color:rgb(181,196,223) currentcolor currentcolor; font-family:Aptos; font-size:12pt; color:black">
<b>From: </b>Joachim Jenke via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Date: </b>Thursday, August 14, 2025 at 12:21 PM<br>
<b>To: </b>Sam Austin <<a href="mailto:sam.austin.p@gmail.com" target="_blank">sam.austin.p@gmail.com</a>><br>
<b>Cc: </b>Joachim Jenke <<a href="mailto:jenke@itc.rwth-aachen.de" target="_blank">jenke@itc.rwth-aachen.de</a>>,
<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Subject: </b>Re: [mpich-discuss] MPICH: SHM bandwidth very low on IPC test<br>
<br>
</div>
<div style="font-size:11pt">Hi Sam,<br>
<br>
the 10GB/s stream bandwidth calculation includes the number of<br>
read+written bytes (see lines 190/366).<br>
<br>
I would assume, that your MPI bandwidth calculation only accounts for<br>
the buffer size (i.e., only read or write). In shm communication one<br>
process (and therefore one core) streams/memcopies the data from the<br>
send to the receive buffer. So, when you see 3.5GB send bandwidth, that<br>
actually compares to 7GB of stream Copy bandwidth.<br>
<br>
As a side-effect of shm communication, we have actually seen that the<br>
placement of the copying process can determine the first-touch<br>
allocation of the buffer. Even if the memory is allocated with calloc,<br>
the memory is not paged. A bcast/scatter to node-local processes can<br>
result in paging all buffers to the same socket (what you typically want<br>
to avoid).<br>
<br>
Best<br>
Joachim<br>
<br>
Am 14.08.25 um 07:16 schrieb Sam Austin:<br>
> Hi Joachim,<br>
><br>
> Thanks for this suggestion! I used stream to test the single-core memory<br>
> bandwidth. I am running on a Xeon E5-2699A v4, which has 55MB last level<br>
> cache. So, I ran with 30 million elements per the instructions. It<br>
> appears that I am seeing about 10 GB/s if I'm reading that right? If so,<br>
> I am still not sure why I am only seeing ~3.5 GB/s on shared memory<br>
> performance with MPICH.<br>
><br>
> -------------------------------------------------------------<br>
> STREAM version $Revision: 5.10 $<br>
> -------------------------------------------------------------<br>
> This system uses 8 bytes per array element.<br>
> -------------------------------------------------------------<br>
> Array size = 30000000 (elements), Offset = 0 (elements)<br>
> Memory per array = 228.9 MiB (= 0.2 GiB).<br>
> Total memory required = 686.6 MiB (= 0.7 GiB).<br>
> Each kernel will be executed 10 times.<br>
> The *best* time for each kernel (excluding the first iteration)<br>
> will be used to compute the reported bandwidth.<br>
> -------------------------------------------------------------<br>
> Your clock granularity/precision appears to be 1 microseconds.<br>
> Each test below will take on the order of 30276 microseconds.<br>
> (= 30276 clock ticks)<br>
> Increase the size of the arrays if this shows that<br>
> you are not getting at least 20 clock ticks per test.<br>
> -------------------------------------------------------------<br>
> WARNING -- The above is only a rough guideline.<br>
> For best results, please be sure you know the<br>
> precision of your system timer.<br>
> -------------------------------------------------------------<br>
> Function Best Rate MB/s Avg time Min time Max time<br>
> Copy: 10038.1 0.048342 0.047818 0.050004<br>
> Scale: 10342.2 0.048738 0.046412 0.056605<br>
> Add: 10580.3 0.068542 0.068051 0.069805<br>
> Triad: 10703.0 0.067615 0.067271 0.068143<br>
> -------------------------------------------------------------<br>
> Solution Validates: avg error less than 1.000000e-13 on all three arrays<br>
> -------------------------------------------------------------<br>
><br>
> Thanks,<br>
> Sam<br>
><br>
> On Wed, Aug 13, 2025 at 5:26 PM Jenke, Joachim <<a href="mailto:jenke@itc.rwth-aachen.de" target="_blank">jenke@itc.rwth-aachen.de</a><br>
> <<a href="mailto:jenke@itc.rwth-aachen.de" target="_blank">mailto:jenke@itc.rwth-aachen.de</a>>> wrote:<br>
><br>
> Hi Sam,<br>
><br>
> Can you try out stream to understand the single-core memory<br>
> bandwidth of the system?<br>
><br>
> <a href="https://urldefense.us/v3/__https://www.cs.virginia.edu/stream/ref.html__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCSaCXar3$" originalsrc="https://urldefense.us/v3/__https://www.cs.virginia.edu/stream/ref.html__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCSaCXar3$" target="_blank">
https://www.cs.virginia.edu/stream/ref.html</a> <https://<br>
> <a href="https://urldefense.us/v3/__http://www.cs.virginia.edu/stream/ref.html*3E__;JQ!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCdZcpzNh$" originalsrc="https://urldefense.us/v3/__http://www.cs.virginia.edu/stream/ref.html*3E__;JQ!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCdZcpzNh$" target="_blank">
www.cs.virginia.edu/stream/ref.html></a><br>
><br>
> Copy bandwidth for large junks (exceeding cache sizes) should<br>
> provide you an upper bound for shm communication bandwidth.<br>
><br>
> Best<br>
> Joachim<br>
><br>
> Am 13.08.2025 22:04 schrieb Sam Austin via discuss<br>
> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">mailto:discuss@mpich.org</a>>>:<br>
> Hi all, I am working to configure MPICH and run a few examples on my<br>
> standalone server (single node). Here are the system specs: Server:<br>
> Dell PowerEdge C4130 CPUs: 2x Xeon E5-2699A v4 GPUs: 4x Tesla V100s<br>
> connected with NVLink, tied to motherboard<br>
> ZjQcmQRYFpfptBannerStart<br>
> This Message Is From an External Sender<br>
> This message came from outside your organization.<br>
> ZjQcmQRYFpfptBannerEnd<br>
> Hi all,<br>
><br>
> I am working to configure MPICH and run a few examples on my<br>
> standalone server (single node). Here are the system specs:<br>
> Server: Dell PowerEdge C4130<br>
> CPUs: 2x Xeon E5-2699A v4<br>
> GPUs: 4x Tesla V100s connected with NVLink, tied to motherboard with<br>
> PCIe gen 3<br>
> OS: Ubuntu 24.04 LTS<br>
> I intend to use this system to develop multi-process programs for<br>
> eventual execution in a large, distributed HPC environment. I ran a<br>
> few tests with and without CUDA support; here is my mpichversion output:<br>
><br>
> MPICH Version: 4.3.1<br>
> MPICH Release date: Fri Jun 20 09:24:41 AM CDT 2025<br>
> MPICH ABI: 17:1:5<br>
> MPICH Device: ch4:ofi<br>
> MPICH configure: --prefix=/opt/mpich/4.2.1-cpu --without-cuda<br>
> MPICH CC: gcc -O2<br>
> MPICH CXX: g++ -O2<br>
> MPICH F77: gfortran -O2<br>
> MPICH FC: gfortran -O2<br>
> MPICH features: threadcomm<br>
><br>
> The first example that I ran was a bandwidth test for CPU-CPU and<br>
> GPU-GPU communication. This simple program sends small packets back<br>
> and forth between processes to test the bandwidth over the various<br>
> intra-node networks.<br>
><br>
> The GPU-GPU bandwidth test showed that the GPU interconnect was<br>
> saturating at ~45 GB/s, which is nominal for the NVLink interconnect<br>
> topology present on the node (this was run with a CUDA-aware build<br>
> of MPICH). The problem appears during the CPU-CPU IPC test. In<br>
> theory, this test is pretty vanilla, as it is communicating between<br>
> processes using shared memory, and does not involve traversing any<br>
> of the intra-node networks (PCIe or NVLink). My understanding is<br>
> that the bandwidth observed on the CPU-CPU IPC test should be quite<br>
> high, at least higher than 10 GB/s.<br>
><br>
> However, the intra-node IPC bandwidth appears to be very low, around<br>
> 3.5 GB/s, when running this test. I tried the following fixes in an<br>
> attempt to force MPICH to use shared memory, but to no avail:<br>
> Passing the option to explicitly specify `nemesis` during the build<br>
> configuration: "--with-device=ch3:nemesis --with-cuda"<br>
> Passing the option to explicitly specify shared memory with ch4 to<br>
> the configuration: "--with-ch4-shmmods=posix --with-cuda"<br>
> Rebuilding MPICH without GPU support: "--without-cuda"<br>
> Switching to Open MPI and running the same test<br>
> These results, especially the last one in which I saw the same<br>
> issues when running with Open MPI, makes me think it might be an<br>
> issue with my system configuration. The question is: why is the IPC<br>
> bandwidth so low despite supposedly using the SHM protocol? I'm<br>
> wondering if anyone has encountered this issue before or might be<br>
> able to lend some advice here. Any help would be greatly appreciated!<br>
><br>
> Some interesting observations from the output below: when I run with<br>
> "mpiexec -np 2 -genv FI_PROVIDER=shm ...", the log file reports<br>
> "Opened fabric: shm". However, when I run without "-genv<br>
> FI_PROVIDER=shm", the log file reports "Opened fabric: <a href="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCeHF2XS1$" originalsrc="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCeHF2XS1$" target="_blank">
10.133.0.0/21</a><br>
> <<a href="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS" originalsrc="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS" target="_blank">https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS</a>!<br>
> ZaUD7Nw-<br>
> pSrvdcr4vb0JBtm7m5HhtE6d7G1wb5HakwqLQQenlo0WTl1tkzV3CrJnLwCQ7cVvC-<br>
> kgoP1Dp-C6$>", which I believe means that MPICH is falling back on<br>
> the TCP socket protocol. In this case, my key point of confusion is<br>
> that the observed bandwidth is essentially the same between the SHM<br>
> and TCP protocols. Perhaps my test script isn't set up properly?<br>
><br>
> Thanks,<br>
> Sam<br>
><br>
> The following is attached below:<br>
> Bandwidth test program<br>
> Run script for the program<br>
> Output of the script on my machine<br>
> ----------------------------------------------------------------------------------------------------------------<br>
> In case the attachment doesn't go through, here are the contents of<br>
> my test program, "shmem_check.cpp":<br>
><br>
> // shmem_check.cpp<br>
> //<br>
> // This is a minimal benchmark to test the raw bandwidth of MPI<br>
> communication<br>
> // between two processes on the same node, using only host (CPU) memory.<br>
> // It completely removes CUDA to isolate the performance of the MPI<br>
> library's<br>
> // on-node communication mechanism (e.g., shared memory vs. TCP<br>
> loopback).<br>
> //<br>
> // Compile/run:<br>
> // /opt/mpich/4.2.1-cpu/bin/mpicxx -std=c++17 -I/opt/mpich/4.2.1-<br>
> cpu/include shmem_check.cpp -o shmem_check<br>
> // /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 ./shmem_check<br>
><br>
> #include <iostream><br>
> #include <vector><br>
> #include <numeric><br>
> #include <mpi.h><br>
><br>
> int main(int argc, char* argv[]) {<br>
> MPI_Init(&argc, &argv);<br>
><br>
> int rank, size;<br>
> MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br>
> MPI_Comm_size(MPI_COMM_WORLD, &size);<br>
><br>
> if (size != 2) {<br>
> if (rank == 0) {<br>
> std::cerr << "Error: This program must be run with<br>
> exactly 2 MPI processes." << std::endl;<br>
> }<br>
> MPI_Finalize();<br>
> return 1;<br>
> }<br>
><br>
> const int num_samples = 100;<br>
> const long long packet_size = 1LL << 28; // 256 MB<br>
><br>
> // Allocate standard host memory. 'new' is sufficient.<br>
> char* buffer = new char[packet_size];<br>
><br>
> if (rank == 0) {<br>
> std::cout << "--- Starting Host-to-Host MPI Bandwidth Test<br>
> ---" << std::endl;<br>
> std::cout << "Packet Size: " << (packet_size / (1024*1024))<br>
> << " MB" << std::endl;<br>
> }<br>
><br>
> std::vector<double> timings;<br>
> for (int i = 0; i < num_samples; ++i) {<br>
> MPI_Barrier(MPI_COMM_WORLD);<br>
> double start_time = MPI_Wtime();<br>
><br>
> if (rank == 0) {<br>
> MPI_Send(buffer, packet_size, MPI_CHAR, 1, 0,<br>
> MPI_COMM_WORLD);<br>
> MPI_Recv(buffer, 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD,<br>
> MPI_STATUS_IGNORE); // Wait for confirmation<br>
> } else { // rank == 1<br>
> MPI_Recv(buffer, packet_size, MPI_CHAR, 0, 0,<br>
> MPI_COMM_WORLD, MPI_STATUS_IGNORE);<br>
> MPI_Send(buffer, 1, MPI_CHAR, 0, 1, MPI_COMM_WORLD); //<br>
> Send confirmation<br>
> }<br>
><br>
> double end_time = MPI_Wtime();<br>
> if (i >= 10) { // Discard warmup runs<br>
> timings.push_back(end_time - start_time);<br>
> }<br>
> }<br>
><br>
> if (rank == 0) {<br>
> double total_time = std::accumulate(timings.begin(),<br>
> timings.end(), 0.0);<br>
> double avg_time = total_time / timings.size();<br>
> double bandwidth = (static_cast<double>(packet_size) /<br>
> (1024.0 * 1024.0 * 1024.0)) / avg_time;<br>
><br>
> std::cout <<<br>
> "------------------------------------------------" << std::endl;<br>
> std::cout << "Average Host-to-Host Bandwidth: " <<<br>
> bandwidth << " GB/s" << std::endl;<br>
> std::cout <<<br>
> "------------------------------------------------" << std::endl;<br>
> }<br>
><br>
> // Clean up host memory<br>
> delete[] buffer;<br>
><br>
> MPI_Finalize();<br>
> return 0;<br>
> }<br>
><br>
> ----------------------------------------------------------------------------------------------------------------<br>
> Here is the script to run the test with verbose compilation and the<br>
> `shm` layer forced and unforced:<br>
><br>
> #!/usr/bin/zsh<br>
> source ~/.zshrc<br>
><br>
> # Compile<br>
> /opt/mpich/4.2.1-cpu/bin/mpicxx -std=c++17 -I/opt/mpich/4.2.1-cpu/<br>
> include shmem_check.cpp -o shmem_check<br>
><br>
> # Run with shm forced<br>
> /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 -genv FI_PROVIDER=shm -genv<br>
> FI_LOG_LEVEL=debug ./shmem_check 2> output_shm.txt<br>
><br>
> # Run without shm forced<br>
> /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 -genv FI_LOG_LEVEL=debug ./<br>
> shmem_check 2> output_no_shm.txt<br>
><br>
> echo "Output of script with SHM forced: "<br>
> grep -i "opened fabric" output_shm.txt<br>
><br>
> echo "Output of script with SHM not forced: "<br>
> grep -i "opened fabric" output_no_shm.txt<br>
><br>
> ----------------------------------------------------------------------------------------------------------------<br>
> Here is the output :<br>
><br>
> --- Starting Host-to-Host MPI Bandwidth Test ---<br>
> Packet Size: 256 MB<br>
> ------------------------------------------------<br>
> Average Host-to-Host Bandwidth: 3.35709 GB/s<br>
> ------------------------------------------------<br>
> --- Starting Host-to-Host MPI Bandwidth Test ---<br>
> Packet Size: 256 MB<br>
> ------------------------------------------------<br>
> Average Host-to-Host Bandwidth: 3.54924 GB/s<br>
> ------------------------------------------------<br>
> Output of script with SHM forced:<br>
> libfabric:3174297:1755114546::core:core:fi_fabric_():1503<info><br>
> Opened fabric: shm<br>
> libfabric:3174298:1755114546::core:core:fi_fabric_():1503<info><br>
> Opened fabric: shm<br>
> Output of script with SHM not forced:<br>
> libfabric:3174351:1755114554::core:core:fi_fabric_():1503<info><br>
> Opened fabric: <a href="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCeHF2XS1$" originalsrc="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCeHF2XS1$" target="_blank">
10.133.0.0/21</a> <<a href="https://urldefense.us/v3/" originalsrc="https://urldefense.us/v3/" target="_blank">https://urldefense.us/v3/</a><br>
> __<a href="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCcxTEnIl$" originalsrc="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCcxTEnIl$" target="_blank">http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-</a><br>
> pSrvdcr4vb0JBtm7m5HhtE6d7G1wb5HakwqLQQenlo0WTl1tkzV3CrJnLwCQ7cVvC-<br>
> kgoP1Dp-C6$><br>
> libfabric:3174350:1755114554::core:core:fi_fabric_():1503<info><br>
> Opened fabric: <a href="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCeHF2XS1$" originalsrc="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCeHF2XS1$" target="_blank">
10.133.0.0/21</a> <<a href="https://urldefense.us/v3/" originalsrc="https://urldefense.us/v3/" target="_blank">https://urldefense.us/v3/</a><br>
> __<a href="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCcxTEnIl$" originalsrc="https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCcxTEnIl$" target="_blank">http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-</a><br>
> pSrvdcr4vb0JBtm7m5HhtE6d7G1wb5HakwqLQQenlo0WTl1tkzV3CrJnLwCQ7cVvC-<br>
> kgoP1Dp-C6$><br>
><br>
<br>
<br>
--<br>
Dr. rer. nat. Joachim Jenke<br>
Deputy Group Lead<br>
<br>
IT Center<br>
Group: HPC - Parallelism, Runtime Analysis & Machine Learning<br>
Division: Computational Science and Engineering<br>
RWTH Aachen University<br>
Seffenter Weg 23<br>
D 52074 Aachen (Germany)<br>
Tel: +49 241 80- 24765<br>
Fax: +49 241 80-624765<br>
<a href="mailto:jenke@itc.rwth-aachen.de" target="_blank">jenke@itc.rwth-aachen.de</a><br>
<a href="https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCclNcVg5$" originalsrc="https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!aD-lRdCn7hGKD7zH2zy7Xu9dMPxA6Ww49nwtLe35Q6nnbgIDUcWcv5UzH-9BEb3gC0GN43yKL5mkCclNcVg5$" target="_blank">www.itc.rwth-aachen.de</a><br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>