<!DOCTYPE html>
<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi, I have been building MPICH with the following configure options for a long time, mainly to keep my code “Valgrind-clean”: === ./configure \ --enable-g=dbg,meminit \ --with-device=ch3: sock \ --enable-romio === This setup worked reasonably</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerrzjwt7m" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerrzjwt7m" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerrzjwt7m" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerrzjwt7m" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html>
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerrzjwt7m { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonrzjwt7m:hover, .pfptPrimaryButtonrzjwt7m:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonrzjwt7m:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,<br>
<br>
I have been building MPICH with the following configure options
for a long time, mainly to keep my code “Valgrind-clean”:</p>
<p>===<br>
./configure \<br>
--enable-g=dbg,meminit \<br>
--with-device=ch3:sock \<br>
--enable-romio</p>
<p>===</p>
<p><br>
</p>
<p>This setup worked reasonably well in the past, but recently I’ve
been seeing occasional errors with address-sanitizer or valgrind
(with 4.3.0 on a single node) such as:</p>
<p><br>
</p>
<p>===</p>
<pre
style="font-variant-ligatures:normal; font-variant-caps:normal; orphans:2; text-align:start; widows:2; text-decoration-style:initial; text-decoration-color:initial; white-space:pre-wrap; word-spacing:0px"><span
style="color:black">Fatal error in internal_Allreduce_c: Unknown error class, error stack:</span></pre>
<pre><span style="color:black">internal_Allreduce_c(347)...................: MPI_Allreduce_c(sendbuf=0x7ffdeb0b8e90, recvbuf=0x7ffdeb0b8e98, count=1, dtype=0x4c00083a, MPI_SUM, comm=0x84000003) failed</span></pre>
<pre><span style="color:black">MPIR_Allreduce_impl(4826)...................: </span></pre>
<pre><span style="color:black">MPIR_Allreduce_allcomm_auto(4732)...........: </span></pre>
<pre><span style="color:black">MPIR_Allreduce_intra_recursive_doubling(115): </span></pre>
<pre><span style="color:black">MPIC_Sendrecv(266)..........................: </span></pre>
<pre><span style="color:black">MPIC_Wait(90)...............................: </span></pre>
<pre><span style="color:black">MPIR_Wait(751)..............................: </span></pre>
<pre><span style="color:black">MPIR_Wait_state(708)........................: </span></pre>
<pre><span style="color:black">MPIDI_CH3i_Progress_wait(187)...............: an error occurred while handling an event returned by MPIDI_CH3I_Sock_Wait()</span></pre>
<pre><span style="color:black">MPIDI_CH3I_Progress_handle_sock_event(385)..: </span></pre>
<pre><span style="color:black">MPIDI_CH3I_Socki_handle_read(3647)..........: connection failure (set=0,sock=1,errno=104:Connection reset by peer)</span></pre>
<p>===</p>
<p><br>
</p>
<p>Is CH3 considered legacy?</p>
<p><br>
</p>
<p>I would like to also ask:<br>
<br>
1. What are the recommended configure options in 2025 for
building MPICH in a way that works well with Valgrind?<br>
<br>
2. Is it preferable now to move to CH4 (e.g. ch4:ofi or ch4:shm)
when debugging with Valgrind?<br>
<br>
3. Are there any other options (besides --enable-g=dbg,meminit)
that you would suggest for catching memory errors while keeping
Valgrind reports as clean as possible?</p>
<p><br>
</p>
<p> 4. Is <a
href="https://urldefense.com/v3/__https:/github.com/pmodels/mpich/blob/main/doc/wiki/design/Support_for_Debugging_Memory_Allocation.md__;!!KGKeukY!0uZHEHtZEaga1beOpdYFXpq7WNGp5jNAW8wQaJk8wgYLGwAEf-QD8rrTOQF7SYFYfdxC1lVvpP3XqxhRMeBGqXCTDdN2eE6IFMZP04X4lX-e$">
https://github.com/pmodels/mpich/blob/main/doc/wiki/design/Support_for_Debugging_Memory_Allocation.md</a>
up-to-date?</p>
<p><br>
</p>
<p>Any guidance on the “best practice” configuration for this use
case would be greatly appreciated.</p>
<p><br>
</p>
<p>PETSc guys have some options about debug (<a
href="https://urldefense.com/v3/__https:/gitlab.com/petsc/petsc/-/blob/main/config/BuildSystem/config/packages/MPICH.py*L94__;Iw!!KGKeukY!0uZHEHtZEaga1beOpdYFXpq7WNGp5jNAW8wQaJk8wgYLGwAEf-QD8rrTOQF7SYFYfdxC1lVvpP3XqxhRMeBGqXCTDdN2eE6IFMZP07bPQizu$">https://gitlab.com/petsc/petsc/-/blob/main/config/BuildSystem/config/packages/MPICH.py#L94</a>)
but still uses CH3 by default. However Satish uses the
configuration described above, at least for valgrind CI.</p>
<p><br>
</p>
<p>Thanks a lot,</p>
<p><br>
</p>
<p>Eric</p>
<pre>-- </pre>
<pre>Eric Chamberland, ing., M. Ing</pre>
<pre>Professionnel de recherche</pre>
<pre>GIREF/Université Laval</pre>
<p><br>
</p>
<div class="moz-cite-prefix">On 2025-10-02 08:13, Balay, Satish
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:SJ0PR09MB10678E886E3E68078F1580DA7EDE7A@SJ0PR09MB10678.namprd09.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
We currently use:</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
<br>
</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
balay@petsc-gpu-02:~$
/nfs/gce/projects/petsc/soft/u22.04/mpich-4.3.0-p2-ucx/bin/mpichversion </div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
MPICH Version: 4.3.0</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
MPICH Release date: Mon Feb 3 09:09:47 AM CST 2025</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
MPICH ABI: 17:0:5</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
MPICH Device: ch4:ucx</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
MPICH configure:
--prefix=/nfs/gce/projects/petsc/soft/u22.04/mpich-4.3.0-p2-ucx
--enable-shared --with-device=ch4:ucx --with-pm=hydra
--enable-fast=no --enable-error-messages=all --enable-g=meminit
--disable-java --without-hwloc --disable-opencl --without-cuda
--without-hip</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
MPICH CC: gcc -O0</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
MPICH CXX: g++ -O0</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
MPICH F77: gfortran -O0</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
MPICH FC: gfortran -O0</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
MPICH features: threadcomm</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
<br>
With: </div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
<br>
</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
#MPICH OFI/UCX/valgrind</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
export FI_PROVIDER=^psm3</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
export UCX_SYSV_HUGETLB_MODE=n</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
export UCX_LOG_LEVEL=error</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
<br>
</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
Satish</div>
<hr style="display:inline-block;width:98%" tabindex="-1"></blockquote>
<br>
</body>
</html>