<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
This is mpich-5.0.0rc3 , I will try that, thanks.</div>
<div class="elementToProof" id="Signature">
<div class="elementToProof" id="divtagdefaultwrapper">
<p style="direction: ltr; margin-top: 1em; margin-bottom: 1em;" class="elementToProof">
<span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">--<br>
</span></p>
<p style="direction: ltr; margin-top: 1em; margin-bottom: 1em;" class="elementToProof">
<span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Justin M Wozniak<br>
</span></p>
<p style="direction: ltr; margin-top: 1em; margin-bottom: 1em;" class="elementToProof">
<span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></p>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Raffenetti, Ken <raffenet@anl.gov><br>
<b>Sent:</b> Tuesday, March 31, 2026 11:30<br>
<b>To:</b> devel@mpich.org <devel@mpich.org>; Harms, Kevin <harms@alcf.anl.gov><br>
<b>Cc:</b> Wozniak, Justin M. <woz@anl.gov><br>
<b>Subject:</b> Re: MPICH with SYCL on Aurora</font>
<div> </div>
</div>
<div>
<div style="direction:ltr; font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt">
<span style="color:rgb(0,0,0)">Which version of MPI is this? This might be a known issue in CMA support (fixed here
<a href="https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G_uCfscf7eWS!cUiPT3Nmpearus7dfptbS5Cz3Q8WaE29of-5eDNCJqQMBj0uuKoeZlwEDkmOTVRQzaomHJtR$" originalsrc="https://github.com/pmodels/mpich/pull/7743" data-outlook-id="5a1e4086-512a-4d44-826c-8cf955b853c0">
https://github.com/pmodels/mpich/pull/7743</a>). You can try disabling CMA with MPIR_CVAR_CH4_CMA_ENABLE=0 to avoid that path or pull in the fix to your copy and rebuild.</span></div>
<div style="direction:ltr; font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr; font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Ken</div>
<div style="direction:ltr; font-family:Aptos,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_mail-editor-reference-message-container" style="color:inherit; background-color:inherit">
<div class="x_ms-outlook-mobile-reference-message x_skipProofing" style="direction:ltr">
</div>
<div class="x_ms-outlook-mobile-reference-message x_skipProofing" style="text-align:left; padding:3pt 0in 0in; border-width:1pt medium medium; border-style:solid none none; border-color:rgb(181,196,223) currentcolor currentcolor; font-family:Aptos; font-size:12pt; color:black">
<b>From: </b>Wozniak, Justin M. via devel <devel@mpich.org><br>
<b>Date: </b>Tuesday, March 31, 2026 at 11:27 AM<br>
<b>To: </b>Harms, Kevin <harms@alcf.anl.gov>, devel@mpich.org <devel@mpich.org><br>
<b>Cc: </b>Wozniak, Justin M. <woz@anl.gov><br>
<b>Subject: </b>Re: [mpich-devel] MPICH with SYCL on Aurora<br>
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like:</div>
<div class="x_elementToProof" style="direction:ltr; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack:</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
MPIR_Waitall(916)......: </div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
MPIDI_IPC_rndv_cb(172).: </div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
MPIDI_CMA_copy_data(54): </div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
copy_iovs(202).........: process_vm_readv failed (errno 14)</div>
<div class="x_elementToProof" style="direction:ltr; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack:</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
(same)</div>
<div class="x_elementToProof" style="direction:ltr; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" id="x_Signature" style="color:inherit; background-color:inherit">
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time.</div>
<div class="x_elementToProof" style="direction:ltr; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The app looks like:</div>
<div class="x_elementToProof" style="direction:ltr; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
$ ldd =agent</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000)</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000)</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000)</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
(etc.) </div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000)</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000)</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000) </div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000)</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000)</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000)</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libc.so.6 => /lib64/libc.so.6 (0x000014670852a000)</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000)</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
(etc.)</div>
<div class="x_elementToProof" style="direction:ltr; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks</div>
<div class="x_elementToProof" style="direction:ltr; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" id="x_divtagdefaultwrapper" style="color:inherit; background-color:inherit">
<p class="x_elementToProof" style="direction:ltr"><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">--</span></p>
<p class="x_elementToProof" style="direction:ltr"><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">Justin M Wozniak</span></p>
<p class="x_elementToProof" style="direction:ltr"><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></p>
</div>
</div>
<div class="x_ms-outlook-mobile-reference-message x_skipProofing" style="direction:ltr; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" style="color:inherit; background-color:inherit">
<div style="direction:ltr; font-family:Calibri,sans-serif; font-size:11pt; color:rgb(0,0,0)">
<b>From:</b> Harms, Kevin <harms@alcf.anl.gov><br>
<b>Sent:</b> Monday, March 30, 2026 14:50<br>
<b>To:</b> devel@mpich.org <devel@mpich.org><br>
<b>Cc:</b> Wozniak, Justin M. <woz@anl.gov><br>
<b>Subject:</b> Re: MPICH with SYCL on Aurora</div>
<div style="direction:ltr"> </div>
</div>
<div class="x_ms-outlook-mobile-reference-message x_skipProofing" style="font-size:11pt">
Justin,<br>
<br>
can you provide the specific error?<br>
<br>
kevin<br>
<br>
________________________________________<br>
From: Wozniak, Justin M. via devel <devel@mpich.org><br>
Sent: Monday, March 30, 2026 2:24 PM<br>
To: devel@mpich.org<br>
Cc: Wozniak, Justin M.<br>
Subject: [mpich-devel] MPICH with SYCL on Aurora<br>
<br>
Hi<br>
I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI
calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL?<br>
Thanks<br>
<br>
--<br>
<br>
Justin M Wozniak<br>
<br>
</div>
</div>
</div>
</body>
</html>