<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thanks, indeed the env variable fixes the issue and glad to hear its fixed in the latest version.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Best,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Edgar</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Raffenetti, Ken via discuss <discuss@mpich.org><br>
<b>Sent:</b> Thursday, June 8, 2023 3:50 PM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Raffenetti, Ken <raffenet@anl.gov><br>
<b>Subject:</b> Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:#0563C1;
text-decoration:underline}
span.x_xelementtoproof
{}
span.x_EmailStyle20
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal">FWIW, you can workaround the bug in older versions by setting MPIR_CVAR_DEVICE_COLLECTIVES=none in your environment.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Ken</p>
<p class="x_MsoNormal"> </p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt; color:black">From:
</span></b><span style="font-size:12.0pt; color:black">"Raffenetti, Ken via discuss" <discuss@mpich.org><br>
<b>Reply-To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Date: </b>Thursday, June 8, 2023 at 3:45 PM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>"Raffenetti, Ken" <raffenet@anl.gov><br>
<b>Subject: </b>Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
</div>
<p class="x_MsoNormal" style="margin-left:.5in">Hi,</p>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
<p class="x_MsoNormal" style="margin-left:.5in">I believe this bug was recently fixed in
<a href="https://urldefense.com/v3/__https://github.com/pmodels/mpich/pull/6543__;!!DZ3fjg!8AzKEU73uMciZu4bNdH1-uJ_sMFhjhYR4z6BeYLyhi-KiaPNXdrn62dnQZ4iz3VzxMVG5mCaYmCzlj5lcno$">
https://github.com/pmodels/mpich/pull/6543</a>. The fix is part of the MPICH 4.1.2 release just posted to our website and Github. I confirmed that your test program works as expected now vs. an older 4.1 release.</p>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
<p class="x_MsoNormal" style="margin-left:.5in">Ken</p>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin-left:1.0in"><b><span style="font-size:12.0pt; color:black">From:
</span></b><span style="font-size:12.0pt; color:black">"Solomonik, Edgar via discuss" <discuss@mpich.org><br>
<b>Reply-To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Date: </b>Thursday, June 8, 2023 at 3:37 PM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>"Solomonik, Edgar" <solomon2@illinois.edu><br>
<b>Subject: </b>[mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"> </p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span class="x_xelementtoproof"><span style="font-size:12.0pt; color:black; background:white">Hello,</span></span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in; background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in; background:white"><span style="font-size:12.0pt; color:black">Our library's autobuild (CTF, which uses MPI extensively and in relatively sophisticated ways) started failing on multiple architectures after github
workflows moved to later OS versions (and so later MPI versions). I believe I have narrowed the issue to an MPI bug associated with very basic usage of MPI Reduce. The following test code runs into a segmentation fault inside MPI when running with 2 MPI processes
with the latest Ubuntu MPI build and MPI 4.0. It works for smaller values of message size (n) or if the root is rank 0. The usage of MPI_IN_PLACE adheres with the MPI standard.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in; background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in; background:white"><span style="font-size:12.0pt; color:black">Best,</span></p>
</div>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span class="x_xelementtoproof"><span style="font-size:12.0pt; color:black; background:white">Edgar Solomonik</span></span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span class="x_xelementtoproof"><span style="font-size:12.0pt; color:black; background:white">#include <mpi.h>
</span><span style="background:white"></span></span></p>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white">#include <iostream></span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white">int main(int argc, char ** argv){</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> int64_t n = 257;</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> MPI_Init(&argc, &argv);</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> int rank;</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> MPI_Comm_rank(MPI_COMM_WORLD, &rank);</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> double * A = (double*)malloc(sizeof(double)*n);</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> for (int i=0; i<n; i++){</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> A[i] = (double)i;</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> }</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> if (rank == 1){</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> MPI_Reduce(MPI_IN_PLACE, A, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD);</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> } else {</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> MPI_Reduce(A, NULL, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD);</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> }</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> free(A);</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> MPI_Finalize();</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white"> return 0;</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black; background:white">}</span></p>
</div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:1.0in"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
</div>
</div>
</body>
</html>