<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Edric,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I don’t see anything wrong in your pseudo code. I believe it is a correct pattern. I ran some experiments myself on some local machines and could not cause a hang, so if you come up with a reproducer please
send it along.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ken<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"Calibri",sans-serif;color:black">From:
</span></b><span style="font-family:"Calibri",sans-serif;color:black">Edric Ellis via discuss <discuss@mpich.org><br>
<b>Reply-To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Date: </b>Wednesday, April 24, 2024 at 3:05 AM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>Edric Ellis <eellis@mathworks.com><br>
<b>Subject: </b>[mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting
for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3:</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">nemesis. It
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;margin-left:.5in;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0in 12.0pt 0in">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0in 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="margin-left:.5in;mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<pre style="margin-left:.5in;white-space:pre-wrap"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3:nemesis. It works fine on ch4:ofi when the message is small (presumably using the "eager" protocol). When using ch4:ofi (the embedded tcp provider) and the message is large (presumably switching to "rendezvous" protocol), rank 0 never leaves the MPI_Barrier call. (I think the SHM piece of ch4 does not show the problem)<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">Should this work? I cannot find anything in the MPI standard that says it should not, but perhaps I'm not looking in the right place. <o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">I'm using mpich-4.1.2 in both cases, either in "--with-device=ch3:nemesis" mode or "--with-libfabric=embedded --with-device=ch4:ofi:tcp". <o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">Here's a sketch of the problematic section of code, I'll attempt to attach a full reproduction (but I'm not sure if that works?)<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">// setup code...<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">if (rank == 1) {<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"> MPI_Isend(data, count, MPI_INT, 0, TAG, comm, &req);<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">} else {<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"> std::this_thread::sleep_for(std::chrono::seconds(1));<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">}<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">MPI_Barrier(comm);<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">MPI_Barrier(comm);<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">// cleanup code, receive the message etc...<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">Cheers,<o:p></o:p></span></pre>
<pre style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">Edric.<o:p></o:p></span></pre>
</div>
</body>
</html>