<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:"Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:"Segoe UI Light";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1985694289;
mso-list-template-ids:-2042188126;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Ok, thanks for explaining that. I think it should be possible for us to arrange things to avoid calling MPI_Cancel on send requests. We do this currently because we’re using MPI communication in an interactive environment where the user
can tell us to stop at any time – the reproduction steps I posted were rather simplified compared to the actual case we were hitting. We already have mechanisms in place to “flush” messages at the receiving side when necessary, so that covers part of what
we’ll need.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal">Edric.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Zhou, Hui via discuss <discuss@mpich.org>
<br>
<b>Sent:</b> 02 February 2022 00:16<br>
<b>To:</b> discuss@mpich.org<br>
<b>Cc:</b> Zhou, Hui <zhouh@anl.gov>; Edric Ellis <eellis@mathworks.com><br>
<b>Subject:</b> Re: [mpich-discuss] MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hi Edric,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">After some pondering, we are closing
<a href="https://github.com/pmodels/mpich/issues/5775">
https://github.com/pmodels/mpich/issues/5775</a> as </span><code><span style="font-size:10.0pt;color:black">wontfix</span></code><span style="font-size:12.0pt;color:black">. For the following reasons:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<ul type="disc">
<li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
<span style="font-size:12.0pt">Cancelling send is deprecated in MPI-4.<o:p></o:p></span></li><li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
<span style="font-size:12.0pt">The potential fix will be very messy due to current design.<o:p></o:p></span></li><li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
<span style="font-size:12.0pt">On the other hand, it is probably easier to work around in application.
<o:p></o:p></span></li></ul>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Example workaround could be to replace
</span><code><span style="font-size:10.0pt;color:black">MPI_Issend</span></code><span style="font-size:12.0pt;color:black"> with `MPI_Isend` plus separate
</span><code><span style="font-size:10.0pt;color:black">MPI_Irecv</span></code><span style="font-size:12.0pt;color:black"> for acknowledgement. You only need do this for the
</span><code><span style="font-size:10.0pt;color:black">MPI_Issend</span></code><span style="font-size:12.0pt;color:black"> that you may cancel, and I think it is a small inconvenience for more predictable behavior. Internally an
</span><code><span style="font-size:10.0pt;color:black">MPI_Issend</span></code><span style="font-size:12.0pt;color:black"> is not implemented as an
</span><code><span style="font-size:10.0pt;color:black">MPI_Isend </span></code><span style="font-size:12.0pt;color:black">+
</span><code><span style="font-size:10.0pt;color:black">MPI_Irecv</span></code><span style="font-size:12.0pt;color:black"> to be more efficient and to exploit hardware acceleration where we can, but that also made cancelling it very complicated.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">If you have strong reasons to fix this issue, for example, no feasible work around, let's discuss.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">-- <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hui Zhou<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Zhou, Hui <<a href="mailto:zhouh@anl.gov">zhouh@anl.gov</a>><br>
<b>Sent:</b> Wednesday, January 19, 2022 12:28 PM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Cc:</b> Edric Ellis <<a href="mailto:eellis@mathworks.com">eellis@mathworks.com</a>><br>
<b>Subject:</b> Re: MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Thanks for reporting. We'll fix this. Tracking issue:
<a href="https://github.com/pmodels/mpich/issues/5775">
https://github.com/pmodels/mpich/issues/5775</a><o:p></o:p></span></p>
</div>
<div>
<div style="margin-top:12.0pt;margin-bottom:12.0pt;min-width:424px" id="LPBorder_GTaHR0cHM6Ly9naXRodWIuY29tL3Btb2RlbHMvbXBpY2gvaXNzdWVzLzU3NzU.">
<table class="MsoNormalTable" border="1" cellspacing="3" cellpadding="0" width="100%" style="width:100.0%;border:solid #C8C8C8 1.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:9.0pt 27.0pt 9.0pt 9.0pt">
<div style="margin-right:9.0pt;overflow:hidden" id="LPImageContainer383364">
<p class="MsoNormal"><a href="https://github.com/pmodels/mpich/issues/5775" target="_blank"><span style="text-decoration:none"><img border="0" width="240" height="120" style="width:2.5in;height:1.25in" id="_x0000_i1028" src="https://opengraph.githubassets.com/abeb43664f034115c44e99f3e273e49dbfdeb9ebcc478dd9ede9ca4d439f78ab/pmodels/mpich/issues/5775"></span></a><o:p></o:p></p>
</div>
</td>
<td width="100%" valign="top" style="width:100.0%;border:none;padding:9.0pt 27.0pt 9.0pt 9.0pt">
<div style="margin-right:6.0pt;margin-bottom:9.0pt" id="LPTitle383364">
<p class="MsoNormal"><span style="font-size:16.0pt;font-family:"Segoe UI Light",sans-serif"><a href="https://github.com/pmodels/mpich/issues/5775" target="_blank"><span style="text-decoration:none">ch4: MPI_Cancel stalls MPI_Wait
· Issue #5775 · pmodels/mpich</span></a><o:p></o:p></span></p>
</div>
<div style="margin-right:6.0pt;margin-bottom:9.0pt;max-height:100px;overflow:hidden" id="LPDescription383364">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#666666">Reported on discuss mailinglist: ---- quote ---- Hi, Running one of our test programs using MPICH 3.4.3 and ch4:ofi, I notice that MPI_Wait on an MPI_Request
that has been MPI_Cancelled never compl...<o:p></o:p></span></p>
</div>
<div id="LPMetadata383364">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#A6A6A6">github.com<o:p></o:p></span></p>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Edric Ellis via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Sent:</b> Wednesday, January 19, 2022 11:21 AM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Cc:</b> Edric Ellis <<a href="mailto:eellis@mathworks.com">eellis@mathworks.com</a>><br>
<b>Subject:</b> [mpich-discuss] MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Hi,<br>
<br>
Running one of our test programs using MPICH 3.4.3 and ch4:ofi, I notice that MPI_Wait on an MPI_Request that has been MPI_Cancelled never completes (it does when using ch3). (The documentation for MPI_Cancel states "If a communication is marked for cancellation,
then a MPI_WAIT call for that communication is guaranteed to return, irrespective of the activities of other processes (i.e., MPI_WAIT behaves as a local function)")<br>
<br>
Here's a simple example:<br>
<br>
#include <mpi.h><br>
#include <stdio.h><br>
#include <stdlib.h><br>
<br>
void check(int const value) {<br>
if (value != MPI_SUCCESS) {<br>
fprintf(stderr, "Failed.\n");<br>
exit(1);<br>
}<br>
}<br>
<br>
int main(int argc, char** argv) {<br>
MPI_Request r1;<br>
int payload = 42;<br>
int result;<br>
<br>
check(MPI_Init(0,0));<br>
check(MPI_Issend(&payload, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &r1));<br>
check(MPI_Test(&r1, &result, MPI_STATUS_IGNORE));<br>
fprintf(stdout, "MPI_Test result: %d\n", result);<br>
check(MPI_Cancel(&r1));<br>
check(MPI_Wait(&r1, MPI_STATUS_IGNORE));<br>
MPI_Finalize();<br>
return 0;<br>
}<br>
<br>
This stalls in MPI_Wait when executed using "mpiexec -n 1 ./a.out". <br>
<br>
Cheers,<br>
Edric.<br>
<br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>