<div dir="auto">Instead of stalling, it would be better to detect send cancel and throw an error. </div><div dir="auto"><br></div><div dir="auto">Jeff </div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 2, 2022 at 2:16 AM Zhou, Hui via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Edric,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
After some pondering, we are closing <a href="https://github.com/pmodels/mpich/issues/5775" id="m_786450584671995045LPlnk969506" target="_blank" style="font-family:Calibri,Arial,Helvetica,sans-serif">
https://github.com/pmodels/mpich/issues/5775</a> as <code style="font-family:monospace">wontfix</code>. For the following reasons:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<ul dir="auto" style="font-family:Calibri,Arial,Helvetica,sans-serif">
<li style="font-family:Calibri,Arial,Helvetica,sans-serif">Cancelling send is deprecated in MPI-4.</li><li style="font-family:Calibri,Arial,Helvetica,sans-serif">The potential fix will be very messy due to current design.<br>
</li><li style="font-family:Calibri,Arial,Helvetica,sans-serif">On the other hand, it is probably easier to work around in application. <br>
</li></ul>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">Example workaround could be to replace <code style="font-family:monospace">MPI_Issend</code> with `MPI_Isend` plus separate
<code style="font-family:monospace">MPI_Irecv</code> for acknowledgement. You only need do this for the <code style="font-family:monospace">
MPI_Issend</code> that you may cancel, and I think it is a small inconvenience for more predictable behavior. Internally an
<code style="font-family:monospace">MPI_Issend</code> is not implemented as an <code style="font-family:monospace">MPI_Isend </code>+ <code style="font-family:monospace">
MPI_Irecv</code> to be more efficient and to exploit hardware acceleration where we can, but that also made cancelling it very complicated.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">If you have strong reasons to fix this issue, for example, no feasible work around, let's discuss.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">-- <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">Hui Zhou<br>
</div>
<br>
</div>
<div id="m_786450584671995045appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_786450584671995045divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><b style="font-family:Calibri,sans-serif">From:</b> Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank" style="font-family:Calibri,sans-serif">zhouh@anl.gov</a>><br>
<b style="font-family:Calibri,sans-serif">Sent:</b> Wednesday, January 19, 2022 12:28 PM<br>
<b style="font-family:Calibri,sans-serif">To:</b> <a href="mailto:discuss@mpich.org" target="_blank" style="font-family:Calibri,sans-serif">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank" style="font-family:Calibri,sans-serif">discuss@mpich.org</a>><br>
<b style="font-family:Calibri,sans-serif">Cc:</b> Edric Ellis <<a href="mailto:eellis@mathworks.com" target="_blank" style="font-family:Calibri,sans-serif">eellis@mathworks.com</a>><br>
<b style="font-family:Calibri,sans-serif">Subject:</b> Re: MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3</font>
<div> </div>
</div></div><div dir="ltr">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks for reporting. We'll fix this. Tracking issue: <a href="https://github.com/pmodels/mpich/issues/5775" id="m_786450584671995045LPlnkOWALinkPreview" target="_blank" style="font-family:Calibri,Arial,Helvetica,sans-serif">
https://github.com/pmodels/mpich/issues/5775</a><br>
</div>
<div>
<div id="m_786450584671995045LPBorder_GTaHR0cHM6Ly9naXRodWIuY29tL3Btb2RlbHMvbXBpY2gvaXNzdWVzLzU3NzU." style="width:100%;margin-top:16px;margin-bottom:16px;max-width:800px;min-width:424px">
<table id="m_786450584671995045LPContainer383364" role="presentation" style="padding:12px 36px 12px 12px;width:100%;border:1px solid rgb(200,200,200);border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">
<tbody>
<tr valign="top" style="border-spacing:0px">
<td>
<div id="m_786450584671995045LPImageContainer383364" style="margin-right:12px;height:120px;overflow:hidden;width:240px">
<a id="m_786450584671995045LPImageAnchor383364" href="https://github.com/pmodels/mpich/issues/5775" target="_blank"><img id="m_786450584671995045LPThumbnailImageId383364" alt="" width="240" height="120" style="display: block;"></a></div>
</td>
<td style="width:100%">
<div id="m_786450584671995045LPTitle383364" style="font-size:21px;font-weight:300;margin-right:8px;font-family:wf_segoe-ui_light,"Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;margin-bottom:12px">
<a id="m_786450584671995045LPUrlAnchor383364" href="https://github.com/pmodels/mpich/issues/5775" style="text-decoration:none;font-family:wf_segoe-ui_light,"Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif" target="_blank">ch4: MPI_Cancel stalls MPI_Wait · Issue #5775 · pmodels/mpich</a></div>
<div id="m_786450584671995045LPDescription383364" style="font-size:14px;max-height:100px;font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;margin-bottom:12px;margin-right:8px;overflow:hidden;color:rgb(102,102,102)">
Reported on discuss mailinglist: ---- quote ---- Hi, Running one of our test programs using MPICH 3.4.3 and ch4:ofi, I notice that MPI_Wait on an MPI_Request that has been MPI_Cancelled never compl...</div>
<div id="m_786450584671995045LPMetadata383364" style="font-size:14px;font-weight:400;font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;color:rgb(166,166,166)">
<a href="http://github.com" target="_blank" style="font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif">github.com</a></div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<br>
<div id="m_786450584671995045x_appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_786450584671995045x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><b style="font-family:Calibri,sans-serif">From:</b> Edric Ellis via discuss <<a href="mailto:discuss@mpich.org" target="_blank" style="font-family:Calibri,sans-serif">discuss@mpich.org</a>><br>
<b style="font-family:Calibri,sans-serif">Sent:</b> Wednesday, January 19, 2022 11:21 AM<br>
<b style="font-family:Calibri,sans-serif">To:</b> <a href="mailto:discuss@mpich.org" target="_blank" style="font-family:Calibri,sans-serif">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank" style="font-family:Calibri,sans-serif">discuss@mpich.org</a>><br>
<b style="font-family:Calibri,sans-serif">Cc:</b> Edric Ellis <<a href="mailto:eellis@mathworks.com" target="_blank" style="font-family:Calibri,sans-serif">eellis@mathworks.com</a>><br>
<b style="font-family:Calibri,sans-serif">Subject:</b> [mpich-discuss] MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3</font>
<div> </div>
</div>
<div><font size="2" style="color:rgb(0,0,0)"><span style="font-size:11pt">
<div>Hi,<br>
<br>
Running one of our test programs using MPICH 3.4.3 and ch4:ofi, I notice that MPI_Wait on an MPI_Request that has been MPI_Cancelled never completes (it does when using ch3). (The documentation for MPI_Cancel states "If a communication is marked for cancellation,
then a MPI_WAIT call for that communication is guaranteed to return, irrespective of the activities of other processes (i.e., MPI_WAIT behaves as a local function)")<br>
<br>
Here's a simple example:<br>
<br>
#include <mpi.h><br>
#include <stdio.h><br>
#include <stdlib.h><br>
<br>
void check(int const value) {<br>
if (value != MPI_SUCCESS) {<br>
fprintf(stderr, "Failed.\n");<br>
exit(1);<br>
}<br>
}<br>
<br>
int main(int argc, char** argv) {<br>
MPI_Request r1;<br>
int payload = 42;<br>
int result;<br>
<br>
check(MPI_Init(0,0));<br>
check(MPI_Issend(&payload, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &r1));<br>
check(MPI_Test(&r1, &result, MPI_STATUS_IGNORE));<br>
fprintf(stdout, "MPI_Test result: %d\n", result);<br>
check(MPI_Cancel(&r1));<br>
check(MPI_Wait(&r1, MPI_STATUS_IGNORE));<br>
MPI_Finalize();<br>
return 0;<br>
}<br>
<br>
This stalls in MPI_Wait when executed using "mpiexec -n 1 ./a.out". <br>
<br>
Cheers,<br>
Edric.<br>
<br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
</div>
</span></font></div>
</div>
</div>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
</blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>