<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">George, thanks for the idea. With those flags, OMPI mpirun said that “sm” was no longer available, and suggested “vader”. So my flags were<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-mca pml ob1 --mca btl <span style="color:red">vader</span>,self,tcp
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is that still a valid test of OMPI? The errors I have been seeing continued to occur with the flags.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Kurt<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> George Bosilca via discuss <discuss@mpich.org> <br>
<b>Sent:</b> Friday, September 15, 2023 1:34 PM<br>
<b>To:</b> discuss@mpich.org<br>
<b>Cc:</b> George Bosilca <bosilca@icl.utk.edu>; Raffenetti, Ken <raffenet@anl.gov><br>
<b>Subject:</b> [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" align="left" style="border:solid black 1.5pt">
<tbody>
<tr>
<td width="100%" style="width:100.0%;border:none;background:#FFEB9C;padding:3.75pt 3.75pt 3.75pt 3.75pt">
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<b><span style="font-size:10.0pt;color:black">CAUTION:</span></b><span style="color:black">
</span><span style="font-size:10.0pt;color:black">This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.</span><span style="color:black">
</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Kurt, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">There is another common component between current MPICH and Open MPI: UCX, that is handling the low level communications. I suggest to try to change the communication substrate to see if your issue continues to exist. For OMPI add `--mca
pml ob1 --mca btl self,sm,tcp' to your mpirun command. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">George.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Sep 15, 2023 at 11:20 AM Joachim Jenke via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Am 15.09.23 um 17:09 schrieb Tony Curtis via discuss:<br>
> <br>
> <br>
>> On Sep 15, 2023, at 11:07 AM, Raffenetti, Ken via discuss <br>
>> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>> wrote:<br>
>><br>
>> 1. Is there a way to detect this kind of overload with an MPI call?<br>
>><br>
>> If MPI detects an error at runtime, the default behavior is to abort <br>
>> the application. If you application does not abort (and you haven't <br>
>> changed the default error handler), then no error was detected by MPI.<br>
>><br>
> <br>
> There’s a tool called MUST that might help<br>
> <br>
> MUST - RWTH AACHEN UNIVERSITY Lehrstuhl für Informatik 12 - Deutsch <br>
> <<a href="https://www.i12.rwth-aachen.de/go/id/nrbe" target="_blank">https://www.i12.rwth-aachen.de/go/id/nrbe</a>><br>
> <a href="http://i12.rwth-aachen.de/" target="_blank">i12.rwth-aachen.de</a> <<a href="https://www.i12.rwth-aachen.de/go/id/nrbe" target="_blank">https://www.i12.rwth-aachen.de/go/id/nrbe</a>><br>
> apple-touch-icon-180x180.png <<a href="https://www.i12.rwth-aachen.de/go/id/nrbe" target="_blank">https://www.i12.rwth-aachen.de/go/id/nrbe</a>><br>
> <br>
> <<a href="https://www.i12.rwth-aachen.de/go/id/nrbe" target="_blank">https://www.i12.rwth-aachen.de/go/id/nrbe</a>><br>
> <br>
<br>
The current release version can only detect conflicts in buffer usage at <br>
MPI API level. That means, it will only detect buffer conflicts for <br>
in-flight messages as in:<br>
<br>
MPI_Irecv(buf, MPI_INT, 10, ..., req1);<br>
MPI_Irecv(&buf[9], MPI_INT, 10, ..., req2);<br>
MPI_Wait(req1,...);<br>
MPI_Wait(req2,...);<br>
<br>
The upcoming release I was referencing in my other mail would detect <br>
conflicting accesses to in-flight buffers as in:<br>
<br>
MPI_Irecv(buf, MPI_INT, 10, ..., req);<br>
buf[5]=5;<br>
MPI_Wait(req,...);<br>
<br>
> <br>
> (Not affiliated, just happen to have been looking at it)<br>
<br>
Happy to see that people look at the tool :D<br>
<br>
- Joachim<br>
<br>
> <br>
> Tony<br>
> <br>
> <br>
> <br>
> _______________________________________________<br>
> discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
> To manage subscription options or unsubscribe:<br>
> <a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
<br>
-- <br>
Dr. rer. nat. Joachim Jenke<br>
<br>
IT Center<br>
Group: High Performance Computing<br>
Division: Computational Science and Engineering<br>
RWTH Aachen University<br>
Seffenter Weg 23<br>
D 52074 Aachen (Germany)<br>
Tel: +49 241 80- 24765<br>
Fax: +49 241 80-624765<br>
<a href="mailto:jenke@itc.rwth-aachen.de" target="_blank">jenke@itc.rwth-aachen.de</a><br>
<a href="http://www.itc.rwth-aachen.de/" target="_blank">www.itc.rwth-aachen.de</a><br>
<br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</body>
</html>