<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Mangal;
panose-1:0 0 4 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:"Lucida Grande";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Michael, <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am cc’ing this note to the MVAPICH-Core developers. One of the developers will follow-up with you.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks, <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">DK<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Thakur, Rajeev via discuss <discuss@mpich.org>
<br>
<b>Sent:</b> Friday, September 15, 2023 11:42 AM<br>
<b>To:</b> discuss@mpich.org<br>
<b>Cc:</b> Thakur, Rajeev <thakur@anl.gov><br>
<b>Subject:</b> Re: [mpich-discuss] issue/question<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">The mailing list for MVAPICH2 is mvapich-discuss@ lists. osu. edu . Rajeev From: "Michael P. Deignan via discuss" <discuss@ mpich. org> Reply-To: "discuss@ mpich. org"
<discuss@ mpich. org> Date: Friday, September 15, <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande"">The mailing list for MVAPICH2 is
<a href="mailto:mvapich-discuss@lists.osu.edu">mvapich-discuss@lists.osu.edu</a> .<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande"">Rajeev<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande""><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">"Michael P. Deignan via discuss" <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Reply-To: </b>"<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>" <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Date: </b>Friday, September 15, 2023 at 10:08 AM<br>
<b>To: </b>"<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>" <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Cc: </b>"Michael P. Deignan" <<a href="mailto:michael.p.deignan@gmail.com">michael.p.deignan@gmail.com</a>><br>
<b>Subject: </b>[mpich-discuss] issue/question<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas;color:#24292F"><o:p> </o:p></span></p>
<pre style="white-space:pre-wrap;-ms-word-wrap: break-word"><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">I have a user who is running an mpi model using mvapich2, compiled with the Intel OneAPI compiler (2023.1.0) on a Rocky 8.6 OpenHPC cluster. Bear with me for a minute as I lay the foundation to get to rdma-core.<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">Periodically (randomly after X minutes, where X has been as few as 5 and as much as 30) the model will crash and in the error log there will be thousands of messages:<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">[mv2_mcast_resend_window] Failed to post mcast send errno:0<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">Basically this message repeats until the disk fills up and the job terminates with an out of disk space message.<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">I tracked the error to the mvapich2 source code at src/mpid/ch3/channels/common/include/ibv_mcast.h:<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> int ret; \ <o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> ret = ibv_post_send(_mcast_ctx->ud_ctx->qp, \<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> &(_v->desc.u.sr), &(_v->desc.y.bad_sr)); \<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> if (ret) { \<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> PRINT_ERROR("Failed to post mcast send errno:%d\n", \<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> errno); \<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> } \<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> _mcast_ctx->ud_ctx->send_wqes_avail--; \<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> while (_mcast_ctx->ud_ctx->send_wqes_avail <= 0) { \<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> MPIDI_CH3I_Progress(FALSE, NULL); \<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> } <o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">which leads me to the ibv_post_send subroutine in include/infiniband/verbs.h:<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black">/** <o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black">* ibv_post_send - Post a list of work requests to a send queue.<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black">*<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black">* If IBV_SEND_INLINE flag is set, the data buffers can be reused<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black">* immediately after the call returns.<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black">*/<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black">static inline int ibv_post_send(struct ibv_qp *qp, struct ibv_send_wr *wr,<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> struct ibv_send_wr **bad_wr)<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black">{<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"> return qp->context->ops.post_send(qp, wr, bad_wr);<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black">}<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">For some reason, this function call (ibv_post_send) is returning zero back to the caller, and no error code is being set (errno = 0).<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">As the model does run for a random amount of time, this would seem to suggest some type of hardware problem, but everything we've checked (system, ib card, ib switch, etc.,) doesn't show any errors.<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">Can anyone shed some light on under what circumstances a call to ibv_post_send would, in fact, return zero when previously it didn't?<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"><o:p> </o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">I realize I'm probably not giving much data to really help.. I suppose there could be some type of underlying operating system issue but even there I'm not sure how I would determine where the problem is, since everything else is running fine (and it appears only this one model seems to cause this problem) with nothing abnormal in the event logs. I tried running the model on an older Rocks Centos 7.x cluster and likewise received the same error.<o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black"> <o:p></o:p></span></pre>
<pre><span style="font-size:16.0pt;font-family:"Segoe UI",sans-serif;color:black">Having some basic understanding of what would cause the post send call to return zero and not set an error code might help.<o:p></o:p></span></pre>
</div>
</div>
</body>
</html>