<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Anatoly,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Without a reproducible simple case, it will be difficult to pin down the issue. To debug, I would start injecting prints right above the assertion to see what is actually in the
<code>pkt_type</code>. But since mpich-3.1 is very old, can you try a newer release? The latest is mpich 4.0b1 --
<a href="https://www.mpich.org/downloads/" id="LPlnkOWALinkPreview">https://www.mpich.org/downloads/</a><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-- <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui Zhou<br>
</div>
<div class="_Entity _EType_OWALinkPreview _EId_OWALinkPreview _EReadonly_1">
<div id="LPBorder_GTaHR0cHM6Ly93d3cubXBpY2gub3JnL2Rvd25sb2Fkcy8." class="LPBorder579398" style="width: 100%; margin-top: 16px; margin-bottom: 16px; position: relative; max-width: 800px; min-width: 424px;">
<table id="LPContainer579398" role="presentation" style="padding: 12px 36px 12px 12px; width: 100%; border-width: 1px; border-style: solid; border-color: rgb(200, 200, 200); border-radius: 2px;">
<tbody>
<tr style="border-spacing: 0px;" valign="top">
<td>
<div id="LPImageContainer579398" style="position: relative; margin-right: 12px; height: 160px; overflow: hidden;">
<a target="_blank" id="LPImageAnchor579398" href="https://www.mpich.org/downloads/"><img id="LPThumbnailImageId579398" alt="" style="display: none;" height="160" src="https://www.mpich.org/files/2012/10/rnd100_home.jpg1zB"></a></div>
</td>
<td style="width: 100%;">
<div id="LPTitle579398" style="font-size: 21px; font-weight: 300; margin-right: 8px; font-family: "wf_segoe-ui_light", "Segoe UI Light", "Segoe WP Light", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12px;">
<a target="_blank" id="LPUrlAnchor579398" href="https://www.mpich.org/downloads/" style="text-decoration: none; color: var(--themePrimary);">Downloads | MPICH</a></div>
<div id="LPDescription579398" style="font-size: 14px; max-height: 100px; color: rgb(102, 102, 102); font-family: "wf_segoe-ui_normal", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12px; margin-right: 8px; overflow: hidden;">
Downloads MPICH is distributed under a BSD-like license. NOTE: MPICH binary packages are available in many UNIX distributions and for Windows. For example, you can search for it using “yum” (on Fedora), “apt” (Debian/Ubuntu), “pkg_add” (FreeBSD) or “port”/”brew”
(Mac OS).</div>
<div id="LPMetadata579398" style="font-size: 14px; font-weight: 400; color: rgb(166, 166, 166); font-family: "wf_segoe-ui_normal", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif;">
www.mpich.org</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<br>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Anatoly G via discuss <discuss@mpich.org><br>
<b>Sent:</b> Wednesday, November 24, 2021 3:49 AM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Anatoly G <anatolyrishon@gmail.com><br>
<b>Subject:</b> [mpich-discuss] MPI_Waitany got abort signal.</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Hello Mpich,</div>
<div>I use mpich-3.1 on Ubuntu 14.</div>
<div>Each process has complicated logic except process 0.<br>
</div>
<div>Process 0 is used as a router to communicate with an application and broadcast/collect results from other processes.</div>
<div>During night runs, sometimes I see a single failure of process with rank 0.</div>
<div><br>
</div>
<div>From process 0 I get wallowing print:</div>
<div><font size="1">Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 596: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO<br>
internal ABORT - process 0<br>
*** Error in `/export/home/fpd/versions/current_ver/third_party/MPI_Scheduler': double free or corruption (fasttop): 0x00007f27003f59a0 ***<br>
</font></div>
<div><br>
</div>
<div><b><u>Stack trace:</u></b></div>
<font size="1">2d40 /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f2712ff2d40]<br>
gsignal + 57<br>
abort + 328<br>
f394 /lib/x86_64-linux-gnu/libc.so.6(+0x73394) [0x7f271302f394]<br>
b66e /lib/x86_64-linux-gnu/libc.so.6(+0x7f66e) [0x7f271303b66e]<br>
std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() + 31<br>
8259 /lib/x86_64-linux-gnu/libc.so.6(+0x3c259) [0x7f2712ff8259]<br>
82a5 /lib/x86_64-linux-gnu/libc.so.6(+0x3c2a5) [0x7f2712ff82a5]<br>
d049 /export/home/fpd/versions/current_ver/third_party/libMPIServices.so(+0x224049) [0x7f2714bdd049]<br>
MPID_Abort + 103<br>
MPIR_Assert_fail + 37<br>
d598 /export/home/fpd/versions/current_ver/third_party/libMPIServices.so(+0x244598) [0x7f2714bfd598]<br>
MPID_nem_tcp_connpoll + 366<br>
MPIDI_CH3I_Progress + 1408<br>
MPI_Waitany + 1072</font><br>
<div><br>
</div>
<div>From the rest of processes I get:</div>
<div><font size="1">terminate called after throwing an instance of 'int'<br>
s/netmod/tcp/socksm.c at line 596: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO<br>
internal ABORT - process 1<br>
*** Error in `Scheduler': double free or corruption (fasttop): 0x0000000003e16d60 ***</font><br>
</div>
<div><br>
</div>
<div>Unfortunately, I can't reproduce this failure on a simplified system. </div>
<div>Even on a real system failure can happen once in at night. </div>
<div><br>
</div>
<div>We have a memory monitor which shows that we have free memory on the computer.</div>
<div>Can you please advise me, what can be the reason for failure?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Anatoly.</div>
<div><br>
</div>
</div>
</div>
</body>
</html>