<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Anatoly,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I think the failure behavior is yet to be standardized, thus the behavior falls into the undefined behavior category.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I suspect the different outcomes with MPICH 3.1 and 3.4.2 are from changes in the process manager. Could you try using hydra from 3.1 in your test with 3.4.2?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-- <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui Zhou<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Anatoly G via discuss <discuss@mpich.org><br>
<b>Sent:</b> Monday, December 13, 2021 7:18 AM<br>
<b>To:</b> mpich-discuss@mcs.anl.gov <mpich-discuss@mcs.anl.gov><br>
<b>Cc:</b> Anatoly G <anatolyrishon@gmail.com><br>
<b>Subject:</b> [mpich-discuss] Question regarding fault tolerance difference 3.1 & 3.4.2</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi MPICH,
<div>I have a small program which has different outputs on MPICH 3.1 & MPICH 3.4.2<br>
</div>
<div>The program (code attached):</div>
<div>
<ol>
<li>Master executes ping-pong with each one of the slaves (separately)</li><li>Each slave replies to master when it gets a message from master</li><li>One of the slaves simulates failre with command abort().</li><li>Master recognizes that slave fails and continues to work with survived slaves only.</li></ol>
<div><u>I execute my program on ubuntu18.</u><br>
</div>
</div>
<div>I use TCP as a transport layer.<br>
</div>
<div><u>Execute command</u>: </div>
<div>mpiexec.hydra -genvall -disable-auto-cleanup -f MpiConfigMachines.txt -launcher=ssh -n 3 ft_ping_pong<br>
</div>
<div><br>
</div>
<div>
<div>
<div>I expect that "ping pong" will continue till all 20 iterations will finish.</div>
</div>
<div>MPICH 3.1 hydra works as expected only slave 1 fails and ping-pong continues between Master and slave2.</div>
<div>MPICH 3.4.2 hydra process slave 2 fails together with slave 1.</div>
<div><br>
</div>
</div>
<div><u>MPICH 3.1 configuration:</u></div>
<div>./configure --prefix="my directory" --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc<br>
</div>
<div><br>
</div>
<div><u>MPICH 3.4.2 configuration:</u></div>
<div>$ ./configure --prefix="my directory" --enable-error-checking=all --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc --disable-fortran
<b>--with-device=ch3:nemesis</b> --enable-error-messages=all<br>
</div>
<div><br>
</div>
<div>Should I use another device or the behavior was modified between versions?<br>
</div>
<div><br>
</div>
<div>Regards,</div>
<div>Anatoly.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</body>
</html>