<html xmlns="http://www.w3.org/1999/xhtml"><head><title></title></head><body><div>Yes, this is expected. Fault tolerance is an experimental feature and as such is not implemented in all devices. It is currently only compatible with the TCP device.
</div><div><br></div><div>Thanks,
</div><div>Wesley
</div><div><br></div><div><br></div><div class="unibox-signature"></div><div><br></div><blockquote type="cite" class="unibox-hidden"><div>On May 21, 2014, at 8:40 AM, Anatoly G <anatolyrishon@gmail.com> wrote:
</div><div><br></div><div dir="ltr" style="orphans: auto; text-align: start; widows: auto;"><div>Thank you, Pavan.
</div><div>I tried MPICH 3.1, it works good with Fault tolerance.
</div><div>One more question:
</div><div>If I execute my simulation (previous mail) with MPICH 3.1 compiled with --with-device=ch3:sock,
</div><div>I see not stable fault tolerance. Sometimes whole system crashes, but sometimes not.
</div><div>If I use default configuration flags (without --with-device=ch3:sock) whole system is stable.
</div><div><br></div><div>Is this expected behavior?
</div><div><br></div><div>Regards,
</div><div>Anatoly.
</div></div><div class="gmail_extra" style="orphans: auto; text-align: start; widows: auto;"><br><br><div class="gmail_quote">On Mon, May 19, 2014 at 5:03 PM, Balaji, Pavan<span dir="ltr"><<a href="mailto:balaji@anl.gov" target="_blank">balaji@anl.gov</a>></span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><br>Hydra should be compatible between different versions of mpich (or mpich derivatives). However, there’s always a possibility that there was a bug in mpich-3.0.4’s hydra that was fixed in mpich-3.1. So we recommend using the latest version.<br><span class="HOEnZb"><font color="#888888"><br> — Pavan<br></font></span><div class="HOEnZb"><div class="h5"><br>On May 19, 2014, at 1:15 AM, Anatoly G <<a href="mailto:anatolyrishon@gmail.com">anatolyrishon@gmail.com</a>> wrote:<br><br>> Hi Wesley.<br>> Thank you very much for quick response.<br>> I executed your's code. Master can't finish it's execution. It stalled on MPI_Wait on iteration 7.<br>><br>> But if I use MPICH2 hydra, Master process will finish executing by reporting number of times on failure of slave process.<br>><br>> Can you please advice if it's safe to make hybrid system of build with MPICH3.0.4, but using MPICH2 hydra?<br>> Or may be any other solution.<br>> Does MPICH 3.0.4 include all MPICH2 hydra functionality?<br>> May be my configuration of MPICH 3.0.4 is wrong?<br>><br>> Regards,<br>> Anatoly.<br>><br>><br>> On Sun, May 18, 2014 at 8:40 PM, Wesley Bland <<a href="mailto:wbland@anl.gov">wbland@anl.gov</a>> wrote:<br>> Hi Anatoly,<br>><br>> I think the problem may be the way that you're aborting. MPICH catches the system abort call and kills the entire application when it's called. Instead, I suggest using MPI_Abort(MPI_COMM_WORLD, 1); That's what I use in my tests and it works fine. It also seemed to work for your code when I tried. I'll attach my modified version of your code. I switched it to C since I happened to have C++ support disabled on my local install, but that shouldn't change anything.<br>><br>> Thanks,<br>> Wesley<br>><br>><br>> On Sun, May 18, 2014 at 5:18 AM, Anatoly G <<a href="mailto:anatolyrishon@gmail.com">anatolyrishon@gmail.com</a>> wrote:<br>> Dear MPICH2,<br>> Can you please help me with understanding Fault Tolerance in MPICH3.0.4<br>> I have a simple MPI program:<br>> Master calls MPI_Irecv + MPI_Wait in loop.<br>> Single slave: calls MPI_Send x 5 times, then calls abort.<br>><br>> When I execute program with MPICH2 hydra I get multiple times Master process prints about fail in slave. In MPICH3 hydra I get a single message about fail of slave and then Master process enters to endless wait for next Irecv completion.<br>> In both cases I compiled program with MPICH3.0.4<br>><br>> In other words, with MPICH2 hydra each Irecv completes (even if slave died before execution of Irecv) but in MPICH3 hydra not. Causes MPI_Irecv endless wait.<br>><br>> If I compile same program with MPICH2 and use MPICH2 hydra, I get the same result as compiling with MPICH3.0.4 and running with MPICH2 hydra.<br>><br>> Execution command:<br>> mpiexec.hydra -genvall -disable-auto-cleanup -f MpiConfigMachines1.txt -launcher=rsh -n 2 mpi_irecv_ft_simple<br>><br>><br>> Both hydra's configured with:<br>> $ ./configure --prefix=/space/local/mpich-3.0.4/ --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc --no-recursion<br>><br>> $ ./configure --prefix=/space/local/mpich2-1.5b2/ --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc<br>><br>> Can you advice please?<br>><br>> Regards,<br>> Anatoly.<br>><br>><br>> _______________________________________________<br>> discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>> To manage subscription options or unsubscribe:<br>><a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>><br>> _______________________________________________<br>> discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>> To manage subscription options or unsubscribe:<br>><a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br><br>_______________________________________________<br>discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>To manage subscription options or unsubscribe:<br><a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
</div></div>
</blockquote>
</div></div><br class="Apple-interchange-newline"></blockquote></body></html>