<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Folks:</div>
<div class=""><br class="">
</div>
I’m interested in running some fault tolerance experiments using the ULFM extensions in MPICH, but I’m stumbling right out of the gate. When I run the following Python code, rank 0 broadcasts a monotonically increasing value to the other ranks, and it gets
printed to stdout:<br class="">
<br class="">
<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class="">import itertools<br class="">
import time<br class="">
from mpi4py import MPI<br class="">
comm = MPI.COMM_WORLD<br class="">
for count in itertools.count():<br class="">
value = comm.bcast(count, root=0)<br class="">
print(f"{comm.rank}: {value}", flush=True)<br class="">
time.sleep(1)</blockquote>
<br class="">
So far, so good. If I run the code using<br class="">
<br class="">
<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class="">mpiexec -n 3 --disable-auto-cleanup python test.py</blockquote>
<div class=""><br class="">
</div>
<div class="">I get the expected output. If I kill one of the three processes, the others keep running for a few more iterations thanks to --disable-auto-cleanup, until bcast() blocks. My assumption was that MPICH would eventually return an error code, rather
than blocking. Otherwise, it doesn’t seem like my code will ever have the chance to use revoke() and shrink(). What am I missing here? Is there a way to specify timeouts for blocking operations? Am I limited to using async operations with ULFM? Could
you refer me to any ULFM tests or examples that you used in development?</div>
<div class=""><br class="">
</div>
<div class="">Thanks in advance,</div>
<div class="">Tim</div>
<div class=""><br class="">
</div>
<br class="">
<div class="">Timothy M. Shead<br class="">
Sandia National Laboratories<br class="">
<a href="mailto:tshead@sandia.gov" class="">tshead@sandia.gov</a><br class="">
</div>
<br class="">
</body>
</html>