<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Thomas,
<div class=""><br class="">
</div>
<div class="">To summarize issue: MPICH proxy does not reap processes after its exit?
<div class=""><br class="">
</div>
<div class="">I’ll look into the issue. From my peruse of current MPI standards, it does not really specify the behavior after MPI_Finalize. The dynamic process part of MPI support is weak, primary due to lack of application adoption. Regardless, I agree that
the proxy creates the process should be responsible to reap its dead children.</div>
<div class=""><br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
— </div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
Hui Zhou</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
<br class="Apple-interchange-newline">
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 6, 2018, at 3:53 AM, Thomas Pak via discuss <<a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Dear all,</div>
<br class="">
<div class="">Does anyone have any feedback on this issue? The problem still persists in MPICH-3.3 and it seems like it is pointing to a severe flaw in how MPICH handles dynamic process creation. To reiterate, the following short program using MPICH creates
zombie processes indefinitely until no more processes can be created.</div>
<br class="">
<div class="">"""</div>
<div class=""><font style="font-family:monospace" class="">#include <stdio.h></font></div>
<div class=""><font style="font-family:monospace" class="">#include <mpi.h></font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">int main(int argc, char *argv[]) {</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">// Initialize MPI</font></div>
<div class=""><font style="font-family:monospace" class="">MPI_Init(NULL, NULL);</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">// Get parent</font></div>
<div class=""><font style="font-family:monospace" class="">MPI_Comm parent;</font></div>
<div class=""><font style="font-family:monospace" class="">MPI_Comm_get_parent(&parent);</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">// If the process was not spawned</font></div>
<div class=""><font style="font-family:monospace" class="">if (parent == MPI_COMM_NULL) {</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">puts("I was not spawned!");</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">// Spawn child process in loop</font></div>
<div class=""><font style="font-family:monospace" class="">char *cmd = argv[0];</font></div>
<div class=""><font style="font-family:monospace" class="">char **cmd_argv = MPI_ARGV_NULL;</font></div>
<div class=""><font style="font-family:monospace" class="">int maxprocs = 1;</font></div>
<div class=""><font style="font-family:monospace" class="">MPI_Info info = MPI_INFO_NULL;</font></div>
<div class=""><font style="font-family:monospace" class="">int root = 0;</font></div>
<div class=""><font style="font-family:monospace" class="">MPI_Comm comm = MPI_COMM_SELF;</font></div>
<div class=""><font style="font-family:monospace" class="">MPI_Comm intercomm;</font></div>
<div class=""><font style="font-family:monospace" class="">int *array_of_errcodes = MPI_ERRCODES_IGNORE;</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">for (;;) {</font></div>
<div class=""><font style="font-family:monospace" class="">MPI_Comm_spawn(cmd, cmd_argv, maxprocs, info, root, comm,</font></div>
<div class=""><font style="font-family:monospace" class="">&intercomm, array_of_errcodes);</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">MPI_Comm_disconnect(&intercomm);</font></div>
<div class=""><font style="font-family:monospace" class="">}</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">// If process was spawned</font></div>
<div class=""><font style="font-family:monospace" class="">} else {</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">puts("I was spawned!");</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">MPI_Comm_disconnect(&parent);</font></div>
<div class=""><font style="font-family:monospace" class="">}</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">// Finalize</font></div>
<div class=""><font style="font-family:monospace" class="">MPI_Finalize();</font></div>
<br class="">
<div class=""><font style="font-family:monospace" class="">}</font></div>
<div class="">"""</div>
<br class="">
<div class="">Thanks in advance for your help.</div>
<br class="">
<div class="">Best wishes,</div>
<div class="">Thomas Pak</div>
<br class="">
<div class="gmail_quote_attribution">On Oct 16 2018, at 4:57 pm, Thomas Pak <<a href="mailto:thomas.pak@maths.ox.ac.uk" class="">thomas.pak@maths.ox.ac.uk</a>> wrote:</div>
<blockquote class=""><br class="">
<div class="">
<div class="">Dear all,</div>
<br class="">
<div class="">My MPI application spawns a large number of MPI processes using MPI_Comm_spawn over its total lifetime. Unfortunately, I have experienced that this results in zombie processes of hydra_pmi_proxy accumulating over time. As a result, my MPI application
will eventually crash because there are too many zombie processes and no new processes can be created.</div>
<br class="">
<div class="">This issue does not seem to be new; I have found references to similar issues in at least three different places:</div>
<br class="">
<div class="">- <a href="https://lists.mpich.org/pipermail/discuss/2013-March/000599.html" title="https://lists.mpich.org/pipermail/discuss/2013-March/000599.html" class="">
[mpich-discuss] MPI_Comm_Spawn causing zombies of hydra_pmi_proxy</a> (March 2013)</div>
<div class="">- <a href="https://github.com/pmodels/mpich/issues/1677" title="https://github.com/pmodels/mpich/issues/1677" class="">
[GitHub issue] hydra_pmi_proxy zombie MPI_comm_spawn #1677</a> (October 2016)</div>
<div class="">- <a href="https://groups.google.com/forum/#!msg/mpi4py/A9mN-2UkFf8/2gnUEtArDwAJ" title="https://groups.google.com/forum/#!msg/mpi4py/A9mN-2UkFf8/2gnUEtArDwAJ" class="">
[Google Groups] Hanging in spawn in master-slave code and hydra_pmi_proxy zombies</a> (December 2016)</div>
<div class="">In the first reference, a dirty fix was proposed by Silvan Brändli to simply reap all the zombie processes in a loop at the start of the function HYDU_create_process in src/pm/hydra/utils/launch/launch.c. However, I have found that this comes
with its own set of problems and makes my MPI application unstable.</div>
<br class="">
<div class="">In the GitHub issue, the problem was marked as "worksforme". However, this problem persists for me and other machines that I have tried to reproduce it on, so it clearly has not yet been resolved.</div>
<br class="">
<div class="">In the Google Groups thread, the problem was "resolved" by avoiding spawning MPI processes entirely, which is not an option in my case.</div>
<br class="">
<div class="">Pavan Balaji mentions in the 2013 mpi-discuss thread that this issue was a problem "once upon a time". It seems to have risen from the dead again.</div>
<br class="">
<div class="">I have attached a short and self-contained program written in C that reproduces the problem. The program simply spawns child processes using MPI_Comm_spawn in an infinite loop, where each child process exits after writing a message to stdout.
I use MPI_Comm_disconnect to disconnect the Intercommunicator, but the same problem occurs when using MPI_Comm_free instead.</div>
<br class="">
<div class="">I have also attached the logfiles generated while building MPICH, as described in the README, except for the file mpich-3.2.1/src/pm/hydra/tools/topo/hwloc/hwloc/config.log, as that was missing. I am using MPICH version 3.2.1, which is the latest
stable release at the time of writing.</div>
<br class="">
<div class="">Thanks in advance for your help!</div>
<br class="">
<div class="">Best wishes,</div>
<div class="">Thomas Pak</div>
<div class="">_______________________________________________</div>
<div class="">discuss mailing list <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a></div>
<div class="">To manage subscription options or unsubscribe:</div>
<div class=""><a href="https://lists.mpich.org/mailman/listinfo/discuss" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></div>
</div>
</blockquote>
_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" class="">https://lists.mpich.org/mailman/listinfo/discuss</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</body>
</html>