<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Lana,</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
You can try add <code>-disable-auto-cleanup</code> to mpiexec to prevent it kill every other processes when one process exits abnormally.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I usually use a <code>.gdbinit</code> script to get a backtrace for such cases. For example, if you program is
<code>./t</code>, then</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<code>mpirun gdb ./t</code></div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Example <code>.gdbinit</code>:</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
```</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
set $_exitcode = -999</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
run</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
if $_exitcode == -999</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
backtrace</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
end</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
exit $_exitcode</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
```</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hope that helps.</div>
<div id="ms-outlook-mobile-signature">
<p class="MsoNormal">-- <br>
Hui Zhou</p>
<p class="MsoNormal"> </p>
</div>
<div id="mail-editor-reference-message-container">
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="text-align: left; padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>Lana Deere via discuss <discuss@mpich.org><br>
<b>Date: </b>Monday, April 6, 2026 at 10:24 AM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>Lana Deere <lana.deere@gmail.com><br>
<b>Subject: </b>[mpich-discuss] hydra_pmi_proxy sending signal 9 to successful processes<br>
<br>
</div>
<div dir="ltr" id="pfptBannerukxck0k" style="visibility: visible !important; opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; text-align: left !important; margin: 16px 0px !important; padding: 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: rgb(208, 216, 220); border-top-width: 4px !important; border-top-style: solid !important; border-top-color: rgb(144, 164, 174) !important;">
<div id="pfptBannerukxck0k" style="visibility: visible !important; opacity: 1 !important; background-color: rgb(208, 216, 220); max-height: none !important; float: left !important; display: block !important; margin: 0px 0px 1px !important; max-width: 600px !important;">
<div id="pfptBannerukxck0k" style="opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; visibility: visible !important; background-color: rgb(208, 216, 220); color: rgb(0, 0, 0); font-family: Arial, sans-serif !important; font-weight: bold !important; font-size: 14px !important; line-height: 18px !important;">
This Message Is From an External Sender</div>
<div id="pfptBannerukxck0k" style="font-weight: normal; opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; visibility: visible !important; background-color: rgb(208, 216, 220); color: rgb(0, 0, 0); font-family: Arial, sans-serif !important; font-size: 12px !important; line-height: 18px !important; margin-top: 2px !important;">
This message came from outside your organization.</div>
</div>
<div style="line-height: 0; height: 0px; display: block; font-size: 0.01px;"> </div>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
I've got several MPI programs here. The one which is the most complicated started exiting, reporting that a process got signal 9 while cleaning up after a run it reported was successful. Many of the other MPI processes showed truncated outputs as if they too
had received a signal 9. Only that one program has this problem, the other programs don't. I tried reducing the big program to a small testcase which reproduces the issue but was unsuccessful.</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
I did put a gdb onto the hydra_pmi_proxy and discovered that it is the process sending the signal 9 to the various MPI processes,</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
(gdb) where<br>
#0 0x00007f4b17853d7e in killpg () from /lib64/libc.so.6<br>
#1 0x00000000004053e2 in PMIP_bcast_signal (sig=sig@entry=9) at proxy/pmip_pg.c:259<br>
#2 0x0000000000406e60 in pmi_cb (fd=9, events=<optimized out>, userp=<optimized out>)<br>
at proxy/pmip_cb.c:326<br>
#3 0x0000000000421418 in HYDT_dmxu_poll_wait_for_event (wtime=<optimized out>)<br>
at lib/tools/demux/demux_poll.c:75<br>
#4 0x0000000000403ff5 in main (argc=<optimized out>, argv=<optimized out>) at proxy/pmip.c:121<br>
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
At that time I was using mpich 4.3.0, so I upgraded to 5.0.0 hoping the problem would be resolved. 5.0.0 still showed the same symptom. This all is happening on SUSE Linux 15.5.</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
On CentOS7 and Rocky Linux 9 we use mvapich2 2.3.6, so for an experiment I took the mpirun and hydra_pmi_proxy from 2.3.6 and used them instead of the versions from the mpich 5.0.0 release. Now the program works without difficulty. All of this suggests to
me that the hydra_pmi_proxy has incorrectly determined that one of the MPI processes exited with a signal. Any suggestions about what's going on?</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="gmail_signature" style="direction: ltr;"><br>
.. Lana (<a href="mailto:lana.deere@gmail.com" target="_blank" data-outlook-id="eb064d95-5d50-43e1-be95-db6168bc700e">lana.deere@gmail.com</a>)<br>
<br>
<br>
</div>
</div>
</body>
</html>