<div dir="ltr">Hello Tony, <div>Thank you very much for your reply. I am posting you the output of 'ps x', when '<span style="color:rgb(0,0,0);font-family:monospace">mpirun -n 5 mpich-3.0.4/examples/hellow' freezes:</span></div><div><span style="color:rgb(0,0,0);font-family:monospace"><br></span></div><div><span style="color:rgb(0,0,0);font-family:monospace">"</span></div><div>marto@dragonfly% ps x<br> PID TT STAT TIME COMMAND<br> 1025 ?? I6s 0:00.14 /usr/local/bin/dbus-daemon --syslog-only --fork --print-pid 5 --print-address 7 --session<br> 1028 ?? I4s 0:00.08 kdeinit5: Running... (kdeinit5)<br> 1029 ?? I2 0:00.26 /usr/local/lib/libexec/kf5/klauncher --fd=8<br> 1031 ?? I7 0:00.65 kded5<br> 1040 ?? I3 0:00.21 /usr/local/bin/kaccess<br> 1047 ?? I4 0:00.02 /usr/local/libexec/dconf-service<br> 1058 ?? I1 0:00.26 /usr/local/bin/ksmserver<br> 1061 ?? I6 0:00.38 /usr/local/bin/kglobalaccel5<br> 1067 ?? I7 0:00.10 /usr/local/lib/libexec/kf5/kscreen_backend_launcher<br> 1069 ?? I0 0:18.15 /usr/local/bin/kwin_x11 -session 100000000158893236700000019470008_1589011398_40052<br> 1071 ?? I3 0:07.36 /usr/local/bin/plasmashell<br> 1073 ?? I6 0:00.06 /usr/local/bin/xembedsniproxy<br> 1075 ?? I3 0:00.17 /usr/local/lib/libexec/polkit-kde-authentication-agent-1<br> 1080 ?? I6 0:00.05 /usr/local/bin/kwrited<br> 1084 ?? I5 0:00.07 /usr/local/bin/gmenudbusmenuproxy<br> 1093 ?? I2 0:00.24 /usr/local/lib/libexec/DiscoverNotifier -session 1014ce0c7d3000158899951500000013340005_1589011398_16101<br> 1121 ?? I2 0:00.45 /usr/local/bin/korgac -session 100000000158893236700000019470009_1589011398_16368<br> 1128 ?? I2 0:00.00 /usr/local/libexec/at-spi-bus-launcher<br> 1130 ?? I3 0:00.24 /usr/local/lib/libexec/kactivitymanagerd<br> 1131 ?? S5 0:00.11 /usr/local/bin/dbus-daemon --config-file=/usr/local/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3<br> 1134 ?? I2 0:00.07 /usr/local/libexec/at-spi2-registryd --use-gnome-session<br> 1136 ?? I1 0:07.00 /usr/local/lib/thunderbird/thunderbird --sm-client-id 100000000158893248800000019470014<br> 1140 ?? S5 0:01.73 /usr/local/bin/gkrellm --sm-client-id 100000000158893266700000019470017<br> 1142 ?? I6 0:00.54 /usr/local/bin/dolphin -session 100000000158893337700000019470020_1589011398_16282<br> 1144 ?? I7 0:00.35 /usr/local/bin/kmix -session 1014ce0c7d3000158899952000000013340007_1589011398_17041<br> 1146 ?? I0 0:01.01 /usr/local/bin/konsole -session 1014ce0c7d3000158899955500000013340009_1589011398_16527<br> 1152 ?? I1 0:00.24 /usr/local/lib/libexec/org_kde_powerdevil<br> 1214 ?? I4 0:00.00 kdeinit5: file.so file local:/var/run/user/1001/klauncherPKUbmP.1.slave-socket local:/var/run/user/1001/kio_desktopMgyNoK.1.slave-socket (kdeinit5)<br> 1216 ?? Z 0:00.00 (sh)<br> 1217 ?? Z 0:00.00 (sh)<br> 1218 ?? Z 0:00.00 (sh)<br> 1219 ?? Z 0:00.01 (sh)<br> 1220 ?? Z 0:00.00 (sh)<br> 1221 ?? Z 0:00.01 (sh)<br> 1222 ?? Z 0:00.00 (sh)<br> 1223 ?? Z 0:00.00 (sh)<br> 1224 ?? Z 0:00.01 (sh)<br> 1225 ?? Z 0:00.00 (sh)<br> 1226 ?? Z 0:00.00 (sh)<br> 1227 ?? Z 0:00.00 (sh)<br> 1228 ?? Z 0:00.00 (sh)<br> 1229 ?? Z 0:00.00 (sh)<br> 1230 ?? Z 0:00.00 (sh)<br> 1231 ?? Z 0:00.00 (sh)<br> 1232 ?? Z 0:00.01 (sh)<br> 1233 ?? Z 0:00.00 (sh)<br> 1234 ?? Z 0:00.01 (sh)<br> 1235 ?? Z 0:00.00 (sh)<br> 1277 ?? S4 0:00.00 kdeinit5: file.so file local:/var/run/user/1001/klauncherPKUbmP.1.slave-socket local:/var/run/user/1001/kded5wwlydC.1.slave-socket (kdeinit5)<br> 1278 ?? I7 0:05.48 chrome: (chrome)<br> 1281 ?? S1 0:01.06 chrome: --type=utility --field-trial-handle=9865741493051962933,5833454360386249874,131072 --lang=en-US --service-sandbox-type=network --disable-webrtc-apm-in-audio-service --shared<br> 1282 ?? I0 0:02.61 chrome: --type=gpu-process --field-trial-handle=9865741493051962933,5833454360386249874,131072 --gpu-preferences=MAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAAAABgAAAAAAAQAAAAAAAAAAAAAAAAAAAACAAA<br> 1302 ?? I0 0:00.67 chrome: --type=renderer --disable-webrtc-apm-in-audio-service --field-trial-handle=9865741493051962933,5833454360386249874,131072 --disable-gpu-compositing --lang=en-US --enable-aut<br> 1315 ?? I6 0:14.93 chrome: --type=renderer --disable-webrtc-apm-in-audio-service --field-trial-handle=9865741493051962933,5833454360386249874,131072 --disable-gpu-compositing --lang=en-US --enable-aut<br> 1320 ?? I1 0:00.26 chrome: --type=renderer --disable-webrtc-apm-in-audio-service --field-trial-handle=9865741493051962933,5833454360386249874,131072 --disable-gpu-compositing --lang=en-US --enable-aut<br> 1321 ?? I4 0:00.20 chrome: --type=renderer --disable-webrtc-apm-in-audio-service --field-trial-handle=9865741493051962933,5833454360386249874,131072 --disable-gpu-compositing --lang=en-US --enable-aut<br> 1322 ?? I5 0:00.25 chrome: --type=utility --field-trial-handle=9865741493051962933,5833454360386249874,131072 --lang=en-US --service-sandbox-type=audio --disable-webrtc-apm-in-audio-service --shared-f<br> 1422 ?? I2s 0:00.00 /home/marto/WRF/Build_WRF/LIBRARIES/mpich/bin/hydra_pmi_proxy --control-port dragonfly:2416 --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0<br> 1423 ?? R4s 1:11.89 mpich-3.0.4/examples/hellow<br> 1424 ?? I2s 0:00.00 mpich-3.0.4/examples/hellow<br> 1425 ?? R3s 1:11.04 mpich-3.0.4/examples/hellow<br> 1426 ?? R2s 1:11.39 mpich-3.0.4/examples/hellow<br> 1427 ?? R6s 1:11.96 mpich-3.0.4/examples/hellow<br> 972 v0 I0 0:00.00 -tcsh (tcsh)<br> 977 v0 I0+ 0:00.00 /bin/sh /usr/local/bin/startx<br> 1001 v0 I0+ 0:00.00 xinit /home/marto/.xinitrc -- /usr/local/bin/X :0 -auth /home/marto/.serverauth.977<br> 1004 v0 I0 0:00.00 ck-launch-session startplasma-x11<br> 1013 v0 I1 0:00.03 startplasma-x11<br> 1024 v0 I6 0:00.00 dbus-launch --autolaunch 7a25735143fe35ec86d2d9be5eac81cc --binary-syntax --close-stderr<br> 1043 v0 I2 0:00.13 /usr/local/bin/plasma_session<br> 1196 1 I2s 0:00.07 /bin/tcsh<br> 1421 1 I2+ 0:00.00 mpirun -n 5 mpich-3.0.4/examples/hellow (mpiexec.hydra)<br> 1199 2 I5s+ 0:00.04 /bin/tcsh<br> 1194 3 S6s 0:00.04 /bin/tcsh<br> 1429 3 R6+ 0:00.00 ps x <br></div><div><span style="color:rgb(0,0,0);font-family:monospace">"</span></div><div><span style="color:rgb(0,0,0);font-family:monospace"><br></span></div><div><font color="#000000" face="monospace">After that freeze, I killed mpirun with Ctrl + C:</font></div><div><font color="#000000" face="monospace">"</font></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">marto@dragonfly% mpirun -n 5 mpich-3.0.4/examples/hellow </span><br>^C[mpiexec@dragonfly] Sending Ctrl-C to processes as requested
<br>[mpiexec@dragonfly] Press Ctrl-C again to force abort
<br>[proxy:0:0@dragonfly] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:71): assert (!(pollfds[i].revents & ~POLLIN & ~POLLOUT & ~POLLHUP & ~POLLERR)) failed
<br>[proxy:0:0@dragonfly] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event
<br>[mpiexec@dragonfly] control_cb (./pm/pmiserv/pmiserv_cb.c:202): assert (!closed) failed
<br>[mpiexec@dragonfly] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
<br>[mpiexec@dragonfly] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event
<br>[mpiexec@dragonfly] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion<br></span></div><div><font color="#000000" face="monospace">"</font></div><div><font color="#000000" face="monospace"><br></font></div><div><font color="#000000" face="monospace">The next relaunch of mpirun with 5 cores was successful. For completeness, I am attaching the output of 'ps x' after mpirun with 2 cores freezes, which it actually with 2 cores always does.</font></div><div><font color="#000000" face="monospace"><br></font></div><div><font color="#000000" face="monospace">I hope this was helpful. I am looking forward to your reply.</font></div><div><font color="#000000" face="monospace"><br></font></div><div><font color="#000000" face="monospace">Best regards,</font></div><div><font color="#000000" face="monospace">Martin</font></div><span style="font-family:monospace"><br></span></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, May 9, 2020 at 2:15 PM Tony Curtis <<a href="mailto:anthony.curtis@stonybrook.edu">anthony.curtis@stonybrook.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><br><div><br><blockquote type="cite"><div>On May 9, 2020, at 2:24 AM, Martin Ivanov <<a href="mailto:marto1980@gmail.com" target="_blank">marto1980@gmail.com</a>> wrote:</div><br><div><div dir="ltr">Hello Tony,<div>Thank you very much for your reply. I followed your advice and gave the hostname 'dragonfly' to my machine. Then, in /etc/hosts I provided gave the alias 'dragonfly' to localhost as you suggested:</div><div><br></div><div><span style="font-family:monospace"><span>"</span></span></div><div><span style="font-family:monospace"><span>marto@dragonfly% cat /etc/hosts</span><br></span></div><div><span style="font-family:monospace"><span>::1 localhost dragonfly
</span><br>127.0.0.1 localhost dragonfly<br></span></div><div><span style="font-family:monospace">"</span></div><div><br></div><div>I compiled both the icpi and hellow examples. Now mpiexec seems to work, although not as reliably as I might wish. Running any of the examples with one core like this:<br></div><div><br></div><div><span style="font-family:monospace"><span>mpiexec -n 1 mpich-3.0.4/examples/hellow</span><br></span></div><div><span style="font-family:monospace"><span><br></span></span></div><div><span style="font-family:monospace"><span>is always successful. I could never get the command to finish with 2, 3, or 4 cores: it simply hangs. E.g. with 2 cores after running the above command I get:</span></span></div><div><span style="font-family:monospace"><span><br></span></span></div><div><span style="font-family:monospace"><span>"</span></span></div><div><span style="font-family:monospace"><span>marto@dragonfly% ps x | grep hellow
</span><br>166493 ?? I6s 0:00.00 mpich-3.0.4/examples/hellow
<br>166494 ?? I6s 0:00.00 mpich-3.0.4/examples/hellow<br></span></div><div><span style="font-family:monospace">"</span></div><div><span style="font-family:monospace"><span><br></span></span></div></div></div></blockquote><div><br></div><div>Ok, good, that seemed like it was the obvious problem. Can you show all the processes involved in the launch, not just the application (i.e. also the mpirun and anything that is spawned)? This might provide a further clue about what is happening underneath.</div><br></div><div>Tony</div><div><br></div></div></blockquote></div>