<html><head></head><body dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Kurt:<div><br><blockquote type="cite">Am 07.12.2019 um 16:07 schrieb Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov">kurt.e.mccall@nasa.gov</a>>:<br><br>Thanks Reuti. I assume that by the "task manager interface without ssh", you mean<br><br>$ mpiexec -launcher rsh ...<br></blockquote><br>No.<br><br>If the default compilation doesn't include SLURM support in your case, one has to compile MPICH with:<br><br>$ ./configure --with-slurm=[PATH] …<br><br>It should look like:<br><br>$ ps -e f<div>…<br> 16594 ? Sl 0:00 slurmstepd: [166806.batch]<br> 16599 ? S 0:00 \_ /bin/bash /var/spool/slurm/d/job166806/slurm_script<br> 16755 ? S 0:00 \_ mpiexec ./mpihello<br> 16757 ? Ssl 0:00 \_ /bin/srun -N 2 -n 2 --input none /home/reuti/local/mpich-3.3.2/bin/hydra_pmi_pro<br> 16758 ? S 0:00 \_ /bin/srun -N 2 -n 2 --input none /home/reuti/local/mpich-3.3.2/bin/hydra_pmi<br> 16766 ? Sl 0:00 slurmstepd: [166806.0]<br> 16772 ? S 0:00 \_ /home/reuti/local/mpich-3.3.2/bin/hydra_pmi_proxy --control-port node045:3<br> 16773 ? Rs 0:09 \_ ./mpihello<br> 16774 ? Rs 0:09 \_ ./mpihello</div><div>…</div><div>(and on the slave nodes only the second daemon is present)</div><div><br>No SSH, no RSH. Hence it's a tight integration into the queuing system.</div><div><br></div><div><a href="https://slurm.schedmd.com/mpi_guide.html">https://slurm.schedmd.com/mpi_guide.html</a><br><br>In addition, to change to `srun` as startup one might need (not used in the example above):<br><br><a href="https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Note_that_the_default_build_of_MPICH_will_work_fine_in_SLURM_environments._No_extra_steps_are_needed">https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Note_that_the_default_build_of_MPICH_will_work_fine_in_SLURM_environments._No_extra_steps_are_needed</a>.<br><br>It should work with a cluster which has no `ssh` to the nodes implemented at all (I allow `ssh` to the nodes only for admin staff).</div><div><br></div><div>===</div><div><br></div><div>Are the nodes on a private network, i.e. the nodes can't be reached from the Internet? Then one might also discuss pro and cons of allowing `rsh` or not. If even inside a private cluster all communication between the nodes has to be encrypted, I fear neither MPICH nor any other MPI implementation provides this.</div><div><br></div><div>===</div><div><br></div><div>AFAIR "fork" was used in times before Hydra as an alternative to start the MPI tasks local on a single machine only. I didn't check for "fork" or "smpd" for some time since Hydra appeared.</div><div><br>-- Reuti<br><br></div><div><br><br><blockquote type="cite">I can't use rsh on our cluster due to security concerns. Another launcher option is "fork", but when I tried it, the whole job froze. Does "fork" refer to a specific binary like ssh, or does it refer to the Linux system call?<br><br>Thanks,<br>Kurt<br><br><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" class="">Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command.<br></blockquote></blockquote><br><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Then it should be possible to use the task manager interface without `ssh`:<br></blockquote><br>-----Original Message-----<br>From: Reuti via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>> <br>Sent: Tuesday, December 3, 2019 9:02 AM<br>To: Mccall, Kurt E. (MSFC-EV41) via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>Cc: Reuti <<a href="mailto:reuti@staff.uni-marburg.de">reuti@staff.uni-marburg.de</a>><br>Subject: Re: [mpich-discuss] [EXTERNAL] Re: too many ssh connections warning<br><br>Hi:<br><br><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Am 03.12.2019 um 15:45 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>>:<br><br>Reuti,<br><br>Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command.<br></blockquote><br>Then it should be possible to use the task manager interface without `ssh`:<br><br><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.adaptivecomputing.com_torque_4-2D2-2D7_Content_topics_7-2DmessagePassing_MPICH.htm&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=dpTb4yi8w2BnIrMIlCu68U0xSr-qw1uaPJR1KRkVKFw&s=ylKo9O73T5IQufhrd-w_1etlEkRdNKcqrSglXtEam84&e=">https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.adaptivecomputing.com_torque_4-2D2-2D7_Content_topics_7-2DmessagePassing_MPICH.htm&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=dpTb4yi8w2BnIrMIlCu68U0xSr-qw1uaPJR1KRkVKFw&s=ylKo9O73T5IQufhrd-w_1etlEkRdNKcqrSglXtEam84&e=</a> <br><br><br><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""> I'll check with our sysadmins to see if there are firewall issues.<br></blockquote><br>This could also be later an issue if MPICH will connect to other machines directly to talk to the already started daemons.<br><br><br><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""> What is PAM?<br></blockquote><br><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Linux-5FPAM&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=dpTb4yi8w2BnIrMIlCu68U0xSr-qw1uaPJR1KRkVKFw&s=df3HqZTL7frMCuiq-Xw17TmjFNFOVK5RK_cMvVGRuXM&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Linux-5FPAM&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=dpTb4yi8w2BnIrMIlCu68U0xSr-qw1uaPJR1KRkVKFw&s=df3HqZTL7frMCuiq-Xw17TmjFNFOVK5RK_cMvVGRuXM&e=</a> <br><br>Several limits can be set here, depending on your distribution:<br><br>ls /lib64/security/<br><br>will show the available ones which are installed by default and are then used/configured in /etc/pam.d<br><br>-- Reuti<br><br><br><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Hui Zhou,<br><br>What do you expect would be making multiple SSH connections to the node? The creation of inter-communicators? Individual MPI_Iprobe/MPI_Isend/MPI_IRecv commands? If you have a guess, that would help me know how to fix the problem.<br><br>Kurt<br><br><br>-----Original Message-----<br>From: Reuti <<a href="mailto:reuti@staff.uni-marburg.de">reuti@staff.uni-marburg.de</a>> <br>Sent: Monday, December 2, 2019 3:20 PM<br>To: <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>Cc: Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov">kurt.e.mccall@nasa.gov</a>><br>Subject: [EXTERNAL] Re: [mpich-discuss] too many ssh connections warning<br><br><br><blockquote type="cite" class="">Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>>:<br><br>My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning<br><br>[mpiexec@n010.cluster.com] WARNING: too many ssh connections to <a href="http://n009.cluster.com">n009.cluster.com</a>; waiting 6 seconds<br><br>What is the cause of this? I have several guesses:<br><br>1) MPICH has an internal limit on the number of connections<br>2) I’m bumping up against a Linux limit on the number of connections<br>3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely)<br></blockquote><br>4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe.<br><br>Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system?<br><br>-- Reuti<br><br><br><blockquote type="cite" class="">The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped?<br><br>I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0.<br><br><br>_______________________________________________<br>discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>To manage subscription options or unsubscribe:<br><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailman_listinfo_discuss&d=DwIFaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=97JqnCQfN2Iy11xYlubB_AugrnlkH8C8vw4uQg6cJho&s=XXNw4ApjKsaCVdFY88_0_gD-tbjnIn4-0nxojl5hj6Y&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailman_listinfo_discuss&d=DwIFaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=97JqnCQfN2Iy11xYlubB_AugrnlkH8C8vw4uQg6cJho&s=XXNw4ApjKsaCVdFY88_0_gD-tbjnIn4-0nxojl5hj6Y&e=</a> <br></blockquote><br>_______________________________________________<br>discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>To manage subscription options or unsubscribe:<br><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailman_listinfo_discuss&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=dpTb4yi8w2BnIrMIlCu68U0xSr-qw1uaPJR1KRkVKFw&s=GhqzOOFuQP9ajlhwERMBbejcjkCy7zrnpLMbvEQk1wE&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailman_listinfo_discuss&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=dpTb4yi8w2BnIrMIlCu68U0xSr-qw1uaPJR1KRkVKFw&s=GhqzOOFuQP9ajlhwERMBbejcjkCy7zrnpLMbvEQk1wE&e=</a> <br></blockquote><br>_______________________________________________<br>discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>To manage subscription options or unsubscribe:<br><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailman_listinfo_discuss&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=dpTb4yi8w2BnIrMIlCu68U0xSr-qw1uaPJR1KRkVKFw&s=GhqzOOFuQP9ajlhwERMBbejcjkCy7zrnpLMbvEQk1wE&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailman_listinfo_discuss&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=dpTb4yi8w2BnIrMIlCu68U0xSr-qw1uaPJR1KRkVKFw&s=GhqzOOFuQP9ajlhwERMBbejcjkCy7zrnpLMbvEQk1wE&e=</a><br></blockquote><br></div></div></body></html>