<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The spawned process launches the same way the first MPI_COMM_WORLD is launched, i.e. in a round-robin fashion through the list of nodes.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-- <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov><br>
<b>Sent:</b> Friday, April 1, 2022 4:22 PM<br>
<b>To:</b> Zhou, Hui <zhouh@anl.gov>; discuss@mpich.org <discuss@mpich.org><br>
<b>Subject:</b> Re: Hydra WARNING: too many ssh connections</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:#0563C1;
text-decoration:underline}
p.x_xmsonormal, li.x_xmsonormal, div.x_xmsonormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
p.x_xmsolistparagraph, li.x_xmsolistparagraph, div.x_xmsolistparagraph
{margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle23
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
ol
{margin-bottom:0in}
ul
{margin-bottom:0in}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal">Thanks Hui, is the spawned process on the local host, or the remote host or both?</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Kurt</p>
<p class="x_MsoNormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b>From:</b> Zhou, Hui <zhouh@anl.gov> <br>
<b>Sent:</b> Friday, April 1, 2022 4:20 PM<br>
<b>To:</b> discuss@mpich.org<br>
<b>Cc:</b> Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov><br>
<b>Subject:</b> [EXTERNAL] Re: Hydra WARNING: too many ssh connections</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Every time you call MPI_Comm_spawn, hydra will launch a ssh (for each host) to create a proxy. It is certainly not ideal for applications relying on spawning many processes.</span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Mccall, Kurt E. (MSFC-EV41) via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Sent:</b> Friday, April 1, 2022 4:08 PM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>><br>
<b>Cc:</b> Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov">kurt.e.mccall@nasa.gov</a>><br>
<b>Subject:</b> [mpich-discuss] Hydra WARNING: too many ssh connections</span> </p>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xmsonormal">Hi, you provided the following information about the warning “too many ssh connections”:</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal"><span style="font-size:10.0pt; font-family:"Courier New"">The particular warning is issued by hydra, MPICH’s process manager. Following excerpt is the comment in that source code:</span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt; font-family:"Courier New""> </span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt; font-family:"Courier New""> /* ssh has many types of security controls that do not allow a</span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt; font-family:"Courier New""> * user to ssh to the same node multiple times very</span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt; font-family:"Courier New""> * quickly. If this happens, the ssh daemons disables ssh</span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt; font-family:"Courier New""> * connections causing the job to fail. This is basically a</span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt; font-family:"Courier New""> * hack to slow down ssh connections to the same node. We</span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt; font-family:"Courier New""> * check for offset == 0 before applying this hack, so we only</span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt; font-family:"Courier New""> * slow down the cases where ssh is being used, and not the</span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt; font-family:"Courier New""> * cases where we fall back to fork. */</span></p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Is this just during an initial ssh connection attempt? I’m trying to figure out where my code is triggering this warning. Could it be from</p>
<p class="x_xmsonormal"> </p>
<ol start="1" type="1" style="margin-top:0in">
<li class="x_xmsolistparagraph" style="margin-left:0in">MPI_Intercomm_create</li><li class="x_xmsolistparagraph" style="margin-left:0in">MPI_Comm_spawn</li><li class="x_xmsolistparagraph" style="margin-left:0in">others?</li></ol>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">I’m calling mpiexec with the “—launcher ssh” option, MPICH 4.0.1.</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Thanks,</p>
<p class="x_xmsonormal">Kurt</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal"> </p>
</div>
</div>
</div>
</div>
</body>
</html>