<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
The particular warning is issued by hydra, MPICH’s process manager. Following excerpt is the comment in that source code:
<div class=""><br class="">
</div>
<div class="">```</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Courier New"; color: rgb(44, 238, 235); background-color: rgb(0, 0, 0);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> /* ssh has many types of security controls that do not allow a</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Courier New"; color: rgb(44, 238, 235); background-color: rgb(0, 0, 0);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> * user to ssh to the same node multiple times very</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Courier New"; color: rgb(44, 238, 235); background-color: rgb(0, 0, 0);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> * quickly. If this happens, the ssh daemons disables ssh</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Courier New"; color: rgb(44, 238, 235); background-color: rgb(0, 0, 0);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> * connections causing the job to fail. This is basically a</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Courier New"; color: rgb(44, 238, 235); background-color: rgb(0, 0, 0);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> * hack to slow down ssh connections to the same node. We</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Courier New"; color: rgb(44, 238, 235); background-color: rgb(0, 0, 0);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> * check for offset == 0 before applying this hack, so we only</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Courier New"; color: rgb(44, 238, 235); background-color: rgb(0, 0, 0);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> * slow down the cases where ssh is being used, and not the</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Courier New"; color: rgb(44, 238, 235); background-color: rgb(0, 0, 0);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> * cases where we fall back to fork. */</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Courier New"; color: rgb(44, 238, 235); background-color: rgb(0, 0, 0);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">```</span></div>
<div class=""><br class="">
</div>
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
— </div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
Hui Zhou</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<br class="">
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
<br class="Apple-interchange-newline">
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 2, 2019, at 3:14 PM, Mccall, Kurt E. (MSFC-EV41) via discuss <<a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
[<a href="mailto:mpiexec@n010.cluster.com" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">mpiexec@n010.cluster.com</a>] WARNING: too many ssh connections to<span class="Apple-converted-space"> </span><a href="http://n009.cluster.com/" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">n009.cluster.com</a>;
waiting 6 seconds<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
What is the cause of this? I have several guesses:<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;" class="">
<span class="">1)<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span>MPICH
has an internal limit on the number of connections<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;" class="">
<span class="">2)<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span>I’m
bumping up against a Linux limit on the number of connections<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;" class="">
<span class="">3)<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span>Non-blocking
communication using MPI_Isend() creates a temporary ssh connection (not likely)<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped?<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0.<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">discuss
mailing list </span><a href="mailto:discuss@mpich.org" style="color: rgb(149, 79, 114); text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">discuss@mpich.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">To
manage subscription options or unsubscribe:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" style="color: rgb(149, 79, 114); text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>