<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
MPICH currently launches individual proxy for each spawn. I think that probably attributes to the flood of fds on the server. For now, I guess the solution is to ask system admin to increase the fd limit. Feel free to open a issue at
<a href="https://github.com/pmodels/mpich/issues" id="LPlnkOWALinkPreview">https://github.com/pmodels/mpich/issues</a>. We'll prioritize to get it enhanced.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-- <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui<br>
</div>
<div class="_Entity _EType_OWALinkPreview _EId_OWALinkPreview _EReadonly_1">
<div id="LPBorder_GTaHR0cHM6Ly9naXRodWIuY29tL3Btb2RlbHMvbXBpY2gvaXNzdWVz" class="LPBorder331021" style="width: 100%; margin-top: 16px; margin-bottom: 16px; position: relative; max-width: 800px; min-width: 424px;">
<table id="LPContainer331021" role="presentation" style="padding: 12px 36px 12px 12px; width: 100%; border-width: 1px; border-style: solid; border-color: rgb(200, 200, 200); border-radius: 2px;">
<tbody>
<tr style="border-spacing: 0px;" valign="top">
<td>
<div id="LPImageContainer331021" style="position: relative; margin-right: 12px; height: 120px; overflow: hidden; width: 240px;">
<a target="_blank" id="LPImageAnchor331021" href="https://github.com/pmodels/mpich/issues"><img id="LPThumbnailImageId331021" alt="" style="display: block;" width="240" height="120" src="https://opengraph.githubassets.com/7cbf378518c55cfe4549cfe44dfb9b74ceb606f933d674d797658393c7979a52/pmodels/mpich"></a></div>
</td>
<td style="width: 100%;">
<div id="LPTitle331021" style="font-size: 21px; font-weight: 300; margin-right: 8px; font-family: "wf_segoe-ui_light", "Segoe UI Light", "Segoe WP Light", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12px;">
<a target="_blank" id="LPUrlAnchor331021" href="https://github.com/pmodels/mpich/issues" style="text-decoration: none; color: var(--themePrimary);">Issues · pmodels/mpich - GitHub</a></div>
<div id="LPDescription331021" style="font-size: 14px; max-height: 100px; color: rgb(102, 102, 102); font-family: "wf_segoe-ui_normal", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12px; margin-right: 8px; overflow: hidden;">
Official MPICH Repository. Contribute to pmodels/mpich development by creating an account on GitHub.</div>
<div id="LPMetadata331021" style="font-size: 14px; font-weight: 400; color: rgb(166, 166, 166); font-family: "wf_segoe-ui_normal", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif;">
github.com</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<br>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mccall, Kurt E. (MSFC-EV41) via discuss <discuss@mpich.org><br>
<b>Sent:</b> Tuesday, March 22, 2022 12:55 PM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov><br>
<b>Subject:</b> [mpich-discuss] MPICH -- too many open files</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal">My application, which spawns multiple subprocesses via MPI_Comm_spawn, eventually fails on one Slurm cluster as I scale up the number of processes, with the error:</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><b>[mpiexec@n002.cluster.pssclabs.com] HYDU_create_process (../../../../mpich-4.0.1/src/pm/hydra/utils/launch/launch.c:21): pipe error (<span style="color:red">Too many open files</span>)</b></p>
<p class="x_MsoNormal"><b>[mpiexec@n002.cluster.pssclabs.com] HYDT_bscd_common_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/external_common_launch.c:296): create process returned error</b></p>
<p class="x_MsoNormal"><b>free(): invalid pointer</b></p>
<p class="x_MsoNormal"><b>/var/spool/slurm/job235999/slurm_script: line 296: 3778907 Aborted (core dumped)</b></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">It works fine on a different (Torque) cluster for very large job sizes.
</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">“ulimit -n” (number of open files) on both machines returns 1024.
</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I’m hoping that there is some other system setting on the Slurm cluster that would allow larger jobs. I can provide the “-verbose” output file if that would help.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Thanks,</p>
<p class="x_MsoNormal">Kurt</p>
</div>
</div>
</body>
</html>