<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Kurt,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This looks to be an issue allocating resources on infiniband device on the node. MPI_Init should not require any special system settings. Are you able to run infiniband diagnostics without any MPI library? Ibstatus should tell you if the
IB card is online and what state it is in. From there, you could try running an ib_send_bw test across 2 nodes and verify that traffic is flowing.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ken<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">"Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss@mpich.org><br>
<b>Reply-To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Date: </b>Wednesday, October 18, 2023 at 12:36 PM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>"Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall@nasa.gov><br>
<b>Subject: </b>[mpich-discuss] Configuring MPICH 4.1.2 without increasing the locked memory limit<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">Hi,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">I have configured MPICH 4.1.2 with both –with-device=ch4:ofi and –with-device=ch4:ucx. My application fails in both cases when it can’t allocate enough memory. For –with-device=ch4:ofi :
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><b>Unable to create send CQ of size 5080 on mlx5_0: Cannot allocate memory<o:p></o:p></b></p>
<p class="MsoNormal" style="margin-left:.5in">n001.cluster.pssclabs.com:rank0.NeedlesMpiMM: Unable to initialize verbs NIC /sys/class/infiniband/mlx5_0 (unit 0:0)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">n001.cluster.pssclabs.com:rank0: PSM3 can't open nic unit: 0 (err=23)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Abort(606197135): Fatal error in internal_Init: Other MPI error, error stack:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">internal_Init(66)........: MPI_Init(argc=0x7ffc1cbd334c, argv=0x7ffc1cbd3340) failed<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">MPII_Init_thread(234)....:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">MPID_Init(513)...........:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">MPIDI_OFI_init_local(604):<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Configuring using –with-device=ch4:ucx, there was a very similar error involving /sys/class/infiniband/mlx5_0 that explicitly stating that the locked memory limit (ulimit -l) needs to be set to “unlimited”.
Are there any other ch4 device configuration options that don’t require unlimited locked memory?<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Thanks,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Kurt<o:p></o:p></p>
</div>
</body>
</html>