<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
We have seen this issue with Intel MPI as well and the solution was to set the following two variables.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<font face="monospace"><span class="x_uioutputtext"><span style="font-size: 10pt; color: rgb(8, 7, 7);"><span class="x_uioutputtext"><span style="color: rgb(8, 7, 7);"><span class="x_uioutputtext"><span><font face="monospace">export
</font></span></span><font face="monospace"><span class="x_uioutputtext"><span style="color: rgb(8, 7, 7);"><span class="x_uioutputtext"><span style="color: rgb(8, 7, 7);">MPIR_CVAR_CH4_OFI_RANK_BITS=17<br>
export MPIR_CVAR_CH4_OFI_TAG_BITS=$((41 - MPIR_CVAR_CH4_OFI_RANK_BITS))</span></span></span></span></font></span></span></span></span></font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
The total length is 41 and can be adjusted for desired ranks and tags count.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Amit Ruhela<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Zhou, Hui via discuss <discuss@mpich.org><br>
<b>Sent:</b> Thursday, April 8, 2021 10:18 AM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Zhou, Hui <zhouh@anl.gov><br>
<b>Subject:</b> Re: [mpich-discuss] Supporting > 64K ranks in CH4/UCX netmod</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle18
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal">Hi Min, </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I think we can do something about it. We’ll follow-up when we have updates.</p>
<p class="x_MsoNormal"> </p>
<div>
<div>
<div>
<p class="x_MsoNormal">-- <br>
Hui Zhou</p>
</div>
</div>
</div>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin-right:0in; margin-bottom:12.0pt; margin-left:.5in">
<b><span style="font-size:12.0pt; color:black">From: </span></b><span style="font-size:12.0pt; color:black">M Xie via discuss <discuss@mpich.org><br>
<b>Date: </b>Thursday, April 8, 2021 at 12:30 AM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>M Xie <xmxmxie@gmail.com><br>
<b>Subject: </b>[mpich-discuss] Supporting > 64K ranks in CH4/UCX netmod</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in">Hi,</p>
<div>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in">I am using MPICH on CH4/UCX netmod, the version is mpich-3.4.1.</p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in">I noticed that there is a configure parameter "--with-ch4-rank-bits" which can set the value of CH4_RANK_BITS, but seems CH4_RANK_BITS is not used in the code.</p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in">And I also find in the netmod/ucx/ucx_impl.h, _UCX_init_tag()/_UCX_recv_tag() use only 16 bits to set MPI rank in the ucp_tag, but this cannot differentiate correct ucp_tag when MPI ranks exceed 64K.</p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in">In Open MPI, 20 bits is used in pml/ucx module to set rank in ucp_tag, 20 bits for context, 24 bits for MPI tag, thus the maximum ranks in Open MPI can be 1M.</p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in">Is there any plan to support > 64K ranks in MPICH/CH4/UCX?</p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in">Thanks.</p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:.5in">Min</p>
</div>
</div>
</div>
</div>
</body>
</html>