<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
The NIC symmetry test is specific to HPE Cray MPICH. As a workaround, try setting MPICH_OFI_NUM_NICS=1. — Michael Raymond HPE HPC programming environment From: Kevin Buckley via discuss <discuss@ mpich. org> Date: Tuesday, March 10, 2026</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanner51azx7d" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanner51azx7d" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanner51azx7d" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanner51azx7d" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html>
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanner51azx7d { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButton51azx7d:hover, .pfptPrimaryButton51azx7d:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButton51azx7d:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The NIC symmetry test is specific to HPE Cray MPICH.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
As a workaround, try setting MPICH_OFI_NUM_NICS=1.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="ms-outlook-mobile-signature">
<p class="MsoNormal" style="margin: 0in; font-family: Aptos, sans-serif; font-size: 12pt;">
—</p>
<p class="MsoNormal" style="margin: 0in; font-family: Aptos, sans-serif; font-size: 12pt;">
Michael Raymond</p>
<p class="MsoNormal" style="margin: 0in; font-family: Aptos, sans-serif; font-size: 12pt;">
HPE HPC programming environment</p>
</div>
<div id="mail-editor-reference-message-container">
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="text-align: left; padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>Kevin Buckley via discuss <discuss@mpich.org><br>
<b>Date: </b>Tuesday, March 10, 2026 at 00:47<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>Kevin Buckley <kevin.buckley.pawsey.org.au@gmail.com><br>
<b>Subject: </b>Re: [mpich-discuss] How does MPICH dtermine available NICs?<br>
<br>
</div>
<div dir="ltr" id="pfptBanneriupi7sl" style="visibility: visible !important; opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; text-align: left !important; margin: 16px 0px !important; padding: 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: rgb(208, 216, 220); border-top-width: 4px !important; border-top-style: solid !important; border-top-color: rgb(144, 164, 174) !important;">
<div id="pfptBanneriupi7sl" style="visibility: visible !important; opacity: 1 !important; background-color: rgb(208, 216, 220); max-height: none !important; float: left !important; display: block !important; margin: 0px 0px 1px !important; max-width: 600px !important;">
<div id="pfptBanneriupi7sl" style="opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; visibility: visible !important; background-color: rgb(208, 216, 220); color: rgb(0, 0, 0); font-family: Arial, sans-serif !important; font-weight: bold !important; font-size: 14px !important; line-height: 18px !important;">
This Message Is From an External Sender</div>
<div id="pfptBanneriupi7sl" style="font-weight: normal; opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; visibility: visible !important; background-color: rgb(208, 216, 220); color: rgb(0, 0, 0); font-family: Arial, sans-serif !important; font-size: 12px !important; line-height: 18px !important; margin-top: 2px !important;">
This message came from outside your organization.</div>
</div>
<div style="line-height: 0; height: 0px; display: block; font-size: 0.01px;"> </div>
</div>
<pre><div class="ms-outlook-mobile-reference-message skipProofing" style="white-space: pre-wrap; font-family: sans-serif;">On 2026/03/09 23:24, Zhou, Hui wrote:
> Could you try the upstream MPICH?
I followed you suggestion to try the upstream MPICH.
Trying to configure 3.4a2, the version "inside" the cray-mpich,
modulo any vendor updates made to that base release, saw
...
configure: error: The Fortran compiler gfortran will not compile files that\
call the same routine with arguments of different types.
$
$ which gfortran
/opt/cray/pe/gcc-native/14/bin/gfortran
$
Wasn't sure if that meant that the MPICH code is too old to
make correct compiler checks, or that the compiler is too
new to respond them - so I shelved that line of attack.
Was able to configure and build 5.0.0, and have used that to
compile and run the same noddy "connectivity_c.c" example code
that I had been seeing the issue with.
It all seems to work without any warnings (as does compiling/
running within an OpenMPI 5.0.3 environment), in that a 2-node
job spanning a node with just one NIC, and the dual-NIC node
with one NIC disabled, completes, as does a 2-node job spanning
the node with just one NIC, and a dual-NIC node with neither
NIC disabled.
Would you though, expect the vanilla MPICH 5.0.0 to have
warned me about the NIC_SYMMETRY inconsistencies?
Your comment in the original reply,
> The upstream MPICH should be fine with different number of
> NICs on different nodes. By default, the process picks a
> nic that is closest to process's CPU affinity; or if you
> have multiple processes on the same node, each process
> will try pick a different nic in a round-robin fashion.
> Usually a disabled nic won't be selected during init,
would suggest that "later" (than cray-mpich's 3.4a2 starting
point, plus whatver enhancements Cray and/or HPE have since
added) MPICH-s would not bother to give the warning, as those
versions would "just do the right thing" (tm)?
If so, think it's sounding as though the consideration of
the disabled-at-boot-time NIC as somehow being "available"
for MPI traffic, is tied into something that the vendor is
doing: somewhere within their overall MPI+Slingshot stack.
Thanks again for fielding this one,
Kevin M. Buckley
--
Supercomputing Systems Administrator
Pawsey Supercomputing Centre
PERTH
Australia
</div></pre>
</div>
</body>
</html>