<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Please find below : Interestingly , I don’t see verbs provider in the list. root@ dpr740: /mpich/examples# mpirun -n 2 -hosts 10. 118. 91. 158,10. 118. 91. 159 ./a. out ==== Various sizes and limits ==== sizeof(MPIDI_per_vci_t): 192 Required minimum
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanners3lr9tc" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanners3lr9tc" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanners3lr9tc" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanners3lr9tc" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanners3lr9tc { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtons3lr9tc:hover, .pfptPrimaryButtons3lr9tc:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtons3lr9tc:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
p.xmsonormal0, li.xmsonormal0, div.xmsonormal0
{mso-style-name:x_msonormal0;
margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
p.xxxxmsonormal, li.xxxxmsonormal, div.xxxxmsonormal
{mso-style-name:x_xxxmsonormal;
margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;}
p.xxxxmsonormal0, li.xxxxmsonormal0, div.xxxxmsonormal0
{mso-style-name:x_xxxmsonormal0;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
p.xxxxmsonormal00, li.xxxxmsonormal00, div.xxxxmsonormal00
{mso-style-name:x_xxxmsonormal00;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
p.xxxxp1, li.xxxxp1, div.xxxxp1
{mso-style-name:x_xxxp1;
margin:0in;
font-size:13.5pt;
font-family:Menlo;
color:black;}
p.xxxxp2, li.xxxxp2, div.xxxxp2
{mso-style-name:x_xxxp2;
margin:0in;
font-size:13.5pt;
font-family:Menlo;
color:black;}
p.xxxxmsochpdefault, li.xxxxmsochpdefault, div.xxxxmsochpdefault
{mso-style-name:x_xxxmsochpdefault;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;}
p.xp1, li.xp1, div.xp1
{mso-style-name:x_p1;
margin:0in;
font-size:12.0pt;
font-family:Menlo;
color:black;}
p.xp2, li.xp2, div.xp2
{mso-style-name:x_p2;
margin:0in;
font-size:12.0pt;
font-family:Menlo;
color:black;}
p.xmsochpdefault, li.xmsochpdefault, div.xmsochpdefault
{mso-style-name:x_msochpdefault;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:10.0pt;
font-family:"Aptos",sans-serif;}
span.xmsohyperlink
{mso-style-name:x_msohyperlink;
color:#467886;
text-decoration:underline;}
span.xmsohyperlinkfollowed
{mso-style-name:x_msohyperlinkfollowed;
color:purple;
text-decoration:underline;}
span.xxxxmsohyperlink
{mso-style-name:x_xxxmsohyperlink;
color:#467886;
text-decoration:underline;}
span.xxxxmsohyperlinkfollowed
{mso-style-name:x_xxxmsohyperlinkfollowed;
color:purple;
text-decoration:underline;}
span.xxxxemailstyle17
{mso-style-name:x_xxxemailstyle17;
font-family:"Aptos",sans-serif;
color:windowtext;}
span.xxxxemailstyle24
{mso-style-name:x_xxxemailstyle24;
font-family:"Aptos",sans-serif;
color:windowtext;}
span.xxxxemailstyle26
{mso-style-name:x_xxxemailstyle26;
font-family:"Aptos",sans-serif;
color:windowtext;}
span.xemailstyle32
{mso-style-name:x_emailstyle32;
font-family:"Aptos",sans-serif;
color:windowtext;}
span.xs1
{mso-style-name:x_s1;}
span.xapple-converted-space
{mso-style-name:x_apple-converted-space;}
span.xxxxs1
{mso-style-name:x_xxxs1;}
span.xxxxapple-converted-space
{mso-style-name:x_xxxapple-converted-space;}
span.EmailStyle42
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:12.0pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Please find below : <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Interestingly , I don’t see verbs provider in the list.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="p1"><span class="s1">root@dpr740:/mpich/examples# mpirun</span><span class="apple-converted-space">
</span><span class="s1">-n 2 -hosts 10.118.91.158,10.118.91.159 ./a.out</span><o:p></o:p></p>
<p class="p1"><span class="s1">==== Various sizes and limits ====</span><o:p></o:p></p>
<p class="p1"><span class="s1">sizeof(MPIDI_per_vci_t): 192</span><o:p></o:p></p>
<p class="p1"><span class="s1">Required minimum FI_VERSION: 0, current version: 10015</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: shm, score = 4, pref = -2, FI_ADDR_STR [13] - fi_shm://694</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: shm, score = 4, pref = -2, FI_ADDR_STR [13] - fi_shm://694</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::42a6:b7ff:fe28:c008</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::f66b:8cff:fe55:657c</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sm2, score = 3, pref = 0, FI_ADDR_STR [12] - fi_sm2://694</span><o:p></o:p></p>
<p class="p1"><span class="s1">provider: sm2, score = 3, pref = 0, FI_ADDR_STR [12] - fi_sm2://694</span><o:p></o:p></p>
<p class="p1"><span class="s1">Required minimum FI_VERSION: 10005, current version: 10015</span><o:p></o:p></p>
<p class="p1"><span class="s1">==== Capability set configuration ====</span><o:p></o:p></p>
<p class="p1"><span class="s1">libfabric provider: sockets - 192.168.1.0/24</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_DATA: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_AV_TABLE: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_MR_ALLOCATED: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_MR_PROV_KEY: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_TAGGED: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_AM: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_RMA: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_ATOMICS: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_TRIGGERED: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_ENABLE_HMEM: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_NUM_AM_BUFFERS: 8</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_CONTEXT_BITS: 20</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_SOURCE_BITS: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_TAG_BITS: 31</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_VNI_USE_DOMAIN: 1</span><o:p></o:p></p>
<p class="p1"><span class="s1">MAXIMUM SUPPORTED RANKS: 4294967296</span><o:p></o:p></p>
<p class="p1"><span class="s1">MAXIMUM TAG: 2147483648</span><o:p></o:p></p>
<p class="p1"><span class="s1">==== Provider global thresholds ====</span><o:p></o:p></p>
<p class="p1"><span class="s1">max_buffered_send: 255</span><o:p></o:p></p>
<p class="p1"><span class="s1">max_buffered_write: 255</span><o:p></o:p></p>
<p class="p1"><span class="s1">max_msg_size: 9223372036854775807</span><o:p></o:p></p>
<p class="p1"><span class="s1">max_order_raw: -1</span><o:p></o:p></p>
<p class="p1"><span class="s1">max_order_war: -1</span><o:p></o:p></p>
<p class="p1"><span class="s1">max_order_waw: -1</span><o:p></o:p></p>
<p class="p1"><span class="s1">tx_iov_limit: 8</span><o:p></o:p></p>
<p class="p1"><span class="s1">rx_iov_limit: 8</span><o:p></o:p></p>
<p class="p1"><span class="s1">rma_iov_limit: 8</span><o:p></o:p></p>
<p class="p1"><span class="s1">max_mr_key_size: 8</span><o:p></o:p></p>
<p class="p1"><span class="s1">==== Various sizes and limits ====</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_AM_MSG_HEADER_SIZE: 24</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_MAX_AM_HDR_SIZE: 255</span><o:p></o:p></p>
<p class="p1"><span class="s1">sizeof(MPIDI_OFI_am_request_header_t): 416</span><o:p></o:p></p>
<p class="p1"><span class="s1">sizeof(MPIDI_OFI_per_vci_t): 52480</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024</span><o:p></o:p></p>
<p class="p1"><span class="s1">MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384</span><o:p></o:p></p>
<p class="p1"><span class="s1">Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x59a0fc) [0xffffa661a0fc]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x4a6b58) [0xffffa6526b58]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x564740) [0xffffa65e4740]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x546c14) [0xffffa65c6c14]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x4f70cc) [0xffffa65770cc]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x4f9850) [0xffffa6579850]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x3ffd2c) [0xffffa647fd2c]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x4017ec) [0xffffa64817ec]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x3fe384) [0xffffa647e384]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x436a64) [0xffffa64b6a64]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(+0x43700c) [0xffffa64b700c]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/opt/mpich/lib/libmpi.so.0(MPI_Init+0x44) [0xffffa61aeeb4]</span><o:p></o:p></p>
<p class="p1"><span class="s1">./a.out(+0x9c4) [0xaaaac2a209c4]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffffa5ef73fc]</span><o:p></o:p></p>
<p class="p1"><span class="s1">/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffffa5ef74cc]</span><o:p></o:p></p>
<p class="p1"><span class="s1">./a.out(+0x8b0) [0xaaaac2a208b0]</span><o:p></o:p></p>
<p class="p1"><span class="s1">Abort(1) on node 0: Internal error</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">Zhou, Hui <zhouh@anl.gov><br>
<b>Date: </b>Monday, June 17, 2024 at 11:08</span><span style="font-family:"Arial",sans-serif;color:black"> </span><span style="color:black">AM<br>
<b>To: </b>Niyaz Murshed <Niyaz.Murshed@arm.com>, discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>nd <nd@arm.com><br>
<b>Subject: </b>Re: mpi hello-world error<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Could you set env "MPIR_CVAR_DEBUG_SUMMARY=1` and rerun the test?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Hui<o:p></o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Niyaz Murshed <Niyaz.Murshed@arm.com><br>
<b>Sent:</b> Monday, June 17, 2024 11:05 AM<br>
<b>To:</b> Zhou, Hui <zhouh@anl.gov>; discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> nd <nd@arm.com><br>
<b>Subject:</b> Re: mpi hello-world error</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Yes, one of the hosts. I have 2 servers. Hostname1: dpr740/10.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">118.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">91.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">159
Hostname2 : ampere-altra-2-1/10.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">118.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">91.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">158
I am running the application on dpr740 Adding both hosts: root@</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">dpr740:</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">/mpich/examples#
mpirun -n 2 -hosts 10.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">118.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">91.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">158,10.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">118.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">91.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">159
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<div style="border:none;border-top:solid #90A4AE 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#D0D8DC!important;border-top:#90a4ae!important" id="x_pfptBannerdxieyhc">
<div id="x_pfptBannerdxieyhc">
<div id="x_pfptBannerdxieyhc">
<p class="MsoNormal" style="line-height:13.5pt;background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</div>
<div id="x_pfptBannerdxieyhc">
<p class="MsoNormal" style="line-height:13.5pt;background:#D0D8DC"><span style="font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="background:#D0D8DC"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:11.0pt">Yes, one of the hosts.</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt">I have 2 servers. </span><o:p></o:p></p>
<p class="xp1"><span style="font-size:11.0pt;font-family:"Aptos",sans-serif">Hostname1:
</span><span class="xs1">dpr740/10.118.91.159</span><o:p></o:p></p>
<p class="xp1">Hostname2 : <span style="font-size:9.0pt">ampere-altra-2-1/10.118.91.158</span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt">I am running the application on dpr740</span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt">Adding both hosts:</span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"><span class="xs1">root@dpr740:/mpich/examples# mpirun</span><span class="xapple-converted-space">
</span><span class="xs1">-n 2 -hosts 10.118.91.158,10.118.91.159 ./a.out</span><o:p></o:p></p>
<p class="xp1"> <o:p></o:p></p>
<p class="xp1"><span class="xs1">Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x59a0fc) [0xffffa063a0fc]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x4a6b58) [0xffffa0546b58]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x564740) [0xffffa0604740]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x546c14) [0xffffa05e6c14]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x4f70cc) [0xffffa05970cc]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x4f9850) [0xffffa0599850]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x3ffd2c) [0xffffa049fd2c]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x4017ec) [0xffffa04a17ec]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x3fe384) [0xffffa049e384]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x436a64) [0xffffa04d6a64]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(+0x43700c) [0xffffa04d700c]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/opt/mpich/lib/libmpi.so.0(MPI_Init+0x44) [0xffffa01ceeb4]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">./a.out(+0x9c4) [0xaaaacd3309c4]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffff9ff173fc]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffff9ff174cc]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">./a.out(+0x8b0) [0xaaaacd3308b0]</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">Abort(1) on node 0: Internal error</span><o:p></o:p></p>
<p class="xp2"> <o:p></o:p></p>
<p class="xp1"><span class="xs1">[mpiexec@dpr740] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor)</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">[mpiexec@dpr740] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">^C[mpiexec@dpr740] Sending Ctrl-C to processes as requested</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">[mpiexec@dpr740] Press Ctrl-C again to force abort</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">[mpiexec@dpr740] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor)</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">[mpiexec@dpr740] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">[mpiexec@dpr740] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">[mpiexec@dpr740] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">[mpiexec@dpr740] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">[mpiexec@dpr740] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">[mpiexec@dpr740] main (mpiexec/mpiexec.c:260): process manager error waiting for completion</span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt">If I just add the remote host, it will run successfully.</span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"><span class="xs1">root@dpr740:/mpich/examples# mpirun</span><span class="xapple-converted-space">
</span><span class="xs1">-n 2 -hosts 10.118.91.158 ./a.out</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">Hello world from process 0 of 2</span><o:p></o:p></p>
<p class="xp1"><span class="xs1">Hello world from process 1 of 2</span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"><span style="font-size:9.0pt"> </span><o:p></o:p></p>
<p class="xp1"> <o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div id="x_mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xmsonormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">Zhou, Hui <zhouh@anl.gov><br>
<b>Date: </b>Monday, June 17, 2024 at 10:33</span><span style="font-family:"Arial",sans-serif;color:black"> </span><span style="color:black">AM<br>
<b>To: </b>Niyaz Murshed <Niyaz.Murshed@arm.com>, discuss@mpich.org <discuss@mpich.org><br>
<b>Subject: </b>Re: mpi hello-world error</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black">Alright. Let's focus on the case of two fixed nodes running</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black"> mpirun -n 2 -hosts 10.118.91.158,10.118.91.159 ./a.out<br>
<br>
Is the error consistent every time?<br>
<br>
Are you running the command from one of the host? Out of curiosity, why the host names looks like from two different naming systems?</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black">--</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black">Hui</span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Niyaz Murshed <Niyaz.Murshed@arm.com><br>
<b>Sent:</b> Monday, June 17, 2024 10:23 AM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Zhou, Hui <zhouh@anl.gov><br>
<b>Subject:</b> Re: mpi hello-world error</span> <o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-size:1.0pt;color:white">Hi Hui, Apologies for this, I just assumed more logs would give more information. Yes, both servers are on the same network. In the first email, I can run the hello-world application from server1
to server2 and vice versa. Its only when I add </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #90A4AE 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#D0D8DC!important;border-top:#90a4ae!important" id="x_x_pfptBanners07bqrk">
<div id="x_x_pfptBanners07bqrk">
<div id="x_x_pfptBanners07bqrk">
<p class="xmsonormal" style="line-height:13.5pt;background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><o:p></o:p></p>
</div>
<div id="x_x_pfptBanners07bqrk">
<p class="xmsonormal" style="line-height:13.5pt;background:#D0D8DC"><span style="font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xmsonormal" style="background:#D0D8DC"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="xmsonormal">Hi Hui,<o:p></o:p></p>
</div>
<div>
<p class="xmsonormal">Apologies for this, I just assumed more logs would give more information.<o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal">Yes, both servers are on the same network. <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal">In the first email, I can run the hello-world application from server1 to server2 and vice versa.<o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal">Its only when I add both servers in the parameters, the error is seen.<o:p></o:p></p>
</div>
</div>
</div>
<div id="x_x_ms-outlook-mobile-signature">
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
<p class="xmsonormal">Get <a href="https://urldefense.us/v3/__https:/aka.ms/o0ukef__;!!G_uCfscf7eWS!aA6_K_xqXWVqnjCozuSNlnNItijBkb7EDA_6xOPPs1AVXK3mV0yGRzhJT1WJ1N0oqX4ZZThlLzZPSpU2nxU$">
Outlook for iOS</a><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Zhou, Hui via discuss <discuss@mpich.org><br>
<b>Sent:</b> Monday, June 17, 2024 9:41:50 AM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Zhou, Hui <zhouh@anl.gov><br>
<b>Subject:</b> Re: [mpich-discuss] mpi hello-world error</span> <o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal"><span style="color:black">Niyaz,</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black">I am quite lost on the errors you encountered. The three errors seem all over the place. Are the two hosts on the same local network?</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black">--</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black">Hui</span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Niyaz Murshed via discuss <discuss@mpich.org><br>
<b>Sent:</b> Monday, June 17, 2024 1:07 AM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Niyaz Murshed <Niyaz.Murshed@arm.com>; nd <nd@arm.com><br>
<b>Subject:</b> Re: [mpich-discuss] mpi hello-world error</span> <o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-size:1.0pt;color:white">What is the best way to understand this log ? [proxy:</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">1@</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">ampere-altra-2-1]
Sending upstream hdr.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">cmd = CMD_STDERR Abort(680650255) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">:
MPI_Init(argc=(nil), </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #90A4AE 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#D0D8DC!important;border-top:#90a4ae!important" id="x_x_x_x_pfptBannerjs70v1p">
<div id="x_x_x_x_pfptBannerjs70v1p">
<div id="x_x_x_x_pfptBannerjs70v1p">
<p class="xmsonormal" style="line-height:13.5pt;background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><o:p></o:p></p>
</div>
<div id="x_x_x_x_pfptBannerjs70v1p">
<p class="xmsonormal" style="line-height:13.5pt;background:#D0D8DC"><span style="font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xmsonormal" style="background:#D0D8DC"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal">What is the best way to understand this log ? <o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">[</span><span class="xmsohyperlink"><span style="font-size:9.0pt;font-family:Menlo">proxy:1@ampere-altra-2-1</span></span><span style="font-size:9.0pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_STDERR</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">Abort(680650255) on node 1: Fatal error in internal_Init: Other MPI error, error stack:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">internal_Init(70)................: MPI_Init(argc=(nil), argv=(nil)) failed</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPII_Init_thread(268)............: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIR_init_comm_world(34).........: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIR_Comm_commit(823)............: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPID_Comm_commit_post_hook(222)..: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIDI_world_post_init(665).......: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIDI_OFI_init_vcis(851).........: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">check_num_nics(900)..............: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIR_Allreduce_allcomm_auto(4726): </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIC_Sendrecv(301)...............: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPID_Isend(63)...................: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIDI_isend(35)..................: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">(unknown)(): Other MPI error</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">[</span><span class="xmsohyperlink"><span style="font-size:9.0pt;font-family:Menlo">proxy:1@ampere-altra-2-1</span></span><span style="font-size:9.0pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_EXIT_STATUS</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">[</span><span class="xmsohyperlink"><span style="font-size:9.0pt;font-family:Menlo">proxy:0@cesw-amp-gbt-2s-m12830-01</span></span><span style="font-size:9.0pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_STDERR</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">Abort(680650255) on node 0: Fatal error in internal_Init: Other MPI error, error stack:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">internal_Init(70)................: MPI_Init(argc=(nil), argv=(nil)) failed</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPII_Init_thread(268)............: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIR_init_comm_world(34).........: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIR_Comm_commit(823)............: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPID_Comm_commit_post_hook(222)..: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIDI_world_post_init(665).......: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIDI_OFI_init_vcis(851).........: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">check_num_nics(900)..............: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIR_Allreduce_allcomm_auto(4726): </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIC_Sendrecv(301)...............: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPID_Isend(63)...................: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">MPIDI_isend(35)..................: </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">(unknown)(): Other MPI error</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:9.0pt;font-family:Menlo;color:black">[</span><span class="xmsohyperlink"><span style="font-size:9.0pt;font-family:Menlo">proxy:0@cesw-amp-gbt-2s-m12830-01</span></span><span style="font-size:9.0pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_EXIT_STATUS</span><o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<div id="x_x_x_x_mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xxxxmsonormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">Niyaz Murshed via discuss <discuss@mpich.org><br>
<b>Date: </b>Saturday, June 15, 2024 at 10:53</span><span style="font-family:"Arial",sans-serif;color:black"> </span><span style="color:black">AM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>Niyaz Murshed <Niyaz.Murshed@arm.com>, nd <nd@arm.com><br>
<b>Subject: </b>Re: [mpich-discuss] mpi hello-world error</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-size:1.0pt;color:white">Also seeing this error sometimes. root@</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">dpr740:</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">/mpich/examples#
export FI_PROVIDER=tcp root@</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">dpr740:</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">/mpich/examples#
mpirun -verbose -n 2 -hosts 10.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">118.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">91.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">158,10.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">118.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">91.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">159
./a.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">out host: 10.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">118.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">91.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">158
host: 10.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">118.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">91.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">159
[mpiexec@</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">dpr740] Timeout
</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #90A4AE 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#D0D8DC!important;border-top:#90a4ae!important" id="x_x_x_x_pfptBannerfhd5z4i">
<div id="x_x_x_x_pfptBannerfhd5z4i">
<div id="x_x_x_x_pfptBannerfhd5z4i">
<p class="xxxxmsonormal" style="line-height:13.5pt;background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><o:p></o:p></p>
</div>
<div id="x_x_x_x_pfptBannerfhd5z4i">
<p class="xxxxmsonormal" style="line-height:13.5pt;background:#D0D8DC"><span style="font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xxxxmsonormal" style="background:#D0D8DC"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal">Also seeing this error sometimes.<o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">root@dpr740:/mpich/examples# export FI_PROVIDER=tcp</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">root@dpr740:/mpich/examples# mpirun -verbose -n 2 -hosts 10.118.91.158,10.118.91.159 ./a.out </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">host: 10.118.91.158</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">host: 10.118.91.159</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Timeout set to -1 (-1 means infinite)</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">==================================================================================================</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">mpiexec options:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">----------------</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> Base path: /opt/mpich/bin/</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> Launcher: (null)</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> Debug level: 1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> Enable X: -1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> Global environment:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> -------------------</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> HOSTNAME=dpr740</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> PWD=/mpich/examples</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> HOME=/root</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> FI_PROVIDER=tcp</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> LESSCLOSE=/usr/bin/lesspipe %s %s</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> TERM=xterm</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> LESSOPEN=| /usr/bin/lesspipe %s</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> SHLVL=1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> OLDPWD=/</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> _=/opt/mpich/bin/mpirun</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> Hydra internal environment:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> ---------------------------</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> GFORTRAN_UNBUFFERED_PRECONNECTED=y</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> Proxy information:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> *********************</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> [1] proxy: 10.118.91.158 (1 cores)</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> Exec list: ./a.out (1 processes); </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> [2] proxy: 10.118.91.159 (1 cores)</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> Exec list: ./a.out (1 processes); </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">==================================================================================================</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">Proxy launch args: /opt/mpich/bin/hydra_pmi_proxy --control-port 10.118.91.159:35625 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port
0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">Arguments being passed to proxy 0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">--version 4.3.0a1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 10.118.91.158 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup
1 --pmi-kvsname kvs_1151_0_1450155337_dpr740 --pmi-process-mapping (vector,(0,2,1)) --global-inherited-env 15 'PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig' 'HOSTNAME=dpr740' 'HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233' 'PWD=/mpich/examples'
'HOME=/root' 'FI_PROVIDER=tcp' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'SHLVL=1' 'LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin'
'OLDPWD=/' '_=/opt/mpich/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /mpich/examples --exec-args 1 ./a.out </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">Arguments being passed to proxy 1:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">--version 4.3.0a1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 10.118.91.159 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup
1 --pmi-kvsname kvs_1151_0_1450155337_dpr740 --pmi-process-mapping (vector,(0,2,1)) --global-inherited-env 15 'PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig' 'HOSTNAME=dpr740' 'HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233' 'PWD=/mpich/examples'
'HOME=/root' 'FI_PROVIDER=tcp' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'SHLVL=1' 'LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin'
'OLDPWD=/' '_=/opt/mpich/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /mpich/examples --exec-args 1 ./a.out </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Launch arguments: /usr/bin/ssh -x -p 2233 10.118.91.158 "/opt/mpich/bin/hydra_pmi_proxy" --control-port 10.118.91.159:35625 --debug --rmk user --launcher
ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Launch arguments: /opt/mpich/bin/hydra_pmi_proxy --control-port 10.118.91.159:35625 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize
-2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 1 </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_PID_LIST</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=init pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=response_to_init rc=0 pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_maxes</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=maxes rc=0 kvsname_max=256 keylen_max=64 vallen_max=1024</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_appnum</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=appnum rc=0 appnum=0</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_my_kvsname</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=my_kvsname rc=0 kvsname=kvs_1151_0_1450155337_dpr740</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_process_mapping</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_result rc=0 value=(vector,(0,2,1)) found=TRUE</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_hwloc_xmlfile</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_result rc=0 value=/tmp/hydra_hwloc_xmlfile_YPoAhr found=TRUE</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_result rc=1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
we don't understand the response get_result; forwarding downstream</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_PID_LIST</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=init pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=response_to_init rc=0 pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_maxes</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=maxes rc=0 kvsname_max=256 keylen_max=64 vallen_max=1024</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_appnum</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=appnum rc=0 appnum=0</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_my_kvsname</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=my_kvsname rc=0 kvsname=kvs_1151_0_1450155337_dpr740</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_process_mapping</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_result rc=0 value=(vector,(0,2,1)) found=TRUE</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_hwloc_xmlfile</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_result rc=0 value=/tmp/hydra_hwloc_xmlfile_68iqm3 found=TRUE</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_result rc=1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
we don't understand the response get_result; forwarding downstream</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=put kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-0 value=0200937DC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
cached command: -allgather-shm-1-0=0200937DC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=put_result rc=0</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=mput -allgather-shm-1-0=0200937DC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
flushing 1 put command(s) out</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
forwarding command upstream:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">cmd=mput -allgather-shm-1-0=0200937DC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=mput -allgather-shm-1-0=0200937DC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=put kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-1 value=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
cached command: -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=put_result rc=0</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
flushing 1 put command(s) out</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
forwarding command upstream:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">cmd=mput -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=mput -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=mput -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> </span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=keyval_cache -allgather-shm-1-0=0200937DC0A80101[8] -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=keyval_cache -allgather-shm-1-0=0200937DC0A80101[8] -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-0</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_result rc=0 value=0200937DC0A80101[8] found=TRUE</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_result rc=0 value=0A00B381[4]FE80[6]526B4BFFFEFC134208[3] found=TRUE</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_STDERR</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-0</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_result rc=0 value=0200937DC0A80101[8] found=TRUE</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-1</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black"> cmd=get_result rc=0 value=0A00B381[4]FE80[6]526B4BFFFEFC134208[3] found=TRUE</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_STDERR</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_STDERR</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x58c005) [0x7f967920c005]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x491858) [0x7f9679111858]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x55428c) [0x7f96791d428c]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x53402d) [0x7f96791b402d]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x4dc71f) [0x7f967915c71f]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x4df09a) [0x7f967915f09a]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x3deab6) [0x7f967905eab6]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x3e0732) [0x7f9679060732]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x3dd075) [0x7f967905d075]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x418215) [0x7f9679098215]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_STDERR</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x4188fa) [0x7f96790988fa]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(MPI_Init+0x34) [0x7f9678d57594]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">./a.out(+0x121a) [0x55b07f1cc21a]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f9678a7cd90]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f9678a7ce40]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">./a.out(+0x1125) [0x55b07f1cc125]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">Abort(1) on node 1: Internal error</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x59a0fc) [0xffff91d0a0fc]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x4a6b58) [0xffff91c16b58]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x564740) [0xffff91cd4740]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x546c14) [0xffff91cb6c14]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x4f70cc) [0xffff91c670cc]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x4f9850) [0xffff91c69850]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x3ffd2c) [0xffff91b6fd2c]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x4017ec) [0xffff91b717ec]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x3fe384) [0xffff91b6e384]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x436a64) [0xffff91ba6a64]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_STDERR</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_STDERR</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(+0x43700c) [0xffff91ba700c]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/opt/mpich/lib/libmpi.so.0(MPI_Init+0x44) [0xffff9189eeb4]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">./a.out(+0x9c4) [0xaaaab5c709c4]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffff915e73fc]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffff915e74cc]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">./a.out(+0x8b0) [0xaaaab5c708b0]</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">Abort(1) on node 0: Internal error</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:1@dpr740</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_EXIT_STATUS</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span style="font-size:13.5pt;font-family:Menlo;color:black">[</span><span class="xxxxmsohyperlink"><span style="font-size:13.5pt;font-family:Menlo">proxy:0@ampere-altra-2-1</span></span><span style="font-size:13.5pt;font-family:Menlo;color:black">]
Sending upstream hdr.cmd = CMD_EXIT_STATUS</span><o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<div id="x_x_x_x_mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xxxxmsonormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">Niyaz Murshed via discuss <discuss@mpich.org><br>
<b>Date: </b>Saturday, June 15, 2024 at 12:10</span><span style="font-family:"Arial",sans-serif;color:black"> </span><span style="color:black">AM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>Niyaz Murshed <Niyaz.Murshed@arm.com>, nd <nd@arm.com><br>
<b>Subject: </b>[mpich-discuss] mpi hello-world error</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-size:1.0pt;color:white">Hello, I am trying to run the example hellow.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">c between 2 servers.
I can run them individually and it works fine. 10.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">118.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">91.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">158
is the machine I am running on. 10.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">118.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">91.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">159
is the remote machine. root@</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">dpr740:</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">/mpich/examples#
mpirun -n 2 -hosts </span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #90A4AE 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#D0D8DC!important;border-top:#90a4ae!important" id="x_x_x_x_pfptBanneriin5kjq">
<div id="x_x_x_x_pfptBanneriin5kjq">
<div id="x_x_x_x_pfptBanneriin5kjq">
<p class="xxxxmsonormal" style="line-height:13.5pt;background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><o:p></o:p></p>
</div>
<div id="x_x_x_x_pfptBanneriin5kjq">
<p class="xxxxmsonormal" style="line-height:13.5pt;background:#D0D8DC"><span style="font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xxxxmsonormal" style="background:#D0D8DC"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal">Hello, <o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal">I am trying to run the example hellow.c between 2 servers.<o:p></o:p></p>
<p class="xxxxmsonormal">I can run them individually and it works fine.<o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal"><span class="xxxxs1">10.118.91.158</span><span class="xxxxapple-converted-space"> is the machine I am running on.</span><o:p></o:p></p>
<p class="xxxxmsonormal"><span class="xxxxs1">10.118.91.159 is the remote machine.</span><o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">root@dpr740:/mpich/examples# mpirun</span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">-n 2 -hosts 10.118.91.158</span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">./a.out</span><span class="xxxxapple-converted-space"> </span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">Hello world from process 0 of 2</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">Hello world from process 1 of 2</span><o:p></o:p></p>
<p class="xxxxp1"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">root@dpr740:/mpich/examples# mpirun</span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">-n 2 -hosts 10.118.91.159</span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">./a.out</span><span class="xxxxapple-converted-space"> </span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">Hello world from process 1 of 2</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">Hello world from process 0 of 2</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1"> </span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">However, when I try to run them on both, I get the below error.</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">realloc(): invalid pointer</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1"> </span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">Is this a known issue ? Any suggestions?</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1"> </span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1"> </span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">root@dpr740:/mpich/examples# mpirun -verbose</span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">-n 2 -hosts 10.118.91.159,10.118.91.158</span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">./a.out</span><span class="xxxxapple-converted-space"> </span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">host: 10.118.91.159</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">host: 10.118.91.158</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Timeout set to -1 (-1 means infinite)</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">==================================================================================================</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">mpiexec options:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">----------------</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">Base path: /opt/mpich/bin/</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">Launcher: (null)</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">Debug level: 1</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">Enable X: -1</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">Global environment:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">-------------------</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">HOSTNAME=dpr740</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">PWD=/mpich/examples</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">HOME=/root</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">LESSCLOSE=/usr/bin/lesspipe %s %s</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">TERM=xterm</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">LESSOPEN=| /usr/bin/lesspipe %s</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">SHLVL=1</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">_=/opt/mpich/bin/mpirun</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">OLDPWD=/</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">Hydra internal environment:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">---------------------------</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">GFORTRAN_UNBUFFERED_PRECONNECTED=y</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">Proxy information:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">*********************</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">[1] proxy: 10.118.91.159 (1 cores)</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">Exec list: ./a.out (1 processes);</span><span class="xxxxapple-converted-space"> </span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">[2] proxy: 10.118.91.158 (1 cores)</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">Exec list: ./a.out (1 processes);</span><span class="xxxxapple-converted-space"> </span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">==================================================================================================</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">Proxy launch args: /opt/mpich/bin/hydra_pmi_proxy --control-port 10.118.91.159:33909 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc
-2 --proxy-id</span><span class="xxxxapple-converted-space"> </span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">Arguments being passed to proxy 0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">--version 4.3.0a1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 10.118.91.159 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_844_0_801938186_dpr740
--pmi-process-mapping (vector,(0,2,1)) --global-inherited-env 14 'PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig' 'HOSTNAME=dpr740' 'HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233' 'PWD=/mpich/examples' 'HOME=/root' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'SHLVL=1' 'LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin'
'_=/opt/mpich/bin/mpirun' 'OLDPWD=/' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /mpich/examples --exec-args 1 ./a.out</span><span class="xxxxapple-converted-space"> </span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">Arguments being passed to proxy 1:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">--version 4.3.0a1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 10.118.91.158 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_844_0_801938186_dpr740
--pmi-process-mapping (vector,(0,2,1)) --global-inherited-env 14 'PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig' 'HOSTNAME=dpr740' 'HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233' 'PWD=/mpich/examples' 'HOME=/root' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'SHLVL=1' 'LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin'
'_=/opt/mpich/bin/mpirun' 'OLDPWD=/' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /mpich/examples --exec-args 1 ./a.out</span><span class="xxxxapple-converted-space"> </span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Launch arguments: /opt/mpich/bin/hydra_pmi_proxy --control-port 10.118.91.159:33909 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc
-2 --proxy-id 0</span><span class="xxxxapple-converted-space"> </span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Launch arguments: /usr/bin/ssh -x -p 2233 10.118.91.158 "/opt/mpich/bin/hydra_pmi_proxy" --control-port 10.118.91.159:33909 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize
-2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 1</span><span class="xxxxapple-converted-space"> </span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_PID_LIST</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=init pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=response_to_init rc=0 pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_maxes</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=maxes rc=0 kvsname_max=256 keylen_max=64 vallen_max=1024</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_appnum</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=appnum rc=0 appnum=0</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_my_kvsname</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=my_kvsname rc=0 kvsname=kvs_844_0_801938186_dpr740</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_process_mapping</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_result rc=0 value=(vector,(0,2,1)) found=TRUE</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_hwloc_xmlfile</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_result rc=0 value=/tmp/hydra_hwloc_xmlfile_CeNRJN found=TRUE</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_result rc=1</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] we don't understand the response get_result; forwarding downstream</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_PID_LIST</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=init pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=response_to_init rc=0 pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_maxes</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=maxes rc=0 kvsname_max=256 keylen_max=64 vallen_max=1024</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_appnum</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=appnum rc=0 appnum=0</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_my_kvsname</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=my_kvsname rc=0 kvsname=kvs_844_0_801938186_dpr740</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_process_mapping</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_result rc=0 value=(vector,(0,2,1)) found=TRUE</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_hwloc_xmlfile</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_result rc=0 value=/tmp/hydra_hwloc_xmlfile_xv8EIG found=TRUE</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_result rc=1</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] we don't understand the response get_result; forwarding downstream</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=put kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-0 value=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] [</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=put kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-1 value=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">cached command: -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=put_result rc=0</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] flushing 1 put command(s) out</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] forwarding command upstream:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">cmd=mput -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=mput -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] cached command: -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=mput -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=put_result rc=0</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=mput -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] flushing 1 put command(s) out</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] forwarding command upstream:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=keyval_cache -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=keyval_cache -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">cmd=mput -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=mput -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_in</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-0</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_result rc=0 value=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] found=TRUE</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-1</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_result rc=0 value=0200A8BFC0A80101[8] found=TRUE</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=barrier_out</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-0</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_result rc=0 value=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] found=TRUE</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-1</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending PMI command:</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxapple-converted-space"> </span><span class="xxxxs1">cmd=get_result rc=0 value=0200A8BFC0A80101[8] found=TRUE</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">realloc(): invalid pointer</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_STDERR</span><o:p></o:p></p>
<p class="xxxxp2"> <o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">===================================================================================</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">= </span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">= </span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">PID 2404 RUNNING AT 10.118.91.158</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">= </span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">EXIT CODE: 134</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">= </span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">CLEANING UP REMAINING PROCESSES</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">= </span><span class="xxxxapple-converted-space">
</span><span class="xxxxs1">YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">===================================================================================</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:1@ampere-altra-2-1</span><span class="xxxxs1">] Sending upstream hdr.cmd = CMD_EXIT_STATUS</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:484): assert (!closed) failed</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[</span><span class="xxxxmsohyperlink">proxy:0@dpr740</span><span class="xxxxs1">] main (proxy/pmip.c:122): demux engine error waiting for event</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion</span><o:p></o:p></p>
<p class="xxxxp1"><span class="xxxxs1">[mpiexec@dpr740] main (mpiexec/mpiexec.c:260): process manager error waiting for completion</span><o:p></o:p></p>
<p class="xxxxp1"> <o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal">Regards,<o:p></o:p></p>
<p class="xxxxmsonormal">Niyaz<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="xmsonormal">IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person,
use it for any purpose, or store or copy the information in any medium. Thank you.
<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>