<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Niyaz,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I am quite lost on the errors you encountered. The three errors seem all over the place. Are the two hosts on the same local network?</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Niyaz Murshed via discuss <discuss@mpich.org><br>
<b>Sent:</b> Monday, June 17, 2024 1:07 AM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Niyaz Murshed <Niyaz.Murshed@arm.com>; nd <nd@arm.com><br>
<b>Subject:</b> Re: [mpich-discuss] mpi hello-world error</font>
<div> </div>
</div>
<div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
What is the best way to understand this log ? [proxy: 1@ ampere-altra-2-1] Sending upstream hdr. cmd = CMD_STDERR Abort(680650255) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70). . . . . . . . . . . . . . . . : MPI_Init(argc=(nil),
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="x_pfptBannerjs70v1p" style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#D0D8DC!important; background-color:#D0D8DC; border-top:4px solid #90a4ae!important; border-top:4px solid #90a4ae">
<div id="x_pfptBannerjs70v1p" style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBannerjs70v1p" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBannerjs70v1p" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<style>
<!--
#x_pfptBannerjs70v1p
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#D0D8DC!important;
max-width:none!important;
max-height:none!important}
-->
</style>
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Aptos}
@font-face
{font-family:Menlo}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif}
a:link, span.x_MsoHyperlink
{color:#467886;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
p.x_msonormal0, li.x_msonormal0, div.x_msonormal0
{margin-right:0in;
margin-left:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif}
p.x_msonormal00, li.x_msonormal00, div.x_msonormal00
{margin-right:0in;
margin-left:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif}
p.x_p1, li.x_p1, div.x_p1
{margin:0in;
font-size:13.5pt;
font-family:Menlo;
color:black}
p.x_p2, li.x_p2, div.x_p2
{margin:0in;
font-size:13.5pt;
font-family:Menlo;
color:black}
p.x_msochpdefault, li.x_msochpdefault, div.x_msochpdefault
{margin-right:0in;
margin-left:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif}
span.x_emailstyle17
{font-family:"Aptos",sans-serif;
color:windowtext}
span.x_emailstyle24
{font-family:"Aptos",sans-serif;
color:windowtext}
span.x_s1
{}
span.x_apple-converted-space
{}
span.x_EmailStyle26
{font-family:"Aptos",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div class="x_WordSection1">
<p class="x_MsoNormal">What is the best way to understand this log ? </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">[<a href="">proxy:1@ampere-altra-2-1</a>] Sending upstream hdr.cmd = CMD_STDERR</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">Abort(680650255) on node 1: Fatal error in internal_Init: Other MPI error, error stack:</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">internal_Init(70)................: MPI_Init(argc=(nil), argv=(nil)) failed</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPII_Init_thread(268)............: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIR_init_comm_world(34).........: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIR_Comm_commit(823)............: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPID_Comm_commit_post_hook(222)..: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIDI_world_post_init(665).......: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIDI_OFI_init_vcis(851).........: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">check_num_nics(900)..............: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIR_Allreduce_allcomm_auto(4726): </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIC_Sendrecv(301)...............: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPID_Isend(63)...................: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIDI_isend(35)..................: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">(unknown)(): Other MPI error</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">[<a href="">proxy:1@ampere-altra-2-1</a>] Sending upstream hdr.cmd = CMD_EXIT_STATUS</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">[<a href="">proxy:0@cesw-amp-gbt-2s-m12830-01</a>] Sending upstream hdr.cmd = CMD_STDERR</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">Abort(680650255) on node 0: Fatal error in internal_Init: Other MPI error, error stack:</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">internal_Init(70)................: MPI_Init(argc=(nil), argv=(nil)) failed</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPII_Init_thread(268)............: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIR_init_comm_world(34).........: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIR_Comm_commit(823)............: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPID_Comm_commit_post_hook(222)..: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIDI_world_post_init(665).......: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIDI_OFI_init_vcis(851).........: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">check_num_nics(900)..............: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIR_Allreduce_allcomm_auto(4726): </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIC_Sendrecv(301)...............: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPID_Isend(63)...................: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">MPIDI_isend(35)..................: </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">(unknown)(): Other MPI error</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Menlo; color:black">[<a href="">proxy:0@cesw-amp-gbt-2s-m12830-01</a>] Sending upstream hdr.cmd = CMD_EXIT_STATUS</span></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<div id="x_mail-editor-reference-message-container">
<div>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">Niyaz Murshed via discuss <discuss@mpich.org><br>
<b>Date: </b>Saturday, June 15, 2024 at 10:53</span><span style="font-family:"Arial",sans-serif; color:black"> </span><span style="color:black">AM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>Niyaz Murshed <Niyaz.Murshed@arm.com>, nd <nd@arm.com><br>
<b>Subject: </b>Re: [mpich-discuss] mpi hello-world error</span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<p class="x_MsoNormal" style=""><span style="font-size:1.0pt; color:white">Also seeing this error sometimes. root@</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">dpr740:</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">/mpich/examples#
export FI_PROVIDER=tcp root@</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">dpr740:</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">/mpich/examples#
mpirun -verbose -n 2 -hosts 10.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">118.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">91.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">158,10.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">118.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">91.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">159
./a.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">out host: 10.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">118.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">91.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">158
host: 10.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">118.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">91.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">159
[mpiexec@</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">dpr740] Timeout
</span></p>
</div>
<div>
<p class="x_MsoNormal" style=""><span style="font-size:1.0pt; color:white">ZjQcmQRYFpfptBannerStart</span></p>
</div>
<div id="x_pfptBannerfhd5z4i" style="border:none; border-top:solid #90A4AE 3.0pt; padding:0in 0in 0in 0in; display:block!important; text-align:left!important; margin:0px!important; padding:16px!important; border-radius:4px!important; min-width:200px!important; background-color:#D0D8DC!important; border-top:#90a4ae!important">
<div id="x_pfptBannerfhd5z4i">
<div id="x_pfptBannerfhd5z4i">
<p class="x_MsoNormal" style="line-height:13.5pt; background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif; color:black">This Message Is From an External Sender
</span></b><b><span style="font-size:12.0pt; font-family:"Arial",sans-serif; color:black"></span></b></p>
</div>
<div id="x_pfptBannerfhd5z4i">
<p class="x_MsoNormal" style="line-height:13.5pt; background:#D0D8DC"><span style="font-family:"Arial",sans-serif; color:black">This message came from outside your organization.
</span></p>
</div>
</div>
<div>
<p class="x_MsoNormal" style="background:#D0D8DC"><span style="color:black"> </span></p>
</div>
</div>
<div>
<p class="x_MsoNormal" style=""><span style="font-size:1.0pt; color:white">ZjQcmQRYFpfptBannerEnd</span></p>
</div>
<div>
<p class="x_MsoNormal">Also seeing this error sometimes.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">root@dpr740:/mpich/examples# export FI_PROVIDER=tcp</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">root@dpr740:/mpich/examples# mpirun -verbose -n 2 -hosts 10.118.91.158,10.118.91.159 ./a.out </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">host: 10.118.91.158</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">host: 10.118.91.159</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Timeout set to -1 (-1 means infinite)</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">==================================================================================================</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">mpiexec options:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">----------------</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> Base path: /opt/mpich/bin/</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> Launcher: (null)</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> Debug level: 1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> Enable X: -1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> Global environment:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> -------------------</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> HOSTNAME=dpr740</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> PWD=/mpich/examples</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> HOME=/root</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> FI_PROVIDER=tcp</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> LESSCLOSE=/usr/bin/lesspipe %s %s</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> TERM=xterm</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> LESSOPEN=| /usr/bin/lesspipe %s</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> SHLVL=1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> OLDPWD=/</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> _=/opt/mpich/bin/mpirun</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> Hydra internal environment:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> ---------------------------</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> GFORTRAN_UNBUFFERED_PRECONNECTED=y</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> Proxy information:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> *********************</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> [1] proxy: 10.118.91.158 (1 cores)</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> Exec list: ./a.out (1 processes); </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> [2] proxy: 10.118.91.159 (1 cores)</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> Exec list: ./a.out (1 processes); </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">==================================================================================================</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">Proxy launch args: /opt/mpich/bin/hydra_pmi_proxy --control-port 10.118.91.159:35625 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port
0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">Arguments being passed to proxy 0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">--version 4.3.0a1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 10.118.91.158 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup
1 --pmi-kvsname kvs_1151_0_1450155337_dpr740 --pmi-process-mapping (vector,(0,2,1)) --global-inherited-env 15 'PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig' 'HOSTNAME=dpr740' 'HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233' 'PWD=/mpich/examples'
'HOME=/root' 'FI_PROVIDER=tcp' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'SHLVL=1' 'LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin'
'OLDPWD=/' '_=/opt/mpich/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /mpich/examples --exec-args 1 ./a.out </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">Arguments being passed to proxy 1:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">--version 4.3.0a1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 10.118.91.159 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup
1 --pmi-kvsname kvs_1151_0_1450155337_dpr740 --pmi-process-mapping (vector,(0,2,1)) --global-inherited-env 15 'PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig' 'HOSTNAME=dpr740' 'HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233' 'PWD=/mpich/examples'
'HOME=/root' 'FI_PROVIDER=tcp' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'SHLVL=1' 'LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin'
'OLDPWD=/' '_=/opt/mpich/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /mpich/examples --exec-args 1 ./a.out </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Launch arguments: /usr/bin/ssh -x -p 2233 10.118.91.158 "/opt/mpich/bin/hydra_pmi_proxy" --control-port 10.118.91.159:35625 --debug --rmk user --launcher
ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Launch arguments: /opt/mpich/bin/hydra_pmi_proxy --control-port 10.118.91.159:35625 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize
-2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 1 </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream hdr.cmd = CMD_PID_LIST</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=init pmi_version=1 pmi_subversion=1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=response_to_init rc=0 pmi_version=1 pmi_subversion=1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_maxes</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=maxes rc=0 kvsname_max=256 keylen_max=64 vallen_max=1024</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_appnum</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=appnum rc=0 appnum=0</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_my_kvsname</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=my_kvsname rc=0 kvsname=kvs_1151_0_1450155337_dpr740</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_process_mapping</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_result rc=0 value=(vector,(0,2,1)) found=TRUE</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_hwloc_xmlfile</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_result rc=0 value=/tmp/hydra_hwloc_xmlfile_YPoAhr found=TRUE</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream internal PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_result rc=1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] we don't understand the response get_result; forwarding downstream</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream internal PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream hdr.cmd = CMD_PID_LIST</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=init pmi_version=1 pmi_subversion=1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=response_to_init rc=0 pmi_version=1 pmi_subversion=1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_maxes</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=maxes rc=0 kvsname_max=256 keylen_max=64 vallen_max=1024</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_appnum</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=appnum rc=0 appnum=0</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_my_kvsname</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=my_kvsname rc=0 kvsname=kvs_1151_0_1450155337_dpr740</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_process_mapping</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_result rc=0 value=(vector,(0,2,1)) found=TRUE</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_hwloc_xmlfile</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_result rc=0 value=/tmp/hydra_hwloc_xmlfile_68iqm3 found=TRUE</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_result rc=1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream internal PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] we don't understand the response get_result; forwarding downstream</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_out</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_out</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream internal PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] [<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_out</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_out</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=put kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-0 value=0200937DC0A80101[8]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] cached command: -allgather-shm-1-0=0200937DC0A80101[8]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=put_result rc=0</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=mput -allgather-shm-1-0=0200937DC0A80101[8]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] flushing 1 put command(s) out</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] forwarding command upstream:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">cmd=mput -allgather-shm-1-0=0200937DC0A80101[8]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream internal PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=mput -allgather-shm-1-0=0200937DC0A80101[8]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream internal PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=put kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-1 value=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] cached command: -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=put_result rc=0</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] flushing 1 put command(s) out</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] forwarding command upstream:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">cmd=mput -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream internal PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=mput -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream internal PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=mput -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=keyval_cache -allgather-shm-1-0=0200937DC0A80101[8] -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=keyval_cache -allgather-shm-1-0=0200937DC0A80101[8] -allgather-shm-1-1=0A00B381[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_out</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_out</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_out</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-0</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_result rc=0 value=0200937DC0A80101[8] found=TRUE</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_result rc=0 value=0A00B381[4]FE80[6]526B4BFFFEFC134208[3] found=TRUE</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=barrier_out</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream hdr.cmd = CMD_STDERR</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-0</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_result rc=0 value=0200937DC0A80101[8] found=TRUE</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] got pmi command from downstream 0-0:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get kvsname=kvs_1151_0_1450155337_dpr740 key=-allgather-shm-1-1</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending PMI command:</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black"> cmd=get_result rc=0 value=0A00B381[4]FE80[6]526B4BFFFEFC134208[3] found=TRUE</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream hdr.cmd = CMD_STDERR</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream hdr.cmd = CMD_STDERR</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x58c005) [0x7f967920c005]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x491858) [0x7f9679111858]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x55428c) [0x7f96791d428c]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x53402d) [0x7f96791b402d]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x4dc71f) [0x7f967915c71f]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x4df09a) [0x7f967915f09a]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x3deab6) [0x7f967905eab6]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x3e0732) [0x7f9679060732]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x3dd075) [0x7f967905d075]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x418215) [0x7f9679098215]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream hdr.cmd = CMD_STDERR</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x4188fa) [0x7f96790988fa]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(MPI_Init+0x34) [0x7f9678d57594]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">./a.out(+0x121a) [0x55b07f1cc21a]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f9678a7cd90]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f9678a7ce40]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">./a.out(+0x1125) [0x55b07f1cc125]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">Abort(1) on node 1: Internal error</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x59a0fc) [0xffff91d0a0fc]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x4a6b58) [0xffff91c16b58]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x564740) [0xffff91cd4740]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x546c14) [0xffff91cb6c14]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x4f70cc) [0xffff91c670cc]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x4f9850) [0xffff91c69850]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x3ffd2c) [0xffff91b6fd2c]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x4017ec) [0xffff91b717ec]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x3fe384) [0xffff91b6e384]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x436a64) [0xffff91ba6a64]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream hdr.cmd = CMD_STDERR</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream hdr.cmd = CMD_STDERR</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(+0x43700c) [0xffff91ba700c]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/opt/mpich/lib/libmpi.so.0(MPI_Init+0x44) [0xffff9189eeb4]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">./a.out(+0x9c4) [0xaaaab5c709c4]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffff915e73fc]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffff915e74cc]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">./a.out(+0x8b0) [0xaaaab5c708b0]</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">Abort(1) on node 0: Internal error</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:1@dpr740</span>] Sending upstream hdr.cmd = CMD_EXIT_STATUS</span></p>
<p class="x_MsoNormal"><span style="font-size:13.5pt; font-family:Menlo; color:black">[<span class="x_MsoHyperlink">proxy:0@ampere-altra-2-1</span>] Sending upstream hdr.cmd = CMD_EXIT_STATUS</span></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<div id="x_mail-editor-reference-message-container">
<div>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">Niyaz Murshed via discuss <discuss@mpich.org><br>
<b>Date: </b>Saturday, June 15, 2024 at 12:10</span><span style="font-family:"Arial",sans-serif; color:black"> </span><span style="color:black">AM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>Niyaz Murshed <Niyaz.Murshed@arm.com>, nd <nd@arm.com><br>
<b>Subject: </b>[mpich-discuss] mpi hello-world error</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:1.0pt; color:white">Hello, I am trying to run the example hellow.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">c between 2 servers.
I can run them individually and it works fine. 10.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">118.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">91.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">158
is the machine I am running on. 10.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">118.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">91.</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">159
is the remote machine. root@</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">dpr740:</span><span style="font-size:1.0pt; font-family:"Arial",sans-serif; color:white"> </span><span style="font-size:1.0pt; color:white">/mpich/examples#
mpirun -n 2 -hosts </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:1.0pt; color:white">ZjQcmQRYFpfptBannerStart</span></p>
</div>
<div id="x_pfptBanneriin5kjq" style="border:none; border-top:solid #90A4AE 3.0pt; padding:0in 0in 0in 0in; display:block!important; text-align:left!important; margin:0px!important; padding:16px!important; border-radius:4px!important; min-width:200px!important; background-color:#D0D8DC!important; border-top:#90a4ae!important">
<div id="x_pfptBanneriin5kjq">
<div id="x_pfptBanneriin5kjq">
<p class="x_MsoNormal" style="line-height:13.5pt; background:#D0D8DC"><b><span style="font-family:"Arial",sans-serif; color:black">This Message Is From an External Sender
</span></b></p>
</div>
<div id="x_pfptBanneriin5kjq">
<p class="x_MsoNormal" style="line-height:13.5pt; background:#D0D8DC"><span style="font-family:"Arial",sans-serif; color:black">This message came from outside your organization.
</span></p>
</div>
</div>
<div>
<p class="x_MsoNormal" style="background:#D0D8DC"><span style="color:black"> </span></p>
</div>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:1.0pt; color:white">ZjQcmQRYFpfptBannerEnd</span></p>
</div>
<div>
<p class="x_MsoNormal">Hello, </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I am trying to run the example hellow.c between 2 servers.</p>
<p class="x_MsoNormal">I can run them individually and it works fine.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><span class="x_s1">10.118.91.158</span><span class="x_apple-converted-space"> is the machine I am running on.</span></p>
<p class="x_MsoNormal"><span class="x_s1">10.118.91.159 is the remote machine.</span></p>
<p class="x_MsoNormal"> </p>
<p class="x_p1"><span class="x_s1">root@dpr740:/mpich/examples# mpirun</span><span class="x_apple-converted-space">
</span><span class="x_s1">-n 2 -hosts 10.118.91.158</span><span class="x_apple-converted-space">
</span><span class="x_s1">./a.out</span><span class="x_apple-converted-space"> </span></p>
<p class="x_p1"><span class="x_s1">Hello world from process 0 of 2</span></p>
<p class="x_p1"><span class="x_s1">Hello world from process 1 of 2</span></p>
<p class="x_p1"> </p>
<p class="x_p1"><span class="x_s1">root@dpr740:/mpich/examples# mpirun</span><span class="x_apple-converted-space">
</span><span class="x_s1">-n 2 -hosts 10.118.91.159</span><span class="x_apple-converted-space">
</span><span class="x_s1">./a.out</span><span class="x_apple-converted-space"> </span></p>
<p class="x_p1"><span class="x_s1">Hello world from process 1 of 2</span></p>
<p class="x_p1"><span class="x_s1">Hello world from process 0 of 2</span></p>
<p class="x_p1"><span class="x_s1"> </span></p>
<p class="x_p1"><span class="x_s1">However, when I try to run them on both, I get the below error.</span></p>
<p class="x_p1"><span class="x_s1">realloc(): invalid pointer</span></p>
<p class="x_p1"><span class="x_s1"> </span></p>
<p class="x_p1"><span class="x_s1">Is this a known issue ? Any suggestions?</span></p>
<p class="x_p1"><span class="x_s1"> </span></p>
<p class="x_p1"><span class="x_s1"> </span></p>
<p class="x_p1"><span class="x_s1">root@dpr740:/mpich/examples# mpirun -verbose</span><span class="x_apple-converted-space">
</span><span class="x_s1">-n 2 -hosts 10.118.91.159,10.118.91.158</span><span class="x_apple-converted-space">
</span><span class="x_s1">./a.out</span><span class="x_apple-converted-space"> </span></p>
<p class="x_p1"><span class="x_s1">host: 10.118.91.159</span></p>
<p class="x_p1"><span class="x_s1">host: 10.118.91.158</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Timeout set to -1 (-1 means infinite)</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">==================================================================================================</span></p>
<p class="x_p1"><span class="x_s1">mpiexec options:</span></p>
<p class="x_p1"><span class="x_s1">----------------</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">Base path: /opt/mpich/bin/</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">Launcher: (null)</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">Debug level: 1</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">Enable X: -1</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">Global environment:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">-------------------</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">HOSTNAME=dpr740</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">PWD=/mpich/examples</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">HOME=/root</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">LESSCLOSE=/usr/bin/lesspipe %s %s</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">TERM=xterm</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">LESSOPEN=| /usr/bin/lesspipe %s</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">SHLVL=1</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">_=/opt/mpich/bin/mpirun</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">OLDPWD=/</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">Hydra internal environment:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">---------------------------</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">GFORTRAN_UNBUFFERED_PRECONNECTED=y</span></p>
<p class="x_p2"> </p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">Proxy information:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">*********************</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">[1] proxy: 10.118.91.159 (1 cores)</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">Exec list: ./a.out (1 processes);</span><span class="x_apple-converted-space"> </span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">[2] proxy: 10.118.91.158 (1 cores)</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">Exec list: ./a.out (1 processes);</span><span class="x_apple-converted-space"> </span></p>
<p class="x_p2"> </p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">==================================================================================================</span></p>
<p class="x_p2"> </p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">Proxy launch args: /opt/mpich/bin/hydra_pmi_proxy --control-port 10.118.91.159:33909 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2
--proxy-id</span><span class="x_apple-converted-space"> </span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">Arguments being passed to proxy 0:</span></p>
<p class="x_p1"><span class="x_s1">--version 4.3.0a1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 10.118.91.159 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_844_0_801938186_dpr740 --pmi-process-mapping
(vector,(0,2,1)) --global-inherited-env 14 'PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig' 'HOSTNAME=dpr740' 'HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233' 'PWD=/mpich/examples' 'HOME=/root' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'SHLVL=1' 'LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin'
'_=/opt/mpich/bin/mpirun' 'OLDPWD=/' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /mpich/examples --exec-args 1 ./a.out</span><span class="x_apple-converted-space"> </span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">Arguments being passed to proxy 1:</span></p>
<p class="x_p1"><span class="x_s1">--version 4.3.0a1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 10.118.91.158 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_844_0_801938186_dpr740 --pmi-process-mapping
(vector,(0,2,1)) --global-inherited-env 14 'PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig' 'HOSTNAME=dpr740' 'HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233' 'PWD=/mpich/examples' 'HOME=/root' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'SHLVL=1' 'LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin'
'_=/opt/mpich/bin/mpirun' 'OLDPWD=/' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /mpich/examples --exec-args 1 ./a.out</span><span class="x_apple-converted-space"> </span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Launch arguments: /opt/mpich/bin/hydra_pmi_proxy --control-port 10.118.91.159:33909 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc
-2 --proxy-id 0</span><span class="x_apple-converted-space"> </span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Launch arguments: /usr/bin/ssh -x -p 2233 10.118.91.158 "/opt/mpich/bin/hydra_pmi_proxy" --control-port 10.118.91.159:33909 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port
0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 1</span><span class="x_apple-converted-space"> </span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_PID_LIST</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=init pmi_version=1 pmi_subversion=1</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=response_to_init rc=0 pmi_version=1 pmi_subversion=1</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_maxes</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=maxes rc=0 kvsname_max=256 keylen_max=64 vallen_max=1024</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_appnum</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=appnum rc=0 appnum=0</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_my_kvsname</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=my_kvsname rc=0 kvsname=kvs_844_0_801938186_dpr740</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_process_mapping</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_result rc=0 value=(vector,(0,2,1)) found=TRUE</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_hwloc_xmlfile</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_result rc=0 value=/tmp/hydra_hwloc_xmlfile_CeNRJN found=TRUE</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending upstream internal PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_result rc=1</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] we don't understand the response get_result; forwarding downstream</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_PID_LIST</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=init pmi_version=1 pmi_subversion=1</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=response_to_init rc=0 pmi_version=1 pmi_subversion=1</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_maxes</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=maxes rc=0 kvsname_max=256 keylen_max=64 vallen_max=1024</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_appnum</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=appnum rc=0 appnum=0</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_my_kvsname</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=my_kvsname rc=0 kvsname=kvs_844_0_801938186_dpr740</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_process_mapping</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_result rc=0 value=(vector,(0,2,1)) found=TRUE</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_hwloc_xmlfile</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_result rc=0 value=/tmp/hydra_hwloc_xmlfile_xv8EIG found=TRUE</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_in</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending upstream internal PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_in</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_result rc=1</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream internal PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] we don't understand the response get_result; forwarding downstream</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_out</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_out</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_out</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_in</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream internal PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_in</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_out</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=put kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-0 value=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] [</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=put kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-1 value=0200A8BFC0A80101[8]</span></p>
<p class="x_p1"><span class="x_s1">cached command: -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=put_result rc=0</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_in</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] flushing 1 put command(s) out</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] forwarding command upstream:</span></p>
<p class="x_p1"><span class="x_s1">cmd=mput -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending upstream internal PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=mput -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] cached command: -allgather-shm-1-1=0200A8BFC0A80101[8]</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending upstream internal PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_in</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=mput -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=put_result rc=0</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=mput -allgather-shm-1-1=0200A8BFC0A80101[8]</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_in</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] flushing 1 put command(s) out</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] forwarding command upstream:</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=keyval_cache -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] -allgather-shm-1-1=0200A8BFC0A80101[8]</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=keyval_cache -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] -allgather-shm-1-1=0200A8BFC0A80101[8]</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_out</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_out</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_out</span></p>
<p class="x_p1"><span class="x_s1">cmd=mput -allgather-shm-1-1=0200A8BFC0A80101[8]</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream internal PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=mput -allgather-shm-1-1=0200A8BFC0A80101[8]</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream internal PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_in</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_PMI</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-0</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_result rc=0 value=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] found=TRUE</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-1</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_result rc=0 value=0200A8BFC0A80101[8] found=TRUE</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=barrier_out</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-0</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_result rc=0 value=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] found=TRUE</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] got pmi command from downstream 0-0:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-1</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending PMI command:</span></p>
<p class="x_p1"><span class="x_apple-converted-space"> </span><span class="x_s1">cmd=get_result rc=0 value=0200A8BFC0A80101[8] found=TRUE</span></p>
<p class="x_p1"><span class="x_s1">realloc(): invalid pointer</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_STDERR</span></p>
<p class="x_p2"> </p>
<p class="x_p1"><span class="x_s1">===================================================================================</span></p>
<p class="x_p1"><span class="x_s1">= </span><span class="x_apple-converted-space">
</span><span class="x_s1">BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</span></p>
<p class="x_p1"><span class="x_s1">= </span><span class="x_apple-converted-space">
</span><span class="x_s1">PID 2404 RUNNING AT 10.118.91.158</span></p>
<p class="x_p1"><span class="x_s1">= </span><span class="x_apple-converted-space">
</span><span class="x_s1">EXIT CODE: 134</span></p>
<p class="x_p1"><span class="x_s1">= </span><span class="x_apple-converted-space">
</span><span class="x_s1">CLEANING UP REMAINING PROCESSES</span></p>
<p class="x_p1"><span class="x_s1">= </span><span class="x_apple-converted-space">
</span><span class="x_s1">YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</span></p>
<p class="x_p1"><span class="x_s1">===================================================================================</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:1@ampere-altra-2-1</span><span class="x_s1">] Sending upstream hdr.cmd = CMD_EXIT_STATUS</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:484): assert (!closed) failed</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status</span></p>
<p class="x_p1"><span class="x_s1">[</span><span class="x_MsoHyperlink">proxy:0@dpr740</span><span class="x_s1">] main (proxy/pmip.c:122): demux engine error waiting for event</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion</span></p>
<p class="x_p1"><span class="x_s1">[mpiexec@dpr740] main (mpiexec/mpiexec.c:260): process manager error waiting for completion</span></p>
<p class="x_p1"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Regards,</p>
<p class="x_MsoNormal">Niyaz</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>