<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hello, I am trying to run the example hellow. c between 2 servers. I can run them individually and it works fine. 10. 118. 91. 158 is the machine I am running on. 10. 118. 91. 159 is the remote machine. root@ dpr740: /mpich/examples# mpirun -n 2 -hosts
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanneriin5kjq" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanneriin5kjq" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanneriin5kjq" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanneriin5kjq" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanneriin5kjq { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtoniin5kjq:hover, .pfptPrimaryButtoniin5kjq:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtoniin5kjq:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:13.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0in;
font-size:13.5pt;
font-family:Menlo;
color:black;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello, <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am trying to run the example hellow.c between 2 servers.<o:p></o:p></p>
<p class="MsoNormal">I can run them individually and it works fine.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span class="s1">10.118.91.158</span><span class="apple-converted-space"> is the machine I am running on.<o:p></o:p></span></p>
<p class="MsoNormal"><span class="s1">10.118.91.159 is the remote machine.</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">root@dpr740:/mpich/examples# mpirun</span><span class="apple-converted-space">
</span><span class="s1">-n 2 -hosts 10.118.91.158</span><span class="apple-converted-space">
</span><span class="s1">./a.out</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">Hello world from process 0 of 2</span><o:p></o:p></p>
<p class="p1"><span class="s1">Hello world from process 1 of 2<o:p></o:p></span></p>
<p class="p1"><o:p> </o:p></p>
<p class="p1"><span class="s1">root@dpr740:/mpich/examples# mpirun</span><span class="apple-converted-space">
</span><span class="s1">-n 2 -hosts 10.118.91.159</span><span class="apple-converted-space">
</span><span class="s1">./a.out</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">Hello world from process 1 of 2</span><o:p></o:p></p>
<p class="p1"><span class="s1">Hello world from process 0 of 2<o:p></o:p></span></p>
<p class="p1"><span class="s1"><o:p> </o:p></span></p>
<p class="p1"><span class="s1">However, when I try to run them on both, I get the below error.<o:p></o:p></span></p>
<p class="p1"><span class="s1">realloc(): invalid pointer</span><o:p></o:p></p>
<p class="p1"><span class="s1"><o:p> </o:p></span></p>
<p class="p1"><span class="s1">Is this a known issue ? Any suggestions?<o:p></o:p></span></p>
<p class="p1"><span class="s1"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><o:p> </o:p></span></p>
<p class="p1"><span class="s1">root@dpr740:/mpich/examples# mpirun -verbose</span><span class="apple-converted-space">
</span><span class="s1">-n 2 -hosts 10.118.91.159,10.118.91.158</span><span class="apple-converted-space">
</span><span class="s1">./a.out</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">host: 10.118.91.159</span><o:p></o:p></p>
<p class="p1"><span class="s1">host: 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Timeout set to -1 (-1 means infinite)</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">==================================================================================================</span><o:p></o:p></p>
<p class="p1"><span class="s1">mpiexec options:</span><o:p></o:p></p>
<p class="p1"><span class="s1">----------------</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Base path: /opt/mpich/bin/</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Launcher: (null)</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Debug level: 1</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Enable X: -1</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Global environment:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">-------------------</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">HOSTNAME=dpr740</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">PWD=/mpich/examples</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">HOME=/root</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">LESSCLOSE=/usr/bin/lesspipe %s %s</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">TERM=xterm</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">LESSOPEN=| /usr/bin/lesspipe %s</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">SHLVL=1</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">_=/opt/mpich/bin/mpirun</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">OLDPWD=/</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Hydra internal environment:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">---------------------------</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">GFORTRAN_UNBUFFERED_PRECONNECTED=y</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Proxy information:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">*********************</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">[1] proxy: 10.118.91.159 (1 cores)</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Exec list: ./a.out (1 processes);</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">[2] proxy: 10.118.91.158 (1 cores)</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Exec list: ./a.out (1 processes);</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">==================================================================================================</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">Proxy launch args: /opt/mpich/bin/hydra_pmi_proxy --control-port 10.118.91.159:33909 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">Arguments being passed to proxy 0:</span><o:p></o:p></p>
<p class="p1"><span class="s1">--version 4.3.0a1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 10.118.91.159 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_844_0_801938186_dpr740 --pmi-process-mapping
(vector,(0,2,1)) --global-inherited-env 14 'PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig' 'HOSTNAME=dpr740' 'HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233' 'PWD=/mpich/examples' 'HOME=/root' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'SHLVL=1' 'LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin'
'_=/opt/mpich/bin/mpirun' 'OLDPWD=/' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /mpich/examples --exec-args 1 ./a.out</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">Arguments being passed to proxy 1:</span><o:p></o:p></p>
<p class="p1"><span class="s1">--version 4.3.0a1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 10.118.91.158 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_844_0_801938186_dpr740 --pmi-process-mapping
(vector,(0,2,1)) --global-inherited-env 14 'PKG_CONFIG_PATH=:/opt/libfabric/lib/pkgconfig:/opt/mpich/lib/pkgconfig' 'HOSTNAME=dpr740' 'HYDRA_LAUNCHER_EXTRA_ARGS=-p 2233' 'PWD=/mpich/examples' 'HOME=/root' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'SHLVL=1' 'LD_LIBRARY_PATH=:/opt/libfabric/lib:/opt/fabtests/lib:/opt/mpich/lib' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/libfabric/bin/:/opt/fabtest/bin:/opt/mpich/bin'
'_=/opt/mpich/bin/mpirun' 'OLDPWD=/' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /mpich/examples --exec-args 1 ./a.out</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Launch arguments: /opt/mpich/bin/hydra_pmi_proxy --control-port 10.118.91.159:33909 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc
-2 --proxy-id 0</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Launch arguments: /usr/bin/ssh -x -p 2233 10.118.91.158 "/opt/mpich/bin/hydra_pmi_proxy" --control-port 10.118.91.159:33909 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port
0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 1</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending upstream hdr.cmd = CMD_PID_LIST</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=init pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=response_to_init rc=0 pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_maxes</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=maxes rc=0 kvsname_max=256 keylen_max=64 vallen_max=1024</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_appnum</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=appnum rc=0 appnum=0</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_my_kvsname</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=my_kvsname rc=0 kvsname=kvs_844_0_801938186_dpr740</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_process_mapping</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_result rc=0 value=(vector,(0,2,1)) found=TRUE</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_hwloc_xmlfile</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_result rc=0 value=/tmp/hydra_hwloc_xmlfile_CeNRJN found=TRUE</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_result rc=1</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] we don't understand the response get_result; forwarding downstream</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream hdr.cmd = CMD_PID_LIST</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=init pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=response_to_init rc=0 pmi_version=1 pmi_subversion=1</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_maxes</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=maxes rc=0 kvsname_max=256 keylen_max=64 vallen_max=1024</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_appnum</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=appnum rc=0 appnum=0</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_my_kvsname</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=my_kvsname rc=0 kvsname=kvs_844_0_801938186_dpr740</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_process_mapping</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_result rc=0 value=(vector,(0,2,1)) found=TRUE</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_hwloc_xmlfile</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_result rc=0 value=/tmp/hydra_hwloc_xmlfile_xv8EIG found=TRUE</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_in</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_in</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_result rc=1</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=PMI_mpi_memory_alloc_kinds</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] we don't understand the response get_result; forwarding downstream</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_out</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_out</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_out</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_in</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_in</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_out</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=put kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-0 value=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] [<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=put kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-1 value=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="p1"><span class="s1">cached command: -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=put_result rc=0</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_in</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] flushing 1 put command(s) out</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] forwarding command upstream:</span><o:p></o:p></p>
<p class="p1"><span class="s1">cmd=mput -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=mput -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] cached command: -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_in</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=mput -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3]</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=put_result rc=0</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=mput -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_in</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] flushing 1 put command(s) out</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] forwarding command upstream:</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] [pgid: 0] got PMI command: cmd=barrier_in</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=keyval_cache -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=keyval_cache -allgather-shm-1-0=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:0):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_out</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] Sending internal PMI command (proxy:0:1):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_out</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_out</span><o:p></o:p></p>
<p class="p1"><span class="s1">cmd=mput -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=mput -allgather-shm-1-1=0200A8BFC0A80101[8]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream internal PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_in</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream hdr.cmd = CMD_PMI</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-0</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_result rc=0 value=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] found=TRUE</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-1</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_result rc=0 value=0200A8BFC0A80101[8] found=TRUE</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=barrier_out</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-0</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_result rc=0 value=0A00812D[4]FE80[6]526B4BFFFEFC134208[3] found=TRUE</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] got pmi command from downstream 0-0:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get kvsname=kvs_844_0_801938186_dpr740 key=-allgather-shm-1-1</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending PMI command:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cmd=get_result rc=0 value=0200A8BFC0A80101[8] found=TRUE</span><o:p></o:p></p>
<p class="p1"><span class="s1">realloc(): invalid pointer</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream hdr.cmd = CMD_STDERR</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">===================================================================================</span><o:p></o:p></p>
<p class="p1"><span class="s1">= </span><span class="apple-converted-space"> </span>
<span class="s1">BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</span><o:p></o:p></p>
<p class="p1"><span class="s1">= </span><span class="apple-converted-space"> </span>
<span class="s1">PID 2404 RUNNING AT 10.118.91.158</span><o:p></o:p></p>
<p class="p1"><span class="s1">= </span><span class="apple-converted-space"> </span>
<span class="s1">EXIT CODE: 134</span><o:p></o:p></p>
<p class="p1"><span class="s1">= </span><span class="apple-converted-space"> </span>
<span class="s1">CLEANING UP REMAINING PROCESSES</span><o:p></o:p></p>
<p class="p1"><span class="s1">= </span><span class="apple-converted-space"> </span>
<span class="s1">YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</span><o:p></o:p></p>
<p class="p1"><span class="s1">===================================================================================</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:1@ampere-altra-2-1">proxy:1@ampere-altra-2-1</a>] Sending upstream hdr.cmd = CMD_EXIT_STATUS</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:484): assert (!closed) failed</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status</span><o:p></o:p></p>
<p class="p1"><span class="s1">[<a href="proxy:0@dpr740">proxy:0@dpr740</a>] main (proxy/pmip.c:122): demux engine error waiting for event</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion</span><o:p></o:p></p>
<p class="p1"><span class="s1">[mpiexec@dpr740] main (mpiexec/mpiexec.c:260): process manager error waiting for completion</span><o:p></o:p></p>
<p class="p1"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Niyaz<o:p></o:p></p>
</div>
</body>
</html>