<div dir="ltr">Yes I am successfully able to run cpi program. No such error at all. <br><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jun 28, 2013 at 12:31 PM, Jeff Hammond <span dir="ltr"><<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Can you run the cpi program? If that doesn't run, something is wrong,<br>
because that program is trivial and correct.<br>
<br>
Jeff<br>
<div><div class="h5"><br>
On Thu, Jun 27, 2013 at 10:29 PM, Syed. Jahanzeb Maqbool Hashmi<br>
<<a href="mailto:jahanzeb.maqbool@gmail.com">jahanzeb.maqbool@gmail.com</a>> wrote:<br>
> again that same error:<br>
> Fatal error in PMPI_Wait: A process has failed, error stack:<br>
> PMPI_Wait(180)............: MPI_Wait(request=0xbebb9a1c, status=0xbebb99f0)<br>
> failed<br>
> MPIR_Wait_impl(77)........:<br>
> dequeue_and_set_error(888): Communication error with rank 4<br>
><br>
> here is the verbose output:<br>
><br>
> --------------START------------------<br>
><br>
> host: weiser1<br>
> host: weiser2<br>
><br>
> ==================================================================================================<br>
> mpiexec options:<br>
> ----------------<br>
> Base path: /mnt/nfs/install/mpich-install/bin/<br>
> Launcher: (null)<br>
> Debug level: 1<br>
> Enable X: -1<br>
><br>
> Global environment:<br>
> -------------------<br>
> TERM=xterm<br>
> SHELL=/bin/bash<br>
><br>
> XDG_SESSION_COOKIE=218a1dd8e20ea6d6ec61475b00000019-1372384778.679329-1845893422<br>
> SSH_CLIENT=192.168.0.3 57311 22<br>
> OLDPWD=/mnt/nfs/jahanzeb/bench/hpl/hpl-2.1<br>
> SSH_TTY=/dev/pts/0<br>
> USER=linaro<br>
><br>
> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35<br>
:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:<br>
> LD_LIBRARY_PATH=:/mnt/nfs/install/mpich-install/lib<br>
> MAIL=/var/mail/linaro<br>
><br>
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mnt/nfs/install/mpich-install/bin<br>
> PWD=/mnt/nfs/jahanzeb/bench/hpl/hpl-2.1/bin/armv7-a<br>
> LANG=C.UTF-8<br>
> SHLVL=1<br>
> HOME=/home/linaro<br>
> LOGNAME=linaro<br>
> SSH_CONNECTION=192.168.0.3 57311 192.168.0.101 22<br>
> LESSOPEN=| /usr/bin/lesspipe %s<br>
> LESSCLOSE=/usr/bin/lesspipe %s %s<br>
> _=/mnt/nfs/install/mpich-install/bin/mpiexec<br>
><br>
> Hydra internal environment:<br>
> ---------------------------<br>
> GFORTRAN_UNBUFFERED_PRECONNECTED=y<br>
><br>
><br>
> Proxy information:<br>
> *********************<br>
> [1] proxy: weiser1 (4 cores)<br>
> Exec list: ./xhpl (4 processes);<br>
><br>
> [2] proxy: weiser2 (4 cores)<br>
> Exec list: ./xhpl (4 processes);<br>
><br>
><br>
> ==================================================================================================<br>
><br>
> [mpiexec@weiser1] Timeout set to -1 (-1 means infinite)<br>
> [mpiexec@weiser1] Got a control port string of weiser1:45851<br>
><br>
> Proxy launch args: /mnt/nfs/install/mpich-install/bin/hydra_pmi_proxy<br>
> --control-port weiser1:45851 --debug --rmk user --launcher ssh --demux poll<br>
> --pgid 0 --retries 10 --usize -2 --proxy-id<br>
><br>
> Arguments being passed to proxy 0:<br>
> --version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname<br>
> weiser1 --global-core-map 0,4,8 --pmi-id-map 0,0 --global-process-count 8<br>
> --auto-cleanup 1 --pmi-kvsname kvs_24541_0 --pmi-process-mapping<br>
> (vector,(0,2,4)) --ckpoint-num -1 --global-inherited-env 20 'TERM=xterm'<br>
> 'SHELL=/bin/bash'<br>
> 'XDG_SESSION_COOKIE=218a1dd8e20ea6d6ec61475b00000019-1372384778.679329-1845893422'<br>
> 'SSH_CLIENT=192.168.0.3 57311 22'<br>
> 'OLDPWD=/mnt/nfs/jahanzeb/bench/hpl/hpl-2.1' 'SSH_TTY=/dev/pts/0'<br>
> 'USER=linaro'<br>
> 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;3<br>
5:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'<br>
> 'LD_LIBRARY_PATH=:/mnt/nfs/install/mpich-install/lib'<br>
> 'MAIL=/var/mail/linaro'<br>
> 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mnt/nfs/install/mpich-install/bin'<br>
> 'PWD=/mnt/nfs/jahanzeb/bench/hpl/hpl-2.1/bin/armv7-a' 'LANG=C.UTF-8'<br>
> 'SHLVL=1' 'HOME=/home/linaro' 'LOGNAME=linaro' 'SSH_CONNECTION=192.168.0.3<br>
> 57311 192.168.0.101 22' 'LESSOPEN=| /usr/bin/lesspipe %s'<br>
> 'LESSCLOSE=/usr/bin/lesspipe %s %s'<br>
> '_=/mnt/nfs/install/mpich-install/bin/mpiexec' --global-user-env 0<br>
> --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y'<br>
> --proxy-core-count 4 --exec --exec-appnum 0 --exec-proc-count 4<br>
> --exec-local-env 0 --exec-wdir<br>
> /mnt/nfs/jahanzeb/bench/hpl/hpl-2.1/bin/armv7-a --exec-args 1 ./xhpl<br>
><br>
> Arguments being passed to proxy 1:<br>
> --version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname<br>
> weiser2 --global-core-map 0,4,8 --pmi-id-map 0,4 --global-process-count 8<br>
> --auto-cleanup 1 --pmi-kvsname kvs_24541_0 --pmi-process-mapping<br>
> (vector,(0,2,4)) --ckpoint-num -1 --global-inherited-env 20 'TERM=xterm'<br>
> 'SHELL=/bin/bash'<br>
> 'XDG_SESSION_COOKIE=218a1dd8e20ea6d6ec61475b00000019-1372384778.679329-1845893422'<br>
> 'SSH_CLIENT=192.168.0.3 57311 22'<br>
> 'OLDPWD=/mnt/nfs/jahanzeb/bench/hpl/hpl-2.1' 'SSH_TTY=/dev/pts/0'<br>
> 'USER=linaro'<br>
> 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;3<br>
5:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'<br>
> 'LD_LIBRARY_PATH=:/mnt/nfs/install/mpich-install/lib'<br>
> 'MAIL=/var/mail/linaro'<br>
> 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mnt/nfs/install/mpich-install/bin'<br>
> 'PWD=/mnt/nfs/jahanzeb/bench/hpl/hpl-2.1/bin/armv7-a' 'LANG=C.UTF-8'<br>
> 'SHLVL=1' 'HOME=/home/linaro' 'LOGNAME=linaro' 'SSH_CONNECTION=192.168.0.3<br>
> 57311 192.168.0.101 22' 'LESSOPEN=| /usr/bin/lesspipe %s'<br>
> 'LESSCLOSE=/usr/bin/lesspipe %s %s'<br>
> '_=/mnt/nfs/install/mpich-install/bin/mpiexec' --global-user-env 0<br>
> --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y'<br>
> --proxy-core-count 4 --exec --exec-appnum 0 --exec-proc-count 4<br>
> --exec-local-env 0 --exec-wdir<br>
> /mnt/nfs/jahanzeb/bench/hpl/hpl-2.1/bin/armv7-a --exec-args 1 ./xhpl<br>
><br>
> [mpiexec@weiser1] Launch arguments:<br>
> /mnt/nfs/install/mpich-install/bin/hydra_pmi_proxy --control-port<br>
> weiser1:45851 --debug --rmk user --launcher ssh --demux poll --pgid 0<br>
> --retries 10 --usize -2 --proxy-id 0<br>
> [mpiexec@weiser1] Launch arguments: /usr/bin/ssh -x weiser2<br>
> "/mnt/nfs/install/mpich-install/bin/hydra_pmi_proxy" --control-port<br>
> weiser1:45851 --debug --rmk user --launcher ssh --demux poll --pgid 0<br>
> --retries 10 --usize -2 --proxy-id 1<br>
> [proxy:0:0@weiser1] got pmi command (from 0): init<br>
> pmi_version=1 pmi_subversion=1<br>
> [proxy:0:0@weiser1] PMI response: cmd=response_to_init pmi_version=1<br>
> pmi_subversion=1 rc=0<br>
> [proxy:0:0@weiser1] got pmi command (from 0): get_maxes<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64<br>
> vallen_max=1024<br>
> [proxy:0:0@weiser1] got pmi command (from 15): init<br>
> pmi_version=1 pmi_subversion=1<br>
> [proxy:0:0@weiser1] PMI response: cmd=response_to_init pmi_version=1<br>
> pmi_subversion=1 rc=0<br>
> [proxy:0:0@weiser1] got pmi command (from 15): get_maxes<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64<br>
> vallen_max=1024<br>
> [proxy:0:0@weiser1] got pmi command (from 8): init<br>
> pmi_version=1 pmi_subversion=1<br>
> [proxy:0:0@weiser1] PMI response: cmd=response_to_init pmi_version=1<br>
> pmi_subversion=1 rc=0<br>
> [proxy:0:0@weiser1] got pmi command (from 0): get_appnum<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=appnum appnum=0<br>
> [proxy:0:0@weiser1] got pmi command (from 15): get_appnum<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=appnum appnum=0<br>
> [proxy:0:0@weiser1] got pmi command (from 0): get_my_kvsname<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:0@weiser1] got pmi command (from 8): get_maxes<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64<br>
> vallen_max=1024<br>
> [proxy:0:0@weiser1] got pmi command (from 0): get_my_kvsname<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:0@weiser1] got pmi command (from 6): init<br>
> pmi_version=1 pmi_subversion=1<br>
> [proxy:0:0@weiser1] PMI response: cmd=response_to_init pmi_version=1<br>
> pmi_subversion=1 rc=0<br>
> [proxy:0:0@weiser1] got pmi command (from 15): get_my_kvsname<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:0@weiser1] got pmi command (from 0): get<br>
> kvsname=kvs_24541_0 key=PMI_process_mapping<br>
> [proxy:0:0@weiser1] PMI response: cmd=get_result rc=0 msg=success<br>
> value=(vector,(0,2,4))<br>
> [proxy:0:0@weiser1] got pmi command (from 8): get_appnum<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=appnum appnum=0<br>
> [proxy:0:0@weiser1] got pmi command (from 15): get_my_kvsname<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:0@weiser1] got pmi command (from 8): get_my_kvsname<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:0@weiser1] got pmi command (from 0): put<br>
> kvsname=kvs_24541_0 key=sharedFilename[0]<br>
> value=/dev/shm/mpich_shar_tmpnEZdQ9<br>
> [proxy:0:0@weiser1] cached command:<br>
> sharedFilename[0]=/dev/shm/mpich_shar_tmpnEZdQ9<br>
> [proxy:0:0@weiser1] PMI response: cmd=put_result rc=0 msg=success<br>
> [proxy:0:0@weiser1] got pmi command (from 15): get<br>
> kvsname=kvs_24541_0 key=PMI_process_mapping<br>
> [proxy:0:0@weiser1] PMI response: cmd=get_result rc=0 msg=success<br>
> value=(vector,(0,2,4))<br>
> [proxy:0:0@weiser1] got pmi command (from 0): barrier_in<br>
><br>
> [proxy:0:0@weiser1] got pmi command (from 6): get_maxes<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64<br>
> vallen_max=1024<br>
> [proxy:0:0@weiser1] got pmi command (from 8): get_my_kvsname<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:0@weiser1] got pmi command (from 15): barrier_in<br>
><br>
> [proxy:0:0@weiser1] got pmi command (from 8): get<br>
> kvsname=kvs_24541_0 key=PMI_process_mapping<br>
> [proxy:0:0@weiser1] PMI response: cmd=get_result rc=0 msg=success<br>
> value=(vector,(0,2,4))<br>
> [proxy:0:0@weiser1] got pmi command (from 6): get_appnum<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=appnum appnum=0<br>
> [proxy:0:0@weiser1] got pmi command (from 8): barrier_in<br>
><br>
> [proxy:0:0@weiser1] got pmi command (from 6): get_my_kvsname<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:0@weiser1] got pmi command (from 6): get_my_kvsname<br>
><br>
> [proxy:0:0@weiser1] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:0@weiser1] got pmi command (from 6): get<br>
> kvsname=kvs_24541_0 key=PMI_process_mapping<br>
> [proxy:0:0@weiser1] PMI response: cmd=get_result rc=0 msg=success<br>
> value=(vector,(0,2,4))<br>
> [proxy:0:0@weiser1] got pmi command (from 6): barrier_in<br>
><br>
> [proxy:0:0@weiser1] flushing 1 put command(s) out<br>
> [mpiexec@weiser1] [pgid: 0] got PMI command: cmd=put<br>
> sharedFilename[0]=/dev/shm/mpich_shar_tmpnEZdQ9<br>
> [proxy:0:0@weiser1] forwarding command (cmd=put<br>
> sharedFilename[0]=/dev/shm/mpich_shar_tmpnEZdQ9) upstream<br>
> [proxy:0:0@weiser1] forwarding command (cmd=barrier_in) upstream<br>
> [mpiexec@weiser1] [pgid: 0] got PMI command: cmd=barrier_in<br>
> [proxy:0:1@weiser2] got pmi command (from 7): init<br>
> pmi_version=1 pmi_subversion=1<br>
> [proxy:0:1@weiser2] PMI response: cmd=response_to_init pmi_version=1<br>
> pmi_subversion=1 rc=0<br>
> [proxy:0:1@weiser2] got pmi command (from 5): init<br>
> pmi_version=1 pmi_subversion=1<br>
> [proxy:0:1@weiser2] PMI response: cmd=response_to_init pmi_version=1<br>
> pmi_subversion=1 rc=0<br>
> [proxy:0:1@weiser2] got pmi command (from 7): get_maxes<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=maxes kvsname_max=256 keylen_max=64<br>
> vallen_max=1024<br>
> [proxy:0:1@weiser2] got pmi command (from 4): init<br>
> pmi_version=1 pmi_subversion=1<br>
> [proxy:0:1@weiser2] PMI response: cmd=response_to_init pmi_version=1<br>
> pmi_subversion=1 rc=0<br>
> [proxy:0:1@weiser2] got pmi command (from 7): get_appnum<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=appnum appnum=0<br>
> [proxy:0:1@weiser2] got pmi command (from 4): get_maxes<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=maxes kvsname_max=256 keylen_max=64<br>
> vallen_max=1024<br>
> [proxy:0:1@weiser2] got pmi command (from 7): get_my_kvsname<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:1@weiser2] got pmi command (from 4): get_appnum<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=appnum appnum=0<br>
> [proxy:0:1@weiser2] got pmi command (from 7): get_my_kvsname<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:1@weiser2] got pmi command (from 4): get_my_kvsname<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:1@weiser2] got pmi command (from 7): get<br>
> kvsname=kvs_24541_0 key=PMI_process_mapping<br>
> [proxy:0:1@weiser2] PMI response: cmd=get_result rc=0 msg=success<br>
> value=(vector,(0,2,4))<br>
> [proxy:0:1@weiser2] got pmi command (from 4): get_my_kvsname<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:1@weiser2] got pmi command (from 7): barrier_in<br>
><br>
> [proxy:0:1@weiser2] got pmi command (from 4): get<br>
> kvsname=kvs_24541_0 key=PMI_process_mapping<br>
> [proxy:0:1@weiser2] PMI response: cmd=get_result rc=0 msg=success<br>
> value=(vector,(0,2,4))<br>
> [proxy:0:1@weiser2] got pmi command (from 5): get_maxes<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=maxes kvsname_max=256 keylen_max=64<br>
> vallen_max=1024<br>
> [proxy:0:1@weiser2] got pmi command (from 5): get_appnum<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=appnum appnum=0<br>
> [proxy:0:1@weiser2] got pmi command (from 4): put<br>
> kvsname=kvs_24541_0 key=sharedFilename[4]<br>
> value=/dev/shm/mpich_shar_tmpuKzlSa<br>
> [proxy:0:1@weiser2] cached command:<br>
> sharedFilename[4]=/dev/shm/mpich_shar_tmpuKzlSa<br>
> [proxy:0:1@weiser2] PMI response: cmd=put_result rc=0 msg=success<br>
> [proxy:0:1@weiser2] got pmi command (from 5): get_my_kvsname<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:1@weiser2] got pmi command (from 4): barrier_in<br>
><br>
> [mpiexec@weiser1] [pgid: 0] got PMI command: cmd=put<br>
> sharedFilename[4]=/dev/shm/mpich_shar_tmpuKzlSa<br>
> [mpiexec@weiser1] [pgid: 0] got PMI command: cmd=barrier_in<br>
> [mpiexec@weiser1] PMI response to fd 6 pid 10: cmd=keyval_cache<br>
> sharedFilename[0]=/dev/shm/mpich_shar_tmpnEZdQ9<br>
> sharedFilename[4]=/dev/shm/mpich_shar_tmpuKzlSa<br>
> [mpiexec@weiser1] PMI response to fd 7 pid 10: cmd=keyval_cache<br>
> sharedFilename[0]=/dev/shm/mpich_shar_tmpnEZdQ9<br>
> sharedFilename[4]=/dev/shm/mpich_shar_tmpuKzlSa<br>
> [mpiexec@weiser1] PMI response to fd 6 pid 10: cmd=barrier_out<br>
> [mpiexec@weiser1] PMI response to fd 7 pid 10: cmd=barrier_out<br>
> [proxy:0:1@weiser2] got pmi command (from 5): get_my_kvsname<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:1@weiser2] got pmi command (from 5): get<br>
> kvsname=kvs_24541_0 key=PMI_process_mapping<br>
> [proxy:0:1@weiser2] PMI response: cmd=get_result rc=0 msg=success<br>
> value=(vector,(0,2,4))<br>
> [proxy:0:1@weiser2] got pmi command (from 10): init<br>
> pmi_version=1 pmi_subversion=1<br>
> [proxy:0:1@weiser2] PMI response: cmd=response_to_init pmi_version=1<br>
> pmi_subversion=1 rc=0<br>
> [proxy:0:1@weiser2] got pmi command (from 5): barrier_in<br>
><br>
> [proxy:0:1@weiser2] got pmi command (from 10): get_maxes<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=maxes kvsname_max=256 keylen_max=64<br>
> vallen_max=1024<br>
> [proxy:0:1@weiser2] got pmi command (from 10): get_appnum<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=appnum appnum=0<br>
> [proxy:0:1@weiser2] got pmi command (from 10): get_my_kvsname<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:1@weiser2] got pmi command (from 10): get_my_kvsname<br>
><br>
> [proxy:0:1@weiser2] PMI response: cmd=my_kvsname kvsname=kvs_24541_0<br>
> [proxy:0:1@weiser2] got pmi command (from 10): get<br>
> kvsname=kvs_24541_0 key=PMI_process_mapping<br>
> [proxy:0:1@weiser2] PMI response: cmd=get_result rc=0 msg=success<br>
> value=(vector,(0,2,4))<br>
> [proxy:0:1@weiser2] got pmi command (from 10): barrier_in<br>
><br>
> [proxy:0:1@weiser2] flushing 1 put command(s) out<br>
> [proxy:0:1@weiser2] forwarding command (cmd=put<br>
> sharedFilename[4]=/dev/shm/mpich_shar_tmpuKzlSa) upstream<br>
> [proxy:0:1@weiser2] forwarding command (cmd=barrier_in) upstream<br>
> [proxy:0:0@weiser1] PMI response: cmd=barrier_out<br>
> [proxy:0:0@weiser1] PMI response: cmd=barrier_out<br>
> [proxy:0:0@weiser1] PMI response: cmd=barrier_out<br>
> [proxy:0:0@weiser1] PMI response: cmd=barrier_out<br>
> [proxy:0:0@weiser1] got pmi command (from 6): get<br>
> kvsname=kvs_24541_0 key=sharedFilename[0]<br>
> [proxy:0:0@weiser1] PMI response: cmd=get_result rc=0 msg=success<br>
> value=/dev/shm/mpich_shar_tmpnEZdQ9<br>
> [proxy:0:1@weiser2] PMI response: cmd=barrier_out<br>
> [proxy:0:1@weiser2] PMI response: cmd=barrier_out<br>
> [proxy:0:1@weiser2] PMI response: cmd=barrier_out<br>
> [proxy:0:1@weiser2] PMI response: cmd=barrier_out<br>
> [proxy:0:1@weiser2] got pmi command (from 5): get<br>
> kvsname=kvs_24541_0 key=sharedFilename[4]<br>
> [proxy:0:1@weiser2] PMI response: cmd=get_result rc=0 msg=success<br>
> value=/dev/shm/mpich_shar_tmpuKzlSa<br>
> [proxy:0:1@weiser2] got pmi command (from 7): get<br>
> kvsname=kvs_24541_0 key=sharedFilename[4]<br>
> [proxy:0:1@weiser2] PMI response: cmd=get_result rc=0 msg=success<br>
> value=/dev/shm/mpich_shar_tmpuKzlSa<br>
> [proxy:0:1@weiser2] got pmi command (from 10): get<br>
> kvsname=kvs_24541_0 key=sharedFilename[4]<br>
> [proxy:0:1@weiser2] PMI response: cmd=get_result rc=0 msg=success<br>
> value=/dev/shm/mpich_shar_tmpuKzlSa<br>
> [proxy:0:0@weiser1] got pmi command (from 8): get<br>
> kvsname=kvs_24541_0 key=sharedFilename[0]<br>
> [proxy:0:0@weiser1] PMI response: cmd=get_result rc=0 msg=success<br>
> value=/dev/shm/mpich_shar_tmpnEZdQ9<br>
> [proxy:0:0@weiser1] got pmi command (from 15): get<br>
> kvsname=kvs_24541_0 key=sharedFilename[0]<br>
> [proxy:0:0@weiser1] PMI response: cmd=get_result rc=0 msg=success<br>
> value=/dev/shm/mpich_shar_tmpnEZdQ9<br>
> [proxy:0:0@weiser1] got pmi command (from 0): put<br>
> kvsname=kvs_24541_0 key=P0-businesscard<br>
> value=description#weiser1$port#56190$ifname#192.168.0.101$<br>
> [proxy:0:0@weiser1] cached command:<br>
> P0-businesscard=description#weiser1$port#56190$ifname#192.168.0.101$<br>
> [proxy:0:0@weiser1] PMI response: cmd=put_result rc=0 msg=success<br>
> [proxy:0:0@weiser1] got pmi command (from 8): put<br>
> kvsname=kvs_24541_0 key=P2-businesscard<br>
> value=description#weiser1$port#40019$ifname#192.168.0.101$<br>
> [proxy:0:0@weiser1] cached command:<br>
> P2-businesscard=description#weiser1$port#40019$ifname#192.168.0.101$<br>
> [proxy:0:0@weiser1] PMI response: cmd=put_result rc=0 msg=success<br>
> [proxy:0:0@weiser1] got pmi command (from 15): put<br>
> kvsname=kvs_24541_0 key=P3-businesscard<br>
> value=description#weiser1$port#57150$ifname#192.168.0.101$<br>
> [proxy:0:0@weiser1] cached command:<br>
> P3-businesscard=description#weiser1$port#57150$ifname#192.168.0.101$<br>
> [proxy:0:0@weiser1] PMI response: cmd=put_result rc=0 msg=success<br>
> [proxy:0:0@weiser1] got pmi command (from 0): barrier_in<br>
><br>
> [proxy:0:0@weiser1] got pmi command (from 6): put<br>
> kvsname=kvs_24541_0 key=P1-businesscard<br>
> value=description#weiser1$port#34048$ifname#192.168.0.101$<br>
> [proxy:0:0@weiser1] cached command:<br>
> P1-businesscard=description#weiser1$port#34048$ifname#192.168.0.101$<br>
> [proxy:0:0@weiser1] PMI response: cmd=put_result rc=0 msg=success<br>
> [proxy:0:0@weiser1] got pmi command (from 8): barrier_in<br>
><br>
> [proxy:0:0@weiser1] got pmi command (from 6): barrier_in<br>
><br>
> [proxy:0:0@weiser1] got pmi command (from 15): barrier_in<br>
><br>
> [proxy:0:0@weiser1] flushing 4 put command(s) out<br>
> [mpiexec@weiser1] [pgid: 0] got PMI command: cmd=put<br>
> P0-businesscard=description#weiser1$port#56190$ifname#192.168.0.101$<br>
> P2-businesscard=description#weiser1$port#40019$ifname#192.168.0.101$<br>
> P3-businesscard=description#weiser1$port#57150$ifname#192.168.0.101$<br>
> P1-businesscard=description#weiser1$port#34048$ifname#192.168.0.101$<br>
> [proxy:0:0@weiser1] forwarding command (cmd=put<br>
> P0-businesscard=description#weiser1$port#56190$ifname#192.168.0.101$<br>
> P2-businesscard=description#weiser1$port#40019$ifname#192.168.0.101$<br>
> P3-businesscard=description#weiser1$port#57150$ifname#192.168.0.101$<br>
> P1-businesscard=description#weiser1$port#34048$ifname#192.168.0.101$)<br>
> upstream<br>
> [proxy:0:0@weiser1] forwarding command (cmd=barrier_in) upstream<br>
> [mpiexec@weiser1] [pgid: 0] got PMI command: cmd=barrier_in<br>
> [proxy:0:1@weiser2] got pmi command (from 4): put<br>
> kvsname=kvs_24541_0 key=P4-businesscard<br>
> value=description#weiser2$port#60693$ifname#192.168.0.102$<br>
> [proxy:0:1@weiser2] cached command:<br>
> P4-businesscard=description#weiser2$port#60693$ifname#192.168.0.102$<br>
> [proxy:0:1@weiser2] PMI response: cmd=put_result rc=0 msg=success<br>
> [proxy:0:1@weiser2] got pmi command (from 5): put<br>
> kvsname=kvs_24541_0 key=P5-businesscard<br>
> value=description#weiser2$port#49938$ifname#192.168.0.102$<br>
> [proxy:0:1@weiser2] cached command:<br>
> P5-businesscard=description#weiser2$port#49938$ifname#192.168.0.102$<br>
> [proxy:0:1@weiser2] PMI response: cmd=put_result rc=0 msg=success<br>
> [proxy:0:1@weiser2] got pmi command (from 7): put<br>
> kvsname=kvs_24541_0 key=P6-businesscard<br>
> value=description#weiser2$port#33516$ifname#192.168.0.102$<br>
> [proxy:0:1@weiser2] cached command:<br>
> P6-businesscard=description#weiser2$port#33516$ifname#192.168.0.102$<br>
> [proxy:0:1@weiser2] PMI response: cmd=put_result rc=0 msg=success<br>
> [proxy:0:1@weiser2] got pmi command (from 10): put<br>
> kvsname=kvs_24541_0 key=P7-businesscard<br>
> value=description#weiser2$port#43116$ifname#192.168.0.102$<br>
> [proxy:0:1@weiser2] cached command:<br>
> P7-businesscard=description#weiser2$port#43116$ifname#192.168.0.102$<br>
> [proxy:0:1@weiser2] [mpiexec@weiser1] [pgid: 0] got PMI command: cmd=put<br>
> P4-businesscard=description#weiser2$port#60693$ifname#192.168.0.102$<br>
> P5-businesscard=description#weiser2$port#49938$ifname#192.168.0.102$<br>
> P6-businesscard=description#weiser2$port#33516$ifname#192.168.0.102$<br>
> P7-businesscard=description#weiser2$port#43116$ifname#192.168.0.102$<br>
> PMI response: cmd=put_result rc=0 msg=success<br>
> [proxy:0:1@weiser2] got pmi command (from 4): barrier_in<br>
><br>
> [proxy:0:1@weiser2] got pmi command (from 5): barrier_in<br>
><br>
> [proxy:0:1@weiser2] got pmi command (from 7): barrier_in<br>
> [mpiexec@weiser1] [pgid: 0] got PMI command: cmd=barrier_in<br>
> [mpiexec@weiser1] PMI response to fd 6 pid 10: cmd=keyval_cache<br>
> P0-businesscard=description#weiser1$port#56190$ifname#192.168.0.101$<br>
> P2-businesscard=description#weiser1$port#40019$ifname#192.168.0.101$<br>
> P3-businesscard=description#weiser1$port#57150$ifname#192.168.0.101$<br>
> P1-businesscard=description#weiser1$port#34048$ifname#192.168.0.101$<br>
> P4-businesscard=description#weiser2$port#60693$ifname#192.168.0.102$<br>
> P5-businesscard=description#weiser2$port#49938$ifname#192.168.0.102$<br>
> P6-businesscard=description#weiser2$port#33516$ifname#192.168.0.102$<br>
> P7-businesscard=description#weiser2$port#43116$ifname#192.168.0.102$<br>
> [mpiexec@weiser1] PMI response to fd 7 pid 10: cmd=keyval_cache<br>
> P0-businesscard=description#weiser1$port#56190$ifname#192.168.0.101$<br>
> P2-businesscard=description#weiser1$port#40019$ifname#192.168.0.101$<br>
> P3-businesscard=description#weiser1$port#57150$ifname#192.168.0.101$<br>
> P1-businesscard=description#weiser1$port#34048$ifname#192.168.0.101$<br>
> P4-businesscard=description#weiser2$port#60693$ifname#192.168.0.102$<br>
> P5-businesscard=description#weiser2$port#49938$ifname#192.168.0.102$<br>
> P6-businesscard=description#weiser2$port#33516$ifname#192.168.0.102$<br>
> P7-businesscard=description#weiser2$port#43116$ifname#192.168.0.102$<br>
> [mpiexec@weiser1] PMI response to fd 6 pid 10: cmd=barrier_out<br>
> [mpiexec@weiser1] PMI response to fd 7 pid 10: cmd=barrier_out<br>
> [proxy:0:0@weiser1] PMI response: cmd=barrier_out<br>
> [proxy:0:0@weiser1]<br>
> [proxy:0:1@weiser2] got pmi command (from 10): barrier_in<br>
><br>
> [proxy:0:1@weiser2] flushing 4 put command(s) out<br>
> [proxy:0:1@weiser2] forwarding command (cmd=put<br>
> P4-businesscard=description#weiser2$port#60693$ifname#192.168.0.102$<br>
> P5-businesscard=description#weiser2$port#49938$ifname#192.168.0.102$<br>
> P6-businesscard=description#weiser2$port#33516$ifname#192.168.0.102$<br>
> P7-businesscard=description#weiser2$port#43116$ifname#192.168.0.102$)<br>
> upstream<br>
> [proxy:0:1@weiser2] forwarding command (cmd=barrier_in) upstream<br>
> PMI response: cmd=barrier_out<br>
> [proxy:0:0@weiser1] PMI response: cmd=barrier_out<br>
> [proxy:0:0@weiser1] PMI response: cmd=barrier_out<br>
> [proxy:0:1@weiser2] PMI response: cmd=barrier_out<br>
> [proxy:0:1@weiser2] PMI response: cmd=barrier_out<br>
> [proxy:0:1@weiser2] PMI response: cmd=barrier_out<br>
> [proxy:0:1@weiser2] PMI response: cmd=barrier_out<br>
> [proxy:0:1@weiser2] got pmi command (from 4): get<br>
> kvsname=kvs_24541_0 key=P0-businesscard<br>
> [proxy:0:1@weiser2] PMI response: cmd=get_result rc=0 msg=success<br>
> value=description#weiser1$port#56190$ifname#192.168.0.101$<br>
> ================================================================================<br>
> HPLinpack 2.1 -- High-Performance Linpack benchmark -- October 26, 2012<br>
> Written by A. Petitet and R. Clint Whaley, Innovative Computing Laboratory,<br>
> UTK<br>
> Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK<br>
> Modified by Julien Langou, University of Colorado Denver<br>
> ================================================================================<br>
><br>
> An explanation of the input/output parameters follows:<br>
> T/V : Wall time / encoded variant.<br>
> N : The order of the coefficient matrix A.<br>
> NB : The partitioning blocking factor.<br>
> P : The number of process rows.<br>
> Q : The number of process columns.<br>
> Time : Time in seconds to solve the linear system.<br>
> Gflops : Rate of execution for solving the linear system.<br>
><br>
> The following parameter values will be used:<br>
><br>
> N : 14616<br>
> NB : 168<br>
> PMAP : Row-major process mapping<br>
> P : 2<br>
> Q : 4<br>
> PFACT : Right<br>
> NBMIN : 4<br>
> NDIV : 2<br>
> RFACT : Crout<br>
> BCAST : 1ringM<br>
> DEPTH : 1<br>
> SWAP : Mix (threshold = 64)<br>
> L1 : transposed form<br>
> U : transposed form<br>
> EQUIL : yes<br>
> ALIGN : 8 double precision words<br>
><br>
> --------------------------------------------------------------------------------<br>
><br>
> - The matrix A is randomly generated for each test.<br>
> - The following scaled residual check will be computed:<br>
> ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )<br>
> - The relative machine precision (eps) is taken to be<br>
> 1.110223e-16<br>
> [proxy:0:0@weiser1] got pmi command (from 6): get<br>
> - Computational tests pass if scaled residuals are less than<br>
> 16.0<br>
><br>
> kvsname=kvs_24541_0 key=P5-businesscard<br>
> [proxy:0:0@weiser1] PMI response: cmd=get_result rc=0 msg=success<br>
> value=description#weiser2$port#49938$ifname#192.168.0.102$<br>
> [proxy:0:0@weiser1] got pmi command (from 15): get<br>
> kvsname=kvs_24541_0 key=P7-businesscard<br>
> [proxy:0:0@weiser1] PMI response: cmd=get_result rc=0 msg=success<br>
> value=description#weiser2$port#43116$ifname#192.168.0.102$<br>
> [proxy:0:0@weiser1] got pmi command (from 8): get<br>
> kvsname=kvs_24541_0 key=P6-businesscard<br>
> [proxy:0:0@weiser1] PMI response: cmd=get_result rc=0 msg=success<br>
> value=description#weiser2$port#33516$ifname#192.168.0.102$<br>
> [proxy:0:1@weiser2] got pmi command (from 5): get<br>
> kvsname=kvs_24541_0 key=P1-businesscard<br>
> [proxy:0:1@weiser2] PMI response: cmd=get_result rc=0 msg=success<br>
> value=description#weiser1$port#34048$ifname#192.168.0.101$<br>
><br>
> ===================================================================================<br>
> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>
> = EXIT CODE: 9<br>
> = CLEANING UP REMAINING PROCESSES<br>
> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>
> ===================================================================================<br>
><br>
><br>
> ----------- END --------------<br>
><br>
> if that can help :(<br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Fri, Jun 28, 2013 at 12:24 PM, Pavan Balaji <<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a>> wrote:<br>
>><br>
>><br>
>> Looks like your application aborted for some reason.<br>
>><br>
>> -- Pavan<br>
>><br>
>><br>
>> On 06/27/2013 10:21 PM, Syed. Jahanzeb Maqbool Hashmi wrote:<br>
>>><br>
>>> My bad, I just found out that there was a duplicate entry like:<br>
>>> weiser1 127.0.1.1<br>
>>> weiser1 192.168.0.101<br>
>>> so i removed teh 127.x.x.x. entry and kept the hostfile contents similar<br>
>>> on both nodes. Now previous error is reduced to this one:<br>
>>><br>
>>> ------ START OF OUTPUT -------<br>
>>><br>
>>> ....some HPL startup string (no final result)<br>
>>> ...skip.....<br>
>>><br>
>>><br>
>>> ===================================================================================<br>
>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>
>>> = EXIT CODE: 9<br>
>>> = CLEANING UP REMAINING PROCESSES<br>
>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>
>>><br>
>>> ===================================================================================<br>
>>> [proxy:0:0@weiser1] HYD_pmcd_pmip_control_cmd_cb<br>
>>> (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed<br>
>>> [proxy:0:0@weiser1] HYDT_dmxu_poll_wait_for_event<br>
>>> (./tools/demux/demux_poll.c:77): callback returned error status<br>
>>> [proxy:0:0@weiser1] main (./pm/pmiserv/pmip.c:206): demux engine error<br>
>>> waiting for event<br>
>>> [mpiexec@weiser1] HYDT_bscu_wait_for_completion<br>
>>> (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes<br>
>>> terminated badly; aborting<br>
>>> [mpiexec@weiser1] HYDT_bsci_wait_for_completion<br>
>>> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting<br>
>>> for completion<br>
>>> [mpiexec@weiser1] HYD_pmci_wait_for_completion<br>
>>> (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for<br>
>>> completion<br>
>>> [mpiexec@weiser1] main (./ui/mpich/mpiexec.c:331): process manager error<br>
>>> waiting for completion<br>
>>><br>
>>> ------ END OF OUTPUT -------<br>
>>><br>
>>><br>
>>><br>
>>> On Fri, Jun 28, 2013 at 12:12 PM, Pavan Balaji <<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a><br>
>>> <mailto:<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a>>> wrote:<br>
>>><br>
>>><br>
>>> On 06/27/2013 10:08 PM, Syed. Jahanzeb Maqbool Hashmi wrote:<br>
>>><br>
>>><br>
>>> P4-businesscard=description#__weiser2$port#57651$ifname#192.__168.0.102$<br>
>>><br>
>>> P5-businesscard=description#__weiser2$port#52622$ifname#192.__168.0.102$<br>
>>><br>
>>> P6-businesscard=description#__weiser2$port#55935$ifname#192.__168.0.102$<br>
>>><br>
>>> P7-businesscard=description#__weiser2$port#54952$ifname#192.__168.0.102$<br>
>>><br>
>>> P0-businesscard=description#__weiser1$port#41958$ifname#127.__0.1.1$<br>
>>><br>
>>> P2-businesscard=description#__weiser1$port#35049$ifname#127.__0.1.1$<br>
>>><br>
>>> P1-businesscard=description#__weiser1$port#39634$ifname#127.__0.1.1$<br>
>>><br>
>>> P3-businesscard=description#__weiser1$port#51802$ifname#127.__0.1.1$<br>
>>><br>
>>><br>
>>><br>
>>> I have two concerns with your output. Let's start with the first.<br>
>>><br>
>>> Did you look at this question on the FAQ page?<br>
>>><br>
>>> "Is your /etc/hosts file consistent across all nodes? Unless you are<br>
>>> using an external DNS server, the /etc/hosts file on every machine<br>
>>> should contain the correct IP information about all hosts in the<br>
>>> system."<br>
>>><br>
>>><br>
>>> -- Pavan<br>
>>><br>
>>> --<br>
>>> Pavan Balaji<br>
>>> <a href="http://www.mcs.anl.gov/~balaji" target="_blank">http://www.mcs.anl.gov/~balaji</a><br>
>>><br>
>>><br>
>><br>
>> --<br>
>> Pavan Balaji<br>
>> <a href="http://www.mcs.anl.gov/~balaji" target="_blank">http://www.mcs.anl.gov/~balaji</a><br>
><br>
><br>
><br>
</div></div>> _______________________________________________<br>
> discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
> To manage subscription options or unsubscribe:<br>
> <a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
--<br>
Jeff Hammond<br>
<a href="mailto:jeff.science@gmail.com">jeff.science@gmail.com</a><br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
</font></span></blockquote></div><br></div>