[mpich-discuss] MPI_Info key host error in SLURM

Iker Martín Álvarez martini at uji.es
Mon Mar 1 13:37:43 CST 2021


Hi Ken,

Thanks for your reply.
Here are attached two files with the arg "-v". *BatchOutput.txt *has the
output which is giving the error message when the code is executed with the
command *sbatch*, while the file *InteractiveOutput.txt* is the one
executed interactively in the node and work as expected. Both of them has
been compiled and executed with mpich 3.4.1.

Thanks, Iker

El vie, 26 feb 2021 a las 16:15, Raffenetti, Kenneth J. (<
raffenet at mcs.anl.gov>) escribió:

> Hi,
>
> Could you add "-v" to your mpiexec command and provide the output? The
> "host" info key is handled by the process manager when executing the spawn
> command.
>
> Ken
>
> On 2/24/21, 11:55 AM, "Iker Martín Álvarez via discuss" <
> discuss at mpich.org> wrote:
>
>     Hi,
>     I have been working around the MPI_Comm_spawn function with the key
> "host" along a value in a system with the SLURM resource manager (slurm-wlm
> 17.11.2). The function works as expected, but when I send to execute the
> code with the sbatch command, an error arises. This does not happen when I
> execute directly the code in the same machine SLURM decided to execute it
> when it was sended with sbatch. In both cases with the key "host", as when
> I do not use the key, it works just fine.
>
>     The same code has been tested with MPICH 3.3.2 and 3.4.1, which gives
> different errors. Also, I tried it with other implementations (OpenMPI and
> Intel MPI), which works as expected creating the processes in the indicated
> host.
>
>     I would like to create processes by MPI_Comm_spawn in an assigned
> host, so if there are other key values for the Info argument, I could try
> them, but I have not found any looking through MPICH documentation.
>
>     Here is the code I have been using:
>
>     int main(int argc, char ** argv) {
>
>       int myId, numP;
>       MPI_Info info;
>       MPI_Comm comm;
>
>       MPI_Init(&argc, &argv);
>       MPI_Comm_rank(MPI_COMM_WORLD, &myId);
>       MPI_Comm_size(MPI_COMM_WORLD, &numP);
>
>       MPI_Comm comm_par;
>       MPI_Comm_get_parent(&comm_par);
>       if(comm_par != MPI_COMM_NULL ) {
>         if(myId == ROOT) {
>           printf("SONS\n"); fflush(stdout);
>         }
>       } else {
>         if(myId == ROOT) {
>           printf("FATHERS\n"); fflush(stdout);
>         }
>         MPI_Info_create(&info);
>         MPI_Info_set(info, "host", "n00");
>         MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numP, info, ROOT,
> MPI_COMM_WORLD, &comm, MPI_ERRCODES_IGNORE);
>       }
>       MPI_Finalize();
>       return 0;
>     }
>
>
>     For MPICH 3.4.1 there is sometimes no error and the code stops working
> at MPI_Comm_spawn function, other times this error is shown:
>     Abort(1615120) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init:
> Internal MPI error!, error stack:
>     MPIR_Init_thread(152)...:
>     MPID_Init(562)..........:
>     MPIDU_Init_shm_init(195):
>     Init_shm_barrier(94)....: Internal MPI error!  barrier not initialized
>
>
>     Also, the error code for MPICH 3.3.2:
>
>     Assertion failed in file
> src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank
> < pg->size
>     Assertion failed in file
> src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank
> < pg->size
>     Assertion failed in file
> src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank
> < pg->size
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     libbacktrace: no debug info in ELF executable
>     internal ABORT - process 0
>
>     Thanks, Iker
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20210301/4a166ead/attachment-0001.html>
-------------- next part --------------
Test
host: n00

==================================================================================================
mpiexec options:
----------------
  Base path: /home/martini/MPI/mpich-3.4.1_INST/bin/
  Launcher: (null)
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
    LD_LIBRARY_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib
    SLURM_TASK_PID=8625
    SSH_CONNECTION=150.128.98.104 37392 150.128.80.83 22
    LESSCLOSE=/usr/bin/lesspipe %s %s
    SLURM_PRIO_PROCESS=0
    LANG=es_ES.UTF-8
    SLURM_SUBMIT_DIR=/home/martini/tfm/TestSpawn/Threads/PingPong
    HOSTNAME=n00
    LD_RUN_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1
    OLDPWD=/home/martini/TestsSergio/slurm-job-expansion-test
    ENVIRONMENT=BATCH
    PATH_modshare=/usr/sbin:1:/bin:1:/usr/bin:1:/usr/games:1:/snap/bin:1:/home/martini/MPI/mpich-3.4.1_INST/bin:1:/usr/local/sbin:1:/usr/local/bin:1:/sbin:1:/usr/local/games:1
    LOADEDMODULES_modshare=/home/martini/MODULES/modulefiles/mpich3.4:1
    MPI_RUN=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun
    SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint
    SLURM_JOB_GID=1002
    SLURMD_NODENAME=n00
    LD_LIBRARY_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1
    XDG_SESSION_ID=2265
    MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl
    USER=martini
    ENV=/usr/share/modules/init/profile.sh
    PWD=/home/martini/tfm/TestSpawn/Threads/PingPong
    SLURM_JOB_NODELIST=n00
    HOME=/home/martini
    SLURM_CLUSTER_NAME=nasp
    SSH_CLIENT=150.128.98.104 37392 22
    SLURM_JOB_CPUS_PER_NODE=20
    BASH_ENV=/usr/share/modules/init/bash
    XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
    SLURM_TOPOLOGY_ADDR=n00
    _LMFILES__modshare=/home/martini/MODULES/modulefiles/mpich3.4:1
    SLURM_WORKING_CLUSTER=nasp:nasp:6817:8192
    SLURM_JOB_NAME=runBase.sh
    TMPDIR=/tmp
    LOADEDMODULES=/home/martini/MODULES/modulefiles/mpich3.4
    SLURM_NODE_ALIASES=(null)
    SLURM_TOPOLOGY_ADDR_PATTERN=node
    SSH_TTY=/dev/pts/1
    MAIL=/var/mail/martini
    SLURM_JOB_NUM_NODES=1
    MPI_HOME=/home/martini/MPI/mpich-3.4.1_INST
    SHELL=/bin/bash
    TERM=xterm
    SLURM_JOB_UID=1002
    MANPATH_modshare=:1:/home/martini/MPI/mpich-3.4.1_INST/share/man/:1
    SLURM_JOB_PARTITION=P1
    LD_RUN_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib
    SLURM_JOB_USER=martini
    SHLVL=2
    SLURM_SUBMIT_HOST=nasp
    MANPATH=:/home/martini/MPI/mpich-3.4.1_INST/share/man/
    MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles
    SLURM_GTIDS=0
    LOGNAME=martini
    XDG_RUNTIME_DIR=/run/user/1002
    MODULEPATH_modshare=/usr/share/modules/$MODULE_VERSION/modulefiles:1:/etc/environment-modules/modules:1:/usr/share/modules/modulefiles:1:/usr/share/modules/versions:1
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/martini/MPI/mpich-3.4.1_INST/bin
    SLURM_JOB_ID=2177
    _LMFILES_=/home/martini/MODULES/modulefiles/mpich3.4
    MODULESHOME=/usr/share/modules
    LESSOPEN=| /usr/bin/lesspipe %s
    BASH_FUNC_module%%=() {  unset _mlre _mlIFS _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 _mlre="eval ${_mlre}";
 fi;
 eval `${_mlre:-}/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl bash $*`;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlre _mlv _mlrv _mlIFS _mlshdbg;
 return $_mlstatus
}
    BASH_FUNC_switchml%%=() {  typeset swfound=1;
 if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
 typeset swname='main';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd.tcl ]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd-compat ]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [ $swfound -eq 0 ]; then
 echo "Switching to Modules $swname version";
 source /usr/share/modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
}
    BASH_FUNC__moduleraw%%=() {  unset _mlre _mlIFS _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 _mlre="eval ${_mlre}";
 fi;
 eval `${_mlre:-}/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl bash $*`;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlre _mlv _mlrv _mlIFS _mlshdbg;
 return $_mlstatus
}
    _=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun

  Hydra internal environment:
  ---------------------------
    GFORTRAN_UNBUFFERED_PRECONNECTED=y


    Proxy information:
    *********************
      [1] proxy: n00 (20 cores)
      Exec list: ./batch5.out (4 processes); 


==================================================================================================

[mpiexec at n00] Timeout set to -1 (-1 means infinite)
[mpiexec at n00] Got a control port string of n00:45129

Proxy launch args: /home/martini/MPI/mpich-3.4.1_INST/bin/hydra_pmi_proxy --control-port n00:45129 --debug --rmk slurm --launcher slurm --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 

Arguments being passed to proxy 0:
--version 3.4.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname n00 --global-core-map 0,20,20 --pmi-id-map 0,0 --global-process-count 4 --auto-cleanup 1 --pmi-kvsname kvs_8631_0_15624359_n00 --pmi-process-mapping (vector,(0,1,20)) --global-inherited-env 67 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'LD_LIBRARY_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib' 'SLURM_TASK_PID=8625' 'SSH_CONNECTION=150.128.98.104 37392 150.128.80.83 22' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'SLURM_PRIO_PROCESS=0' 'LANG=es_ES.UTF-8' 'SLURM_SUBMIT_DIR=/home/martini/tfm/TestSpawn/Threads/PingPong' 'HOSTNAME=n00' 'LD_RUN_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1' 'OLDPWD=/home/martini/TestsSergio/slurm-job-expansion-test' 'ENVIRONMENT=BATCH' 'PATH_modshare=/usr/sbin:1:/bin:1:/usr/bin:1:/usr/games:1:/snap/bin:1:/home/martini/MPI/mpich-3.4.1_INST/bin:1:/usr/local/sbin:1:/usr/local/bin:1:/sbin:1:/usr/local/games:1' 'LOADEDMODULES_modshare=/home/martini/MODULES/modulefiles/mpich3.4:1' 'MPI_RUN=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun' 'SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint' 'SLURM_JOB_GID=1002' 'SLURMD_NODENAME=n00' 'LD_LIBRARY_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1' 'XDG_SESSION_ID=2265' 'MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl' 'USER=martini' 'ENV=/usr/share/modules/init/profile.sh' 'PWD=/home/martini/tfm/TestSpawn/Threads/PingPong' 'SLURM_JOB_NODELIST=n00' 'HOME=/home/martini' 'SLURM_CLUSTER_NAME=nasp' 'SSH_CLIENT=150.128.98.104 37392 22' 'SLURM_JOB_CPUS_PER_NODE=20' 'BASH_ENV=/usr/share/modules/init/bash' 'XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop' 'SLURM_TOPOLOGY_ADDR=n00' '_LMFILES__modshare=/home/martini/MODULES/modulefiles/mpich3.4:1' 'SLURM_WORKING_CLUSTER=nasp:nasp:6817:8192' 'SLURM_JOB_NAME=runBase.sh' 'TMPDIR=/tmp' 'LOADEDMODULES=/home/martini/MODULES/modulefiles/mpich3.4' 'SLURM_NODE_ALIASES=(null)' 'SLURM_TOPOLOGY_ADDR_PATTERN=node' 'SSH_TTY=/dev/pts/1' 'MAIL=/var/mail/martini' 'SLURM_JOB_NUM_NODES=1' 'MPI_HOME=/home/martini/MPI/mpich-3.4.1_INST' 'SHELL=/bin/bash' 'TERM=xterm' 'SLURM_JOB_UID=1002' 'MANPATH_modshare=:1:/home/martini/MPI/mpich-3.4.1_INST/share/man/:1' 'SLURM_JOB_PARTITION=P1' 'LD_RUN_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib' 'SLURM_JOB_USER=martini' 'SHLVL=2' 'SLURM_SUBMIT_HOST=nasp' 'MANPATH=:/home/martini/MPI/mpich-3.4.1_INST/share/man/' 'MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles' 'SLURM_GTIDS=0' 'LOGNAME=martini' 'XDG_RUNTIME_DIR=/run/user/1002' 'MODULEPATH_modshare=/usr/share/modules/$MODULE_VERSION/modulefiles:1:/etc/environment-modules/modules:1:/usr/share/modules/modulefiles:1:/usr/share/modules/versions:1' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/martini/MPI/mpich-3.4.1_INST/bin' 'SLURM_JOB_ID=2177' '_LMFILES_=/home/martini/MODULES/modulefiles/mpich3.4' 'MODULESHOME=/usr/share/modules' 'LESSOPEN=| /usr/bin/lesspipe %s' 'BASH_FUNC_module%%=() {  unset _mlre _mlIFS _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 _mlre="eval ${_mlre}";
 fi;
 eval `${_mlre:-}/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl bash $*`;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlre _mlv _mlrv _mlIFS _mlshdbg;
 return $_mlstatus
}' 'BASH_FUNC_switchml%%=() {  typeset swfound=1;
 if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
 typeset swname='main';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd.tcl ]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd-compat ]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [ $swfound -eq 0 ]; then
 echo "Switching to Modules $swname version";
 source /usr/share/modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
}' 'BASH_FUNC__moduleraw%%=() {  unset _mlre _mlIFS _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 _mlre="eval ${_mlre}";
 fi;
 eval `${_mlre:-}/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl bash $*`;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlre _mlv _mlrv _mlIFS _mlshdbg;
 return $_mlstatus
}' '_=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 20 --exec --exec-appnum 0 --exec-proc-count 4 --exec-local-env 0 --exec-wdir /home/martini/tfm/TestSpawn/Threads/PingPong --exec-args 1 ./batch5.out 

[mpiexec at n00] Launch arguments: /usr/bin/srun -N 1 -n 1 --input none /home/martini/MPI/mpich-3.4.1_INST/bin/hydra_pmi_proxy --control-port n00:45129 --debug --rmk slurm --launcher slurm --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id -1 
[proxy:0:0 at n00] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0 at n00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[mpiexec at n00] [pgid: 0] got PMI command: cmd=put -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D7033395355686E
[mpiexec at n00] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec at n00] PMI response to fd 7 pid 4: cmd=keyval_cache -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D7033395355686E 
[mpiexec at n00] PMI response to fd 7 pid 4: cmd=barrier_out
[proxy:0:0 at n00] got pmi command (from 6): get_maxes

[proxy:0:0 at n00] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at n00] got pmi command (from 6): get_appnum

[proxy:0:0 at n00] PMI response: cmd=appnum appnum=0
[proxy:0:0 at n00] got pmi command (from 6): get_my_kvsname

[proxy:0:0 at n00] PMI response: cmd=my_kvsname kvsname=kvs_8631_0_15624359_n00
[proxy:0:0 at n00] got pmi command (from 6): get
kvsname=kvs_8631_0_15624359_n00 key=PMI_process_mapping 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,20))
[proxy:0:0 at n00] got pmi command (from 6): barrier_in

[proxy:0:0 at n00] got pmi command (from 5): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0 at n00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at n00] got pmi command (from 5): get_maxes

[proxy:0:0 at n00] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at n00] got pmi command (from 5): get_appnum

[proxy:0:0 at n00] PMI response: cmd=appnum appnum=0
[proxy:0:0 at n00] got pmi command (from 5): get_my_kvsname

[proxy:0:0 at n00] PMI response: cmd=my_kvsname kvsname=kvs_8631_0_15624359_n00
[proxy:0:0 at n00] got pmi command (from 5): get
kvsname=kvs_8631_0_15624359_n00 key=PMI_process_mapping 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,20))
[proxy:0:0 at n00] got pmi command (from 5): barrier_in

[proxy:0:0 at n00] got pmi command (from 9): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0 at n00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at n00] got pmi command (from 9): get_maxes

[proxy:0:0 at n00] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at n00] got pmi command (from 9): get_appnum

[proxy:0:0 at n00] PMI response: cmd=appnum appnum=0
[proxy:0:0 at n00] got pmi command (from 9): get_my_kvsname

[proxy:0:0 at n00] PMI response: cmd=my_kvsname kvsname=kvs_8631_0_15624359_n00
[proxy:0:0 at n00] got pmi command (from 9): get
kvsname=kvs_8631_0_15624359_n00 key=PMI_process_mapping 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,20))
[proxy:0:0 at n00] got pmi command (from 9): barrier_in

[proxy:0:0 at n00] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0 at n00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at n00] got pmi command (from 4): get_maxes

[proxy:0:0 at n00] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at n00] got pmi command (from 4): get_appnum

[proxy:0:0 at n00] PMI response: cmd=appnum appnum=0
[proxy:0:0 at n00] got pmi command (from 4): get_my_kvsname

[proxy:0:0 at n00] PMI response: cmd=my_kvsname kvsname=kvs_8631_0_15624359_n00
[proxy:0:0 at n00] got pmi command (from 4): get
kvsname=kvs_8631_0_15624359_n00 key=PMI_process_mapping 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,20))
[proxy:0:0 at n00] got pmi command (from 4): put
kvsname=kvs_8631_0_15624359_n00 key=-bcast-1-0 value=2F6465762F73686D2F6D706963685F736861725F746D7033395355686E 
[proxy:0:0 at n00] cached command: -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D7033395355686E
[proxy:0:0 at n00] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at n00] got pmi command (from 4): barrier_in

[proxy:0:0 at n00] flushing 1 put command(s) out
[proxy:0:0 at n00] forwarding command (cmd=put -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D7033395355686E) upstream
[proxy:0:0 at n00] forwarding command (cmd=barrier_in) upstream
[proxy:0:0 at n00] PMI response: cmd=barrier_out
[proxy:0:0 at n00] PMI response: cmd=barrier_out
[proxy:0:0 at n00] PMI response: cmd=barrier_out
[proxy:0:0 at n00] PMI response: cmd=barrier_out
[proxy:0:0 at n00] got pmi command (from 5): get
kvsname=kvs_8631_0_15624359_n00 key=-bcast-1-0 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F6D706963685F736861725F746D7033395355686E
[proxy:0:0 at n00] got pmi command (from 6): get
kvsname=kvs_8631_0_15624359_n00 key=-bcast-1-0 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F6D706963685F736861725F746D7033395355686E
[proxy:0:0 at n00] got pmi command (from 9): get
kvsname=kvs_8631_0_15624359_n00 key=-bcast-1-0 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F6D706963685F736861725F746D7033395355686E
[proxy:0:0 at n00] got pmi command (from 4): put
kvsname=kvs_8631_0_15624359_n00 key=-allgather-shm-1-0 value=0200A7D1C0A8020A0000000000000000 
[proxy:0:0 at n00] cached command: -allgather-shm-1-0=0200A7D1C0A8020A0000000000000000
[proxy:0:0 at n00] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at n00] got pmi command (from 5): put
kvsname=kvs_8631_0_15624359_n00 key=-allgather-shm-1-1 value=02009D61C0A8020A0000000000000000 
[proxy:0:0 at n00] cached command: -allgather-shm-1-1=02009D61C0A8020A0000000000000000
[proxy:0:0 at n00] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at n00] got pmi command (from 6): put
kvsname=kvs_8631_0_15624359_n00 key=-allgather-shm-1-2 value=02008285C0A8020A0000000000000000 
[proxy:0:0 at n00] cached command: -allgather-shm-1-2=02008285C0A8020A0000000000000000
[proxy:0:0 at n00] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at n00] got pmi command (from 9): put
kvsname=kvs_8631_0_15624359_n00 key=-allgather-shm-1-3 value=02008981C0A8020A0000000000000000 
[proxy:0:0 at n00] cached command: -allgather-shm-1-3=02008981C0A8020A0000000000000000
[proxy:0:0 at n00] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at n00] got pmi command (from 4): barrier_in

[proxy:0:0 at n00] got pmi command (from 5): barrier_in

[proxy:0:0 at n00] got pmi command (from 6): barrier_in

[mpiexec at n00] [pgid: 0] got PMI command: cmd=put -allgather-shm-1-0=0200A7D1C0A8020A0000000000000000 -allgather-shm-1-1=02009D61C0A8020A0000000000000000 -allgather-shm-1-2=02008285C0A8020A0000000000000000 -allgather-shm-1-3=02008981C0A8020A0000000000000000
[proxy:0:0 at n00] got pmi command (from 9): barrier_in

[proxy:0:0 at n00] flushing 4 put command(s) out
[proxy:0:0 at n00] forwarding command (cmd=put -allgather-shm-1-0=0200A7D1C0A8020A0000000000000000 -allgather-shm-1-1=02009D61C0A8020A0000000000000000 -allgather-shm-1-2=02008285C0A8020A0000000000000000 -allgather-shm-1-3=02008981C0A8020A0000000000000000) upstream
[proxy:0:0 at n00] forwarding command (cmd=barrier_in) upstream
[mpiexec at n00] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec at n00] PMI response to fd 7 pid 9: cmd=keyval_cache -allgather-shm-1-0=0200A7D1C0A8020A0000000000000000 -allgather-shm-1-1=02009D61C0A8020A0000000000000000 -allgather-shm-1-2=02008285C0A8020A0000000000000000 -allgather-shm-1-3=02008981C0A8020A0000000000000000 
[mpiexec at n00] PMI response to fd 7 pid 9: cmd=barrier_out
[proxy:0:0 at n00] PMI response: cmd=barrier_out
[proxy:0:0 at n00] PMI response: cmd=barrier_out
[proxy:0:0 at n00] PMI response: cmd=barrier_out
[proxy:0:0 at n00] PMI response: cmd=barrier_out
[proxy:0:0 at n00] got pmi command (from 4): get
kvsname=kvs_8631_0_15624359_n00 key=-allgather-shm-1-0 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=0200A7D1C0A8020A0000000000000000
[proxy:0:0 at n00] got pmi command (from 5): get
kvsname=kvs_8631_0_15624359_n00 key=-allgather-shm-1-1 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=02009D61C0A8020A0000000000000000
[proxy:0:0 at n00] got pmi command (from 6): get
kvsname=kvs_8631_0_15624359_n00 key=-allgather-shm-1-2 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=02008285C0A8020A0000000000000000
[proxy:0:0 at n00] got pmi command (from 9): get
kvsname=kvs_8631_0_15624359_n00 key=-allgather-shm-1-3 
[proxy:0:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=02008981C0A8020A0000000000000000
PADRES
Name n00
Name n00
Name n00
Name n00
[mpiexec at n00] [pgid: 0] got PMI command: mcmd=spawn
nprocs=4
execname=./batch5.out
totspawns=1
spawnssofar=1
argcnt=0
preput_num=1
preput_key_0=PARENT_ROOT_PORT_NAME
preput_val_0=tag#0$connentry#0200A7D1C0A8020A0000000000000000$
info_num=1
info_key_0=host
info_val_0=n00
endcmd
[mpiexec at n00] Got a control port string of n00:44475

Proxy launch args: /home/martini/MPI/mpich-3.4.1_INST/bin/hydra_pmi_proxy --control-port n00:44475 --debug --rmk slurm --launcher slurm --demux poll --pgid 1 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 

Arguments being passed to proxy 0:
--version 3.4.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname n00 --global-core-map 0,1,20 --pmi-id-map 0,0 --global-process-count 4 --auto-cleanup 1 --pmi-kvsname kvs_8631_1_1985138167_n00 --pmi-spawner-kvsname kvs_8631_0_15624359_n00 --pmi-process-mapping (vector,(0,1,1)) --global-inherited-env 67 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'LD_LIBRARY_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib' 'SLURM_TASK_PID=8625' 'SSH_CONNECTION=150.128.98.104 37392 150.128.80.83 22' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'SLURM_PRIO_PROCESS=0' 'LANG=es_ES.UTF-8' 'SLURM_SUBMIT_DIR=/home/martini/tfm/TestSpawn/Threads/PingPong' 'HOSTNAME=n00' 'LD_RUN_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1' 'OLDPWD=/home/martini/TestsSergio/slurm-job-expansion-test' 'ENVIRONMENT=BATCH' 'PATH_modshare=/usr/sbin:1:/bin:1:/usr/bin:1:/usr/games:1:/snap/bin:1:/home/martini/MPI/mpich-3.4.1_INST/bin:1:/usr/local/sbin:1:/usr/local/bin:1:/sbin:1:/usr/local/games:1' 'LOADEDMODULES_modshare=/home/martini/MODULES/modulefiles/mpich3.4:1' 'MPI_RUN=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun' 'SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint' 'SLURM_JOB_GID=1002' 'SLURMD_NODENAME=n00' 'LD_LIBRARY_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1' 'XDG_SESSION_ID=2265' 'MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl' 'USER=martini' 'ENV=/usr/share/modules/init/profile.sh' 'PWD=/home/martini/tfm/TestSpawn/Threads/PingPong' 'SLURM_JOB_NODELIST=n00' 'HOME=/home/martini' 'SLURM_CLUSTER_NAME=nasp' 'SSH_CLIENT=150.128.98.104 37392 22' 'SLURM_JOB_CPUS_PER_NODE=20' 'BASH_ENV=/usr/share/modules/init/bash' 'XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop' 'SLURM_TOPOLOGY_ADDR=n00' '_LMFILES__modshare=/home/martini/MODULES/modulefiles/mpich3.4:1' 'SLURM_WORKING_CLUSTER=nasp:nasp:6817:8192' 'SLURM_JOB_NAME=runBase.sh' 'TMPDIR=/tmp' 'LOADEDMODULES=/home/martini/MODULES/modulefiles/mpich3.4' 'SLURM_NODE_ALIASES=(null)' 'SLURM_TOPOLOGY_ADDR_PATTERN=node' 'SSH_TTY=/dev/pts/1' 'MAIL=/var/mail/martini' 'SLURM_JOB_NUM_NODES=1' 'MPI_HOME=/home/martini/MPI/mpich-3.4.1_INST' 'SHELL=/bin/bash' 'TERM=xterm' 'SLURM_JOB_UID=1002' 'MANPATH_modshare=:1:/home/martini/MPI/mpich-3.4.1_INST/share/man/:1' 'SLURM_JOB_PARTITION=P1' 'LD_RUN_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib' 'SLURM_JOB_USER=martini' 'SHLVL=2' 'SLURM_SUBMIT_HOST=nasp' 'MANPATH=:/home/martini/MPI/mpich-3.4.1_INST/share/man/' 'MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles' 'SLURM_GTIDS=0' 'LOGNAME=martini' 'XDG_RUNTIME_DIR=/run/user/1002' 'MODULEPATH_modshare=/usr/share/modules/$MODULE_VERSION/modulefiles:1:/etc/environment-modules/modules:1:/usr/share/modules/modulefiles:1:/usr/share/modules/versions:1' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/martini/MPI/mpich-3.4.1_INST/bin' 'SLURM_JOB_ID=2177' '_LMFILES_=/home/martini/MODULES/modulefiles/mpich3.4' 'MODULESHOME=/usr/share/modules' 'LESSOPEN=| /usr/bin/lesspipe %s' 'BASH_FUNC_module%%=() {  unset _mlre _mlIFS _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 _mlre="eval ${_mlre}";
 fi;
 eval `${_mlre:-}/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl bash $*`;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlre _mlv _mlrv _mlIFS _mlshdbg;
 return $_mlstatus
}' 'BASH_FUNC_switchml%%=() {  typeset swfound=1;
 if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
 typeset swname='main';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd.tcl ]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd-compat ]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [ $swfound -eq 0 ]; then
 echo "Switching to Modules $swname version";
 source /usr/share/modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
}' 'BASH_FUNC__moduleraw%%=() {  unset _mlre _mlIFS _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 _mlre="eval ${_mlre}";
 fi;
 eval `${_mlre:-}/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl bash $*`;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlre _mlv _mlrv _mlIFS _mlshdbg;
 return $_mlstatus
}' '_=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 4 --exec-local-env 1 'PMI_SPAWNED=1' --exec-wdir /home/martini/tfm/TestSpawn/Threads/PingPong --exec-args 1 ./batch5.out 

[mpiexec at n00] Launch arguments: /usr/bin/srun --nodelist n00 -N 1 -n 1 --input none /home/martini/MPI/mpich-3.4.1_INST/bin/hydra_pmi_proxy --control-port n00:44475 --debug --rmk slurm --launcher slurm --demux poll --pgid 1 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id -1 
[mpiexec at n00] PMI response to fd 7 pid 4: cmd=spawn_result rc=0
[proxy:0:0 at n00] got pmi command (from 4): spawn
nprocs=4 execname=./batch5.out totspawns=1 spawnssofar=1 argcnt=0 preput_num=1 preput_key_0=PARENT_ROOT_PORT_NAME preput_val_0=tag#0$connentry#0200A7D1C0A8020A0000000000000000$ info_num=1 info_key_0=host info_val_0=n00 endcmd 
[proxy:0:0 at n00] we don't understand this command spawn; forwarding upstream
[proxy:0:0 at n00] we don't understand the response spawn_result; forwarding downstream
[proxy:1:0 at n00] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1 
[proxy:1:0 at n00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[mpiexec at n00] [pgid: 1] got PMI command: cmd=put -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D70396F597A524C
[mpiexec at n00] [pgid: 1] got PMI command: cmd=barrier_in
[mpiexec at n00] PMI response to fd 11 pid 5: cmd=keyval_cache PARENT_ROOT_PORT_NAME=tag#0$connentry#0200A7D1C0A8020A0000000000000000$ -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D70396F597A524C 
[mpiexec at n00] PMI response to fd 11 pid 5: cmd=barrier_out
[mpiexec at n00] [pgid: 1] got PMI command: cmd=abort exitcode=1615120
-------------- next part --------------
host: n06

==================================================================================================
mpiexec options:
----------------
  Base path: /home/martini/MPI/mpich-3.4.1_INST/bin/
  Launcher: (null)
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    LD_LIBRARY_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib
    LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
    SSH_CONNECTION=192.168.1.1 54318 192.168.1.16 22
    LESSCLOSE=/usr/bin/lesspipe %s %s
    LANG=es_ES.UTF-8
    LD_RUN_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1
    OLDPWD=/home/martini
    PATH_modshare=/usr/sbin:1:/bin:1:/usr/bin:1:/usr/games:1:/snap/bin:1:/home/martini/MPI/mpich-3.4.1_INST/bin:1:/usr/local/sbin:1:/usr/local/bin:1:/sbin:1:/usr/local/games:1
    LOADEDMODULES_modshare=/home/martini/MODULES/modulefiles/mpich3.4:1
    MPI_RUN=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun
    LD_LIBRARY_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1
    XDG_SESSION_ID=1858
    MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl
    USER=martini
    ENV=/usr/share/modules/init/profile.sh
    PWD=/home/martini/tfm/TestSpawn/Threads/PingPong
    HOME=/home/martini
    SSH_CLIENT=192.168.1.1 54318 22
    BASH_ENV=/usr/share/modules/init/bash
    XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
    _LMFILES__modshare=/home/martini/MODULES/modulefiles/mpich3.4:1
    LOADEDMODULES=/home/martini/MODULES/modulefiles/mpich3.4
    SSH_TTY=/dev/pts/0
    MAIL=/var/mail/martini
    MPI_HOME=/home/martini/MPI/mpich-3.4.1_INST
    TERM=xterm
    SHELL=/bin/bash
    MANPATH_modshare=:1:/home/martini/MPI/mpich-3.4.1_INST/share/man/:1
    LD_RUN_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib
    SHLVL=1
    MANPATH=:/home/martini/MPI/mpich-3.4.1_INST/share/man/
    MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles
    LOGNAME=martini
    XDG_RUNTIME_DIR=/run/user/1002
    MODULEPATH_modshare=/usr/share/modules/$MODULE_VERSION/modulefiles:1:/etc/environment-modules/modules:1:/usr/share/modules/modulefiles:1:/usr/share/modules/versions:1
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/martini/MPI/mpich-3.4.1_INST/bin
    _LMFILES_=/home/martini/MODULES/modulefiles/mpich3.4
    MODULESHOME=/usr/share/modules
    LESSOPEN=| /usr/bin/lesspipe %s
    BASH_FUNC_module%%=() {  _moduleraw "$*" 2>&1
}
    BASH_FUNC_switchml%%=() {  typeset swfound=1;
 if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
 typeset swname='main';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd.tcl ]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd-compat ]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [ $swfound -eq 0 ]; then
 echo "Switching to Modules $swname version";
 source /usr/share/modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
}
    BASH_FUNC__moduleraw%%=() {  unset _mlre _mlIFS _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 _mlre="eval ${_mlre}";
 fi;
 eval `${_mlre:-}/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl bash $*`;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlre _mlv _mlrv _mlIFS _mlshdbg;
 return $_mlstatus
}
    _=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun

  Hydra internal environment:
  ---------------------------
    GFORTRAN_UNBUFFERED_PRECONNECTED=y


    Proxy information:
    *********************
      [1] proxy: n06 (1 cores)
      Exec list: ./batch5.out (4 processes); 


==================================================================================================

[mpiexec at n06] Timeout set to -1 (-1 means infinite)
[mpiexec at n06] Got a control port string of n06:37519

Proxy launch args: /home/martini/MPI/mpich-3.4.1_INST/bin/hydra_pmi_proxy --control-port n06:37519 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 

Arguments being passed to proxy 0:
--version 3.4.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname n06 --global-core-map 0,1,1 --pmi-id-map 0,0 --global-process-count 4 --auto-cleanup 1 --pmi-kvsname kvs_25318_0_67517667_n06 --pmi-process-mapping (vector,(0,1,1)) --global-inherited-env 43 'LD_LIBRARY_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'SSH_CONNECTION=192.168.1.1 54318 192.168.1.16 22' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'LANG=es_ES.UTF-8' 'LD_RUN_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1' 'OLDPWD=/home/martini' 'PATH_modshare=/usr/sbin:1:/bin:1:/usr/bin:1:/usr/games:1:/snap/bin:1:/home/martini/MPI/mpich-3.4.1_INST/bin:1:/usr/local/sbin:1:/usr/local/bin:1:/sbin:1:/usr/local/games:1' 'LOADEDMODULES_modshare=/home/martini/MODULES/modulefiles/mpich3.4:1' 'MPI_RUN=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun' 'LD_LIBRARY_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1' 'XDG_SESSION_ID=1858' 'MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl' 'USER=martini' 'ENV=/usr/share/modules/init/profile.sh' 'PWD=/home/martini/tfm/TestSpawn/Threads/PingPong' 'HOME=/home/martini' 'SSH_CLIENT=192.168.1.1 54318 22' 'BASH_ENV=/usr/share/modules/init/bash' 'XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop' '_LMFILES__modshare=/home/martini/MODULES/modulefiles/mpich3.4:1' 'LOADEDMODULES=/home/martini/MODULES/modulefiles/mpich3.4' 'SSH_TTY=/dev/pts/0' 'MAIL=/var/mail/martini' 'MPI_HOME=/home/martini/MPI/mpich-3.4.1_INST' 'TERM=xterm' 'SHELL=/bin/bash' 'MANPATH_modshare=:1:/home/martini/MPI/mpich-3.4.1_INST/share/man/:1' 'LD_RUN_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib' 'SHLVL=1' 'MANPATH=:/home/martini/MPI/mpich-3.4.1_INST/share/man/' 'MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles' 'LOGNAME=martini' 'XDG_RUNTIME_DIR=/run/user/1002' 'MODULEPATH_modshare=/usr/share/modules/$MODULE_VERSION/modulefiles:1:/etc/environment-modules/modules:1:/usr/share/modules/modulefiles:1:/usr/share/modules/versions:1' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/martini/MPI/mpich-3.4.1_INST/bin' '_LMFILES_=/home/martini/MODULES/modulefiles/mpich3.4' 'MODULESHOME=/usr/share/modules' 'LESSOPEN=| /usr/bin/lesspipe %s' 'BASH_FUNC_module%%=() {  _moduleraw "$*" 2>&1
}' 'BASH_FUNC_switchml%%=() {  typeset swfound=1;
 if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
 typeset swname='main';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd.tcl ]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd-compat ]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [ $swfound -eq 0 ]; then
 echo "Switching to Modules $swname version";
 source /usr/share/modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
}' 'BASH_FUNC__moduleraw%%=() {  unset _mlre _mlIFS _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 _mlre="eval ${_mlre}";
 fi;
 eval `${_mlre:-}/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl bash $*`;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlre _mlv _mlrv _mlIFS _mlshdbg;
 return $_mlstatus
}' '_=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 4 --exec-local-env 0 --exec-wdir /home/martini/tfm/TestSpawn/Threads/PingPong --exec-args 1 ./batch5.out 

[mpiexec at n06] Launch arguments: /home/martini/MPI/mpich-3.4.1_INST/bin/hydra_pmi_proxy --control-port n06:37519 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 0 
[proxy:0:0 at n06] got pmi command (from 8): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0 at n06] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at n06] got pmi command (from 8): get_maxes

[proxy:0:0 at n06] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at n06] got pmi command (from 8): get_appnum

[proxy:0:0 at n06] PMI response: cmd=appnum appnum=0
[proxy:0:0 at n06] got pmi command (from 8): get_my_kvsname

[proxy:0:0 at n06] PMI response: cmd=my_kvsname kvsname=kvs_25318_0_67517667_n06
[proxy:0:0 at n06] got pmi command (from 8): get
kvsname=kvs_25318_0_67517667_n06 key=PMI_process_mapping 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,1))
[proxy:0:0 at n06] got pmi command (from 8): barrier_in

[proxy:0:0 at n06] got pmi command (from 13): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0 at n06] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at n06] got pmi command (from 13): get_maxes

[proxy:0:0 at n06] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at n06] got pmi command (from 13): get_appnum

[proxy:0:0 at n06] PMI response: cmd=appnum appnum=0
[proxy:0:0 at n06] got pmi command (from 13): get_my_kvsname

[proxy:0:0 at n06] PMI response: cmd=my_kvsname kvsname=kvs_25318_0_67517667_n06
[proxy:0:0 at n06] got pmi command (from 13): get
kvsname=kvs_25318_0_67517667_n06 key=PMI_process_mapping 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,1))
[proxy:0:0 at n06] got pmi command (from 13): barrier_in

[proxy:0:0 at n06] got pmi command (from 0): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0 at n06] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at n06] got pmi command (from 0): get_maxes

[proxy:0:0 at n06] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at n06] got pmi command (from 0): get_appnum

[proxy:0:0 at n06] PMI response: cmd=appnum appnum=0
[proxy:0:0 at n06] got pmi command (from 0): get_my_kvsname

[proxy:0:0 at n06] PMI response: cmd=my_kvsname kvsname=kvs_25318_0_67517667_n06
[proxy:0:0 at n06] got pmi command (from 0): get
kvsname=kvs_25318_0_67517667_n06 key=PMI_process_mapping 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,1))
[proxy:0:0 at n06] got pmi command (from 0): put
kvsname=kvs_25318_0_67517667_n06 key=-bcast-1-0 value=2F6465762F73686D2F6D706963685F736861725F746D7075664F353959 
[proxy:0:0 at n06] cached command: -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D7075664F353959
[proxy:0:0 at n06] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at n06] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0 at n06] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at n06] got pmi command (from 0): barrier_in

[proxy:0:0 at n06] got pmi command (from 6): get_maxes

[proxy:0:0 at n06] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at n06] got pmi command (from 6): get_appnum

[proxy:0:0 at n06] PMI response: cmd=appnum appnum=0
[proxy:0:0 at n06] got pmi command (from 6): get_my_kvsname

[proxy:0:0 at n06] PMI response: cmd=my_kvsname kvsname=kvs_25318_0_67517667_n06
[proxy:0:0 at n06] got pmi command (from 6): get
kvsname=kvs_25318_0_67517667_n06 key=PMI_process_mapping 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,1))
[proxy:0:0 at n06] got pmi command (from 6): barrier_in

[proxy:0:0 at n06] flushing 1 put command(s) out
[mpiexec at n06] [pgid: 0] got PMI command: cmd=put -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D7075664F353959
[proxy:0:0 at n06] forwarding command (cmd=put -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D7075664F353959) upstream
[proxy:0:0 at n06] forwarding command (cmd=barrier_in) upstream
[mpiexec at n06] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec at n06] PMI response to fd 6 pid 6: cmd=keyval_cache -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D7075664F353959 
[mpiexec at n06] PMI response to fd 6 pid 6: cmd=barrier_out
[proxy:0:0 at n06] PMI response: cmd=barrier_out
[proxy:0:0 at n06] PMI response: cmd=barrier_out
[proxy:0:0 at n06] PMI response: cmd=barrier_out
[proxy:0:0 at n06] PMI response: cmd=barrier_out
[proxy:0:0 at n06] got pmi command (from 6): get
kvsname=kvs_25318_0_67517667_n06 key=-bcast-1-0 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F6D706963685F736861725F746D7075664F353959
[proxy:0:0 at n06] got pmi command (from 8): get
kvsname=kvs_25318_0_67517667_n06 key=-bcast-1-0 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F6D706963685F736861725F746D7075664F353959
[proxy:0:0 at n06] got pmi command (from 13): get
kvsname=kvs_25318_0_67517667_n06 key=-bcast-1-0 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F6D706963685F736861725F746D7075664F353959
[proxy:0:0 at n06] got pmi command (from 0): put
kvsname=kvs_25318_0_67517667_n06 key=-allgather-shm-1-0 value=020087B3C0A802100000000000000000 
[proxy:0:0 at n06] cached command: -allgather-shm-1-0=020087B3C0A802100000000000000000
[proxy:0:0 at n06] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at n06] got pmi command (from 6): put
kvsname=kvs_25318_0_67517667_n06 key=-allgather-shm-1-1 value=02009D0FC0A802100000000000000000 
[proxy:0:0 at n06] cached command: -allgather-shm-1-1=02009D0FC0A802100000000000000000
[proxy:0:0 at n06] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at n06] got pmi command (from 8): put
kvsname=kvs_25318_0_67517667_n06 key=-allgather-shm-1-2 value=0200ABFFC0A802100000000000000000 
[proxy:0:0 at n06] cached command: -allgather-shm-1-2=0200ABFFC0A802100000000000000000
[proxy:0:0 at n06] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at n06] got pmi command (from 13): put
kvsname=kvs_25318_0_67517667_n06 key=-allgather-shm-1-3 value=0200A64FC0A802100000000000000000 
[proxy:0:0 at n06] cached command: -allgather-shm-1-3=0200A64FC0A802100000000000000000
[proxy:0:0 at n06] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at n06] got pmi command (from 0): barrier_in

[proxy:0:0 at n06] got pmi command (from 6): barrier_in

[proxy:0:0 at n06] got pmi command (from 8): barrier_in

[proxy:0:0 at n06] got pmi command (from 13): barrier_in

[proxy:0:0 at n06] flushing 4 put command(s) out
[proxy:0:0 at n06] forwarding command (cmd=put -allgather-shm-1-0=020087B3C0A802100000000000000000 -allgather-shm-1-1=02009D0FC0A802100000000000000000 -allgather-shm-1-2=0200ABFFC0A802100000000000000000 -allgather-shm-1-3=0200A64FC0A802100000000000000000) upstream
[mpiexec at n06] [pgid: 0] got PMI command: cmd=put -allgather-shm-1-0=020087B3C0A802100000000000000000 -allgather-shm-1-1=02009D0FC0A802100000000000000000 -allgather-shm-1-2=0200ABFFC0A802100000000000000000 -allgather-shm-1-3=0200A64FC0A802100000000000000000
[proxy:0:0 at n06] forwarding command (cmd=barrier_in) upstream
[mpiexec at n06] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec at n06] PMI response to fd 6 pid 13: cmd=keyval_cache -allgather-shm-1-0=020087B3C0A802100000000000000000 -allgather-shm-1-1=02009D0FC0A802100000000000000000 -allgather-shm-1-2=0200ABFFC0A802100000000000000000 -allgather-shm-1-3=0200A64FC0A802100000000000000000 
[mpiexec at n06] PMI response to fd 6 pid 13: cmd=barrier_out
[proxy:0:0 at n06] PMI response: cmd=barrier_out
[proxy:0:0 at n06] PMI response: cmd=barrier_out
[proxy:0:0 at n06] PMI response: cmd=barrier_out
[proxy:0:0 at n06] PMI response: cmd=barrier_out
[proxy:0:0 at n06] got pmi command (from 0): get
kvsname=kvs_25318_0_67517667_n06 key=-allgather-shm-1-0 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=020087B3C0A802100000000000000000
[proxy:0:0 at n06] got pmi command (from 6): get
kvsname=kvs_25318_0_67517667_n06 key=-allgather-shm-1-1 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=02009D0FC0A802100000000000000000
[proxy:0:0 at n06] got pmi command (from 8): get
kvsname=kvs_25318_0_67517667_n06 key=-allgather-shm-1-2 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=0200ABFFC0A802100000000000000000
[proxy:0:0 at n06] got pmi command (from 13): get
kvsname=kvs_25318_0_67517667_n06 key=-allgather-shm-1-3 
[proxy:0:0 at n06] PMI response: cmd=get_result rc=0 msg=success value=0200A64FC0A802100000000000000000
PADRES
Name n06
Name n06
Name n06
Name n06
[proxy:0:0 at n06] got pmi command (from 0): spawn
nprocs=4 execname=./batch5.out totspawns=1 spawnssofar=1 argcnt=0 preput_num=1 preput_key_0=PARENT_ROOT_PORT_NAME preput_val_0=tag#0$connentry#020087B3C0A802100000000000000000$ info_num=1 info_key_0=host info_val_0=n00 endcmd 
[proxy:0:0 at n06] we don't understand this command spawn; forwarding upstream
[mpiexec at n06] [pgid: 0] got PMI command: mcmd=spawn
nprocs=4
execname=./batch5.out
totspawns=1
spawnssofar=1
argcnt=0
preput_num=1
preput_key_0=PARENT_ROOT_PORT_NAME
preput_val_0=tag#0$connentry#020087B3C0A802100000000000000000$
info_num=1
info_key_0=host
info_val_0=n00
endcmd
[mpiexec at n06] Got a control port string of n06:45957

Proxy launch args: /home/martini/MPI/mpich-3.4.1_INST/bin/hydra_pmi_proxy --control-port n06:45957 --debug --rmk user --launcher ssh --demux poll --pgid 1 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 

Arguments being passed to proxy 0:
--version 3.4.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname n00 --global-core-map 0,1,1 --pmi-id-map 0,0 --global-process-count 4 --auto-cleanup 1 --pmi-kvsname kvs_25318_1_247201886_n06 --pmi-spawner-kvsname kvs_25318_0_67517667_n06 --pmi-process-mapping (vector,(0,1,1)) --global-inherited-env 43 'LD_LIBRARY_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'SSH_CONNECTION=192.168.1.1 54318 192.168.1.16 22' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'LANG=es_ES.UTF-8' 'LD_RUN_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1' 'OLDPWD=/home/martini' 'PATH_modshare=/usr/sbin:1:/bin:1:/usr/bin:1:/usr/games:1:/snap/bin:1:/home/martini/MPI/mpich-3.4.1_INST/bin:1:/usr/local/sbin:1:/usr/local/bin:1:/sbin:1:/usr/local/games:1' 'LOADEDMODULES_modshare=/home/martini/MODULES/modulefiles/mpich3.4:1' 'MPI_RUN=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun' 'LD_LIBRARY_PATH_modshare=/home/martini/MPI/mpich-3.4.1_INST/lib:1' 'XDG_SESSION_ID=1858' 'MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl' 'USER=martini' 'ENV=/usr/share/modules/init/profile.sh' 'PWD=/home/martini/tfm/TestSpawn/Threads/PingPong' 'HOME=/home/martini' 'SSH_CLIENT=192.168.1.1 54318 22' 'BASH_ENV=/usr/share/modules/init/bash' 'XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop' '_LMFILES__modshare=/home/martini/MODULES/modulefiles/mpich3.4:1' 'LOADEDMODULES=/home/martini/MODULES/modulefiles/mpich3.4' 'SSH_TTY=/dev/pts/0' 'MAIL=/var/mail/martini' 'MPI_HOME=/home/martini/MPI/mpich-3.4.1_INST' 'TERM=xterm' 'SHELL=/bin/bash' 'MANPATH_modshare=:1:/home/martini/MPI/mpich-3.4.1_INST/share/man/:1' 'LD_RUN_PATH=/home/martini/MPI/mpich-3.4.1_INST/lib' 'SHLVL=1' 'MANPATH=:/home/martini/MPI/mpich-3.4.1_INST/share/man/' 'MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles' 'LOGNAME=martini' 'XDG_RUNTIME_DIR=/run/user/1002' 'MODULEPATH_modshare=/usr/share/modules/$MODULE_VERSION/modulefiles:1:/etc/environment-modules/modules:1:/usr/share/modules/modulefiles:1:/usr/share/modules/versions:1' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/martini/MPI/mpich-3.4.1_INST/bin' '_LMFILES_=/home/martini/MODULES/modulefiles/mpich3.4' 'MODULESHOME=/usr/share/modules' 'LESSOPEN=| /usr/bin/lesspipe %s' 'BASH_FUNC_module%%=() {  _moduleraw "$*" 2>&1
}' 'BASH_FUNC_switchml%%=() {  typeset swfound=1;
 if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
 typeset swname='main';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd.tcl ]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd-compat ]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [ $swfound -eq 0 ]; then
 echo "Switching to Modules $swname version";
 source /usr/share/modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
}' 'BASH_FUNC__moduleraw%%=() {  unset _mlre _mlIFS _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 _mlre="eval ${_mlre}";
 fi;
 eval `${_mlre:-}/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl bash $*`;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlre _mlv _mlrv _mlIFS _mlshdbg;
 return $_mlstatus
}' '_=/home/martini/MPI/mpich-3.4.1_INST/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 4 --exec-local-env 1 'PMI_SPAWNED=1' --exec-wdir /home/martini/tfm/TestSpawn/Threads/PingPong --exec-args 1 ./batch5.out 

[mpiexec at n06] Launch arguments: /usr/bin/ssh -x n00 "/home/martini/MPI/mpich-3.4.1_INST/bin/hydra_pmi_proxy" --control-port n06:45957 --debug --rmk user --launcher ssh --demux poll --pgid 1 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 0 
[mpiexec at n06] PMI response to fd 6 pid 0: cmd=spawn_result rc=0
[proxy:0:0 at n06] we don't understand the response spawn_result; forwarding downstream
[proxy:1:0 at n00] got pmi command (from 9): init
pmi_version=1 pmi_subversion=1 
[proxy:1:0 at n00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:1:0 at n00] got pmi command (from 9): get_maxes

[proxy:1:0 at n00] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:1:0 at n00] got pmi command (from 9): get_appnum

[proxy:1:0 at n00] PMI response: cmd=appnum appnum=0
[proxy:1:0 at n00] got pmi command (from 9): get_my_kvsname

[proxy:1:0 at n00] PMI response: cmd=my_kvsname kvsname=kvs_25318_1_247201886_n06
[proxy:1:0 at n00] got pmi command (from 9): get
kvsname=kvs_25318_1_247201886_n06 key=PMI_process_mapping 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,1))
[proxy:1:0 at n00] got pmi command (from 9): barrier_in

[proxy:1:0 at n00] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1 
[proxy:1:0 at n00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:1:0 at n00] got pmi command (from 4): get_maxes

[proxy:1:0 at n00] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:1:0 at n00] got pmi command (from 4): get_appnum

[proxy:1:0 at n00] PMI response: cmd=appnum appnum=0
[proxy:1:0 at n00] got pmi command (from 4): get_my_kvsname

[proxy:1:0 at n00] PMI response: cmd=my_kvsname kvsname=kvs_25318_1_247201886_n06
[proxy:1:0 at n00] got pmi command (from 4): get
kvsname=kvs_25318_1_247201886_n06 key=PMI_process_mapping 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,1))
[proxy:1:0 at n00] got pmi command (from 4): put
kvsname=kvs_25318_1_247201886_n06 key=-bcast-1-0 value=2F6465762F73686D2F6D706963685F736861725F746D70743631524334 
[proxy:1:0 at n00] cached command: -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D70743631524334
[proxy:1:0 at n00] PMI response: cmd=put_result rc=0 msg=success
[proxy:1:0 at n00] got pmi command (from 4): barrier_in

[proxy:1:0 at n00] got pmi command (from 5): init
pmi_version=1 pmi_subversion=1 
[proxy:1:0 at n00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:1:0 at n00] got pmi command (from 5): get_maxes

[proxy:1:0 at n00] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:1:0 at n00] got pmi command (from 5): get_appnum

[proxy:1:0 at n00] PMI response: cmd=appnum appnum=0
[proxy:1:0 at n00] got pmi command (from 5): get_my_kvsname

[proxy:1:0 at n00] PMI response: cmd=my_kvsname kvsname=kvs_25318_1_247201886_n06
[proxy:1:0 at n00] got pmi command (from 5): get
kvsname=kvs_25318_1_247201886_n06 key=PMI_process_mapping 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,1))
[proxy:1:0 at n00] got pmi command (from 5): barrier_in

[proxy:1:0 at n00] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1 
[proxy:1:0 at n00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:1:0 at n00] got pmi command (from 6): get_maxes

[proxy:1:0 at n00] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:1:0 at n00] got pmi command (from 6): get_appnum

[proxy:1:0 at n00] PMI response: cmd=appnum appnum=0
[proxy:1:0 at n00] got pmi command (from 6): get_my_kvsname

[proxy:1:0 at n00] PMI response: cmd=my_kvsname kvsname=kvs_25318_1_247201886_n06
[proxy:1:0 at n00] got pmi command (from 6): get
kvsname=kvs_25318_1_247201886_n06 key=PMI_process_mapping 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,1))
[mpiexec at n06] [pgid: 1] got PMI command: cmd=put -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D70743631524334
[proxy:1:0 at n00] got pmi command (from 6): barrier_in

[proxy:1:0 at n00] flushing 1 put command(s) out
[proxy:1:0 at n00] forwarding command (cmd=put -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D70743631524334) upstream
[proxy:1:0 at n00] forwarding command (cmd=barrier_in) upstream
[mpiexec at n06] [pgid: 1] got PMI command: cmd=barrier_in
[mpiexec at n06] PMI response to fd 9 pid 6: cmd=keyval_cache PARENT_ROOT_PORT_NAME=tag#0$connentry#020087B3C0A802100000000000000000$ -bcast-1-0=2F6465762F73686D2F6D706963685F736861725F746D70743631524334 
[mpiexec at n06] PMI response to fd 9 pid 6: cmd=barrier_out
[proxy:1:0 at n00] PMI response: cmd=barrier_out
[proxy:1:0 at n00] PMI response: cmd=barrier_out
[proxy:1:0 at n00] PMI response: cmd=barrier_out
[proxy:1:0 at n00] PMI response: cmd=barrier_out
[proxy:1:0 at n00] got pmi command (from 5): get
kvsname=kvs_25318_1_247201886_n06 key=-bcast-1-0 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F6D706963685F736861725F746D70743631524334
[proxy:1:0 at n00] got pmi command (from 6): get
kvsname=kvs_25318_1_247201886_n06 key=-bcast-1-0 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F6D706963685F736861725F746D70743631524334
[proxy:1:0 at n00] got pmi command (from 9): get
kvsname=kvs_25318_1_247201886_n06 key=-bcast-1-0 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F6D706963685F736861725F746D70743631524334
[proxy:1:0 at n00] got pmi command (from 4): put
kvsname=kvs_25318_1_247201886_n06 key=-allgather-shm-1-0 value=02008269C0A8020A0000000000000000 
[proxy:1:0 at n00] cached command: -allgather-shm-1-0=02008269C0A8020A0000000000000000
[proxy:1:0 at n00] PMI response: cmd=put_result rc=0 msg=success
[proxy:1:0 at n00] got pmi command (from 5): put
kvsname=kvs_25318_1_247201886_n06 key=-allgather-shm-1-1 value=02009BB1C0A8020A0000000000000000 
[proxy:1:0 at n00] cached command: -allgather-shm-1-1=02009BB1C0A8020A0000000000000000
[proxy:1:0 at n00] PMI response: cmd=put_result rc=0 msg=success
[proxy:1:0 at n00] got pmi command (from 6): put
kvsname=kvs_25318_1_247201886_n06 key=-allgather-shm-1-2 value=0200848BC0A8020A0000000000000000 
[proxy:1:0 at n00] cached command: -allgather-shm-1-2=0200848BC0A8020A0000000000000000
[proxy:1:0 at n00] PMI response: cmd=put_result rc=0 msg=success
[proxy:1:0 at n00] got pmi command (from 9): put
kvsname=kvs_25318_1_247201886_n06 key=-allgather-shm-1-3 value=02008E1FC0A8020A0000000000000000 
[proxy:1:0 at n00] cached command: -allgather-shm-1-3=02008E1FC0A8020A0000000000000000
[proxy:1:0 at n00] PMI response: cmd=put_result rc=0 msg=success
[proxy:1:0 at n00] got pmi command (from 4): barrier_in

[proxy:1:0 at n00] got pmi command (from 5): barrier_in

[proxy:1:0 at n00] got pmi command (from 6): barrier_in

[mpiexec at n06] [pgid: 1] got PMI command: cmd=put -allgather-shm-1-0=02008269C0A8020A0000000000000000 -allgather-shm-1-1=02009BB1C0A8020A0000000000000000 -allgather-shm-1-2=0200848BC0A8020A0000000000000000 -allgather-shm-1-3=02008E1FC0A8020A0000000000000000
[proxy:1:0 at n00] got pmi command (from 9): barrier_in

[proxy:1:0 at n00] flushing 4 put command(s) out
[proxy:1:0 at n00] forwarding command (cmd=put -allgather-shm-1-0=02008269C0A8020A0000000000000000 -allgather-shm-1-1=02009BB1C0A8020A0000000000000000 -allgather-shm-1-2=0200848BC0A8020A0000000000000000 -allgather-shm-1-3=02008E1FC0A8020A0000000000000000) upstream
[proxy:1:0 at n00] forwarding command (cmd=barrier_in) upstream
[mpiexec at n06] [pgid: 1] got PMI command: cmd=barrier_in
[mpiexec at n06] PMI response to fd 9 pid 9: cmd=keyval_cache -allgather-shm-1-0=02008269C0A8020A0000000000000000 -allgather-shm-1-1=02009BB1C0A8020A0000000000000000 -allgather-shm-1-2=0200848BC0A8020A0000000000000000 -allgather-shm-1-3=02008E1FC0A8020A0000000000000000 
[mpiexec at n06] PMI response to fd 9 pid 9: cmd=barrier_out
[proxy:1:0 at n00] PMI response: cmd=barrier_out
[proxy:1:0 at n00] PMI response: cmd=barrier_out
[proxy:1:0 at n00] PMI response: cmd=barrier_out
[proxy:1:0 at n00] PMI response: cmd=barrier_out
[proxy:1:0 at n00] got pmi command (from 4): get
kvsname=kvs_25318_1_247201886_n06 key=-allgather-shm-1-0 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=02008269C0A8020A0000000000000000
[proxy:1:0 at n00] got pmi command (from 5): get
kvsname=kvs_25318_1_247201886_n06 key=-allgather-shm-1-1 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=02009BB1C0A8020A0000000000000000
[proxy:1:0 at n00] got pmi command (from 6): get
kvsname=kvs_25318_1_247201886_n06 key=-allgather-shm-1-2 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=0200848BC0A8020A0000000000000000
[proxy:1:0 at n00] got pmi command (from 9): get
kvsname=kvs_25318_1_247201886_n06 key=-allgather-shm-1-3 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=02008E1FC0A8020A0000000000000000
[proxy:1:0 at n00] got pmi command (from 4): get
kvsname=kvs_25318_1_247201886_n06 key=PARENT_ROOT_PORT_NAME 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=tag#0$connentry#020087B3C0A802100000000000000000$
[proxy:1:0 at n00] got pmi command (from 6): get
kvsname=kvs_25318_1_247201886_n06 key=PARENT_ROOT_PORT_NAME 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=tag#0$connentry#020087B3C0A802100000000000000000$
[proxy:1:0 at n00] got pmi command (from 9): get
kvsname=kvs_25318_1_247201886_n06 key=PARENT_ROOT_PORT_NAME 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=tag#0$connentry#020087B3C0A802100000000000000000$
[proxy:1:0 at n00] got pmi command (from 5): get
kvsname=kvs_25318_1_247201886_n06 key=PARENT_ROOT_PORT_NAME 
[proxy:1:0 at n00] PMI response: cmd=get_result rc=0 msg=success value=tag#0$connentry#020087B3C0A802100000000000000000$
HIJOS
A 0 con Grp=1
A 2 con Grp=1
A 1 con Grp=1
A 3 con Grp=1
A 0 con Grp=0
A 1 con Grp=0
A 2 con Grp=0
A 3 con Grp=0
[proxy:0:0 at n06] got pmi command (from 0): finalize

[proxy:0:0 at n06] PMI response: cmd=finalize_ack
[proxy:0:0 at n06] got pmi command (from 6): finalize

[proxy:0:0 at n06] PMI response: cmd=finalize_ack
[proxy:0:0 at n06] got pmi command (from 8): finalize

[proxy:0:0 at n06] PMI response: cmd=finalize_ack
[proxy:1:0 at n00] got pmi command (from 4): finalize

[proxy:1:0 at n00] PMI response: cmd=finalize_ack
[proxy:0:0 at n06] got pmi command (from 13): finalize

[proxy:0:0 at n06] PMI response: cmd=finalize_ack
[proxy:1:0 at n00] got pmi command (from 5): finalize

[proxy:1:0 at n00] PMI response: cmd=finalize_ack
B 0 con Grp=0
[proxy:1:0 at n00] got pmi command (from 6): finalize

[proxy:1:0 at n00] PMI response: cmd=finalize_ack
B 3 con Grp=0
[proxy:1:0 at n00] got pmi command (from 9): finalize

[proxy:1:0 at n00] PMI response: cmd=finalize_ack
B 1 con Grp=0
B 0 con Grp=1
B 1 con Grp=1
B 2 con Grp=0
B 2 con Grp=1
B 3 con Grp=1


More information about the discuss mailing list