[mpich-discuss] All processes only find themselfs

Wed Jul 3 14:10:27 CDT 2024

I believe there has been an error in packaging, possibly resulting from a change in configure options, that has rendered the MPICH library incompatible with Hydra's mpiexec. ☹

I have no good solution besides building your own MPICH from source. I will reach out to the Ubuntu maintainer about the issue. Thanks for reporting.

Ken

-----Original Message-----
From: Joachim Jenke <jenke at itc.rwth-aachen.de <mailto:jenke at itc.rwth-aachen.de>>
Date: Wednesday, July 3, 2024 at 12:20 PM
To: "Raffenetti, Ken" <raffenet at anl.gov <mailto:raffenet at anl.gov>>, "discuss at mpich.org <mailto:discuss at mpich.org>" <discuss at mpich.org <mailto:discuss at mpich.org>>
Subject: Re: [mpich-discuss] All processes only find themselfs

$ env MPIR_CVAR_PMI_VERSION=1 mpirun -np 2 -l env 
MPIR_CVAR_PMI_VERSION=1 ./a.out
[0] Hello, I am 0 of 1 tasks.
[1] Hello, I am 0 of 1 tasks.

still the same. I was not sure whether that option is for mpiexec or the 
mpi library, so I set it twice just to be sure.

Am 03.07.24 um 16:54 schrieb Raffenetti, Ken:
> Hm, there is a PMIx dependency that is presumably coming from the MPICH library. MPICH's mpiexec is not compatible with an external PMIx, so I suspect that is the issue. Can you try setting MPIR_CVAR_PMI_VERSION=1 and see if that works?
> 
> Ken
> 
> -----Original Message-----
> From: Joachim Jenke <jenke at itc.rwth-aachen.de <mailto:jenke at itc.rwth-aachen.de> <mailto:jenke at itc.rwth-aachen.de <mailto:jenke at itc.rwth-aachen.de>>>
> Date: Wednesday, July 3, 2024 at 9:36 AM
> To: "Raffenetti, Ken" <raffenet at anl.gov <mailto:raffenet at anl.gov> <mailto:raffenet at anl.gov <mailto:raffenet at anl.gov>>>, "discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>>" <discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>>>
> Subject: Re: [mpich-discuss] All processes only find themselfs
> 
> 
> $ ldd a.out
> linux-vdso.so.1 (0x00007ffee8145000)
> libmpich.so.12 => /lib/x86_64-linux-gnu/libmpich.so.12 (0x00007f15d3200000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f15d2e00000)
> libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007f15d5a13000)
> libpmix.so.2 => /lib/x86_64-linux-gnu/libpmix.so.2 (0x00007f15d2a00000)
> libucp.so.0 => /lib/x86_64-linux-gnu/libucp.so.0 (0x00007f15d309d000)
> libucs.so.0 => /lib/x86_64-linux-gnu/libucs.so.0 (0x00007f15d59ab000)
> /lib64/ld-linux-x86-64.so.2 (0x00007f15d5aa3000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f15d2d17000)
> libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f15d5978000)
> libmunge.so.2 => /lib/x86_64-linux-gnu/libmunge.so.2 (0x00007f15d5970000)
> libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7
> (0x00007f15d3068000)
> libevent_pthreads-2.1.so.7 =>
> /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f15d5969000)
> libuct.so.0 => /lib/x86_64-linux-gnu/libuct.so.0 (0x00007f15d3022000)
> libucm.so.0 => /lib/x86_64-linux-gnu/libucm.so.0 (0x00007f15d594c000)
> libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f15d3015000)
> 
> 
> $ apt-cache policy libmpich-dev
> libmpich-dev:
> Installed: 4.2.0-5build3
> Candidate: 4.2.0-5build3
> Version table:
> *** 4.2.0-5build3 500
> 500 https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42Oko3kzpT1$  <https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42Oko3kzpT1$ > <https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42Oko3kzpT1$ > <https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu>__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkjdqNFpx$ > noble/universe amd64
> Packages
> 100 /var/lib/dpkg/status
> 
> 
> 
> 
> 
> 
> $ mpirun -np 2 --verbose ./a.out
> host: ITC19206
> [mpiexec at ITC19206] Timeout set to -1 (-1 means infinite)
> 
> 
> ==================================================================================================
> mpiexec options:
> ----------------
> Base path: /usr/bin/
> Launcher: (null)
> Debug level: 1
> Enable X: -1
> 
> 
> Global environment:
> -------------------
> SHELL=/bin/bash
> 
> 
> SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236
> QT_ACCESSIBILITY=1
> COLORTERM=truecolor
> XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg
> XDG_MENU_PREFIX=gnome-
> GNOME_DESKTOP_SESSION_ID=this-is-deprecated
> MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
> GNOME_SHELL_SESSION_MODE=ubuntu
> SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
> MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
> XMODIFIERS=@im=ibus
> DESKTOP_SESSION=ubuntu
> GTK_MODULES=gail:atk-bridge
> XDG_SESSION_DESKTOP=ubuntu
> XDG_SESSION_TYPE=x11
> MODULESHOME=/usr/share/modules
> GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1
> SYSTEMD_EXEC_PID=3274
> XAUTHORITY=/run/user/1000/gdm/Xauthority
> WINDOWPATH=2
> LANG=de_DE.UTF-8
> 
> 
> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
> XDG_CURRENT_DESKTOP=ubuntu:GNOME
> 
> 
> MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user at 1000.serv <mailto:user at 1000.serv> <mailto:user at 1000.serv <mailto:user at 1000.serv>>ice/session.slice/org.gnome.Shell at x11.serv <mailto:org.gnome.Shell at x11.serv> <mailto:org.gnome.Shell at x11.serv <mailto:org.gnome.Shell at x11.serv>>ice/memory.pressure
> VTE_VERSION=7600
> 
> 
> GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc
> QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/
> LESSCLOSE=/usr/bin/lesspipe %s %s
> XDG_SESSION_CLASS=user
> TERM=xterm-256color
> DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
> LESSOPEN=| /usr/bin/lesspipe %s
> GNOME_TERMINAL_SERVICE=:1.95
> LOADEDMODULES=
> SHLVL=1
> GSM_SKIP_SSH_AGENT_WORKAROUND=true
> QT_IM_MODULE=ibus
> XDG_RUNTIME_DIR=/run/user/1000
> __MODULES_LMINIT=module use --append
> /etc/environment-modules/modules:module use --append
> /usr/share/modules/versions:module use --append
> /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append
> /usr/share/modules/modulefiles
> DEBUGINFOD_URLS=https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkpkcRunx$  <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkpkcRunx$ > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkpkcRunx$ > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com>__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkrjhIqm_$ >
> 
> 
> XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
> 
> 
> PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
> 
> 
> MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles
> GDMSESSION=ubuntu
> DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
> MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl
> BASH_FUNC_ml%%=() { module ml "$@"
> }
> BASH_FUNC_module%%=() { local _mlredir=1;
> if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then
> if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then
> _mlredir=0;
> else
> if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then
> _mlredir=1;
> fi;
> fi;
> fi;
> case " $@ " in
> *' --no-redirect '*)
> _mlredir=0
> ;;
> *' --redirect '*)
> _mlredir=1
> ;;
> esac;
> if [ $_mlredir -eq 0 ]; then
> _module_raw "$@";
> else
> _module_raw "$@" 2>&1;
> fi
> }
> BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6
> '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")";
> _mlstatus=$?;
> return $_mlstatus
> }
> _=/usr/bin/mpirun
> 
> 
> Hydra internal environment:
> ---------------------------
> GFORTRAN_UNBUFFERED_PRECONNECTED=y
> 
> 
> 
> 
> Proxy information:
> *********************
> [1] proxy: ITC19206 (1 cores)
> Exec list: ./a.out (2 processes);
> 
> 
> 
> 
> ==================================================================================================
> 
> 
> 
> 
> Proxy launch args: /usr/bin/hydra_pmi_proxy --control-port
> ITC19206:39745 --debug --rmk user --launcher ssh --demux poll --pgid 0
> --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2
> --gpu-subdevs-per-proc -2 --proxy-id
> 
> 
> Arguments being passed to proxy 0:
> --version 4.2.0 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME
> --hostname ITC19206 --global-core-map 0,1,1 --pmi-id-map 0,0
> --global-process-count 2 --auto-cleanup 1 --pmi-kvsname
> kvs_718180_0_155951620_ITC19206 --pmi-process-mapping (vector,(0,1,2))
> --global-inherited-env 57 'SHELL=/bin/bash'
> 'SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236'
> 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor'
> 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-'
> 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated'
> 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path'
> 'GNOME_SHELL_SESSION_MODE=ubuntu'
> 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh'
> 'MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA='
> 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu'
> 'GTK_MODULES=gail:atk-bridge' 'XDG_SESSION_DESKTOP=ubuntu'
> 'XDG_SESSION_TYPE=x11' 'MODULESHOME=/usr/share/modules'
> 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1'
> 'SYSTEMD_EXEC_PID=3274' 'XAUTHORITY=/run/user/1000/gdm/Xauthority'
> 'WINDOWPATH=2' 'LANG=de_DE.UTF-8'
> 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:'
> 'XDG_CURRENT_DESKTOP=ubuntu:GNOME'
> 'MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user at 1000.serv <mailto:user at 1000.serv> <mailto:user at 1000.serv <mailto:user at 1000.serv>>ice/session.slice/org.gnome.Shell at x11.serv <mailto:org.gnome.Shell at x11.serv> <mailto:org.gnome.Shell at x11.serv <mailto:org.gnome.Shell at x11.serv>>ice/memory.pressure'
> 'VTE_VERSION=7600'
> 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc'
> 'QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/'
> 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user'
> 'TERM=xterm-256color'
> 'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path' 'LESSOPEN=|
> /usr/bin/lesspipe %s' 'GNOME_TERMINAL_SERVICE=:1.95' 'LOADEDMODULES='
> 'SHLVL=1' 'GSM_SKIP_SSH_AGENT_WORKAROUND=true' 'QT_IM_MODULE=ibus'
> 'XDG_RUNTIME_DIR=/run/user/1000' '__MODULES_LMINIT=module use --append
> /etc/environment-modules/modules:module use --append
> /usr/share/modules/versions:module use --append
> /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append
> /usr/share/modules/modulefiles'
> 'DEBUGINFOD_URLS=https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkpkcRunx$  <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkpkcRunx$ > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkpkcRunx$ > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com>__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkrjhIqm_$ > '
> 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop'
> 'PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin'
> 'MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles'
> 'GDMSESSION=ubuntu'
> 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus'
> 'MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl' 'BASH_FUNC_ml%%=()
> { module ml "$@"
> }' 'BASH_FUNC_module%%=() { local _mlredir=1;
> if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then
> if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then
> _mlredir=0;
> else
> if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then
> _mlredir=1;
> fi;
> fi;
> fi;
> case " $@ " in
> *' --no-redirect '*)
> _mlredir=0
> ;;
> *' --redirect '*)
> _mlredir=1
> ;;
> esac;
> if [ $_mlredir -eq 0 ]; then
> _module_raw "$@";
> else
> _module_raw "$@" 2>&1;
> fi
> }' 'BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6
> '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")";
> _mlstatus=$?;
> return $_mlstatus
> }' '_=/usr/bin/mpirun' --global-user-env 0 --global-system-env 1
> 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir [...]
> --exec-args 1 ./a.out
> 
> 
> [mpiexec at ITC19206] Launch arguments: /usr/bin/hydra_pmi_proxy
> --control-port ITC19206:39745 --debug --rmk user --launcher ssh --demux
> poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2
> --gpu-subdevs-per-proc -2 --proxy-id 0
> [proxy:0 at ITC19206] Sending upstream hdr.cmd = CMD_PID_LIST
> [proxy:0 at ITC19206] Sending upstream hdr.cmd = CMD_STDOUT
> [proxy:0 at ITC19206] Sending upstream hdr.cmd = CMD_STDOUT
> Hello, I am 0 of 1 tasks.
> Hello, I am 0 of 1 tasks.
> [proxy:0 at ITC19206] Sending upstream hdr.cmd = CMD_EXIT_STATUS
> 
> 
> 
> 
> 
> 
> Am 03.07.24 um 16:25 schrieb Raffenetti, Ken:
>> Can you confirm which MPI library you binary is linked with using ldd? Adding -v to the mpirun command should also provide some clues in the PMI client/server traffic.
>>
>> Ken
>>
>> -----Original Message-----
>> From: Joachim Jenke via discuss <discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>> <mailto:discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>>>>
>> Reply-To: "discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>> <mailto:discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>>>" <discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>> <mailto:discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>>>>
>> Date: Wednesday, July 3, 2024 at 5:16 AM
>> To: "discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>> <mailto:discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>>>" <discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>> <mailto:discuss at mpich.org <mailto:discuss at mpich.org> <mailto:discuss at mpich.org <mailto:discuss at mpich.org>>>>
>> Cc: Joachim Jenke <jenke at itc.rwth-aachen.de <mailto:jenke at itc.rwth-aachen.de> <mailto:jenke at itc.rwth-aachen.de <mailto:jenke at itc.rwth-aachen.de>> <mailto:jenke at itc.rwth-aachen.de <mailto:jenke at itc.rwth-aachen.de> <mailto:jenke at itc.rwth-aachen.de <mailto:jenke at itc.rwth-aachen.de>>>>
>> Subject: [mpich-discuss] All processes only find themselfs
>>
>>
>> Hi all,
>>
>>
>> after upgrading my notebook to Ubuntu 24.04, each processes thinks that
>> MPI_COMM_WORLD only consists of its own process:
>>
>>
>> $ mpirun -np 2 -l ./a.out
>> [1] Hello, I am 0 of 1 tasks.
>> [0] Hello, I am 0 of 1 tasks.
>>
>>
>> Any suggestions what could go wrong?
>>
>>
>>
>>
>> $ mpirun --version
>> HYDRA build details:
>> Version: 4.2.0
>> Release Date: Fri Feb 9 12:29:21 CST 2024
>> CC: gcc -Wdate-time
>> -D_FORTIFY_SOURCE=3 -g -O2 -fno-omit-frame-pointer
>> -mno-omit-leaf-frame-pointer
>> -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto
>> -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection
>> -Wformat -Werror=format-security -fcf-protection
>> -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3
>> -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro
>> Configure options: '--with-hwloc-prefix=/usr'
>> '--with-device=ch4:ucx' 'FFLAGS=-O2 -fno-omit-frame-pointer
>> -mno-omit-leaf-frame-pointer
>> -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto
>> -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection
>> -fcf-protection
>> -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3
>> -fallow-invalid-boz -fallow-argument-mismatch' '--prefix=/usr'
>> 'CFLAGS=-g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer
>> -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto
>> -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection
>> -Wformat -Werror=format-security -fcf-protection
>> -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3'
>> 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects
>> -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=3'
>> Process Manager: pmi
>> Launchers available: ssh rsh fork slurm ll lsf
>> sge manual persist
>> Topology libraries available: hwloc
>> Resource management kernels available: user slurm ll lsf sge pbs
>> cobalt
>> Demux engines available: poll select
>>
>>
>> Best
>> Joachim
>>
>>
> 
> 

-- 
Dr. rer. nat. Joachim Jenke

IT Center
Group: High Performance Computing
Division: Computational Science and Engineering
RWTH Aachen University
Seffenter Weg 23
D 52074 Aachen (Germany)
Tel: +49 241 80- 24765
Fax: +49 241 80-624765
jenke at itc.rwth-aachen.de <mailto:jenke at itc.rwth-aachen.de>
https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42Okjv_YRzx$ 