<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Dear MPICH users,</div>
<div class=""><br class="">
</div>
<div class="">I'm having an issue with, apparently, communication between nodes of the local cluster we're using. To fix ideas, the cluster is made of 8 different nodes and it was set-up with now outdated versions of the Intel compilers and MPI libraries. I
did an mpbboot/mpdringtest and it seems to work fine.</div>
<div class=""><br class="">
</div>
<div class="">Now, I would like to use my application code with MPICH 3.3 installed from PETSc and with gcc-8 installed from linuxbrew on the different nodes.</div>
<div class=""><br class="">
</div>
<div class="">I tested a simple Fortran program:</div>
<div class=""><br class="">
</div>
<div class=""><font face="Courier" class="">program test</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class=""> use mpi</font></div>
<div class=""><font face="Courier" class=""> use ISO_fortran_env, only: output_unit</font></div>
<div class=""><font face="Courier" class=""> </font></div>
<div class=""><font face="Courier" class=""> implicit none</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class=""> integer :: irank, nproc, ierr</font></div>
<div class=""><font face="Courier" class=""> character(len=80) :: hostname</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class=""> call MPI_INIT(ierr)</font></div>
<div class=""><font face="Courier" class=""> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,ierr)</font></div>
<div class=""><font face="Courier" class=""> call MPI_COMM_RANK(MPI_COMM_WORLD,irank,ierr)</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class=""> if (irank == 0) THEN</font></div>
<div class=""><font face="Courier" class=""> WRITE(output_unit,'(1X,A,I3,A)') 'Started test program with', nproc, ' MPI processes'</font></div>
<div class=""><font face="Courier" class=""> end if</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class=""> call MPI_BARRIER(MPI_COMM_WORLD,ierr) </font></div>
<div class=""><br class="">
</div>
<div class=""><font face="Courier" class=""> call HOSTNM(hostname,ierr)</font></div>
<div class=""><font face="Courier" class=""> WRITE(output_unit,'(1X,A,I3,A)') 'I am processor #', irank, ' running on '//hostname</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class=""> call MPI_FINALIZE(ierr)</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class="">end program test</font></div>
<div class=""><br class="">
</div>
<div class="">It works fine on the local host. But when I try to launch it on all the nodes with<font face="Courier" class=""> "mpiexec -f path_to_hostfile -n 8 path_to_my_program"</font> I get:</div>
<div class=""><br class="">
</div>
<div class=""><font face="Courier" class=""> Started test program with 8 MPI processes</font></div>
<div class=""><font face="Courier" class="">Fatal error in PMPI_Barrier: Unknown error class, error stack:</font></div>
<div class=""><font face="Courier" class="">PMPI_Barrier(287)...........................: MPI_Barrier(MPI_COMM_WORLD) failed</font></div>
<div class=""><font face="Courier" class="">PMPI_Barrier(273)...........................: </font></div>
<div class=""><font face="Courier" class="">MPIR_Barrier_impl(173)......................: </font></div>
<div class=""><font face="Courier" class="">MPIR_Barrier_intra_auto(108)................: </font></div>
<div class=""><font face="Courier" class="">MPIR_Barrier_intra_recursive_doubling(47)...: </font></div>
<div class=""><font face="Courier" class="">MPIC_Sendrecv(347)..........................: </font></div>
<div class=""><font face="Courier" class="">MPIC_Wait(73)...............................: </font></div>
<div class=""><font face="Courier" class="">MPIDI_CH3i_Progress_wait(242)...............: an error occurred while handling an event returned by MPIDI_CH3I_Sock_Wait()</font></div>
<div class=""><font face="Courier" class="">MPIDI_CH3I_Progress_handle_sock_event(698)..: </font></div>
<div class=""><font face="Courier" class="">MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to connnect to remote process</font></div>
<div class=""><font face="Courier" class="">MPIDI_CH3I_Sock_post_connect_ifaddr(1774)...: unexpected operating system error (set=0,sock=5,errno=101:Network is unreachable)</font></div>
<div class=""><br class="">
</div>
<div class="">Note that when I comment the call to MPI_BARRIER, it works fine as well. Therefore, communication between nodes seems to be an issue. All the nodes see and have access to the same mpiexec/mpifort executables: I checked $PATH and 'which mpiexec'/'which
mpifort'.</div>
<div class=""> </div>
<div class="">Could you give me ways for an efficient diagnostic of what's possibly wrong?</div>
<div class=""><br class="">
</div>
<div class=""><font face="Courier" class="">mpiexec --version </font>gives</div>
<div class=""><br class="">
</div>
<div class=""><font face="Courier" class="">HYDRA build details:</font></div>
<div class=""><font face="Courier" class=""> Version: 3.3b1</font></div>
<div class=""><font face="Courier" class=""> Release Date: Mon Feb 5 10:16:15 CST 2018</font></div>
<div class=""><font face="Courier" class=""> CC: gcc-8 -fPIC -fstack-protector -O3 -march=native </font></div>
<div class=""><font face="Courier" class=""> CXX: g++-8 -fstack-protector -O3 -march=native -fPIC </font></div>
<div class=""><font face="Courier" class=""> F77: gfortran-8 -fPIC -ffree-line-length-0 -O3 -march=native </font></div>
<div class=""><font face="Courier" class=""> F90: gfortran-8 -fPIC -ffree-line-length-0 -O3 -march=native </font></div>
<div class=""><font face="Courier" class=""> Configure options: '--disable-option-checking' '--prefix=/home/petsc/icm_cplx' 'MAKE=/usr/bin/make' '--libdir=/home/petsc/icm_cplx/lib' 'CC=gcc-8' 'CFLAGS=-fPIC -fstack-protector -O3 -march=native
-O2' 'AR=/usr/bin/ar' 'ARFLAGS=cr' 'CXX=g++-8' 'CXXFLAGS=-fstack-protector -O3 -march=native -fPIC -O2' 'F77=gfortran-8' 'FFLAGS=-fPIC -ffree-line-length-0 -O3 -march=native -O2' 'FC=gfortran-8' 'FCFLAGS=-fPIC -ffree-line-length-0 -O3 -march=native -O2' '--enable-shared'
'--with-device=ch3:sock' '--with-pm=hydra' '--enable-g=meminit' '--cache-file=/dev/null' '--srcdir=.' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/mpl/include -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/mpl/include
-I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/openpa/src -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/openpa/src -D_REENTRANT -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/mpi/romio/include' 'MPLLIBNAME=mpl'</font></div>
<div class=""><font face="Courier" class=""> Process Manager: pmi</font></div>
<div class=""><font face="Courier" class=""> Launchers available: ssh rsh fork slurm ll lsf sge manual persist</font></div>
<div class=""><font face="Courier" class=""> Topology libraries available: hwloc</font></div>
<div class=""><font face="Courier" class=""> Resource management kernels available: user slurm ll lsf sge pbs cobalt</font></div>
<div class=""><font face="Courier" class=""> Checkpointing libraries available: </font></div>
<div class=""><font face="Courier" class=""> Demux engines available: poll select</font></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Thank you,</div>
<div class=""><br class="">
</div>
<div class="">Thibaut</div>
</body>
</html>