<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Thibaut,<br>
<br>
In order to help us isolate this problem, could you please try the following options:<br>
<br>
- Use MPICH 3.3rc instead of MPICH 3.3b1<br>
- Use ch3:tcp instead of ch3:sock (tcp is used by default, so please just delete `--with-device=ch3:sock` when you configure)<br>
- Try `mpiexec -f path_to_hostfile -n 8 hostname`<br>
<br>
Best regards,<br>
Min<br>
<div class="moz-cite-prefix">On 2018/11/13 7:30, Appel, Thibaut via discuss wrote:<br>
</div>
<blockquote type="cite" cite="mid:96C3FCF8-FCFA-4D7F-BEC6-967A565EBEB3@ic.ac.uk">
<div class="">Dear MPICH users,</div>
<div class=""><br class="">
</div>
<div class="">I'm having an issue with, apparently, communication between nodes of the local cluster we're using. To fix ideas, the cluster is made of 8 different nodes and it was set-up with now outdated versions of the Intel compilers and MPI libraries. I
did an mpbboot/mpdringtest and it seems to work fine.</div>
<div class=""><br class="">
</div>
<div class="">Now, I would like to use my application code with MPICH 3.3 installed from PETSc and with gcc-8 installed from linuxbrew on the different nodes.</div>
<div class=""><br class="">
</div>
<div class="">I tested a simple Fortran program:</div>
<div class=""><br class="">
</div>
<div class=""><font class="" face="Courier">program test</font></div>
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class=""><font class="" face="Courier"> use mpi</font></div>
<div class=""><font class="" face="Courier"> use ISO_fortran_env, only: output_unit</font></div>
<div class=""><font class="" face="Courier"> </font></div>
<div class=""><font class="" face="Courier"> implicit none</font></div>
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class=""><font class="" face="Courier"> integer :: irank, nproc, ierr</font></div>
<div class=""><font class="" face="Courier"> character(len=80) :: hostname</font></div>
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class=""><font class="" face="Courier"> call MPI_INIT(ierr)</font></div>
<div class=""><font class="" face="Courier"> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,ierr)</font></div>
<div class=""><font class="" face="Courier"> call MPI_COMM_RANK(MPI_COMM_WORLD,irank,ierr)</font></div>
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class=""><font class="" face="Courier"> if (irank == 0) THEN</font></div>
<div class=""><font class="" face="Courier"> WRITE(output_unit,'(1X,A,I3,A)') 'Started test program with', nproc, ' MPI processes'</font></div>
<div class=""><font class="" face="Courier"> end if</font></div>
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class=""><font class="" face="Courier"> call MPI_BARRIER(MPI_COMM_WORLD,ierr) </font></div>
<div class=""><br class="">
</div>
<div class=""><font class="" face="Courier"> call HOSTNM(hostname,ierr)</font></div>
<div class=""><font class="" face="Courier"> WRITE(output_unit,'(1X,A,I3,A)') 'I am processor #', irank, ' running on '//hostname</font></div>
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class=""><font class="" face="Courier"> call MPI_FINALIZE(ierr)</font></div>
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class=""><font class="" face="Courier">end program test</font></div>
<div class=""><br class="">
</div>
<div class="">It works fine on the local host. But when I try to launch it on all the nodes with<font class="" face="Courier"> "mpiexec -f path_to_hostfile -n 8 path_to_my_program"</font> I get:</div>
<div class=""><br class="">
</div>
<div class=""><font class="" face="Courier"> Started test program with 8 MPI processes</font></div>
<div class=""><font class="" face="Courier">Fatal error in PMPI_Barrier: Unknown error class, error stack:</font></div>
<div class=""><font class="" face="Courier">PMPI_Barrier(287)...........................: MPI_Barrier(MPI_COMM_WORLD) failed</font></div>
<div class=""><font class="" face="Courier">PMPI_Barrier(273)...........................: </font></div>
<div class=""><font class="" face="Courier">MPIR_Barrier_impl(173)......................: </font></div>
<div class=""><font class="" face="Courier">MPIR_Barrier_intra_auto(108)................: </font></div>
<div class=""><font class="" face="Courier">MPIR_Barrier_intra_recursive_doubling(47)...: </font></div>
<div class=""><font class="" face="Courier">MPIC_Sendrecv(347)..........................: </font></div>
<div class=""><font class="" face="Courier">MPIC_Wait(73)...............................: </font></div>
<div class=""><font class="" face="Courier">MPIDI_CH3i_Progress_wait(242)...............: an error occurred while handling an event returned by MPIDI_CH3I_Sock_Wait()</font></div>
<div class=""><font class="" face="Courier">MPIDI_CH3I_Progress_handle_sock_event(698)..: </font></div>
<div class=""><font class="" face="Courier">MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to connnect to remote process</font></div>
<div class=""><font class="" face="Courier">MPIDI_CH3I_Sock_post_connect_ifaddr(1774)...: unexpected operating system error (set=0,sock=5,errno=101:Network is unreachable)</font></div>
<div class=""><br class="">
</div>
<div class="">Note that when I comment the call to MPI_BARRIER, it works fine as well. Therefore, communication between nodes seems to be an issue. All the nodes see and have access to the same mpiexec/mpifort executables: I checked $PATH and 'which mpiexec'/'which
mpifort'.</div>
<div class=""> </div>
<div class="">Could you give me ways for an efficient diagnostic of what's possibly wrong?</div>
<div class=""><br class="">
</div>
<div class=""><font class="" face="Courier">mpiexec --version </font>gives</div>
<div class=""><br class="">
</div>
<div class=""><font class="" face="Courier">HYDRA build details:</font></div>
<div class=""><font class="" face="Courier"> Version: 3.3b1</font></div>
<div class=""><font class="" face="Courier"> Release Date: Mon Feb 5 10:16:15 CST 2018</font></div>
<div class=""><font class="" face="Courier"> CC: gcc-8 -fPIC -fstack-protector -O3 -march=native </font></div>
<div class=""><font class="" face="Courier"> CXX: g++-8 -fstack-protector -O3 -march=native -fPIC </font></div>
<div class=""><font class="" face="Courier"> F77: gfortran-8 -fPIC -ffree-line-length-0 -O3 -march=native </font></div>
<div class=""><font class="" face="Courier"> F90: gfortran-8 -fPIC -ffree-line-length-0 -O3 -march=native </font></div>
<div class=""><font class="" face="Courier"> Configure options: '--disable-option-checking' '--prefix=/home/petsc/icm_cplx' 'MAKE=/usr/bin/make' '--libdir=/home/petsc/icm_cplx/lib' 'CC=gcc-8' 'CFLAGS=-fPIC -fstack-protector -O3 -march=native
-O2' 'AR=/usr/bin/ar' 'ARFLAGS=cr' 'CXX=g++-8' 'CXXFLAGS=-fstack-protector -O3 -march=native -fPIC -O2' 'F77=gfortran-8' 'FFLAGS=-fPIC -ffree-line-length-0 -O3 -march=native -O2' 'FC=gfortran-8' 'FCFLAGS=-fPIC -ffree-line-length-0 -O3 -march=native -O2' '--enable-shared'
'--with-device=ch3:sock' '--with-pm=hydra' '--enable-g=meminit' '--cache-file=/dev/null' '--srcdir=.' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/mpl/include -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/mpl/include
-I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/openpa/src -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/openpa/src -D_REENTRANT -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/mpi/romio/include' 'MPLLIBNAME=mpl'</font></div>
<div class=""><font class="" face="Courier"> Process Manager: pmi</font></div>
<div class=""><font class="" face="Courier"> Launchers available: ssh rsh fork slurm ll lsf sge manual persist</font></div>
<div class=""><font class="" face="Courier"> Topology libraries available: hwloc</font></div>
<div class=""><font class="" face="Courier"> Resource management kernels available: user slurm ll lsf sge pbs cobalt</font></div>
<div class=""><font class="" face="Courier"> Checkpointing libraries available: </font></div>
<div class=""><font class="" face="Courier"> Demux engines available: poll select</font></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Thank you,</div>
<div class=""><br class="">
</div>
<div class="">Thibaut</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset> <br>
<pre wrap="">_______________________________________________
discuss mailing list <a class="moz-txt-link-abbreviated" href="mailto:discuss@mpich.org">discuss@mpich.org</a>
To manage subscription options or unsubscribe:
<a class="moz-txt-link-freetext" href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a>
</pre>
</blockquote>
<br>
</body>
</html>