[mpich-discuss] Excessive disk activity in MPI processes

[Mccall, Kurt E. (MSFC-EV41)]

Sorry, I should have mentioned that our file system is NFS hosted on the cluster head node, with infiniband from the compute nodes to the head.

******

I am attempting to run an MPI Monte Carlo: multiple instances of a simulation "Mav" under the control of MPI.    Mav does a huge amount of output to disk, but the MPI processes that oversee the Mav instances are lightweight and only wake up every 5 seconds.

The MPI Monte Carlo is much slower than running each Mav as an independent PBS Torque job, and according to "iotop", the MPI-supervised Mav instances regularly become blocked waiting for I/O.   iotop often shows the Mavs's spending 99% of their time waiting for I/O.   The independent Torque jobs don't do that.

I got some significant improvement in the MPI performance by telling qsub not to save the stdout and stderr output (qusb -k n) to a file.  My question is: are there any other optional types of disk activity performed by MPI that I can disable?  What else could be happening here?

I'm using MPICH 3.3.2 compiled with pgc++,   PBS Torque 5.1.1,   CentOS 3.10.0.

Here are my MPICH configure options:

'--disable-option-checking' '--prefix=/opt/mpich_pgc' '--with-pbs=/opt/torque' 'CC=pgcc' 'CXX=pgc++' '-enable-debuginfo' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/openpa/src -I/home/kmccall/mpich-3.3.2/src/openpa/src -D_REENTRANT -I/home/kmccall/mpich-3.3.2/src/mpi/romio/include' 'MPLLIBNAME=mpl'

Thanks,
Kurt


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20200602/1372ffb4/attachment.html>