<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText">Hi Ken,<br>
<br>
I tried again to create a minimal demonstration, and I think I now have such an example. Code attached.<br>
<br>
<br>
Notes:<br>
<br>
- Each MPI rank creates its own log file to print results to, for easy examination of data from the different processes.<br>
<br>
- The “BufferExchange” object is designed for easy sharing of arbitrary data buffers between a specified set of peer ranks.<br>
<br>
- I added memory check code around various aspects of the buffer exchanges ( see BufferExchange::FullExchange() ), and I print the memory changes at the end of each FullExchange() call (these are in KB on OSX).<br>
<br>
- The most interesting value is “final_waitall”, the measured resident set memory increase due to calling the final MPI_Waitall() ( see BufferExchange::FullExchange() ).<br>
<br>
- I tested both fixed-size buffers, and variable-sized buffers (either 100 integers per buffer, or buffers of the same size as the loop iteration number)<br>
<br>
- The memory increases seem to be nondeterministic: running the same program several times gives me different values. I think this might suggest it’s due to the OS.<br>
<br>
<br>
I built and ran the example using:<br>
<br>
mpicxx -Wall -Wextra base.cpp BufferExchange.cpp main.cpp<br>
<br>
mpiexec -np 4 ./a.out<br>
<br>
… and the output files were therefore “log.0” to “log.3”.<br>
<br>
Apologies for any obvious bugs in the code</div>
</span></font></div>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText">, it was something of a rush job.<br>
<br>
Cheers,<br>
<br>
J.<br>
<br>
<br>
On Jan 2, 2014, at 9:53 PM, Kenneth Raffenetti <raffenet@mcs.anl.gov> wrote:<br>
<br>
> Can you provide a minimal demonstration of this behavior? That way we can work from the same reference point to identify possible causes.<br>
> <br>
> On 01/01/2014 03:05 PM, John Grime wrote:<br>
>> Hi all,<br>
>> <br>
>> I was wondering if someone could help me out with trying to understand what’s gong on with an MPI_Waitall() call with regards to resident set memory usage. Platform information at the end of the email.<br>
>> <br>
>> If I run a 4 process MPI job on my desktop machine, I get a steady increase in memory usage over the course of the program. Apart from the places where I would reasonably expect expect memory usage to increase, I also see regular increases when I call MPI_Waitall().<br>
>> <br>
>> I measure this behavior via:<br>
>> <br>
>> getmem( &rss0, &vs );<br>
>> <br>
>> result = MPI_Waitall( n_buffer_reqs, buffer_reqs, MPI_STATUSES_IGNORE );<br>
>> if( result != MPI_SUCCESS ) ERROR_MACRO( "MPI_Waitall" );<br>
>> <br>
>> getmem( &rss1, &vs );<br>
>> <br>
>> … where getmem() queries the OS kernel to return the resident set size (“rss") and virtual memory size (“vs", ignored). The getmem() function does not allocate any memory itself, so I can hopefully see any difference in the resident set memory used before
and after the MPI_Waitall() call via (rss1-rss0).<br>
>> <br>
>> After MPI_Waitall() returns, I very often see an increase in the resident set size ranging from 4096 bytes to 65536 bytes.<br>
>> <br>
>> Running an “identical” 4-process job on a single node of the Blue Waters compute cluster (Cray XE system, GCC 4.8.1, Cray mpich 6.0.1) does not seem to produce this behaviour.<br>
>> <br>
>> Switching the MPI_Waitall() for a loop over MPI_Waitany() calls also produces resident set increases.<br>
>> <br>
>> Does anyone have any comments or suggestions as to how I might understand whether this behavior is errant, and how I might track down the cause? It’s irritating me as I’d really like to produce code with as low a memory overhead as possible!<br>
>> <br>
>> Cheers,<br>
>> <br>
>> J.<br>
>> <br>
>> ------------------<br>
>> Platform info:<br>
>> ------------------<br>
>> <br>
>> OS: Apple OSX 10.9.1<br>
>> <br>
>> mpicxx —version:<br>
>> <br>
>> Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)<br>
>> Target: x86_64-apple-darwin13.0.0<br>
>> Thread model: posix<br>
>> <br>
>> mpiexec —version<br>
>> <br>
>> HYDRA build details:<br>
>> Version: 3.0.4<br>
>> Release Date: Wed Apr 24 10:08:10 CDT 2013<br>
>> CC: /usr/bin/clang -pipe -arch x86_64 -Wl,-headerpad_max_install_names -arch x86_64<br>
>> CXX: /usr/bin/clang++ -pipe -arch x86_64 -Wl,-headerpad_max_install_names -arch x86_64<br>
>> F77: /opt/local/bin/gfortran-mp-4.8 -pipe -m64 -Wl,-headerpad_max_install_names -arch x86_64<br>
>> F90: /opt/local/bin/gfortran-mp-4.8 -pipe -m64 -Wl,-headerpad_max_install_names -arch x86_64<br>
>> Configure options: '--disable-option-checking' '--prefix=/opt/local' '--disable-dependency-tracking' '--disable-silent-rules' '--enable-base-cache' '--enable-cache' '--enable-cxx' '--enable-fast=O2' '--enable-shared' '--enable-smpcoll'
'--with-device=ch3:nemesis' '--with-pm=hydra' '--with-thread-package=posix' '--enable-versioning' 'F90FLAGS=' 'F90=' '--enable-timer-type=mach_absolute_time' '--libdir=/opt/local/lib/mpich-mp' '--sysconfdir=/opt/local/etc/mpich-mp' '--program-suffix=-mp' '--enable-f77'
'--enable-fc' 'CC=/usr/bin/clang' 'CFLAGS=-pipe -arch x86_64 -O2' 'LDFLAGS=-Wl,-headerpad_max_install_names -arch x86_64 ' 'CXX=/usr/bin/clang++' 'CXXFLAGS=-pipe -arch x86_64 -O2' 'F77=/opt/local/bin/gfortran-mp-4.8' 'FFLAGS=-pipe -m64 -O2' 'FC=/opt/local/bin/gfortran-mp-4.8'
'FCFLAGS=-pipe -m64 -O2' '--cache-file=/dev/null' '--srcdir=.' 'LIBS=-lpthread ' 'CPPFLAGS= -I/opt/local/var/macports/build/_opt_mports_dports_science_mpich/mpich-default/work/mpich-3.0.4<br>
> /src/mpl/include -I/opt/local/var/macports/build/_opt_mports_dports_science_mpich/mpich-default/work/mpich-3.0.4/src/mpl/include -I/opt/local/var/macports/build/_opt_mports_dports_science_mpich/mpich-default/work/mpich-3.0.4/src/openpa/src -I/opt/local/var/macports/build/_opt_mports_dports_science_mpich/mpich-default/work/mpich-3.0.4/src/openpa/src
-I/opt/local/var/macports/build/_opt_mports_dports_science_mpich/mpich-default/work/mpich-3.0.4/src/mpi/romio/include'<br>
>> Process Manager: pmi<br>
>> Launchers available: ssh rsh fork slurm ll lsf sge manual persist<br>
>> Topology libraries available: hwloc<br>
>> Resource management kernels available: user slurm ll lsf sge pbs cobalt<br>
>> Checkpointing libraries available:<br>
>> Demux engines available: poll select<br>
>> <br>
>> _______________________________________________<br>
>> discuss mailing list discuss@mpich.org<br>
>> To manage subscription options or unsubscribe:<br>
>> <a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
>> <br>
> _______________________________________________<br>
> discuss mailing list discuss@mpich.org<br>
> To manage subscription options or unsubscribe:<br>
> <a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
<br>
</div>
</span></font></div>
</body>
</html>