[mpich-discuss] Mpich2's behavior when too many messages are sent

[Jeff Hammond]

This is a quality of implementation issue that lies outside the MPI
standard.  There is no requirement that an arbitrarily large number of
nonblocking messages be allowed to be outstanding at one time since
this is not possible with finite resources, but obviously a
high-quality implementation will strive to make the allowed number as
large as possible.

I suspect that MPICH will let you enqueue nonblocking sends and
receives until you reach resource exhaustion, i.e. run out of memory
for internal state.  I ran a simple test (see below) that shows you
can post at least 1M messages (true for MPICH 3.0rc1 on my Macbook
Pro) without any problems (although at 1M it gets a bit slow).

Best,

Jeff

$ mpichversion
MPICH Version:    	3.0rc1
MPICH Release date:	Mon Nov 12 10:31:40 CST 2012
MPICH Device:    	ch3:nemesis
MPICH configure: 	--prefix=/opt/local --disable-dependency-tracking
--disable-f77 --disable-fc --disable-silent-rules --enable-base-cache
--enable-cache --enable-cxx --enable-fast=O2 --enable-shared
--enable-smpcoll --includedir=/opt/local/include/mpich
--with-device=ch3:nemesis --with-pm=hydra --with-thread-package=posix
F90FLAGS= F90= MPICHLIB_CPPFLAGS=-I/opt/local/include
--enable-timer-type=mach_absolute_time
MPICH CC: 	/usr/bin/clang -pipe -arch x86_64
MPICH CXX: 	/usr/bin/clang++ -pipe -arch x86_64  -O2
MPICH F77: 	 -pipe -m64
MPICH FC: 	 -pipe -m64

compile with:

mpicc -g -O2 -Wall outstanding.c -o outstanding.x

execute with:

mpiexec -n 2 ./outstanding.x 1000000

$ cat outstanding.c
===========================
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

int main(int argc, char* argv[])
{
    int rank, size;

    MPI_Init(&argc, &argv);

    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    if (size<2)
    {
        printf("run on 2 or more ranks! \n");
        MPI_Abort(MPI_COMM_WORLD, 1);
    }

    int n = (argc>1) ? atoi(argv[1]) : 1e6;

    if (rank==0)
        printf("attempting to call MPI_I{send,recv} %d times before
completing any of them \n", n);

    MPI_Request * reqs = (MPI_Request*) malloc(n*sizeof(MPI_Request));
    if (reqs==NULL)
    {
        printf("malloc of %ld bytes failed! \n", (long)n*sizeof(MPI_Request));
        MPI_Abort(MPI_COMM_WORLD, 1);
    }

    if (rank==0)
        for (int i=0; i<n; i++)
            MPI_Irecv(NULL, 0, MPI_BYTE, 1, i, MPI_COMM_WORLD, &(reqs[i]) );

    else if (rank==1)
        for (int i=0; i<n; i++)
            MPI_Isend(NULL, 0, MPI_BYTE, 0, i, MPI_COMM_WORLD, &(reqs[i]) );

    if (rank==0)
        printf("all MPI_I{send,recv} have been posted \n");

    MPI_Barrier(MPI_COMM_WORLD);

    MPI_Waitall(n, reqs, MPI_STATUSES_IGNORE);

    if (rank==0)
        printf("all MPI_I{send,recv} have been completed \n");

    free(reqs);

    if (rank==0)
        printf("SUCCESS \n");

    MPI_Finalize();

    return 0;
}
===========================

$ mpiexec -n 2 ./outstanding.x 100
attempting to call MPI_I{send,recv} 100 times before completing any of them
all MPI_I{send,recv} have been posted
all MPI_I{send,recv} have been completed
SUCCESS

$ mpiexec -n 2 ./outstanding.x 1000
attempting to call MPI_I{send,recv} 1000 times before completing any of them
all MPI_I{send,recv} have been posted
all MPI_I{send,recv} have been completed
SUCCESS

$ mpiexec -n 2 ./outstanding.x 10000
attempting to call MPI_I{send,recv} 10000 times before completing any of them
all MPI_I{send,recv} have been posted
all MPI_I{send,recv} have been completed
SUCCESS

$ mpiexec -n 2 ./outstanding.x 100000
attempting to call MPI_I{send,recv} 100000 times before completing any of them
all MPI_I{send,recv} have been posted
all MPI_I{send,recv} have been completed
SUCCESS

$ mpiexec -n 2 ./outstanding.x 1000000
attempting to call MPI_I{send,recv} 1000000 times before completing any of them
all MPI_I{send,recv} have been posted
all MPI_I{send,recv} have been completed
SUCCESS


On Tue, Nov 27, 2012 at 3:50 AM, Matthieu Dorier
<matthieu.dorier at irisa.fr> wrote:
> Hello,
>
> I have a program that include two processes, one is doing a lot of
> MPI_Isend, the other one loops, receives the messages using MPI_Irecv and
> treats them.
> I'd like to know what the MPI-2 standard says, and what Mpich2 does, if the
> first process is too fast and the second process cannot keep up with the
> number of messages sent. Is the first process supposed to eventually block,
> waiting for the messages to be received, or will the application crashes?
>
> Thanks
>
> Matthieu Dorier
> PhD student at ENS Cachan Brittany and IRISA
> http://people.irisa.fr/Matthieu.Dorier
>
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss



-- 
Jeff Hammond
Argonne Leadership Computing Facility
University of Chicago Computation Institute
jhammond at alcf.anl.gov / (630) 252-5381
http://www.linkedin.com/in/jeffhammond
https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond