[mpich-discuss] MPI_Send and recv

Fri Dec 6 07:10:15 CST 2013

For large messages, MPI_Send may behave like MPI_Ssend on Blue Gene
and probably other platforms.  You have to specify the message sizes
your interested in for me to provide more accurate information.

If you don't want data transfer to block, just use very small
Send-Recv for control and move data with RMA.

Jeff

On Fri, Dec 6, 2013 at 12:46 AM, Matthieu Dorier
<matthieu.dorier at irisa.fr> wrote:
> Thanks Jeff for your answer.
> By "systematically" I mean "whether a remote buffer is full or not" (i.e.
> "every time"), as opposed to an MPI_Send that "may block under some
> conditions, but not always".
>
> Matthieu
>
> ________________________________
>
> De: "Jeff Hammond" <jeff.science at gmail.com>
> À: discuss at mpich.org
> Envoyé: Vendredi 6 Décembre 2013 03:34:00
>
> Objet: Re: [mpich-discuss] MPI_Send and recv
>
> Blue Gene does not always block on mpi_send. I don't know what you mean by
> "systematically"...
>
> If you are sending small messages (less than ~1KB), they will use eager
> protocol and not block until you blow up the remote buffer.
>
> Rendezvous very likely blocks until the recv is posted.
>
> There are slides online about Cray MPI protocols that may show similar
> behavior.
>
> If you want to be sure you don't block on send, you need to use isend. That
> much should be obvious.
>
> Jeff
>
> Sent from my iPhone
>
> On Dec 5, 2013, at 2:59 PM, Matthieu Dorier <matthieu.dorier at irisa.fr>
> wrote:
>
> Hi,
>
> I come back to this old discussion, regarding MPI_Send potentially blocking
> until a matching MPI_Recv is posted.
>
> I happen to write a master-worker type program where workers periodically
> send progress status to the master through two MPI_Send.
> The master has an MPI_Irecv posted for the first one and when this one
> completes, it posts an MPI_Recv for the second one. Those messages being
> small and the worker not expecting any answer before continuing to work, you
> can imagine that a systematically blocking MPI_Send could degrade the
> performance if the master is busy doing something else.
>
> So my question is: to your knowledge, do implementations of MPI (in
> particular Mpich, and those running on Cray Linux and BlueGene/P and Q)
> systematically block on MPI_Send? (I would understand if it blocked because
> too many MPI_Send have not been matched on the receiver)
>
> Otherwise I might have to use MPI_Isend everywhere...
>
> Thanks!
>
> Matthieu
>
> ________________________________
>
> De: "Wesley Bland" <wbland at mcs.anl.gov>
> À: discuss at mpich.org
> Envoyé: Mercredi 3 Juillet 2013 15:40:11
> Objet: Re: [mpich-discuss] MPI_Send and recv
>
> On Jul 2, 2013, at 9:03 PM, Matthieu Dorier <matthieu.dorier at irisa.fr>
> wrote:
>
> Hi,
>
> These messages piqued my curiosity;
>
> If I have two processes, process 0 does an MPI_Send to process 1 and
> eventually, process 1 posts a matching MPI_Recv or MPI_Irecv; does process 0
> waits for the MPI_Recv/Irecv to be posted before returning from the
> MPI_Send? (in other words if process 0 does 2 MPI_Send and process 1 does 2
> MPI_Recv, is the linearization "Send; Send; Recv; Recv" possible?)
>
>
> It is possible that process 0 could return from the MPI_Send before process
> 1 posts the corresponding MPI_Recv, but it's not guaranteed. The only thing
> that is guaranteed from the standard is that when MPI_Send returns, the
> buffer is available to use again.  It is valid to wait for the MPI_Recv to
> post as well. So to answer your example, S - S - R - R is valid, as is S - R
> - S - R.
>
>
> Second question: in the situation described by Sufeng, where a process does
> a MPI_Send to itself, MPI has enough information to see that the process is
> sending to itself while it has not posted any matching receive, why would
> MPI block instead of printing an error and aborting? Is it because in a
> multithreaded program, the matching receive could be posted by another
> thread? (Yet in that case I always thought it was unsafe to use the same
> communicator from multiple threads…)
>
>
> There are probably many reasons why MPI doesn't try to do internal error
> checking. The first is performance. If a program is correct, then checking
> for all of the cases that might cause a deadlock would be not only very
> difficult, but also expensive. Therefore, we just say that the user needs to
> write a correct program and MPI makes no guarantees that anything smart will
> happen in the background. It's possible that someone could write Nice-MPI
> that does lots of error checking and verification, but I would think that
> most users wouldn't want it as it would add overhead.
>
> Second, it's not always unsafe to use the same communicator in multiple
> threads. If you use different tags for each thread, then it shouldn't be a
> problem to communicate between them.
>
> Wesley
>
>
> Thanks,
>
> Matthieu
>
> ________________________________
>
> De: "Sufeng Niu" <sniu at hawk.iit.edu>
> À: "Pavan Balaji" <balaji at mcs.anl.gov>
> Cc: discuss at mpich.org
> Envoyé: Mardi 2 Juillet 2013 17:20:25
> Objet: Re: [mpich-discuss] MPI_Send and recv
>
> Thank you so much! Now I make sense
>
> Sufeng
>
>
> On Tue, Jul 2, 2013 at 5:19 PM, Pavan Balaji <balaji at mcs.anl.gov> wrote:
>>
>>
>> On 07/02/2013 05:18 PM, Sufeng Niu wrote:
>>>
>>> Thanks a lot, that is most confusing part for me in MPI. I would like to
>>> clear this type of problems.
>>>
>>> So the blocking begins to send message to itself, but there is no
>>> receive posted. Thus, the MPI block it. Am I right here?
>>
>>
>> Correct.
>>
>>
>>> Thus, I need a nonblocking receive to wait the message here. Can I use
>>> blocking receive?
>>
>>
>> If you use blocking receive, you'll block in the receive; how will you
>> reach the send?
>>
>>  -- Pavan
>>
>>
>> --
>> Pavan Balaji
>> http://www.mcs.anl.gov/~balaji
>
>
>
>
> --
> Best Regards,
> Sufeng Niu
> ECASP lab, ECE department, Illinois Institute of Technology
> Tel: 312-731-7219
>
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
>
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
>
>
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
>
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
>
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
>
>
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss

-- 
Jeff Hammond
jeff.science at gmail.com