<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"><div dir="ltr"><div>I did more experiments, and results agrees with this. Thanks a lot for your help!</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr">Best regards,<div>Zhen</div></div></div></div>
<br><div class="gmail_quote">On Thu, Nov 6, 2014 at 11:55 AM, Junchao Zhang <span dir="ltr"><<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">My understanding is that: In eager mode, sender does not need a Ack message from receiver (to know recv buf is ready), but in <span style="font-family:arial,sans-serif;font-size:13px">rendezvous mode, it does.</span><div><font face="arial, sans-serif">In your case, if it is in rendezvous mode, Isend() issues the request. But since it is a nonblocking call, before </font>sender<font face="arial, sans-serif"> gets the Ack message, it goes into sleep. The asynchronous progress engine does not progress until MPI_Wait() is called. At that time, the Ack message is got, and message passing happens. Therefore, MPI_Recv() finishes after MPI_Wait().</font></div></div><div class="gmail_extra"><span class="HOEnZb"><font color="#888888"><br clear="all"><div><div><div dir="ltr">--Junchao Zhang</div></div></div></font></span><div><div class="h5">
<br><div class="gmail_quote">On Thu, Nov 6, 2014 at 10:33 AM, Zhen Wang <span dir="ltr"><<a href="mailto:toddwz@gmail.com" target="_blank">toddwz@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Junchao,<div><br></div><div>Thanks for your reply. It works! I digged deeper into the eager and rendezvous modes, and got confused..<br></div><div><br></div><div>The docs I read were <a href="https://computing.llnl.gov/tutorials/mpi_performance/#Protocols" target="_blank">https://computing.llnl.gov/tutorials/mpi_performance/#Protocols</a> and <a href="http://www-01.ibm.com/support/knowledgecenter/SSFK3V_1.3.0/com.ibm.cluster.pe.v1r3.pe400.doc/am106_eagermess.htm" target="_blank">http://www-01.ibm.com/support/knowledgecenter/SSFK3V_1.3.0/com.ibm.cluster.pe.v1r3.pe400.doc/am106_eagermess.htm</a>.</div><div><br></div><div>My understanding is:</div><div><br></div><div>In eager mode, the receiver allocates a buffer and an additional copy is required to copy the data from the buffer to user allocated space. This is good for small messages, not for large ones.</div><div><br></div><div>In rendezvous mode, the user allocated space on the receiver must be ready to receive the data. But if the space is ready, the MPI_Recv() should be completed almost immediately (there's a handshake between the send and receiver, and a copy operation).</div><div><br></div><div>If I was right, in my code example, MPI_Recv() should finish after MPI_Isend() no matter in eager or rendezvous modes. The memory has been allocated, and the handshake should take like no time. Am I missing something?</div><div><br></div><div>Thanks a lot.</div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr">Best regards,<div>Zhen</div></div></div></div><div><div>
<br><div class="gmail_quote">On Wed, Nov 5, 2014 at 2:28 PM, Junchao Zhang <span dir="ltr"><<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I think it is because MPICH just crosses over the eager to rendezvous mode threshold, when n goes from 9999 to 99999. OpenMPI certainly uses a different threshold than MPICH.<div>When you install MPICH, a utility program mpivars is also installed. Type 'mpivars | grep EAGER', you will get default values for various eager thresholds. </div><div><br></div><div>In your case, export MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZE=5000000 and you will get the same result as OpenMPI. </div></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr">--Junchao Zhang</div></div></div>
<br><div class="gmail_quote"><div><div>On Wed, Nov 5, 2014 at 11:20 AM, Zhen Wang <span dir="ltr"><<a href="mailto:toddwz@gmail.com" target="_blank">toddwz@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi MPIers,<br><br>I have some questions regarding MPI_Isend() and MPI_Recv(). MPICH-3.1.3 is used to compile and run the attached code on Red Hat Enterprise Linux 6.3 (a shared memory machine). While n = 9999, the MPI_Recv() finishes immediately after MPI_Isend(): (This is what I understand and expect)<br><br>MPI 1: Recv started at 09:53:53.<br>MPI 0: Isend started at 09:53:53.<br>MPI 1: Recv finished at 09:53:53.<br>MPI 0: Isend finished at 09:53:58.<br><br>When n = 99999, I get the following. The MPI_Recv() finishes after MPI_Wait():<br><br>MPI 1: Recv started at 09:47:56.<br>MPI 0: Isend started at 09:47:56.<br>MPI 0: Isend finished at 09:48:01.<br>MPI 1: Recv finished at 09:48:01.<br><br>But with OpenMPI 1.8 and n = 99999, MPI_Recv() finishes immediately after MPI_Isend():<br><br>MPI 0: Isend started at 09:55:28.<br>MPI 1: Recv started at 09:55:28.<br>MPI 1: Recv finished at 09:55:28.<br>MPI 0: Isend finished at 09:55:33.<div><br></div><div>Am I misunderstanding something here? In case the attached code is dropped, the code is included. Thanks in advance.</div><div><br></div><div><br></div><div><div><font face="courier new, monospace">#include "mpi.h"</font></div><div><font face="courier new, monospace">#include <unistd.h></font></div><div><font face="courier new, monospace">#include <stdio.h></font></div><div><font face="courier new, monospace">#include "vector"</font></div><div><font face="courier new, monospace">#include <time.h></font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">int main(int argc, char* argv[])</font></div><div><font face="courier new, monospace">{</font></div><div><font face="courier new, monospace"> MPI_Init(&argc, &argv);</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> int rank;</font></div><div><font face="courier new, monospace"> MPI_Comm_rank(MPI_COMM_WORLD, &rank);</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> int n = 9999;</font></div><div><font face="courier new, monospace"> std::vector<int> vec(n);</font></div><div><font face="courier new, monospace"> MPI_Request mpiRequest;</font></div><div><font face="courier new, monospace"> MPI_Status mpiStatus;</font></div><div><font face="courier new, monospace"> char tt[9] = {0};</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> MPI_Barrier(MPI_COMM_WORLD);</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> if (rank == 0)</font></div><div><font face="courier new, monospace"> {</font></div><div><font face="courier new, monospace"> MPI_Isend(&vec[0], n, MPI_INT, 1, 0, MPI_COMM_WORLD, &mpiRequest);</font></div><div><font face="courier new, monospace"> time_t t = time(0);</font></div><div><font face="courier new, monospace"> strftime(tt, 9, "%H:%M:%S", localtime(&t));</font></div><div><font face="courier new, monospace"> printf("MPI %d: Isend started at %s.\n", rank, tt);</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> //int done = 0;</font></div><div><font face="courier new, monospace"> //while (done == 0)</font></div><div><font face="courier new, monospace"> //{</font></div><div><font face="courier new, monospace"> // MPI_Test(&mpiRequest, &done, &mpiStatus);</font></div><div><font face="courier new, monospace"> //}</font></div><div><font face="courier new, monospace"> sleep(5);</font></div><div><font face="courier new, monospace"> MPI_Wait(&mpiRequest, &mpiStatus);</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> t = time(0);</font></div><div><font face="courier new, monospace"> strftime(tt, 9, "%H:%M:%S", localtime(&t));</font></div><div><font face="courier new, monospace"> printf("MPI %d: Isend finished at %s.\n", rank, tt);</font></div><div><font face="courier new, monospace"> }</font></div><div><font face="courier new, monospace"> else</font></div><div><font face="courier new, monospace"> {</font></div><div><font face="courier new, monospace"> time_t t = time(0);</font></div><div><font face="courier new, monospace"> strftime(tt, 9, "%H:%M:%S", localtime(&t));</font></div><div><font face="courier new, monospace"> printf("MPI %d: Recv started at %s.\n", rank, tt);</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> MPI_Recv(&vec[0], n, MPI_INT, 0, 0, MPI_COMM_WORLD, &mpiStatus);</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> t = time(0);</font></div><div><font face="courier new, monospace"> strftime(tt, 9, "%H:%M:%S", localtime(&t));</font></div><div><font face="courier new, monospace"> printf("MPI %d: Recv finished at %s.\n", rank, tt);</font></div><div><font face="courier new, monospace"> }</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> MPI_Finalize();</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"> return 0;</font></div><div><font face="courier new, monospace">}</font></div><div><br></div><br><br>Best regards,<br>Zhen<br><div>
</div></div></div>
<br></div></div>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div>
<br>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div>