<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">Hi Huiwei and jeff,<div><br></div><div>Thanks a lot. I will try those ideas you guys suggested on some test codes first and do profiling to verify it at the same time. When I get a chance to run my code on our infiniband nodes, I will repeat those tests and let you guys know the difference. Thanks again!<br><img src="https://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/c029941c3640754cf757dbb224b286ed/spacer.gif" style="border:0; width:0; height:0; overflow:hidden;" width="0" height="0"><img src="http://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/c029941c3640754cf757dbb224b286ed/spacer.gif" style="border:0; width:0; height:0; overflow:hidden;" width="0" height="0"><font face="yw-d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9-c029941c3640754cf757dbb224b286ed--to" style=""></font></div><div>Best,</div><div><br></div><div>Lei Shi</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 7, 2015 at 10:12 AM, Huiwei Lu <span dir="ltr"><<a href="mailto:huiweilu@mcs.anl.gov" target="_blank">huiweilu@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Lei,<div><br></div><div>Your profiling is correct. MPI_Isend/Irecv only start the communication but the real progress is made in MPI_Waitall. If you want to overlap communication with <span style="color:rgb(0,0,0);font-size:medium;white-space:pre-wrap">ResFromDivInvisFlux, you can put MPI_Test inside </span><span style="color:rgb(0,0,0);font-size:medium;white-space:pre-wrap">ResFromDivInvisFlux to force MPI to make progress </span><font color="#000000" size="3"><span style="white-space:pre-wrap">periodically when MPI_Test is called.</span></font></div><div><font color="#000000" size="3"><span style="white-space:pre-wrap"><br></span></font></div><div><font color="#000000" size="3"><span style="white-space:pre-wrap">Another option is to use </span></font><span style="color:rgb(0,0,0);font-size:medium;white-space:pre-wrap">asynchronous progress thread</span><span style="white-space:pre-wrap;color:rgb(0,0,0);font-size:medium">. Check </span><font color="#000000" size="3"><span style="white-space:pre-wrap">MPIR_CVAR_ASYNC_PROGRESS for more information.</span></font></div><div><font color="#000000" size="3"><span style="white-space:pre-wrap"><br></span></font></div><div><font color="#000000" size="3"><span style="white-space:pre-wrap">Best regards,</span></font></div><div class="gmail_extra"><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr">--</div><div dir="ltr">Huiwei</div></div></div></div></div></div></div></div><div><div class="h5">
<br><div class="gmail_quote">On Tue, Apr 7, 2015 at 2:48 AM, Lei Shi <span dir="ltr"><<a href="mailto:lshi@ku.edu" target="_blank">lshi@ku.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"> Here is my pure MPI overlap version. I use intel traceanalyzer, the profiling shows that right now, communication only proceed when I call mpi_waitall on nodes with 10g network. <div><br></div><div><pre style="color:rgb(0,0,0);font-size:medium"><span style="color:rgb(105,105,105)">/** pure mpi overlap **/</span><span>
<span style="color:rgb(128,0,0);font-weight:bold">template</span><span style="color:rgb(128,0,128)"><</span><span style="color:rgb(128,0,0);font-weight:bold">typename</span> T<span style="color:rgb(128,0,128)">></span>
<span style="color:rgb(128,0,0);font-weight:bold">void</span> CPR_NS_3D_Solver<span style="color:rgb(128,0,128)"><</span>T<span style="color:rgb(128,0,128)">></span><span style="color:rgb(128,0,128)">::</span>UpdateRes<span style="color:rgb(128,128,48)">(</span>T<span style="color:rgb(128,128,48)">*</span><span style="color:rgb(128,128,48)">*</span>q<span style="color:rgb(128,128,48)">,</span> T<span style="color:rgb(128,128,48)">*</span><span style="color:rgb(128,128,48)">*</span>res<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span></span>
<span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span>_n_proc<span style="color:rgb(128,128,48)">></span><span style="color:rgb(0,140,0)">1</span><span style="color:rgb(128,128,48)">)</span>
SendInterfaceSol<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span> <span style="color:rgb(105,105,105)">//call isend/irecv to send msg 1</span>
ResFromDivInvisFlux<span style="color:rgb(128,128,48)">(</span>q<span style="color:rgb(128,128,48)">,</span>res<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span> <span style="color:rgb(105,105,105)">//do local jobs</span>
<span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span>_n_proc<span style="color:rgb(128,128,48)">></span><span style="color:rgb(0,140,0)">1</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span>
RevInterfaceSol<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span> <span style="color:rgb(105,105,105)">//mpi_waitall for msg 1</span>
<span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span>vis_mode_<span style="color:rgb(128,128,48)">)</span>
SendInterfaceCorrGrad<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span> <span style="color:rgb(105,105,105)">//depends on msg 1 then snd msg 2</span>
<span style="color:rgb(128,0,128)">}</span>
<span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span>vis_mode_<span style="color:rgb(128,128,48)">)</span>
ResFromDivVisFlux<span style="color:rgb(128,128,48)">(</span>q<span style="color:rgb(128,128,48)">,</span>res<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span> <span style="color:rgb(105,105,105)">//computing, which depends on msg 1</span>
<span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span>_n_proc<span style="color:rgb(128,128,48)">></span><span style="color:rgb(0,140,0)">1</span> <span style="color:rgb(128,128,48)">&</span><span style="color:rgb(128,128,48)">&</span> vis_mode_<span style="color:rgb(128,128,48)">)</span>
RevInterfaceCorrGrad<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span> <span style="color:rgb(105,105,105)">//mpi_waitall for msg 2</span>
ResFromFluxCorrection<span style="color:rgb(128,128,48)">(</span>q<span style="color:rgb(128,128,48)">,</span>res<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span> <span style="color:rgb(105,105,105)">//computing, which depends on msg 1 and 2</span>
<span style="color:rgb(128,0,128)">}</span></pre><img src="https://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/f1bad39d2bbcb42c866a4c6e2850704d/spacer.gif" style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><img src="http://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/f1bad39d2bbcb42c866a4c6e2850704d/spacer.gif" style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><font face="yw-d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9-f1bad39d2bbcb42c866a4c6e2850704d--to"></font></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 7, 2015 at 2:39 AM, Lei Shi <span dir="ltr"><<a href="mailto:lshi@ku.edu" target="_blank">lshi@ku.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><img src="https://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/a506656af4ef843fa1843a06cb1cd0a5/spacer.gif" style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><img src="http://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/a506656af4ef843fa1843a06cb1cd0a5/spacer.gif" style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><font face="yw-d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9-a506656af4ef843fa1843a06cb1cd0a5--to"></font></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 7, 2015 at 2:37 AM, Lei Shi <span dir="ltr"><<a href="mailto:leishi@ku.edu" target="_blank">leishi@ku.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi Huiwei and Jeff,<div><br></div><div>I use hybrid OpenMP/MPI to do overlap communication. So I put all communication in one dedicated OpenMP thread and computation in the other thread. For this case, I'm using intel MPI library. Probably I did some mistakes </div><div><br></div><div>One version of my code using one dedicated thread to do messaging is like this</div><div><pre style="color:rgb(0,0,0);font-size:medium"><span style="color:rgb(105,105,105)">/* hybrid mpi/openmp overlap **/</span>
<span style="color:rgb(128,0,0);font-weight:bold">template</span><span style="color:rgb(128,0,128)"><</span><span style="color:rgb(128,0,0);font-weight:bold">typename</span> T<span style="color:rgb(128,0,128)">></span>
<span style="color:rgb(128,0,0);font-weight:bold">void</span> CPR_NS_3D_Solver<span style="color:rgb(128,0,128)"><</span>T<span style="color:rgb(128,0,128)">></span><span style="color:rgb(128,0,128)">::</span>UpdateRes<span style="color:rgb(128,128,48)">(</span>T<span style="color:rgb(128,128,48)">*</span><span style="color:rgb(128,128,48)">*</span>q<span style="color:rgb(128,128,48)">,</span> T<span style="color:rgb(128,128,48)">*</span><span style="color:rgb(128,128,48)">*</span>res<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span>
<span style="color:rgb(128,0,0);font-weight:bold">int</span> thread_id<span style="color:rgb(128,128,48)">,</span>n_thread<span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,0);font-weight:bold">int</span> sol_rev_flag<span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,128,48)">,</span>grad_rev_flag<span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(105,105,105)">// Explicitly disable dynamic teams</span>
omp_set_dynamic<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(105,105,105)">// Use 2 threads for all consecutive parallel regions </span>
omp_set_num_threads<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(0,140,0)">2</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(0,74,67)">#</span><span style="color:rgb(0,74,67);font-weight:bold">pragma </span><span style="color:rgb(187,121,119);font-weight:bold">omp parallel default(shared) private(thread_id)</span>
<span style="color:rgb(128,0,128)">{</span>
thread_id<span style="color:rgb(128,128,48)">=</span>omp_get_thread_num<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span>
n_thread<span style="color:rgb(128,128,48)">=</span>omp_get_num_threads<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(63,95,191)">/** communication thread **/</span>
<span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span>thread_id<span style="color:rgb(128,128,48)">=</span><span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">1</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span>
SendInterfaceSol<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span>
RevInterfaceSol<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(0,74,67)">#</span><span style="color:rgb(0,74,67);font-weight:bold">pragma </span><span style="color:rgb(187,121,119);font-weight:bold">omp flush</span>
sol_rev_flag<span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">1</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(0,74,67)">#</span><span style="color:rgb(0,74,67);font-weight:bold">pragma </span><span style="color:rgb(187,121,119);font-weight:bold">omp flush(sol_rev_flag)</span>
<span style="color:rgb(128,0,128)">}</span>
<span style="color:rgb(63,95,191)">/** computation thread **/</span>
<span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span>thread_id<span style="color:rgb(128,128,48)">=</span><span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span>
ResFromDivInvisFlux<span style="color:rgb(128,128,48)">(</span>q<span style="color:rgb(128,128,48)">,</span>res<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span> <span style="color:rgb(105,105,105)">//local computation</span>
<span style="color:rgb(0,74,67)">#</span><span style="color:rgb(0,74,67);font-weight:bold">pragma </span><span style="color:rgb(187,121,119);font-weight:bold">omp flush(sol_rev_flag)</span>
<span style="color:rgb(128,0,0);font-weight:bold">while</span><span style="color:rgb(128,128,48)">(</span>sol_rev_flag<span style="color:rgb(128,128,48)">!</span><span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">1</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span>
<span style="color:rgb(0,74,67)"> #</span><span style="color:rgb(0,74,67);font-weight:bold">pragma </span><span style="color:rgb(187,121,119);font-weight:bold">omp flush(sol_rev_flag)</span>
<span style="color:rgb(128,0,128)">}</span>
<span style="color:rgb(0,74,67)">#</span><span style="color:rgb(0,74,67);font-weight:bold">pragma </span><span style="color:rgb(187,121,119);font-weight:bold">omp flush</span>
ResFromFluxCorrection<span style="color:rgb(128,128,48)">(</span>q<span style="color:rgb(128,128,48)">,</span>res<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span> <span style="color:rgb(105,105,105)">//depends on interface sol</span>
<span style="color:rgb(128,0,128)">}</span>
<span style="color:rgb(128,0,128)">}</span><span style="color:rgb(105,105,105)">//end of omp</span>
<span style="color:rgb(128,0,128)">}</span></pre><pre style="color:rgb(0,0,0);font-size:medium"><span style="font-family:arial,sans-serif;color:rgb(128,0,0);font-weight:bold">template</span><span style="font-family:arial,sans-serif;color:rgb(128,0,128)"><</span><span style="font-family:arial,sans-serif;color:rgb(128,0,0);font-weight:bold">typename</span><span style="font-family:arial,sans-serif"> T</span><span style="font-family:arial,sans-serif;color:rgb(128,0,128)">></span><br></pre><pre style="font-size:medium"><pre style="color:rgb(0,0,0)"> <span style="color:rgb(128,0,0);font-weight:bold">void</span> CPR_NS_3D_Solver<span style="color:rgb(128,0,128)"><</span>T<span style="color:rgb(128,0,128)">></span><span style="color:rgb(128,0,128)">::</span>SendInterfaceSol<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span>
uint <span style="color:rgb(128,128,48)">*</span>n_if_to_proc<span style="color:rgb(128,128,48)">=</span><span style="color:rgb(128,0,0);font-weight:bold">this</span><span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span>grid_<span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span>num_iface_proc<span style="color:rgb(128,0,128)">;</span>
uint <span style="color:rgb(128,128,48)">*</span><span style="color:rgb(128,128,48)">*</span>if_to_proc<span style="color:rgb(128,128,48)">=</span><span style="color:rgb(128,0,0);font-weight:bold">this</span><span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span>grid_<span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span>snd_iface_proc<span style="color:rgb(128,0,128)">;</span>
uint <span style="color:rgb(128,128,48)">*</span><span style="color:rgb(128,128,48)">*</span>rev_if_to_f<span style="color:rgb(128,128,48)">=</span><span style="color:rgb(128,0,0);font-weight:bold">this</span><span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span>grid_<span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span>rev_iface_proc<span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,0);font-weight:bold">int</span> tag<span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">52</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,0);font-weight:bold">for</span><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,0,0);font-weight:bold">int</span> p2<span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,0,128)">;</span>p2<span style="color:rgb(128,128,48)"><</span>_n_proc<span style="color:rgb(128,0,128)">;</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span>p2<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span>
<span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span>p2<span style="color:rgb(128,128,48)">!</span><span style="color:rgb(128,128,48)">=</span>_proc_id<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span>
<span style="color:rgb(128,0,0);font-weight:bold">int</span> nif<span style="color:rgb(128,128,48)">=</span>n_if_to_proc<span style="color:rgb(128,128,48)">[</span>p2<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(105,105,105)">//pack data to send ....</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif"> </span></pre><pre><font color="#000000"> </font><span style="color:rgb(128,0,128)">}</span><font color="#000000">
</font><span style="color:rgb(128,0,128)">}</span><font color="#000000">
</font><span style="color:rgb(63,95,191)">/** </span><font color="#000000">
</font><span style="color:rgb(63,95,191)"> * Exchange interface sol</span><font color="#000000">
</font><span style="color:rgb(63,95,191)"> **/</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">int</span><font color="#000000"> n_proc_exchange</font><span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,0,128)">;</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">for</span><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,0,0);font-weight:bold">int</span><font color="#000000"> z</font><span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,0,128)">;</span><font color="#000000">z</font><span style="color:rgb(128,128,48)"><</span><font color="#000000">_n_proc</font><span style="color:rgb(128,0,128)">;</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><font color="#000000">z</font><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">int</span><font color="#000000"> nif</font><span style="color:rgb(128,128,48)">=</span><font color="#000000">n_if_to_proc</font><span style="color:rgb(128,128,48)">[</span><font color="#000000">z</font><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span><font color="#000000">
</font><span style="color:rgb(105,105,105)">//send data</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span><font color="#000000">nif</font><span style="color:rgb(128,128,48)">></span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span><font color="#000000">
</font><font color="#000000"> MPI_Isend</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">&</span><font color="#000000">snd_buf_</font><span style="color:rgb(128,128,48)">[</span><font color="#000000">z</font><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,128,48)">,</span><font color="#000000">n_</font><font color="#000000">buf_</font><span style="color:rgb(128,128,48)">[</span><font color="#000000">z</font><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,128,48)">,</span><font color="#000000">MPI_DOUBLE</font><span style="color:rgb(128,128,48)">,</span><font color="#000000">z</font><span style="color:rgb(128,128,48)">,</span><font color="#000000">tag</font><span style="color:rgb(128,128,48)">,</span><font color="#000000"> MPI_COMM_WORLD</font><span style="color:rgb(128,128,48)">,</span><font color="#000000"> </font><span style="color:rgb(128,128,48)">&</span><font color="#000000">s_sol_req_</font><span style="color:rgb(128,128,48)">[</span><font color="#000000">n_proc_exchange</font><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span><font color="#000000">
MPI_Irecv</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">&</span><font color="#000000">rev_buf_</font><span style="color:rgb(128,128,48)">[</span><font color="#000000">z</font><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,128,48)">,</span><font color="#000000">n_</font><font color="#000000">buf_</font><span style="color:rgb(128,128,48)">[</span><font color="#000000">z</font><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,128,48)">,</span><font color="#000000">MPI_DOUBLE</font><span style="color:rgb(128,128,48)">,</span><font color="#000000">z</font><span style="color:rgb(128,128,48)">,</span><font color="#000000">tag</font><span style="color:rgb(128,128,48)">,</span><font color="#000000"> MPI_COMM_WORLD</font><span style="color:rgb(128,128,48)">,</span><font color="#000000"> </font><span style="color:rgb(128,128,48)">&</span><font color="#000000">r_sol_req_</font><span style="color:rgb(128,128,48)">[</span><font color="#000000">n_proc_exchange</font><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span><font color="#000000">
</font><font color="#000000"> n_proc_exchange</font><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,0,128)">;</span><font color="#000000">
</font><span style="color:rgb(128,0,128)">}</span><font color="#000000">
</font><span style="color:rgb(128,0,128)">}</span><font color="#000000">
</font><span style="color:rgb(128,0,128)">}</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">template</span><span style="color:rgb(128,0,128)"><</span><span style="color:rgb(128,0,0);font-weight:bold">typename</span><font color="#000000"> T</font><span style="color:rgb(128,0,128)">></span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">void</span><font color="#000000"> CPR_NS_3D_Solver</font><span style="color:rgb(128,0,128)"><</span><font color="#000000">T</font><span style="color:rgb(128,0,128)">></span><span style="color:rgb(128,0,128)">::</span><font color="#000000">RevInterf</font><font color="#000000">aceSol</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span><font color="#000000">
</font><font color="#000000"> uint </font><span style="color:rgb(128,128,48)">*</span><font color="#000000">n_if_to_proc</font><span style="color:rgb(128,128,48)">=</span><span style="color:rgb(128,0,0);font-weight:bold">this</span><span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span><font color="#000000">grid_</font><span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span><font color="#000000">num</font><font color="#000000">_iface_proc</font><span style="color:rgb(128,0,128)">;</span><font color="#000000">
uint </font><span style="color:rgb(128,128,48)">*</span><span style="color:rgb(128,128,48)">*</span><font color="#000000">if_to_proc</font><span style="color:rgb(128,128,48)">=</span><span style="color:rgb(128,0,0);font-weight:bold">this</span><span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span><font color="#000000">grid_</font><span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span><font color="#000000">snd_</font><font color="#000000">iface_proc</font><span style="color:rgb(128,0,128)">;</span><font color="#000000">
uint </font><span style="color:rgb(128,128,48)">*</span><span style="color:rgb(128,128,48)">*</span><font color="#000000">rev_if_to_f</font><span style="color:rgb(128,128,48)">=</span><span style="color:rgb(128,0,0);font-weight:bold">this</span><span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span><font color="#000000">grid_</font><span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">></span><font color="#000000">rev</font><font color="#000000">_iface_proc</font><span style="color:rgb(128,0,128)">;</span><font color="#000000">
</font><span style="color:rgb(105,105,105)">//wait</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span><font color="#000000">n_proc_exchange_</font><span style="color:rgb(128,128,48)">></span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span><font color="#000000">
</font><font color="#000000"> </font><span style="color:rgb(105,105,105)">MPI_Waitall(n_proc_exchange_,s_sol_req_,MPI_STATUS_IGNORE);</span><font color="#000000">
</font><font color="#000000"> </font><span style="color:rgb(105,105,105)">MPI_Waitall(n_proc_exchange_,r_sol_req_,MPI_STATUS_IGNORE);</span><font color="#000000">
</font><font color="#000000"> </font><span style="color:rgb(128,0,128)">}</span><font color="#000000">
</font><span style="color:rgb(63,95,191)">/** store to local data structure **/</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">for</span><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,0,0);font-weight:bold">int</span><font color="#000000"> z</font><span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,0,128)">;</span><font color="#000000">z</font><span style="color:rgb(128,128,48)"><</span><font color="#000000">_n_proc</font><span style="color:rgb(128,0,128)">;</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><font color="#000000">z</font><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">int</span><font color="#000000"> nif</font><span style="color:rgb(128,128,48)">=</span><font color="#000000">n_if_to_proc</font><span style="color:rgb(128,128,48)">[</span><font color="#000000">z</font><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">if</span><span style="color:rgb(128,128,48)">(</span><font color="#000000">nif</font><span style="color:rgb(128,128,48)">></span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">{</span></pre><pre> //unpacking ....
<font color="#000000"> </font><span style="color:rgb(128,0,128)">}</span><font color="#000000">
</font><span style="color:rgb(128,0,128)">}</span><font color="#000000">
</font><span style="color:rgb(128,0,128)">}</span></pre></pre></div><div><br></div><div><br></div><div><br></div><div><br></div><div class="gmail_extra"><span><br clear="all"><div><div>Sincerely Yours,<br><br>Lei Shi <br>---------</div></div>
<br></span><div><div><div class="gmail_quote">On Fri, Apr 3, 2015 at 4:37 PM, Jeff Hammond <span dir="ltr"><<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">As far as I know, Ethernet is not good at making asynchronous progress in hardware the way e.g. InfiniBand is. I would have thought that a dedicated progress thread would help, but it seems you tried that. Did you use your own progress thread or MPICH_ASYNC_PROGRESS=1?<br><br>Jeff<br><div><div><div><br><div class="gmail_quote">On Fri, Apr 3, 2015 at 10:10 AM, Lei Shi <span dir="ltr"><<a href="mailto:lshi@ku.edu" target="_blank">lshi@ku.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Huiwei,<div><br></div><div>Thanks for your email. Your answer leads to my another question about <span style="font-size:12.8000001907349px">asynchronous </span>MPI communication.</div><div><br></div><div>I'm trying to do an overlapped communication/computing to speedup my MPI code. I read some papers comparing some different approaches to do the overlapped communication. The "naive" overlapped communication implementation, which only use non-blocking mpi Isend/Irecv and the hybrid approach using OpenMP and MPI together. In the hybrid approach, a separated thread is use to do all non-blocking communications. Just exactly as you said, the results indicate that <span style="font-size:12.8000001907349px">current MPI implementations do not support true asynchronous communication.</span></div><div><br></div><div>If I use the naive approach, my code with non-blocking or blocking send/recv gives me almost the same performance in term of Wtime. All communications are postponed to MPI_Wait.</div><div><br></div><div>I have tried calling mpi_test to push library to do communication during iterations. And try to use a dedicated thread to do communication and the other thread to do computing only. However, the performance gains are very small or no gain at all. I'm wondering it is due to the hardware. The cluster I tested uses 10G Ethernet card.</div><div><br></div><div><br></div><div>Best,</div><div><br></div><div>Lei Shi</div><div><br></div><div><br></div><div><br></div><div><br><img style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><img style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 3, 2015 at 8:49 AM, Huiwei Lu <span dir="ltr"><<a href="mailto:huiweilu@mcs.anl.gov" target="_blank">huiweilu@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi Lei,<div><br></div><div>As far as I know, all current MPI implementations do not support true asynchronous communication for now. i.e., If there is no MPI calls in your iterations, MPICH will not be able to make progress on communication.</div><div><br></div><div>One solution is to poll the MPI runtime regularly to make progress by inserting MPI_Test to your iteration (even though you do not want to check the data).</div><div><br></div><div>Another solution is to enable MPI's asynchronous progress thread to make progress for you.</div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">--</div><div dir="ltr">Huiwei</div></div></div></div></div></div><div><div>
<br><div class="gmail_quote">On Thu, Apr 2, 2015 at 11:44 PM, Lei Shi <span dir="ltr"><<a href="mailto:lshi@ku.edu" target="_blank">lshi@ku.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><span style="font-size:12.8000001907349px">Hi Junchao,</span><div style="font-size:12.8000001907349px"><br></div><div style="font-size:12.8000001907349px">Thanks for your reply. For my case, I don't want to check the data have been received or not. So I don't want to call MPI_Test or any function to verify it. But my problem is like if I ignore calling the MPI_Wait, just call Isend/Irev, my program freezes for several sec and then continues to run. My guess is probably I messed up the MPI library internal buffer by doing this. </div><img style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><img style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 2, 2015 at 7:25 PM, Junchao Zhang <span dir="ltr"><<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Does MPI_Test fit your needs?</div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr">--Junchao Zhang</div></div></div>
<br><div class="gmail_quote"><div><div>On Thu, Apr 2, 2015 at 7:16 PM, Lei Shi <span dir="ltr"><<a href="mailto:lshi@ku.edu" target="_blank">lshi@ku.edu</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><div dir="ltr"><div style="font-size:12.8000001907349px">I want to use non-blocking send/rev MPI_Isend/MPI_Irev to do communication. But in my case, I don't really care what kind of data I get or it is ready to use or not. So I don't want to waste my time to do any synchronization by calling MPI_Wait or etc API. </div><div style="font-size:12.8000001907349px"><br></div><div style="font-size:12.8000001907349px">But when I avoid calling MPI_Wait, my program is freezed several secs after running some iterations (after multiple MPI_Isend/Irev callings), then continues. It takes even more time than the case with MPI_Wait. So my question is how to do a "true" non-blocking communication without waiting for the data ready or not. Thanks.</div><img style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><img style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"></div>
<br></div></div>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div>
<br>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div></div></div></div>
<br>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div>Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</font></span></div>
<br>_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div></div></div><img src="https://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/2b67d6832d81ffc52d70f73174c57af7/spacer.gif" style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><img src="http://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/2b67d6832d81ffc52d70f73174c57af7/spacer.gif" style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><font face="yw-d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9-2b67d6832d81ffc52d70f73174c57af7--to"></font></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>