<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class="">We use the mailing list for MPICH-related questions, not general MPI questions. With respect to <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a>, I get those emails and the list has more people who might be able to help you as
well. Cc’ed the list. Please send MPICH-related emails to that list. Please send general “how do I write an MPI program” sort of questions to stack overflow or equivalent.</div>
<div class=""><br class="">
</div>
<div class="">I’ll respond to your email this time, but please avoid sending us general MPI questions in the future.</div>
<div class=""><br class="">
</div>
<div class="">I’m not sure what you mean by using a single buffer. Do you mean that, once you send the data to your “send_to" neighbor, you don’t need that buffer anymore and would like to overwrite that with the data coming in from the “recv_from” neighbor?</div>
<div class=""><br class="">
</div>
<div class="">Assuming that’s what you are looking for, there’s no good function to directly do what you are asking for. However, pipelining the data transfer might help the memory usage problem. Specifically, don’t allocate a full receive buffer, but a smaller
buffer for a part of the array. Once you receive the data and the corresponding data in your local array is full, you can overwrite it from the receive buffer.</div>
<div class=""><br class="">
</div>
<div class="">Something like this:</div>
<div class=""><br class="">
</div>
<div class="">#define CHUNK_SIZE (1024)</div>
<div class="">#define TOTAL_ARRAY_SIZE (CHUNK_SIZE * SOME_LARGE_NUMBER)</div>
<div class=""><br class="">
</div>
<div class="">struct foobar main_buffer[TOTAL_ARRAY_SIZE];</div>
<div class="">struct foobar temp_buffer[CHUNK_SIZE];</div>
<div class="">MPI_Request req[CHUNK_SIZE * 2];</div>
<div class=""><br class="">
</div>
<div class="">for (j = 0; j < SOME_LARGE_NUMBER; j++) {</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>for (i = 0; i < CHUNK_SIZE; i++) {</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>Isend(… data from main buffer …, &req[2 * i]);</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>Irecv(… data into temp buffer …, &req[2 * i + 1]);</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>}</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>WAITALL(…)</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>memcpy(&main_buffer[j * CHUNK_SIZE], &temp_buffer, …);</div>
<div class="">}</div>
<div class=""><br class="">
</div>
<div class="">You can make this a little better by not waiting for all chunks, but testing any of the chunks to complete and pipelining the copy as well.</div>
<div class=""><br class="">
</div>
<div class=""> — Pavan</div>
<div class=""><br class="">
</div>
<div>
<blockquote type="cite" class="">
<div class="">On Apr 25, 2015, at 5:44 PM, Params <<a href="mailto:praman1@ucsc.edu" class="">praman1@ucsc.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Hi Pavan,
<div class=""><br class="">
</div>
<div class="">Sorry for directly emailing you about this, but I am a new user to the mpich2 forums and I was not sure if my message to the mailing group reached you (please let me know about that too in case I need to use the forum again). I have a question
about mpich usage and would be great if any of you guys could help me out.</div>
<div class=""><br class="">
</div>
<div class="">
<div style="font-size:12.8000001907349px" class="">I have a use-case where I need to transmit an Array of Structures (AoS) among a group of processors in a circular ("ring") fashion.</div>
<div style="font-size:12.8000001907349px" class="">I do that currently by using a non-blocking send and receive, and using two auxiliary buffers on the senders and receivers side respectively.</div>
<div style="font-size:12.8000001907349px" class=""><br class="">
</div>
<div style="font-size:12.8000001907349px" class="">Since the data I am transferring is extremely huge (~60 gb before partitioning), allocating two buffers seems inefficient to me. My question is -- Is there a better way to do it by avoiding "two buffers"? (Perhaps
just using one buffer in some way?).</div>
<div style="font-size:12.8000001907349px" class=""><br class="">
</div>
<div style="font-size:12.8000001907349px" class="">Roughly my communication procedure looks like this:</div>
<div style="font-size:12.8000001907349px" class="">
<div class=""><br class="">
</div>
<div class="">//Pack AoS into send buffer (which is a vector<char>) </div>
<div class=""> char *msg_ptr = &(msg_send_buf_.front());</div>
<div class=""> for (auto& param : local_primal_params_) {</div>
<div class=""> param.pack_to(msg_ptr);</div>
<div class=""> msg_ptr += PARAM_BYTENUM;</div>
<div class=""> }</div>
<div class=""><br class="">
</div>
<div class="">//Perform non-blocking send receive<br class="">
</div>
<div class=""> {</div>
<div class=""> MPI_Request send_request;</div>
<div class=""> MPI_Status send_stat, recv_stat;</div>
<div class=""> MPI_Isend(&(msg_send_buf_.front()), local_primal_params_.size() * PARAM_BYTENUM, MPI_CHAR,</div>
<div class=""> send_to, 0,</div>
<div class=""> MPI_COMM_WORLD, &send_request);</div>
<div class=""> MPI_Recv(&(msg_recv_buf_.front()), local_primal_params_.size() * PARAM_BYTENUM, MPI_CHAR,</div>
<div class=""> recv_from, 0,</div>
<div class=""> MPI_COMM_WORLD, &recv_stat);</div>
<div class=""> MPI_Wait(&send_request, &send_stat);</div>
<div class=""> }</div>
<div class=""><br class="">
</div>
<div class="">//Unpack receive buffer back into AoS on receiver side<br class="">
</div>
<div class=""> {</div>
<div class=""> char *msg_ptr = &(msg_recv_buf_.front());</div>
<div class=""> for (auto& param : local_primal_params_) {</div>
<div class=""> param.unpack_from(msg_ptr);</div>
<div class=""> msg_ptr += PARAM_BYTENUM;</div>
<div class=""> }</div>
<div class=""> }</div>
</div>
<div style="font-size:12.8000001907349px" class="">
<div class=""><br class="">
</div>
<div class="">Any recommendations will be very helpful.</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Parameswaran Raman</div>
<div class="">PhD Student, UC Santa Cruz</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>