<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">
<html><head><meta name="qrichtext" content="1" /><style type="text/css">
p, li { white-space: pre-wrap; }
</style></head><body style=" font-family:'Sans Serif'; font-size:9pt; font-weight:400; font-style:normal;">
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Hi Luiz,</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Your error trace indicates that the receiver went out of memory due to a too large amount (261,895) of eager unexpected messages received, i.e., small messages received without a matching receive operation. Whenever this happens, the receiver allocates a temporary buffer to hold the received message. This exhausted the available memory in the computer where the receiver was executing.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">To avoid this, try to pre-post receives before messages arrive. Indeed, this is far more efficient. Maybe you could do an MPI_IRecv per worker in your writer process, and process them after an MPI_Waitany. You may also consider having multiple writer processes if your use case permits and the volume of received messages is too high to be processed by a single writer.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> Antonio</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">On Wednesday, October 23, 2013 05:27:27 PM Luiz Carlos da Costa Junior wrote:<br /></p>
<p style=" margin-top:12px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Hi,</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">I am getting the following error when running my parallel application:</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050;">MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060, MPI_CHARACTE</span><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">R, src=24, </span><span style=" font-family:'courier new, monospace'; background-color:#ffffff;">tag</span><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">=</span><span style=" font-family:'courier new, monospace'; background-color:#ffffff;">94</span><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">, comm=0x84000002, status=0x125fcff0) failed</span> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">MPIDI_CH3I_Progress(402)...........: </span> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">MPID_nem_mpich2_blocking_recv(905).: </span> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">MPID_nem_tcp_connpoll(1838)........: </span> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">state_commrdy_handler(1676)........: </span> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">MPID_nem_tcp_recv_handler(1564)....: </span> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">MPID_nem_handle_pkt(636)...........: </span> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">MPIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for an unexpected message. 261895 unexpected messages queued.</span> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">Fatal error in MPI_Send: Other MPI error, error stack:</span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060, MPI_CHARACTER, dest=0, </span><span style=" font-family:'courier new, monospace'; background-color:#ffffff;">tag</span><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">=</span><span style=" font-family:'courier new, monospace'; background-color:#ffffff;">94</span><span style=" font-family:'courier new, monospace'; color:#535050; background-color:#ffffff;">, comm=0x8</span><span style=" font-family:'courier new, monospace'; color:#535050;">4000004) failed</span> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'courier new, monospace'; color:#535050;">MPID_nem_tcp_connpoll(1826): Communication error with rank 1: Connection reset by peer</span> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">I went to MPICH's FAQ (<a href="http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_am_I_getting_so_many_unexpected_messages.3F"><span style=" text-decoration: underline; color:#0057ae;">http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_am_I_getting_so_many_unexpected_messages.3F</span></a>). </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">It says that most likely the receiver process can't cope to process the high number of messages it is receiving.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">In my application, the worker processes perform a very large number of small computations and, after some computation is complete, they sent the data to a special "writer" process that is responsible to write the output to disk. </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">This scheme use to work in a very reasonable fashion, until we faced some new data with larger parameters that caused the problem above.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Even though we can redesign the application, for example, by creating a pool of writer process we still have only one hard disk, so the bottleneck would not be solved. So, this doesn't seem to be a good approach. </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">As far as I understood, MPICH saves the content of every MPI_SEND in a internal buffer (I don't know where the buffer in located, sender or receiver?) to allow asynchronous sender's computation while the messages are being received. </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">The problem is that buffer has been exhausted due some resource limitation.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">It is very interesting to have a buffer but if the buffer in the writer process is close to its limit the workers processes should stop and wait until it frees some space to restart sending new data to be written to disk. </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Is it possible to check this buffer in MPICH? Or is it possible to check the number of messages to be received?</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Can anyone suggest a better (easy to implement) solution?<br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Thanks in advance.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Regards,<br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Luiz</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p></body></html>