[mpich-discuss] Error: failed to allocate memory for an unexpected message

[XingFENG]

Hi all,

I am running a MPI program on two machines. I got errors as follows:


====================================================================
Fatal error in MPI_Test: Other MPI error, error stack:
MPI_Test(153)......................: MPI_Test(request=0xa0a088,
flag=0x7fff470e86fc,  status=0x7fff470e86e0) failed
MPIDI_CH3I_Progress(150)...........:
MPID_nem_mpich2_test_recv(800).....:
MPID_nem_tcp_connpoll(1720)........:
state_commrdy_handler(1556)........:
MPID_nem_tcp_recv_handler(1459)....:
MPID_nem_handle_pkt(493)...........:
MPIDI_CH3_PktHandler_EagerSend(589): Failed to allocate memory for an
unexpected message. 261892 unexpected messages queued.
Fatal error in MPI_Test: Other MPI error, error stack:
MPI_Test(153)......................: MPI_Test(request=0xadb128,
flag=0x7fff33cba448, status=0x7fff33cba430) failed
MPIDI_CH3I_Progress(150)...........:
MPID_nem_mpich2_test_recv(800).....:
MPID_nem_tcp_connpoll(1720)........:
state_commrdy_handler(1556)........:
MPID_nem_tcp_recv_handler(1459)....:
MPID_nem_handle_pkt(493)...........:
MPIDI_CH3_PktHandler_EagerSend(589): Failed to allocate memory for an
unexpected message. 261890 unexpected messages queued.
rank 1 in job 11  slave_36134   caused collective abort of all ranks
  exit status of rank 1: killed by signal 9

====================================================================


I have never seen such errors before. What is the cause of this error? Is
it an out of memory error? ( There is 20% remaining memory on machines )

Any help would be greatly appreciated. Thanks in advance!


-- 
Best Regards.
---
Xing FENG
PhD Candidate
Database Research Group

School of Computer Science and Engineering
University of New South Wales
NSW 2052, Sydney

Phone: (+61) 413 857 288
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20140928/3bf0f66c/attachment.html>
-------------- next part --------------
_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss