<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body data-blackberry-caret-color="#00a8df" style="background-color: rgb(255, 255, 255); line-height: initial;">
<div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">
Well yes, the job runs on host2 locally, but parallel execution does the same thing like on host1. </div>
<div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">
<span style="font-size: initial; text-align: initial; line-height: initial;">Someone here said that if all the computers have the same ip direction it won't work..</span></div>
<div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">
<span style="font-size: initial; text-align: initial; line-height: initial;">Well, every node has 127.0.1.1 As ip, and all of them had same host name till I changed the two of them. Hydra is the default launcher. </span></div>
<div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">
<span style="font-size: initial; text-align: initial; line-height: initial;">I also forgot to mencione that ping host2 and same, ping host1 works fine...</span></div>
<div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">
Sent from my BlackBerry 10 smartphone.</div>
<table width="100%" style="background-color:white;border-spacing:0px;">
<tbody>
<tr>
<td colspan="2" style="font-size: initial; text-align: initial; background-color: rgb(255, 255, 255);">
<div id="_persistentHeader" style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in; font-family: Tahoma, 'BB Alpha Sans', 'Slate Pro'; font-size: 10pt;">
<div><b>From: </b><span style="font-size: smaller; text-align: initial; line-height: initial;"></span><span style="font-size: smaller; text-align: initial; line-height: initial;">discuss-request@mpich.org</span></div>
<div><b>Sent: </b><span style="font-size: smaller; text-align: initial; line-height: initial;"></span><span style="font-size: smaller; text-align: initial; line-height: initial;">Wednesday, 23 October 2013 22:42</span></div>
<div><b>To: </b>discuss@mpich.org</div>
<div><b>Reply To: </b>discuss@mpich.org</div>
<div><b>Subject: </b>discuss Digest, Vol 12, Issue 13</div>
</div>
</td>
</tr>
</tbody>
</table>
<div style="border-style: solid none none; border-top-color: rgb(186, 188, 209); border-top-width: 1pt; font-size: initial; text-align: initial; background-color: rgb(255, 255, 255);">
</div>
<br>
<div class="BodyFragment">
<div class="PlainText">Send discuss mailing list submissions to<br>
discuss@mpich.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
discuss-request@mpich.org<br>
<br>
You can reach the person managing the list at<br>
discuss-owner@mpich.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: running parallel job issue! (Reuti)<br>
2. Re: running parallel job issue! (Ricardo Rom?n Brenes)<br>
3. Failed to allocate memory for an unexpected message<br>
(Luiz Carlos da Costa Junior)<br>
4. Re: Failed to allocate memory for an unexpected message<br>
(Antonio J. Pe?a)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Wed, 23 Oct 2013 17:00:53 +0200<br>
From: Reuti <reuti@staff.uni-marburg.de><br>
To: discuss@mpich.org<br>
Subject: Re: [mpich-discuss] running parallel job issue!<br>
Message-ID:<br>
<071A0111-ECEF-437E-9D3C-7412FDCC84D5@staff.uni-marburg.de><br>
Content-Type: text/plain; charset=iso-8859-1<br>
<br>
Am 23.10.2013 um 16:51 schrieb Ricardo Rom?n Brenes:<br>
<br>
> Hi. I'm not sure how it's done on that version of mpich, but you need to be sure that the mpd daemon is running on both hosts (mpdboot or maybe mpiexec starts it ).<br>
<br>
No. The `mpdboot` is gone for some time - nowadays Hydra is used (since MPICH2 v1.3).<br>
<br>
-- Reuti<br>
<br>
<br>
> Does cpi runs in hosts 2 locally?<br>
> <br>
> On Oct 23, 2013 8:47 AM, "Alexandra Betouni" <alexandra_99@windowslive.com> wrote:<br>
> <br>
> <br>
> <br>
> <br>
> Hey there, I am trying to set up a parallel invironment with 14 machines, running Linux XUbuntu, connected via ethernet.<br>
> They all have same IP's and same hostnames. Well I started installing mpich-3.0.4 on a single machine, I run the cpi example on localhost by giving mpiexec -host localhost -n 4 ./examples/cpi and everything worked fine!<br>
> So I continued changing the hostnames of 2 pc's for a start, and setting up the ssh in these two, also I installed the mpich-3.0.4 on the other machine too.<br>
> By giving the ssh <othermachine> date commant , I get the date of the other host without giving a password, so I think I passed that step too.<br>
> Next step was to check if the mpich-3.0.4 runs parallel, so I created a machine file (I made a text file giving the hostnames of the two computers , host1 and host2), and save it in my mpich-3.0.4 build directory. Though when I am trying to parallel run
the cpi code by giving mpiexec -n 4 -f machinefile ./examples/cpi on my working directory, I get NO errors but neither parallel job...<br>
> All processes still running on host1 which is my work station. <br>
> What am I doing wrong?<br>
> Thanks<br>
> <br>
> <br>
> _______________________________________________<br>
> discuss mailing list discuss@mpich.org<br>
> To manage subscription options or unsubscribe:<br>
> <a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
> _______________________________________________<br>
> discuss mailing list discuss@mpich.org<br>
> To manage subscription options or unsubscribe:<br>
> <a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 23 Oct 2013 09:04:29 -0600<br>
From: Ricardo Rom?n Brenes <roman.ricardo@gmail.com><br>
To: discuss@mpich.org<br>
Subject: Re: [mpich-discuss] running parallel job issue!<br>
Message-ID:<br>
<CAG-vK_yyAQ_CNFhYBxqoMniE01HVMA2YxJH=wC79m9KkBDVWFg@mail.gmail.com><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
Then the mpiexec should handle it... sorry, I use an outdated version that<br>
is in centos repos<br>
On Oct 23, 2013 9:01 AM, "Reuti" <reuti@staff.uni-marburg.de> wrote:<br>
<br>
> Am 23.10.2013 um 16:51 schrieb Ricardo Rom?n Brenes:<br>
><br>
> > Hi. I'm not sure how it's done on that version of mpich, but you need to<br>
> be sure that the mpd daemon is running on both hosts (mpdboot or maybe<br>
> mpiexec starts it ).<br>
><br>
> No. The `mpdboot` is gone for some time - nowadays Hydra is used (since<br>
> MPICH2 v1.3).<br>
><br>
> -- Reuti<br>
><br>
><br>
> > Does cpi runs in hosts 2 locally?<br>
> ><br>
> > On Oct 23, 2013 8:47 AM, "Alexandra Betouni" <<br>
> alexandra_99@windowslive.com> wrote:<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Hey there, I am trying to set up a parallel invironment with 14<br>
> machines, running Linux XUbuntu, connected via ethernet.<br>
> > They all have same IP's and same hostnames. Well I started installing<br>
> mpich-3.0.4 on a single machine, I run the cpi example on localhost by<br>
> giving mpiexec -host localhost -n 4 ./examples/cpi and everything worked<br>
> fine!<br>
> > So I continued changing the hostnames of 2 pc's for a start, and setting<br>
> up the ssh in these two, also I installed the mpich-3.0.4 on the other<br>
> machine too.<br>
> > By giving the ssh <othermachine> date commant , I get the date of the<br>
> other host without giving a password, so I think I passed that step too.<br>
> > Next step was to check if the mpich-3.0.4 runs parallel, so I created a<br>
> machine file (I made a text file giving the hostnames of the two computers<br>
> , host1 and host2), and save it in my mpich-3.0.4 build directory. Though<br>
> when I am trying to parallel run the cpi code by giving mpiexec -n 4 -f<br>
> machinefile ./examples/cpi on my working directory, I get NO errors but<br>
> neither parallel job...<br>
> > All processes still running on host1 which is my work station.<br>
> > What am I doing wrong?<br>
> > Thanks<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > discuss mailing list discuss@mpich.org<br>
> > To manage subscription options or unsubscribe:<br>
> > <a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
> > _______________________________________________<br>
> > discuss mailing list discuss@mpich.org<br>
> > To manage subscription options or unsubscribe:<br>
> > <a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
><br>
> _______________________________________________<br>
> discuss mailing list discuss@mpich.org<br>
> To manage subscription options or unsubscribe:<br>
> <a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.mpich.org/pipermail/discuss/attachments/20131023/fdf9d820/attachment-0001.html">http://lists.mpich.org/pipermail/discuss/attachments/20131023/fdf9d820/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Wed, 23 Oct 2013 17:27:27 -0200<br>
From: Luiz Carlos da Costa Junior <lcjunior@ufrj.br><br>
To: MPICH Discuss <mpich-discuss@mcs.anl.gov><br>
Subject: [mpich-discuss] Failed to allocate memory for an unexpected<br>
message<br>
Message-ID:<br>
<CAOv4ofRY4ajVZecZcDN3d3tdENV=XBMd=5i1TjX3310ZnEFUdg@mail.gmail.com><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
Hi,<br>
<br>
I am getting the following error when running my parallel application:<br>
<br>
MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060,<br>
MPI_CHARACTER, src=24, tag=94, comm=0x84000002, status=0x125fcff0) failed<br>
MPIDI_CH3I_Progress(402)...........:<br>
MPID_nem_mpich2_blocking_recv(905).:<br>
MPID_nem_tcp_connpoll(1838)........:<br>
state_commrdy_handler(1676)........:<br>
MPID_nem_tcp_recv_handler(1564)....:<br>
MPID_nem_handle_pkt(636)...........:<br>
MPIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for an<br>
unexpected message. 261895 unexpected messages queued.<br>
Fatal error in MPI_Send: Other MPI error, error stack:<br>
MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060,<br>
MPI_CHARACTER, dest=0, tag=94, comm=0x84000004) failed<br>
MPID_nem_tcp_connpoll(1826): Communication error with rank 1: Connection<br>
reset by peer<br>
<br>
<br>
I went to MPICH's FAQ (<br>
<a href="http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_am_I_getting_so_many_unexpected_messages.3F">http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_am_I_getting_so_many_unexpected_messages.3F</a><br>
).<br>
It says that most likely the receiver process can't cope to process the<br>
high number of messages it is receiving.<br>
<br>
In my application, the worker processes perform a very large number of<br>
small computations and, after some computation is complete, they sent the<br>
data to a special "writer" process that is responsible to write the output<br>
to disk.<br>
This scheme use to work in a very reasonable fashion, until we faced some<br>
new data with larger parameters that caused the problem above.<br>
<br>
Even though we can redesign the application, for example, by creating a<br>
pool of writer process we still have only one hard disk, so the bottleneck<br>
would not be solved. So, this doesn't seem to be a good approach.<br>
<br>
As far as I understood, MPICH saves the content of every MPI_SEND in a<br>
internal buffer (I don't know where the buffer in located, sender or<br>
receiver?) to allow asynchronous sender's computation while the messages<br>
are being received.<br>
The problem is that buffer has been exhausted due some resource limitation.<br>
<br>
It is very interesting to have a buffer but if the buffer in the writer<br>
process is close to its limit the workers processes should stop and wait<br>
until it frees some space to restart sending new data to be written to disk.<br>
<br>
Is it possible to check this buffer in MPICH? Or is it possible to check<br>
the number of messages to be received?<br>
Can anyone suggest a better (easy to implement) solution?<br>
<br>
Thanks in advance.<br>
<br>
Regards,<br>
Luiz<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.mpich.org/pipermail/discuss/attachments/20131023/3a02fa51/attachment-0001.html">http://lists.mpich.org/pipermail/discuss/attachments/20131023/3a02fa51/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Wed, 23 Oct 2013 14:42:15 -0500<br>
From: Antonio J. Pe?a <apenya@mcs.anl.gov><br>
To: discuss@mpich.org<br>
Cc: MPICH Discuss <mpich-discuss@mcs.anl.gov><br>
Subject: Re: [mpich-discuss] Failed to allocate memory for an<br>
unexpected message<br>
Message-ID: <1965559.SsluspJNke@localhost.localdomain><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
<br>
Hi Luiz,<br>
<br>
Your error trace indicates that the receiver went out of memory due to a <br>
too large amount (261,895) of eager unexpected messages received, i.e., <br>
small messages received without a matching receive operation. Whenever <br>
this happens, the receiver allocates a temporary buffer to hold the <br>
received message. This exhausted the available memory in the computer <br>
where the receiver was executing.<br>
<br>
To avoid this, try to pre-post receives before messages arrive. Indeed, this <br>
is far more efficient. Maybe you could do an MPI_IRecv per worker in your <br>
writer process, and process them after an MPI_Waitany. You may also <br>
consider having multiple writer processes if your use case permits and the <br>
volume of received messages is too high to be processed by a single <br>
writer.<br>
<br>
Antonio<br>
<br>
<br>
On Wednesday, October 23, 2013 05:27:27 PM Luiz Carlos da Costa Junior <br>
wrote:<br>
<br>
<br>
Hi,<br>
<br>
<br>
I am getting the following error when running my parallel application:<br>
<br>
<br>
MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060, <br>
MPI_CHARACTER, src=24, tag=94, comm=0x84000002, status=0x125fcff0) <br>
failed <br>
MPIDI_CH3I_Progress(402)...........: <br>
MPID_nem_mpich2_blocking_recv(905).: <br>
MPID_nem_tcp_connpoll(1838)........: <br>
state_commrdy_handler(1676)........: <br>
MPID_nem_tcp_recv_handler(1564)....: <br>
MPID_nem_handle_pkt(636)...........: <br>
MPIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for an <br>
unexpected message. 261895 unexpected messages queued. <br>
Fatal error in MPI_Send: Other MPI error, error stack:<br>
MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060, <br>
MPI_CHARACTER, dest=0, tag=94, comm=0x84000004) failed <br>
MPID_nem_tcp_connpoll(1826): Communication error with rank 1: <br>
Connection reset by peer <br>
<br>
<br>
I went to MPICH's FAQ <br>
(<a href=""></a>http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Wh<br>
y_am_I_getting_so_many_unexpected_messages.3F[1]). <br>
It says that most likely the receiver process can't cope to process the high <br>
number of messages it is receiving.<br>
<br>
<br>
In my application, the worker processes perform a very large number of <br>
small computations and, after some computation is complete, they sent <br>
the data to a special "writer" process that is responsible to write the <br>
output to disk. <br>
This scheme use to work in a very reasonable fashion, until we faced some <br>
new data with larger parameters that caused the problem above.<br>
<br>
<br>
Even though we can redesign the application, for example, by creating a <br>
pool of writer process we still have only one hard disk, so the bottleneck <br>
would not be solved. So, this doesn't seem to be a good approach. <br>
<br>
<br>
As far as I understood, MPICH saves the content of every MPI_SEND in a <br>
internal buffer (I don't know where the buffer in located, sender or <br>
receiver?) to allow asynchronous sender's computation while the <br>
messages are being received. <br>
The problem is that buffer has been exhausted due some resource <br>
limitation.<br>
<br>
<br>
It is very interesting to have a buffer but if the buffer in the writer process <br>
is close to its limit the workers processes should stop and wait until it <br>
frees some space to restart sending new data to be written to disk. <br>
<br>
<br>
Is it possible to check this buffer in MPICH? Or is it possible to check the <br>
number of messages to be received?<br>
Can anyone suggest a better (easy to implement) solution?<br>
<br>
<br>
<br>
<br>
Thanks in advance.<br>
<br>
<br>
Regards,<br>
<br>
<br>
Luiz<br>
<br>
<br>
<br>
--------<br>
[1] <br>
<a href="http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why">http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why</a><br>
_am_I_getting_so_many_unexpected_messages.3F<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.mpich.org/pipermail/discuss/attachments/20131023/ff1485b7/attachment.html">http://lists.mpich.org/pipermail/discuss/attachments/20131023/ff1485b7/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
discuss mailing list<br>
discuss@mpich.org<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
<br>
End of discuss Digest, Vol 12, Issue 13<br>
***************************************<br>
</div>
</div>
</body>
</html>