<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 22, 2014, at 11:42 PM, Anatoly G <<a href="mailto:anatolyrishon@gmail.com" class="">anatolyrishon@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Thank you very much.<div class=""><br class=""><div class="">I'm using MPICH 3.1.</div><div class=""><br class=""></div><div class="">The application purpose is continue execution until at least Master process is live. The assumption is that processes can fail, but no more than single failure at a time. In this case survived processes must continue "like a Terminator". (-:<br class=""></div><div class="">In run-time I can't execute "manually" MPI_Test on all requests, it's too heavy. I have a network traffic ~ 400-500 Mb/s.</div><div class=""><br class=""></div><div class="">Implementation:</div><div class="">I set -disable-auto-cleanup flag to activate Fault tolerance.</div><div class="">All my processes activate MPI_Irecv on other processes and then execute MPI_Waitany on all active requests. When data arrives correctly, I process it and execute new MPI_Irecv on source process. If MPI_Waitany returns error (some process failed) I recognize failed rank and stop communication with it on application level (no more Sends & Recvs to it). In this mode system continues execution with survived processes. I don't use any collective operations, I simulate them using MPI_Irecv & MPI_Isend + MPI_Waitany or MPI_Waitall (returns error if some process failed).</div><div class=""><br class=""></div><div class="">I think it's ugly solution, but I can't think on any more elegant solution.</div><div class="">Any other solution at all would be welcome.</div></div></div></div></blockquote><div><br class=""></div>You could use MPI_Testall/any if you’d like, but of course that may not fit your application usage. Overall, I don’t see an issue with your model.</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><br class=""></div><div class="">The problem in "ready" state, when there is almost only waiting 3-4 messages may be received by process. In this state all CPUs are busy by executing polling mechanism.</div></div></div></div></blockquote><div><br class=""></div>That’s to be expected. While waiting for messages using MPI_Wait, it’s expected that nothing else is happening (unless you’re using threads as well, which it appears that you aren’t). That puts MPI into a busy wait to receive the message. As I mentioned previously, if you want to poll less frequently, you’ll have to use some flavor of MPI_Test.</div><div><br class=""></div><div>Out of curiosity, what is the issue with using 100% of the CPU? If you’re not using it for your application (which it appears that you aren’t since you’re calling MPI_Wait), what difference does it make if MPI uses all of it?</div><div><br class=""></div><div>Thanks,</div><div>Wesley</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Anatoly.</div><div class=""><br class=""></div></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Mon, Sep 22, 2014 at 4:46 PM, Wesley Bland <span dir="ltr" class=""><<a href="mailto:wbland@anl.gov" target="_blank" class="">wbland@anl.gov</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">Which version of MPICH are you using?<div class=""><br class=""></div><div class="">Which fault tolerance features are you using? Fault tolerance is currently undergoing some changes and has different features than it used to have.</div><div class=""><br class=""></div><div class="">AFAIK, neither version of FT has been tested with ch3:sock. It’s possible that it will work, but FT is still a very experimental feature and hasn’t been widely tested.</div><div class=""><br class=""></div><div class="">If you want to avoid polling more, you can use non-blocking receive calls to post a receive and poll the system yourself periodically (using MPI_TEST). This will give your application an opportunity to do something else while waiting for the receives to complete.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Wesley</div><div class=""><div class="h5"><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Sep 22, 2014, at 8:30 AM, Anatoly G <<a href="mailto:anatolyrishon@gmail.com" target="_blank" class="">anatolyrishon@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Dear MPICH.<div class="">I have a problem with poling MPICH mechanism.</div><div class="">I'm working on cluster. There are 2-4 processes on each computer (I can't execute single process per computer because of application requirements).</div><div class="">My system has 2 states:</div><div class="">Ready - slaves listen to master (but no data flow)</div><div class="">Run - masters start communication, then there is data flow.</div><div class="">When system in ready state (all processes except master executed MPI_Recv requests on master) but Master process still net sending data I see CPU usage > 100% (more than 1 core used) per process. When 4 processes are in ready state (waiting for data) computer begins to slow down other processes, I think because of polling.</div><div class="">I tried to build MPICH with <span style="font-family:arial,sans-serif;font-size:12.666666984558105px" class=""> </span><span style="font-family:arial,sans-serif;font-size:12.666666984558105px" class="">--with-device=ch3:sock, then I get 0% CPU usage in ready state, but I have a problem with Fault tolerance feature.</span></div><div class=""><span style="font-family:arial,sans-serif;font-size:12.666666984558105px" class="">My questions are:</span></div><div class=""><font face="arial, sans-serif" class=""><span style="font-size:12.6666669845581px" class="">1) Is such behavior expected that build with </span></font><span style="font-family:arial,sans-serif;font-size:12.666666984558105px" class="">--with-device=ch3:sock causes Fault tolerance not work? Does Fault tolerance based on polling mechanism?</span></div><div class=""><span style="font-family:arial,sans-serif;font-size:12.666666984558105px" class="">2) Can I change polling rate to reduce CPU payload? I understand that penalty is transfer rate slow down.</span></div><div class=""><span style="font-family:arial,sans-serif;font-size:12.666666984558105px" class="">3) Can I use any other MPI APIs to check if message from master is arrived w/o activating polling mechanism?</span></div><div class=""><span style="font-family:arial,sans-serif;font-size:12.666666984558105px" class=""><br class=""></span></div><div class=""><span style="font-family:arial,sans-serif;font-size:12.666666984558105px" class="">Regards,</span></div><div class=""><span style="font-family:arial,sans-serif;font-size:12.666666984558105px" class="">Anatoly. </span></div><div class=""><span style="font-family:arial,sans-serif;font-size:12.666666984558105px" class=""><br class=""></span></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Thu, May 8, 2014 at 3:57 PM, Balaji, Pavan <span dir="ltr" class=""><<a href="mailto:balaji@anl.gov" target="_blank" class="">balaji@anl.gov</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="">
This is expected. Currently, the only way to not have MPICH poll is to configure with --with-device=ch3:sock. Please note that this can cause performance loss (the polling is helpful for performance in the common case).<br class="">
<br class="">
We are planning to allow this in the default build as well in the future.<br class="">
<br class="">
— Pavan<br class="">
<div class=""><div class=""><br class="">
On May 8, 2014, at 7:54 AM, Anatoly G <<a href="mailto:anatolyrishon@gmail.com" target="_blank" class="">anatolyrishon@gmail.com</a>> wrote:<br class="">
<br class="">
> Dear MPICH forum.<br class="">
> I created an endless MPI program.<br class="">
> In this program each process calls MPI_Recv from other process, w/o any MPI_Send.<br class="">
> When I execute this program I see each process takes ~ 100% CPU core.<br class="">
> Is this behavior (I suppose polling) is normal?<br class="">
> May I reduce MPI_Recv CPU penalty?<br class="">
><br class="">
> Regards,<br class="">
> Anatoly.<br class="">
</div></div>> <mpi_polling.cpp>_______________________________________________<br class="">
> discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank" class="">discuss@mpich.org</a><br class="">
> To manage subscription options or unsubscribe:<br class="">
> <a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank" class="">https://lists.mpich.org/mailman/listinfo/discuss</a><br class="">
<br class="">
_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank" class="">https://lists.mpich.org/mailman/listinfo/discuss</a><br class="">
</blockquote></div><br class=""></div>
_______________________________________________<br class="">discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank" class="">discuss@mpich.org</a><br class="">To manage subscription options or unsubscribe:<br class=""><a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></div></blockquote></div><br class=""></div></div></div></div><br class="">_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank" class="">https://lists.mpich.org/mailman/listinfo/discuss</a><br class=""></blockquote></div><br class=""></div>
_______________________________________________<br class="">discuss mailing list <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a><br class="">To manage subscription options or unsubscribe:<br class=""><a href="https://lists.mpich.org/mailman/listinfo/discuss" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></div></blockquote></div><br class=""></body></html>