<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">Thanks, Halim. I have now enabled asynchronous progress in MPICH (can't find something similar in openmpi) and now all ranks acquire the lock and the program finish as expected. However if I put a while(1) loop around the acquire-release code in main.c it will fail again at random and go into an infinite loop. The simple unfair lock does not have this problem.<br><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 7, 2017 at 12:44 AM, Halim Amer <span dir="ltr"><<a target="_blank" href="mailto:aamer@anl.gov">aamer@anl.gov</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">My understanding is that this code assumes asynchronous progress.<br>
An example of when the processes hang is as follows:<br>
<br>
1) P0 Finishes MCSLockAcquire()<br>
2) P1 is busy waiting in MCSLockAcquire() at<br>
do {<br>
MPI_Win_sync(win);<br>
} while (lmem[blocked] == 1);<br>
3) P0 executes MCSLockRelease()<br>
4) P0 waits on MPI_Win_lock_all() inside MCSLockRlease()<br>
<br>
Hang!<br>
<br>
For P1 to get out of the loop, P0 has to get out of MPI_Win_lock_all() and executes its Compare_and_swap().<br>
<br>
For P0 to get out MPI_Win_lock_all(), it needs an ACK from P1 that it got the lock.<br>
<br>
P1 does not make communication progress because MPI_Win_sync is not required to do so. It only synchronizes private and public copies.<br>
<br>
For this hang to disappear, one can either trigger progress manually by using heavy-duty synchronization calls instead of Win_sync (e.g., Win_unlock_all + Win_lock_all), or enable asynchronous progress.<br>
<br>
To enable asynchronous progress in MPICH, set the MPIR_CVAR_ASYNC_PROGRESS env var to 1.<br>
<br>
Halim<br>
<a target="_blank" rel="noreferrer" href="http://www.mcs.anl.gov/%7Eaamer">www.mcs.anl.gov/~aamer</a><div class="gmail-HOEnZb"><div class="gmail-h5"><br>
<br>
On 3/6/17 1:11 PM, Ask Jakobsen wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
I am testing on x86_64 platform.<br>
<br>
I have tried to built both the mpich and the mcs lock code with -O0 to<br>
avoid agressive optimization. After your suggestion I have also tried to<br>
make volatile int *pblocked pointing to lmem[blocked] in the MCSLockAcquire<br>
function and volatile int *pnextrank pointing to lmem[nextRank] in<br>
MCSLockRelease, but it does not appear to make a difference.<br>
<br>
On suggestion from Richard Warren I have also tried building the code using<br>
openmpi-2.0.2 without any luck (however it appears to acquire the lock a<br>
couple of extra times before failing) which I find troubling.<br>
<br>
I think I will give up using local load/stores and will see if I can figure<br>
out if rewrite using MPI calls like MPI_Fetch_and_op as you suggest.<br>
Thanks for your help.<br>
<br>
On Mon, Mar 6, 2017 at 7:20 PM, Jeff Hammond <<a target="_blank" href="mailto:jeff.science@gmail.com">jeff.science@gmail.com</a>> wrote:<br>
<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
What processor architecture are you testing?<br>
<br>
Maybe set lmem to volatile or read it with MPI_Fetch_and_op rather than a<br>
load. MPI_Win_sync cannot prevent the compiler from caching *lmem in a<br>
register.<br>
<br>
Jeff<br>
<br>
On Sat, Mar 4, 2017 at 12:30 AM, Ask Jakobsen <<a target="_blank" href="mailto:afj@qeye-labs.com">afj@qeye-labs.com</a>> wrote:<br>
<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
Hi,<br>
<br>
I have downloaded the source code for the MCS lock from the excellent<br>
book "Using Advanced MPI" from <a target="_blank" rel="noreferrer" href="http://www.mcs.anl.gov/researc">http://www.mcs.anl.gov/researc</a><br>
h/projects/mpi/usingmpi/exampl<wbr>es-advmpi/rma2/mcs-lock.c<br>
<br>
I have made a very simple piece of test code for testing the MCS lock but<br>
it works at random and often never escapes the busy loops in the acquire<br>
and release functions (see attached source code). The code appears<br>
semantically correct to my eyes.<br>
<br>
#include <stdio.h><br>
#include <mpi.h><br>
#include "mcs-lock.h"<br>
<br>
int main(int argc, char *argv[])<br>
{<br>
MPI_Win win;<br>
MPI_Init( &argc, &argv );<br>
<br>
MCSLockInit(MPI_COMM_WORLD, &win);<br>
<br>
int rank, size;<br>
MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br>
MPI_Comm_size(MPI_COMM_WORLD, &size);<br>
<br>
printf("rank: %d, size: %d\n", rank, size);<br>
<br>
<br>
MCSLockAcquire(win);<br>
printf("rank %d aquired lock\n", rank); fflush(stdout);<br>
MCSLockRelease(win);<br>
<br>
<br>
MPI_Win_free(&win);<br>
MPI_Finalize();<br>
return 0;<br>
}<br>
<br>
<br>
I have tested on several hardware platforms and mpich-3.2 and mpich-3.3a2<br>
but with no luck.<br>
<br>
It appears that the MPI_Win_Sync are not "refreshing" the local data or I<br>
have a bug I can't spot.<br>
<br>
A simple unfair lock like <a target="_blank" rel="noreferrer" href="http://www.mcs.anl.gov/researc">http://www.mcs.anl.gov/researc</a><br>
h/projects/mpi/usingmpi/exampl<wbr>es-advmpi/rma2/ga_mutex1.c works perfectly.<br>
<br>
Best regards, Ask Jakobsen<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a target="_blank" href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a target="_blank" rel="noreferrer" href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailma<wbr>n/listinfo/discuss</a><br>
<br>
</blockquote>
<br>
<br>
<br>
--<br>
Jeff Hammond<br>
<a target="_blank" href="mailto:jeff.science@gmail.com">jeff.science@gmail.com</a><br>
<a target="_blank" rel="noreferrer" href="http://jeffhammond.github.io/">http://jeffhammond.github.io/</a><br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a target="_blank" href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a target="_blank" rel="noreferrer" href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailma<wbr>n/listinfo/discuss</a><br>
<br>
</blockquote>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a target="_blank" href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a target="_blank" rel="noreferrer" href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailma<wbr>n/listinfo/discuss</a><br>
<br>
</blockquote>
______________________________<wbr>_________________<br>
discuss mailing list <a target="_blank" href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a target="_blank" rel="noreferrer" href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailma<wbr>n/listinfo/discuss</a><br>
</div></div></blockquote></div></div></div></div></div>