No subject


Tue Jun 18 13:52:11 CDT 2019 

loads and stores in the MPI_Win_lock_all epochs using MPI_Fetch_and_op (see=
 attached files).<br>
<br>
This version behaves very similar to the original code and also fails from =
time to time. Putting a sleep into the acquire busy loop (usleep(100)) will=
 make the code "much more robust" (I hack, I know, but indicating=
 some underlying race condition?!). Let me know if you see any problems in =
the way I am using MPI_Fetch_and_op in a busy loop. Flushing or syncing is =
not necessary in this case, right?<br>
<br>
All work is done with export MPIR_CVAR_ASYNC_PROGRESS=3D1 on mpich-3.2 and =
mpich-3.3a2<br>
<br>
On Wed, Mar 8, 2017 at 4:21 PM, Halim Amer <<a href=3D"mailto:aamer at anl.=
gov" target=3D"_blank">aamer at anl.gov</a>> wrote:<br>
I cannot claim that I thoroughly verified the correctness of that code, so =
take it with a grain of salt. Please keep in mind that it is a test code fr=
om a tutorial book; those codes are meant for learning purposes not for dep=
loyment.<br>
<br>
If your goal is to have a high performance RMA lock, I suggest you to look =
into the recent HPDC'16 paper: "High-Performance Distributed RMA Locks=
".<br>
<br>
Halim<br>
<a href=3D"http://www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=3D"_bla=
nk">www.mcs.anl.gov/~aamer</a><br>
<br>
On 3/8/17 3:06 AM, Ask Jakobsen wrote:<br>
You are absolutely correct, Halim. Removing the test lmem[nextRank] =3D=3D =
-1<br>
in release fixes the problem. Great work. Now I will try to understand why<=
br>
you are right. I hope the authors of the book will credit you for<br>
discovering the bug.<br>
<br>
So in conclusion you need to remove the above mentioned test AND enable<br>
asynchronous progression using the environment variable<br>
MPIR_CVAR_ASYNC_PROGRESS=3D1 in MPICH (BTW I still can't get the code to wo=
rk<br>
in openmpi).<br>
<br>
On Tue, Mar 7, 2017 at 5:37 PM, Halim Amer <<a href=3D"mailto:aamer at anl.=
gov" target=3D"_blank">aamer at anl.gov</a>> wrote:<br>
<br>
detect that another process is being or already enqueued in the MCS<br>
queue.<br>
<br>
Actually the problem occurs only when the waiting process already enqueued<=
br>
itself, i.e., the accumulate operation on the nextRank field succeeded.<br>
<br>
Halim<br>
<a href=3D"http://www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=3D"_bla=
nk">www.mcs.anl.gov/~aamer</a> <<a href=3D"http://www.mcs.anl.gov/%7Eaam=
er" rel=3D"noreferrer" target=3D"_blank">http://www.mcs.anl.gov/%7Eaam<wbr>=
er</a>><br>
<br>
<br>
On 3/7/17 10:29 AM, Halim Amer wrote:<br>
<br>
In the Release protocol, try removing this test:<br>
<br>
if (lmem[nextRank] =3D=3D -1) {<br>
  If-Block;<br>
}<br>
<br>
but keep the If-Block.<br>
<br>
The hang occurs because the process releasing the MCS lock fails to<br>
detect that another process is being or already enqueued in the MCS queue.<=
br>
<br>
Halim<br>
<a href=3D"http://www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=3D"_bla=
nk">www.mcs.anl.gov/~aamer</a> <<a href=3D"http://www.mcs.anl.gov/%7Eaam=
er" rel=3D"noreferrer" target=3D"_blank">http://www.mcs.anl.gov/%7Eaam<wbr>=
er</a>><br>
<br>
<br>
On 3/7/17 6:43 AM, Ask Jakobsen wrote:<br>
<br>
Thanks, Halim. I have now enabled asynchronous progress in MPICH (can't<br>
find something similar in openmpi) and now all ranks acquire the lock and<b=
r>
the program finish as expected. However if I put a while(1) loop<br>
around the<br>
acquire-release code in main.c it will fail again at random and go<br>
into an<br>
infinite loop. The simple unfair lock does not have this problem.<br>
<br>
On Tue, Mar 7, 2017 at 12:44 AM, Halim Amer <<a href=3D"mailto:aamer at anl=
.gov" target=3D"_blank">aamer at anl.gov</a>> wrote:<br>
<br>
My understanding is that this code assumes asynchronous progress.<br>
An example of when the processes hang is as follows:<br>
<br>
1) P0 Finishes MCSLockAcquire()<br>
2) P1 is busy waiting in MCSLockAcquire() at<br>
do {<br>
     MPI_Win_sync(win);<br>
  } while (lmem[blocked] =3D=3D 1);<br>
3) P0 executes MCSLockRelease()<br>
4) P0 waits on MPI_Win_lock_all() inside MCSLockRlease()<br>
<br>
Hang!<br>
<br>
For P1 to get out of the loop, P0 has to get out of<br>
MPI_Win_lock_all() and<br>
executes its Compare_and_swap().<br>
<br>
For P0 to get out MPI_Win_lock_all(), it needs an ACK from P1 that it<br>
got<br>
the lock.<br>
<br>
P1 does not make communication progress because MPI_Win_sync is not<br>
required to do so. It only synchronizes private and public copies.<br>
<br>
For this hang to disappear, one can either trigger progress manually by<br>
using heavy-duty synchronization calls instead of Win_sync (e.g.,<br>
Win_unlock_all + Win_lock_all), or enable asynchronous progress.<br>
<br>
To enable asynchronous progress in MPICH, set the<br>
MPIR_CVAR_ASYNC_PROGRESS<br>
env var to 1.<br>
<br>
Halim<br>
<a href=3D"http://www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=3D"_bla=
nk">www.mcs.anl.gov/~aamer</a> <<a href=3D"http://www.mcs.anl.gov/%7Eaam=
er" rel=3D"noreferrer" target=3D"_blank">http://www.mcs.anl.gov/%7Eaam<wbr>=
er</a>> <<br>
<a href=3D"http://www.mcs.anl.gov/%7Eaamer" rel=3D"noreferrer" target=3D"_b=
lank">http://www.mcs.anl.gov/%7Eaame<wbr>r</a>><br>
<br>
<br>
On 3/6/17 1:11 PM, Ask Jakobsen wrote:<br>
<br>
I am testing on x86_64 platform.<br>
<br>
I have tried to built both the mpich and the mcs lock code with -O0 to<br>
avoid agressive optimization. After your suggestion I have also<br>
tried to<br>
make volatile int *pblocked pointing to lmem[blocked] in the<br>
MCSLockAcquire<br>
function and volatile int *pnextrank pointing to lmem[nextRank] in<br>
MCSLockRelease, but it does not appear to make a difference.<br>
<br>
On suggestion from Richard Warren I have also tried building the code<br>
using<br>
openmpi-2.0.2 without any luck (however it appears to acquire the<br>
lock a<br>
couple of extra times before failing) which I find troubling.<br>
<br>
I think I will give up using local load/stores and will see if I can<br>
figure<br>
out if rewrite using MPI calls like MPI_Fetch_and_op  as you suggest.<=
br>
Thanks for your help.<br>
<br>
On Mon, Mar 6, 2017 at 7:20 PM, Jeff Hammond <<a href=3D"mailto:jeff.sci=
ence at gmail.com" target=3D"_blank">jeff.science at gmail.com</a>><br>
wrote:<br>
<br>
What processor architecture are you testing?<br>
<br>
<br>
Maybe set lmem to volatile or read it with MPI_Fetch_and_op rather<br>
than a<br>
load.  MPI_Win_sync cannot prevent the compiler from caching *lmem<br>
in a<br>
register.<br>
<br>
Jeff<br>
<br>
On Sat, Mar 4, 2017 at 12:30 AM, Ask Jakobsen <<a href=3D"mailto:afj at qey=
e-labs.com" target=3D"_blank">afj at qeye-labs.com</a>><br>
wrote:<br>
<br>
Hi,<br>
<br>
<br>
I have downloaded the source code for the MCS lock from the excellent<br>
book "Using Advanced MPI" from <a href=3D"http://www.mcs.anl.gov/=
researc" rel=3D"noreferrer" target=3D"_blank">http://www.mcs.anl.gov/resear=
c</a><br>
h/projects/mpi/usingmpi/exampl<wbr>es-advmpi/rma2/mcs-lock.c<br>
<br>
I have made a very simple piece of test code for testing the MCS lock<br>
but<br>
it works at random and often never escapes the busy loops in the<br>
acquire<br>
and release functions (see attached source code). The code appears<br>
semantically correct to my eyes.<br>
<br>
#include <stdio.h><br>
#include <mpi.h><br>
#include "mcs-lock.h"<br>
<br>
int main(int argc, char *argv[])<br>
{<br>
 MPI_Win win;<br>
 MPI_Init( &argc, &argv );<br>
<br>
 MCSLockInit(MPI_COMM_WORLD, &win);<br>
<br>
 int rank, size;<br>
 MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br>
 MPI_Comm_size(MPI_COMM_WORLD, &size);<br>
<br>
 printf("rank: %d, size: %d\n", rank, size);<br>
<br>
<br>
 MCSLockAcquire(win);<br>
 printf("rank %d aquired lock\n", rank);   fflush(=
stdout);<br>
 MCSLockRelease(win);<br>
<br>
<br>
 MPI_Win_free(&win);<br>
 MPI_Finalize();<br>
 return 0;<br>
}<br>
<br>
<br>
I have tested on several hardware platforms and mpich-3.2 and<br>
mpich-3.3a2<br>
but with no luck.<br>
<br>
It appears that the MPI_Win_Sync are not "refreshing" the local<b=
r>
data or<br>
I<br>
have a bug I can't spot.<br>
<br>
A simple unfair lock like <a href=3D"http://www.mcs.anl.gov/researc" rel=3D=
"noreferrer" target=3D"_blank">http://www.mcs.anl.gov/researc</a><br>
h/projects/mpi/usingmpi/exampl<wbr>es-advmpi/rma2/ga_mutex1.c works<br>
perfectly.<br>
<br>
Best regards, Ask Jakobsen<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
<br>
--<br>
Jeff Hammond<br>
<a href=3D"mailto:jeff.science at gmail.com" target=3D"_blank">jeff.science at gm=
ail.com</a><br>
<a href=3D"http://jeffhammond.github.io/" rel=3D"noreferrer" target=3D"_bla=
nk">http://jeffhammond.github.io/</a><br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
______________________________<wbr>_________________<br>
<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<main.c><mcs-lock-fop.c><mcs-l<wbr>ock.h>________________=
________<wbr>_______________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
</blockquote>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
--<br>
Ask Jakobsen<br>
R&D<br>
<br>
Qeye Labs<br>
Lers=C3=B8 Parkall=C3=A9 107<br>
2100 Copenhagen =C3=98<br>
Denmark<br>
<br>
mobile: <a href=3D"tel:%2B45%202834%206936" value=3D"+4528346936" targe=
t=3D"_blank">+45 2834 6936</a><br>
email: afj at Qeye-Labs.com<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
</blockquote>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
</blockquote>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
</blockquote>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto:discuss at mpich.org=
" target=3D"_blank">discuss at mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href=3D"https://lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https://lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a></div></div></blockquote></div><br><br clear=3D"all"><div><br></div>--=
 <br><div class=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div=
 dir=3D"ltr"><div><div dir=3D"ltr"><font size=3D"1"><b>Ask Jakobsen</b><br>=
R&D<br><br><span style=3D"color:rgb(255,153,102)">Q</span>eye Labs<br>L=
ers=C3=B8 Parkall=C3=A9 107<br>2100 Copenhagen =C3=98 <br>Denmark<br><br>mo=
bile: +45 2834 6936<br>email: afj at Qeye-Labs.com<br></font></div></div><=
/div></div>
</div>

--94eb2c13e2e428ae25054aa11616--

--===============4174043369255602394==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss
--===============4174043369255602394==--

More information about the discuss mailing list