<div dir="ltr"><div dir="ltr">Thank you for the answer!<div><br></div><div>Second solution just suits me. I wanted to implement simple release-acquire semantics through atomic flag. For example, two processes include two threads, first of which waits for the flag by MPI_Get_accumulate and the other one remotely sets flag by MPI_Accumulate on the other process. In order to keep atomicity of operations, I have to use one window. Now all works fine:<br></div><div><br></div><div>MPI_Win win;<br>int flag = 0;<br>int nproc = 0;<br>int myrank = 0;<br><br>void *thread(void *arg)<br>{<br> int flag_get = 0;<br><br> do {<br> MPI_Get_accumulate(NULL, 0, MPI_INT,<br> &flag_get, 1, MPI_INT,<br> myrank, 0, 1, MPI_INT, MPI_NO_OP, win);<br> MPI_Win_flush(myrank, win);<br> } while (flag_get == 0);<br><br> printf("%d \t I got a flag!\n", myrank);<br><br> return NULL;<br>}<br></div><div><br></div><div>int main(int argc, char *argv[])<br>{<br> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, NULL);<br><br> MPI_Comm_size(MPI_COMM_WORLD, &nproc);<br> MPI_Comm_rank(MPI_COMM_WORLD, &myrank);<br> assert(nproc == 2);<br><br> int *buf = NULL;<br> const int bufsize = 1;<br> MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &buf);<br> MPI_Win_create(buf, 1, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win);<br><br> MPI_Win_lock_all(0, win);<br><br> pthread_t tid;<br><br> pthread_create(&tid, NULL, thread, NULL);<br><br> int peer = 1 - myrank;<br> int flag_put = 1;<br><br> MPI_Accumulate(&flag_put, 1, MPI_INT, peer, 0, 1, MPI_INT, MPI_REPLACE, win);<br> MPI_Win_flush(peer, win);<br><br> pthread_join(tid, NULL);<br><br> MPI_Win_unlock_all(win);<br><br> MPI_Win_free(&win);<br><br> MPI_Finalize();<br><br> return 0;<br>}<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 6, 2019 at 23:18 Jeff Hammond wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Your threads are trying to lock a window that is already locked or unlock a window that is not locked. That's because you are using one window with multiple threads.</div><div><br></div><div>If you want to use multiple threads like this, do one of two things:</div><div>1) use a window per thread</div><div>2) synchronize with flush(_all). only lock and unlock the window once (immediately after construction and immediately before destruction, respectively)</div><div><br></div><div>Jeff</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 6, 2019 at 12:30 PM Alexey Paznikov via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I have multithreaded MPI program (MPI_THREAD_MULTIPLE mode) with RMA calls.This is a simplified example:</div><div><br></div><div>#include <stdio.h><br>#include <mpi.h><br>#include <pthread.h><br><br>MPI_Win win;<br>pthread_mutex_t lock;<br><br>void *thread(void *arg)<br>{<br> MPI_Win_lock_all(0, win);<br> MPI_Win_unlock_all(win);<br> return NULL;<br>}<br><br>int main(int argc, char *argv[])<br>{<br> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, NULL);<br> pthread_mutex_init(&lock, NULL);<br><br> int *buf = NULL;<br> const int bufsize = 1;<br> MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &buf);<br> MPI_Win_create(buf, 1, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win);<br><br> pthread_t tid;<br> pthread_create(&tid, NULL, thread, NULL);<br><br> MPI_Win_lock_all(0, win);<br> MPI_Win_unlock_all(win);<br><br> pthread_join(tid, NULL);<br><br> MPI_Win_free(&win);<br><br> MPI_Finalize();<br><br> return 0;<br>}<br></div><div><br></div><div>If I run such program, it crashes with an error message:</div><div><br></div><div>Fatal error in PMPI_Win_lock_all: Wrong synchronization of RMA calls , error stack:<br>PMPI_Win_lock_all(149).: MPI_Win_lock_all(assert=0, win=0xa0000000) failed<br>MPID_Win_lock_all(1522): Wrong synchronization of RMA calls<br></div><div><br></div><div>If I replace MPI_Win_lock_all with MPI_Win_lock the problem remains:</div><div><br></div><div>Fatal error in PMPI_Win_lock: Wrong synchronization of RMA calls , error stack:<br>PMPI_Win_lock(157).: MPI_Win_lock(lock_type=234, rank=0, assert=0, win=0xa0000000) failed<br>MPID_Win_lock(1163): Wrong synchronization of RMA calls<br></div><div><br></div><div>(this message is repeated many times)</div><div><br></div><div>If I protect RMA operations with a mutex, the problem disappears.<br></div><div><br></div><div>MPICH version 3.3. Similar problem is also in MVAPICH2 2.3.1.</div><div><br></div><div>Are the RMA operations not thread safe at the moment? Could you tell me how to deal with this problem?<br></div><div><br></div></div>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_4406475999987504268m_3854148835739910419m_-6556198184052715640gmail-m_7757999064680259408gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div></div>
</blockquote></div></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_4406475999987504268m_3854148835739910419gmail-m_-6556198184052715640gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Alexey Paznikov</div></div></div></div></div></div></div></div></div></div>
</div>