<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 16, 2016 at 7:18 AM, Mark Davis <span dir="ltr"><<a href="mailto:markdavisinboston@gmail.com" target="_blank">markdavisinboston@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello, I'm hitting a data race and a deconstruction error when using<br>
MPI_Bcast in an MPI application that also uses multiple threads per<br>
process. I'll go into detail below, but the fundamental question I<br>
have is: what can I assume, if anything, about the state of an MPI<br>
buffer during an MPI call? (I do realize that the buffers are not<br>
declared const in MPI_Bcast, whereas in other situations, such as<br>
MPI_Send, they are.)<br></blockquote><div><br></div><div>Nothing, unless the buffer pointer has the const attribute on it.</div><div><br></div><div>At the non-root processes, MPI_Bcast behaves like a MPI_Recv call, and thus you cannot touch the buffer until the function returned. Multiple threads cannot use the same buffer to multiple MPI_Bcast calls.</div><div><br></div><div>Jeff</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Background: running on Linux 3.13.0-52 / Ubuntu x86_64. gcc 5.3. MPICH<br>
3.2 release version with thread support enabled. I am MPI_Initing with<br>
thread multiple support. I'm also using thread sanitizer on my<br>
application (although not on MPICH itself). C++11 standard.<br>
<br>
Problem description: I'm running into a SIGABRT at the end of the<br>
program when MPICH seems to be running its deconstructors. My program<br>
is running NPROC MPI processes, each with T C++11 threads (so, NPROC *<br>
T threads total). This simplified application (which I've created to<br>
exhibit this problem) simply has one root looping through H times,<br>
doing a MPI_Bcast to other processes each time. Only one thread per<br>
process participates in the MPI_Bcast. Thread sanitizer is also<br>
detecting a data race. In fact, thread sanitizer is detecting that<br>
MPI_Bcast is *writing* to the root's buffer, even though in a<br>
broadcast, at least semantically, the root should only ever be sending<br>
data, not receiving data. Taking a quick look at MPI_Bcast (which<br>
ultimately calls to MPIDI_CH3U_Receive_data_found which invokes the<br>
memcpy), it does seem that depending on the message size (in my case,<br>
it's just 10 MPI_INTs per message), scatter and allgather can be used<br>
or a binomial tree algorithm. I haven't dug in to see which one is<br>
being used in my case, but this indicates that there's at least a<br>
possibility that the root's buffer can be received into during a<br>
MPI_Bcast. My program is producing the "right answer" but could just<br>
be luck.<br>
<br>
<br>
Here's the basic structure of the program:<br>
<br>
if root process<br>
if root thread<br>
A: signal using condition variable to (T-1) non-root threads<br>
that it can read the buffer (via shared memory)<br>
B: MPI_Bcast to other procs<br>
else if not root thread<br>
wait on condition variable until it's ok to read buffer<br>
read buffer<br>
else if not root process<br>
// AFAIK there are no problems with non-root processes<br>
MPI_Bcast as a non-root<br>
<br>
<br>
<br>
Thread sanitizer is detecting that on the root process, MPI_Bcast is<br>
writing to the buffer that is being broadcast. Simultaneously,<br>
non-root threads in the root process are reading the buffer being<br>
broadcast *while the MPI_Bcast is happening*. When I change the order<br>
of statements A and B above, the data race goes away. So, it seems to<br>
me that my assumption (that on the root node, the buffer being<br>
broadcast is being written, or at least it's a possibility that the<br>
MPI call will write the buffer) is incorrect. (FYI, the "writing"<br>
that's happening in MPI_Bcast is coming from<br>
MPIDI_CH3U_Receive_data_found<br>
src/mpid/ch3/src/ch3u_handle_<wbr>recv_pkt.c:152.)<br>
<br>
Also, at the end of the program, I'm hitting a SIGABRT during the<br>
deconstruction of something in libmpi (__do_global_dtors_aux). Full<br>
backtrace below. This issue also goes away when I reverse the order of<br>
statements A and B. I imagine I'm corrupting some state in MPICH, but<br>
I'm not sure.<br>
<br>
I should point out that I'd prefer to have the ordering of A and B as<br>
above, so some threads can make progress while the MPI_Bcast is<br>
happening.<br>
<br>
So, my questions are:<br>
<br>
1. what assumptions can be made about buffers in general in MPI during<br>
MPI operations? Does the standard specify anything about this? No<br>
reading non-const buffers during MPI operations, or can I do this in<br>
some situations?<br>
<br>
2. Is there any way for me to achieve what I'm trying to do (above,<br>
where the reading of the buffer is happening simultaneously with an<br>
MPI operation that shouldn't need to write the user buffer but does)?<br>
This would be very helpful to know.<br>
<br>
Thank you.<br>
<br>
<br>
<br>
Program received signal SIGABRT, Aborted.<br>
[Switching to Thread 0x7ff937bf8700 (LWP 32001)]<br>
0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at<br>
../nptl/sysdeps/unix/sysv/<wbr>linux/raise.c:56<br>
<br>
(gdb) bt full<br>
#0 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at<br>
../nptl/sysdeps/unix/sysv/<wbr>linux/raise.c:56<br>
resultvar = 0<br>
pid = 31987<br>
selftid = 32001<br>
#1 0x00007ff93fd1b028 in __GI_abort () at abort.c:89<br>
save_stage = 2<br>
act = {__sigaction_handler = {sa_handler = 0xbea6495,<br>
sa_sigaction = 0xbea6495}, sa_mask = {__val = {140708540140376,<br>
140708494113584, 134272, 0, 140708494597155, 134240,<br>
140708497979320, 131072, 140708494598025, 140708358448000,<br>
140708521085690, 140708539470768, 0, 140708540266640,<br>
140708551504800, 1}}, sa_flags = 934869088,<br>
sa_restorer = 0x1}<br>
sigs = {__val = {32, 0 <repeats 15 times>}}<br>
#2 0x00007ff93fd62dfa in malloc_printerr (ptr=<optimized out>,<br>
str=<optimized out>, action=<optimized out>) at malloc.c:5000<br>
No locals.<br>
#3 free_check (mem=<optimized out>, caller=<optimized out>) at hooks.c:298<br>
p = <optimized out><br>
#4 0x00007ff93fd1d53a in __cxa_finalize (d=0x7ff941207708) at cxa_finalize.c:56<br>
check = 97<br>
cxafn = <optimized out><br>
cxaarg = <optimized out><br>
f = 0x7ff9400a1230 <initial+944><br>
funcs = 0x7ff9400a0e80 <initial><br>
#5 0x00007ff940ff7123 in __do_global_dtors_aux () from<br>
mpich/debug/lib/libmpicxx.so.<wbr>12<br>
No symbol table info available.<br>
#6 0x00007ff937b8e410 in ?? ()<br>
No symbol table info available.<br>
#7 0x00007ff9426e270a in _dl_fini () at dl-fini.c:252<br>
array = 0x7ff941205238<br>
i = 0<br>
nmaps = 32761<br>
nloaded = <optimized out><br>
i = 4<br>
l = 0x7ff9428d7a00<br>
ns = 140708494300474<br>
maps = 0x7ff937b8e330<br>
maps_size = 140708497985152<br>
do_audit = 1116568064<br>
__PRETTY_FUNCTION__ = "_dl_fini"<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/<wbr>mailman/listinfo/discuss</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div></div>