[mpich-discuss] Reading buffers during MPI call in multithreaded application

[Halim Amer]

Mark,

I think we should at least fix the issue of overwriting the send buffer 
at the root process. Whether this will fix all your problems is still 
unsure.

I created a ticket to track this issue 
https://trac.mpich.org/projects/mpich/ticket/2349#ticket.

Please feel free to to provide more details. In particular, if you could 
attach a toy program that reproduces the problem, it will help us verify 
the fix and potentially uncover other issues.

Thanks,
--Halim

www.mcs.anl.gov/~aamer

On 8/16/16 6:11 PM, Mark Davis wrote:
> Rajeev, I think you are probably right that it's the allgather that is
> writing to the root's buffer, even though it is overwriting the buffer
> with the same values that are already there. I could get comfortable
> with this, but I have two concerns:
>
> 1. I'm still getting the SIGABRT related to __do_global_dtors_aux (see
> details in my first message in this thread). It's possible that this
> is unrelated to reading the buffer on the root process while the
> MPI_Bcast is happening, although this issue does go away (along with
> the data race warning) when I don't read concurrently with the
> MPI_Bcast.
>
> 2. Would there be any transient / undefined states of the buffer
> during the actual memcpy, even though the new values are the same as
> the old values? My understanding of memcpy is that there are no thread
> safety guarantees, although maybe in this particular case it's ok.
>
> I have no particular reason to believe that #2 is necessarily a
> problem; however, #1 is currently a problem that I'm curious to
> understand better.
>
> Thank you.
>
> On Tue, Aug 16, 2016 at 4:53 PM, Thakur, Rajeev <thakur at mcs.anl.gov> wrote:
>> I guess what is happening is the allgather in the scatter-allgather algorithm is happening everywhere, even at the root. The root must participate in the allgather, but the copying into the local buffer at the root can be avoided since the data is already there. Since all this is within a blocking MPI_Bcast operation, it was not considered a problem. I don’t know if the buffer at the root is modified during the operation or just over-written with the same values. If the latter, I don’t know whey there would be a problem even in the multithreaded case.
>>
>> Rajeev
>>
>>
>>
>>
>>> On Aug 16, 2016, at 3:47 PM, Mark Davis <markdavisinboston at gmail.com> wrote:
>>>
>>> I'm glad that is the intention of MPI_Bcast. That agrees with my
>>> intuition. However, unless I'm mistaken, there may be a bug here as it
>>> seems that the buffer is being written to on the root process (see my
>>> original post for more details).
>>>
>>> Given that there's only one buffer argument in MPI_Bcast, it does seem
>>> that it would be impossible to mark it as const as it would only be
>>> const on the root and non-const on the non-roots. But, it would be
>>> nice if it was effectively const for the root process.
>>>
>>> On Tue, Aug 16, 2016 at 4:34 PM, Balaji, Pavan <balaji at anl.gov> wrote:
>>>>
>>>>> On Aug 16, 2016, at 3:12 PM, Mark Davis <markdavisinboston at gmail.com> wrote:
>>>>>
>>>>>> At the non-root processes, MPI_Bcast behaves like a MPI_Recv call, and thus
>>>>>> you cannot touch the buffer until the function returned.
>>>>>
>>>>> That part makes sense. I'm not allowing the buffer to be read or
>>>>> otherwise used on non-root threads. It makes sense to me that this
>>>>> acts as a MPI_Recv call.
>>>>>
>>>>> The thing that I'm confused by is on the root process, as it seems
>>>>> that the root process' buffer is also written to during the course of
>>>>> the MPI_Bcast; it should act like an MPI_Send. It seems like this is
>>>>> just an implementation detail, and as you pointed out, since the
>>>>> MPI_Bcast buffer is not marked const, anything could happen.
>>>>
>>>> As of MPI-2.2, we allowed the root process to read from the buffer while communication is going on for point-to-point and collective operations.  In MPI-3.0, we removed that wording when we added "const" attributes, and might have accidentally removed the wording in collectives also even though we didn't add const in all cases.  I didn't get a chance to look through the MPI-3.1 standard, but if we missed that, we should fix it.
>>>>
>>>> The intention is that MPI_Bcast allows the user to read from the buffer at the root process, but not write to it.  So, for example, you can do multiple MPI_Ibcasts from the same buffer.
>>>>
>>>>  -- Pavan
>>>>
>>>> _______________________________________________
>>>> discuss mailing list     discuss at mpich.org
>>>> To manage subscription options or unsubscribe:
>>>> https://lists.mpich.org/mailman/listinfo/discuss
>>> _______________________________________________
>>> discuss mailing list     discuss at mpich.org
>>> To manage subscription options or unsubscribe:
>>> https://lists.mpich.org/mailman/listinfo/discuss
>>
>> _______________________________________________
>> discuss mailing list     discuss at mpich.org
>> To manage subscription options or unsubscribe:
>> https://lists.mpich.org/mailman/listinfo/discuss
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss