[mpich-devel] code path for non-blocking operations that are deferred until synchronization

[Jeff Hammond]

On Fri, Jan 6, 2023 at 4:43 PM Zhou, Hui <zhouh at anl.gov> wrote:

> Requests are performance-critical objects and thus are allocated and
> managed internally by MPICH. You may not have access to it. It is possible
> to have user pre-allocated the request objects, but it will add much
> complications that we'd very much like to avoid.
>
>
The implementation impact is negligible.  As noted on
https://github.com/mpi-forum/mpi-issues/issues/664, request attributes
require adding a single pointer value to the internal request object, which
does not change its size noticeably, given what is already in there.

struct MPIR_Request { ... struct MPIR_Attribute * attributes; ... }


> Anyway, I pretty much support that we not to bother with the cases of
> allreduce on different input/output layout. I don't think it is working
> with any implementations, so technically it is not even breaking anything.
> If we want to support this, the straightforward approach is to offer an
> extension that allows allreduce with different in/out datatypes and assume
> per-element operations.
>

Nothing from the standard is implemented until somebody does it, and MPICH
is almost always first.  I am trying to figure out whether it is feasible
to implement all of the Fortran 2018 features in MPI-3 to know if we should
amend the standard.

Jeff


> --
> Hui
> ------------------------------
> *From:* Jeff Hammond <jeff.science at gmail.com>
> *Sent:* Friday, January 6, 2023 2:17 AM
> *To:* Zhou, Hui <zhouh at anl.gov>
> *Cc:* MPICH <devel at mpich.org>
> *Subject:* Re: [mpich-devel] code path for non-blocking operations that
> are deferred until synchronization
>
> Thanks.  This is helpful.
>
> In order to avoid ABI issues with the status object, I was thinking
> instead we could define attributes on requests.  That is sufficient to
> solve all sorts of problems, because one can just get the attributes off
> the request and decide to clean up as appropriate.
>
> Do you think adding attributes to requests causes any implementation
> headaches?  My expectation is that it does not slow down any critical path
> because the only time the attribute fields are touched are when the user
> accesses them.  The internal object behind a request would need an
> additional field, but that's a small change to MPIR_Request.
>
> However, if we end up standardizing a 256-bit status object in MPI-5
> (hypothetically), then we have an option to put hidden fields for this in
> status.  It's debatable if MPI_Status_get_buffer_address(MPI_Status *
> status) is significantly better than a request attribute lookup.
>
> In any case, I figured out that I can do the deferred implementation using
> generalized requests.  It requires Pthreads and MPI_THREAD_MULTIPLE but I
> can live with that.
>
> Jeff
>
> On Fri, Jan 6, 2023 at 12:39 AM Zhou, Hui <zhouh at anl.gov> wrote:
>
> Hi Jeff,
>
> The nonblocking cases can also do the same support by copying to and from
> contiguous buffers if we stick the temporary buffer pointer in the request
> object and free it upon completion. But if you are looking at this from the
> Fortran binding layer, which does not have access to MPICH internals, then
> it is difficult to do. The straight-forward solution would be ask MPICH to
> add an extension, so the user can stick the buffer pointer somewhere, for
> example, the status object.
>
> I wonder if there is a code path in MPICH associated with nonblocking
> communication that is deferred until the request is synchronized.  For
> example, is there code that merely copies all the arguments passed to the
> call into some object and then processes the whole operation when the
> request is synchronized?
>
> I am not exactly sure what do you mean here, but the persistent request
> most resembles your idea of merely hold all the arguments and only process
> it at start time. But again, if you are not allowed to access MPICH
> internals, then that is out-of-reach.
>
> --
> Hui
>
> ------------------------------
> *From:* Jeff Hammond via devel <devel at mpich.org>
> *Sent:* Thursday, January 5, 2023 4:50 AM
> *To:* MPICH <devel at mpich.org>
> *Cc:* Jeff Hammond <jeff.science at gmail.com>
> *Subject:* [mpich-devel] code path for non-blocking operations that are
> deferred until synchronization
>
> This is related to https://github.com/mpi-forum/mpi-issues/issues/663 but
> there are other scenarios where nonblocking communication with
> non-contiguous Fortran subarrays is extremely difficult to implement.  The
> blocking cases are easy to support by copying to and from contiguous
> buffers.
>
> I wonder if there is a code path in MPICH associated with nonblocking
> communication that is deferred until the request is synchronized.  For
> example, is there code that merely copies all the arguments passed to the
> call into some object and then processes the whole operation when the
> request is synchronized?
>
> Critically for my case, the Fortran arguments would need to be copied,
> meaning the CFI_cdesc_t pointer not just the associated base_addr, which I
> doubt is there based on what I've read so far.
>
> Thanks,
>
> Jeff
>
> --
> Jeff Hammond
> jeff.science at gmail.com
> http://jeffhammond.github.io/
>
>
>
> --
> Jeff Hammond
> jeff.science at gmail.com
> http://jeffhammond.github.io/
>


-- 
Jeff Hammond
jeff.science at gmail.com
http://jeffhammond.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/devel/attachments/20230107/a9120d72/attachment.html>