[mpich-discuss] MPI_Send over ibverbs

[Raffenetti, Ken]

This is the post of the recv for the acknowledgement message: https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/824b5b37c8d2e413ead3474f098f7b252f45a3f8/src/mpid/ch4/netmod/ofi/ofi_send.h*L58-L89__;Iw!!G_uCfscf7eWS!YylbxBKmO8eM6ASEGLYSJSAZSrLvlTQXu_ZpRrKsULtBQgNtfOb0TsW6cfcRzzrB513Ig3wVmiMU7kie$ 

This is where the ack message is sent: https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/824b5b37c8d2e413ead3474f098f7b252f45a3f8/src/mpid/ch4/netmod/ofi/ofi_impl.h*L1084__;Iw!!G_uCfscf7eWS!YylbxBKmO8eM6ASEGLYSJSAZSrLvlTQXu_ZpRrKsULtBQgNtfOb0TsW6cfcRzzrB513Ig3wVmmMqXrW_$ 

Ken

From: Niyaz Murshed <Niyaz.Murshed at arm.com>
Date: Tuesday, October 8, 2024 at 11:32 AM
To: Raffenetti, Ken <raffenet at anl.gov>, discuss at mpich.org <discuss at mpich.org>
Cc: nd <nd at arm.com>
Subject: Re: [mpich-discuss] MPI_Send over ibverbs
Thank you Ken. Do you know where I can see the code of these acks ; message sequences. From: Raffenetti, Ken <raffenet@ anl. gov> Date: Tuesday, October 8, 2024 at 11: 10 AM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd
Thank you Ken.
Do you know where I can see the code of these acks ; message sequences.

From: Raffenetti, Ken <raffenet at anl.gov>
Date: Tuesday, October 8, 2024 at 11:10 AM
To: Niyaz Murshed <Niyaz.Murshed at arm.com>, discuss at mpich.org <discuss at mpich.org>
Cc: nd <nd at arm.com>
Subject: Re: [mpich-discuss] MPI_Send over ibverbs
It’s in MPI scope. To my knowledge, that level of acknowledgement is not exposed from the verbs API.

Ken

From: Niyaz Murshed <Niyaz.Murshed at arm.com>
Date: Tuesday, October 8, 2024 at 11:08 AM
To: Raffenetti, Ken <raffenet at anl.gov>, discuss at mpich.org <discuss at mpich.org>
Cc: nd <nd at arm.com>
Subject: Re: [mpich-discuss] MPI_Send over ibverbs
Are you talking about the ack of RC connection? Or is it MPI’s scope? From: Raffenetti, Ken <raffenet@ anl. gov> Date: Tuesday, October 8, 2024 at 11: 05 AM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>,
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd
Are you talking about the ack of RC connection? Or is it MPI’s scope?

From: Raffenetti, Ken <raffenet at anl.gov>
Date: Tuesday, October 8, 2024 at 11:05 AM
To: discuss at mpich.org <discuss at mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed at arm.com>, nd <nd at arm.com>
Subject: Re: [mpich-discuss] MPI_Send over ibverbs
An internal acknowledgement message is sent from the receiver to the sender indicating that a particular synchronous send operation was matched.

Ken

From: Niyaz Murshed via discuss <discuss at mpich.org>
Date: Tuesday, October 8, 2024 at 10:49 AM
To: discuss at mpich.org <discuss at mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed at arm.com>, nd <nd at arm.com>
Subject: Re: [mpich-discuss] MPI_Send over ibverbs
Thank you for the reply Joachim. ‘MPI_Ssend has the requirement to return after the message was matched at the receiver’ – From the above statement, how do the sender know that the receiver called a matching receive? I am asking from the point
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd
Thank you for the reply Joachim.

‘MPI_Ssend has the requirement to return after the message was matched at
the receiver’ –
>From the above statement, how do the sender know that the receiver called a matching receive?

I am asking from the point of view of MPI_Send over RDMA.
In case of RDMA, internally it will call rdma_send which does not know when a matching rdma_recv was called on destination. How do the sender knows when rdma_recv was called?

From: Joachim Jenke via discuss <discuss at mpich.org>
Date: Tuesday, October 8, 2024 at 9:42 AM
To: discuss at mpich.org <discuss at mpich.org>
Cc: Joachim Jenke <jenke at itc.rwth-aachen.de>
Subject: Re: [mpich-discuss] MPI_Send over ibverbs
Hi,

none of the MPI send calls has a requirement to return after the message
was received. MPI_Send can return as soon as the caller can write to
buffer without impact to the data being transferred. This might be after
the data is put to the network, or the MPI library copied the data to an
internal buffer.

MPI_Ssend has the requirement to return after the message was matched at
the receiver, which implies that the matching Receive was called. But it
still does not require that the message was received at this point.

Best
Joachim

Am 08.10.24 um 04:55 schrieb Niyaz Murshed via discuss:
> Hi,
>
> Is there a way to check the implementation of MPI_Send?
>
> As I understand there is an implicit barrier which allows MPI_Send to
> return only after the buffer is received by the destination.
>
> How is this communication done? How does the destination notify the
> sender that buffer is received?
>
> How is this implemented when using rocev2 ?
>
> Regards,
>
> Niyaz
>
>
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss__;!!G_uCfscf7eWS!YylbxBKmO8eM6ASEGLYSJSAZSrLvlTQXu_ZpRrKsULtBQgNtfOb0TsW6cfcRzzrB513Ig3wVmnq5RhyZ$ <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/listinfo/discuss__;!!G_uCfscf7eWS!aTxlD8Xx22PDuMXv--UZcA68fdpz5fzzNTbFlnm4W8ILFpny-KqRwNDxs2-iX7Ob14g7ReYsSsFBpE5IiC8$>


--
Dr. rer. nat. Joachim Jenke

IT Center
Group: High Performance Computing
Division: Computational Science and Engineering
RWTH Aachen University
Seffenter Weg 23
D 52074  Aachen (Germany)
Tel: +49 241 80- 24765
Fax: +49 241 80-624765
jenke at itc.rwth-aachen.de
https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!YylbxBKmO8eM6ASEGLYSJSAZSrLvlTQXu_ZpRrKsULtBQgNtfOb0TsW6cfcRzzrB513Ig3wVmvEhovAM$ <https://urldefense.us/v3/__http:/www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!aTxlD8Xx22PDuMXv--UZcA68fdpz5fzzNTbFlnm4W8ILFpny-KqRwNDxs2-iX7Ob14g7ReYsSsFBFKt98Uc$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20241010/d793c1a2/attachment-0001.html>