[mpich-discuss] Unexpected behaviour of MPI_Probe + MPI_Get_count
Iker Martín Álvarez
martini at uji.es
Mon Mar 25 12:38:34 CDT 2024
Hello Zhou,
Thanks for the quick reply.
In the attached file you can see the result of running the code with the
environment variable you gave.
Kind regards,
Iker
Missatge de Zhou, Hui <zhouh at anl.gov> del dia dl., 25 de març 2024 a les
17:03:
> Hi Iker,
>
> Could you try reproduce the issue by setting MPIR_CVAR_DEBUG_SUMMARY=1,
> and report the console output? The issue may be in specific provider. The
> log should show that.
>
> --
> Hui
> ------------------------------
> *From:* Iker Martín Álvarez via discuss <discuss at mpich.org>
> *Sent:* Monday, March 25, 2024 6:23 AM
> *To:* discuss at mpich.org <discuss at mpich.org>
> *Cc:* Iker Martín Álvarez <martini at uji.es>
> *Subject:* [mpich-discuss] Unexpected behaviour of MPI_Probe +
> MPI_Get_count
>
> Hello, I recently encountered an unexpected behaviour of the MPI_Probe +
> MPI_Get_count functions under specific conditions. I was hoping that this
> forum could advise me on a solution. Specifically, the application
> performs an MPI_Send communication
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Hello,
>
> I recently encountered an unexpected behaviour of the MPI_Probe +
> MPI_Get_count functions under specific conditions. I was hoping that this
> forum could advise me on a solution.
>
> Specifically, the application performs an MPI_Send communication from the
> root process to process B. Process B doesn't know the size of the message,
> so I use MPI_Probe + MPI_Get_count to discover it. However, as an example,
> if the size of the message is 1000 bytes, process B expects with
> MPI_Get_count function a total of 20 bytes.
>
> The problem only occurs with a specific installation of MPICH and when the
> following conditions are met in my code:
> - The problem only occurs in internode communications.
> - The problem only appears if derived types are used in the communication.
> Specifically a derived type to communicate a vector of integers and a
> vector of reals, both with the same number of elements.
> - None of the MPI functions give an error code. They all return MPI_Sucess.
> - If instead of allocating the amount of bytes returned by
> MPI_Get_count(=20), I allocate the expected value (1000), the message is
> received correctly.
> - The size returned by MPI_Get_count seems to be variable depending on the
> total number of addresses with which the derived type is created.
>
> I have attached the file to reproduce the problem. It can also be accessed
> via the GitLab link below:
> https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_ofi_mpi_probe_bug__;!!G_uCfscf7eWS!bmZo1voFi6MLJirgS2gnNhu09ExasKIejjpWZRyu7G6mWnvs40WALyct7nt2EqVEhI0z3cRkM7pQog$
> <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_ofi_mpi_probe_bug__;!!G_uCfscf7eWS!aVgghOB04ZFIQ9sus7BHy-d5is_qeaeC4HHkojD2AKAz4SjExQRNGSl8AAyhk85tIb_jsqY189JmMw$>
> It is designed to be run with 3 processes, two of them hosted on one node
> and the third on a different one.
>
> As previously mentioned, this problem occurs when using MPICH with ch4:ofi
> without using the embedded option. Specifically, I have tested the
> following installations in which the error appears:
> - MPICH 4.2.0 with config options: '--with-device=ch4:ofi'
> '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1'
> - MPICH 4.0.3 with config options: '--with-device=ch4:ofi'
> '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1'
> - MPICH 4.0.3 with config options: '--with-device=ch4:ofi'
> '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1' '--disable-psm3'
> - MPICH 3.4.1 with config options: '--with-device=ch4:ofi'
> '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1'
>
> However, it does work as expected for the following MPICH installations:
> - MPICH 4.0.3 with config options: '--with-device=ch4:ofi'
> '--with-libfabric=embedded'
> - MPICH 4.0.3 with config options: '--with-device=ch4:ucx'
> '--with-ucx=/soft/gnu/ucx-1.11'
> - MPICH 3.4.1 with config options: '--with-device=ch4:ucx'
> '--with-ucx=/soft/gnu/ucx-1.11'
>
> Although for these installations the code does work, we would like to use
> a different libfabric installation than the embedded one because we get
> better networking performance. In the case of UCX, it is because the
> application in question uses the MPI_Comm_spawn call and MPICH does not
> currently support it with UCX.
>
> Thank you for your help.
> Best regards,
> Iker
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20240325/efb57402/attachment.html>
-------------- next part --------------
martini at c01:~/OtherCodes/Tests/MPIProbe$ mpicc Basecode.c
martini at c01:~/OtherCodes/Tests/MPIProbe$ mpirun -env MPIR_CVAR_DEBUG_SUMMARY 1 -np 3 ./a.out
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10010
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: psm3, score = 4, pref = 0, FI_FORMAT_UNSPEC [32]
provider: psm3, score = 5, pref = 0, FI_FORMAT_UNSPEC [32]
provider: psm3, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxd, score = 4, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 4, pref = 0, FI_FORMAT_UNSPEC [32]
provider: udp;ofi_rxd, score = 4, pref = -1, FI_SOCKADDR_IN [16] 192.168.1.71
provider: udp;ofi_rxd, score = 4, pref = -1, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: udp;ofi_rxd, score = 4, pref = -1, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp;ofi_rxd, score = 4, pref = -1, FI_SOCKADDR_IN6 [28] ::1
provider: shm, score = 3, pref = 0, FI_ADDR_STR [15] - fi_shm://46315
provider: shm, score = 3, pref = 0, FI_ADDR_STR [15] - fi_shm://46315
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: net, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: net, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: net, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: net, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: net, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: net, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: net, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: net, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: net, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.71
provider: net, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::ec4:7aff:fee5:49e4
provider: net, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: net, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
Required minimum FI_VERSION: 10010, current version: 10010
==== Capability set configuration ====
libfabric provider: psm3 - IB/OPA-0xfe80000000000000
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0
MPIDI_OFI_ENABLE_MR_ALLOCATED: 0
MPIDI_OFI_ENABLE_MR_PROV_KEY: 0
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 1
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_CONTEXT_BITS: 20
MPIDI_OFI_SOURCE_BITS: 0
MPIDI_OFI_TAG_BITS: 31
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 4294967296
MAXIMUM TAG: 2147483648
==== Provider global thresholds ====
max_buffered_send: 64
max_buffered_write: 64
max_msg_size: 4294963200
max_order_raw: 4096
max_order_war: 4096
max_order_waw: 4096
tx_iov_limit: 13
rx_iov_limit: 1
rma_iov_limit: 1
max_mr_key_size: 8
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vni_t): 52032
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
==== OFI dynamic settings ====
num_vnis: 1
num_nics: 1
======================================
==== data structure summary ====
sizeof(MPIR_Comm): 1048
sizeof(MPIR_Request): 448
sizeof(MPIR_Datatype): 280
================================
P0 at node c01
P2 at node c01
P2: Total expected counts=9024 elems=752
P2: received message with 752 elements
P1 at node c02
P1: Total expected counts=20 elems=1
Abort(468852750) on node 1 (rank 1 in comm 0): Fatal error in internal_Recv: Message truncated, error stack:
internal_Recv(127): MPI_Recv(buf=0x5555c3177c00, count=1, dtype=USER<struct>, 0, 210, MPI_COMM_WORLD, status=0x7ffe30408b00) failed
(unknown)(): Message truncated
More information about the discuss
mailing list