[mpich-discuss] Hang after MPI_Intercomm_merge with OFI+provider verbs
Iker Martín Álvarez
martini at uji.es
Tue Jun 25 12:02:24 CDT 2024
Hello,
Lately, I have encountered an issue with the usage of MPI_Comm_spawn +
MPI_Intercomm_merge, where in some occasions my application hangs
indefinitely. I was hoping that this forum could advise me on a solution.
For my tests, I run the code with 2 processes and give as an argument the
amount of processes to spawn in MPI_Comm_spawn, where I choose 20. In my
experimentation, 3 to 4 out of 100 executions hang indefinitely at the
function MPI_Barrier. Nevertheless, if I remove both MPI_Intercomm_merge
functions, the code always finalizes.
Also, prior to executing the code I set the environment variable
FI_PROVIDER=verbs.
However, if I change the provider to "tcp", or "udp", the code
doesn't hang, but the performance is lower for other providers different to
"verbs".
The MPICH version used is 4.2.1 with the following configure line:
./configure --prefix=... --with-device=ch4:ofi --disable-psm3
In addition, I have attached a file
<https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_ofi_mpi_intercomm_merge_bug/-/blob/master/log-providers.txt__;!!G_uCfscf7eWS!fbrsIVjsyUNeOENsQntqYTQR8AN6gWXHB_RXhprRM6knmxEqBKQzmHG5XiAj_Fu7WZ1bk9EhFZQ4VQ$ >
with
the output of executing mpirun with "MPIR_CVAR_DEBUG_SUMMARY=1" and
unsetting FI_PROVIDER.
The minimal code
<https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_ofi_mpi_intercomm_merge_bug/-/blob/master/BaseCode.c__;!!G_uCfscf7eWS!fbrsIVjsyUNeOENsQntqYTQR8AN6gWXHB_RXhprRM6knmxEqBKQzmHG5XiAj_Fu7WZ1bk9G9fhVjRg$ >
to reproduce the problem is the following:
===============
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char* argv[]) {
MPI_Init(&argc, &argv);
MPI_Comm spawn, new_comm;
MPI_Comm_get_parent(&spawn);
if(spawn == MPI_COMM_NULL) {
int num_c = atoi(argv[1]);
MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, num_c, MPI_INFO_NULL, 0,
MPI_COMM_WORLD, &spawn, MPI_ERRCODES_IGNORE);
MPI_Intercomm_merge(spawn, 0, &new_comm);
} else {
MPI_Intercomm_merge(spawn, 1, &new_comm);
MPI_Barrier(MPI_COMM_WORLD);
}
MPI_Finalize();
}
===============
Thank you for your help.
Best regards,
Iker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20240625/7f1d40d5/attachment.html>
-------------- next part --------------
MPI
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10012
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 192.168.2.11
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 192.168.1.11
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] ::1
provider: shm, score = 4, pref = -2, FI_ADDR_STR [16] - fi_shm://525604
provider: shm, score = 4, pref = -2, FI_ADDR_STR [16] - fi_shm://525604
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.11
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::63f:7203:d3:dae
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.11
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::6d4:c4ff:fe5c:5166
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
Required minimum FI_VERSION: 10005, current version: 10012
==== Capability set configuration ====
libfabric provider: sockets - 192.168.2.0/24
MPIDI_OFI_ENABLE_DATA: 1
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0
MPIDI_OFI_ENABLE_MR_ALLOCATED: 0
MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 0
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 1
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_TRIGGERED: 0
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 20
MPIDI_OFI_SOURCE_BITS: 0
MPIDI_OFI_TAG_BITS: 31
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 4294967296
MAXIMUM TAG: 2147483648
==== Provider global thresholds ====
max_buffered_send: 255
max_buffered_write: 255
max_msg_size: 9223372036854775807
max_order_raw: -1
max_order_war: -1
max_order_waw: -1
tx_iov_limit: 8
rx_iov_limit: 8
rma_iov_limit: 8
max_mr_key_size: 8
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vci_t): 52480
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
==== OFI dynamic settings ====
num_vcis: 1
num_nics: 1
======================================
error checking : enabled
QMPI : disabled
debugger support : disabled
thread level : MPI_THREAD_SINGLE
thread CS : per-vci
threadcomm : enabled
==== data structure summary ====
sizeof(MPIR_Comm): 1792
sizeof(MPIR_Request): 512
sizeof(MPIR_Datatype): 280
================================
More information about the discuss
mailing list