[mpich-discuss] OFI poll failed error if using more than one cluster node
Stephen Wong
stephen.photond at gmail.com
Mon Aug 5 04:41:59 CDT 2024
Since there is no update to this after posting the Debug messages, I have
since got a new suggestion from someone.
I was told that MPICH was only built with PMIx support whereas
mpiexec.hydra only
provides a PMI server.
Could that be the source of the problem?
On Mon, 29 Jul 2024 at 11:15, Stephen Wong <stephen.photond at gmail.com>
wrote:
> For the build configured with --with-device=ch4:ofi, I got
>
> ==== Various sizes and limits ====
> sizeof(MPIDI_per_vci_t): 192
> Required minimum FI_VERSION: 0, current version: 10014
> provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16]
> 192.168.1.5
> provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] ::1
> provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://4595
> provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://4595
> provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5
> provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28]
> fe80::1ff:fe23:4567:890a
> provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
> provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
> provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://4595
> provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://4595
> Required minimum FI_VERSION: 10005, current version: 10014
> ==== Capability set configuration ====
> libfabric provider: sockets - 192.168.1.0/24
> MPIDI_OFI_ENABLE_DATA: 1
> MPIDI_OFI_ENABLE_AV_TABLE: 1
> MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1
> MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
> MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0
> MPIDI_OFI_ENABLE_MR_ALLOCATED: 0
> MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
> MPIDI_OFI_ENABLE_MR_PROV_KEY: 0
> MPIDI_OFI_ENABLE_TAGGED: 1
> MPIDI_OFI_ENABLE_AM: 1
> MPIDI_OFI_ENABLE_RMA: 1
> MPIDI_OFI_ENABLE_ATOMICS: 1
> MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
> MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
> MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
> MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
> MPIDI_OFI_ENABLE_TRIGGERED: 0
> MPIDI_OFI_ENABLE_HMEM: 0
> MPIDI_OFI_NUM_AM_BUFFERS: 8
> MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
> MPIDI_OFI_CONTEXT_BITS: 20
> MPIDI_OFI_SOURCE_BITS: 0
> MPIDI_OFI_TAG_BITS: 31
> MPIDI_OFI_VNI_USE_DOMAIN: 1
> MAXIMUM SUPPORTED RANKS: 4294967296
> MAXIMUM TAG: 2147483648
> ==== Provider global thresholds ====
> max_buffered_send: 255
> max_buffered_write: 255
> max_msg_size: 9223372036854775807
> max_order_raw: -1
> max_order_war: -1
> max_order_waw: -1
> tx_iov_limit: 8
> rx_iov_limit: 8
> rma_iov_limit: 8
> max_mr_key_size: 8
> ==== Various sizes and limits ====
> MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
> MPIDI_OFI_MAX_AM_HDR_SIZE: 255
> sizeof(MPIDI_OFI_am_request_header_t): 416
> sizeof(MPIDI_OFI_per_vci_t): 52480
> MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
> MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
> ==== OFI dynamic settings ====
> num_vcis: 1
> num_nics: 1
> ======================================
>
> *mpiexec -host host1,host2 -n 2 cpi*
>
> Abort(883025295) on node 1: Fatal error in internal_Init: Other MPI error,
> error stack:
> internal_Init(48306).............: MPI_Init(argc=0x7ffcb8c7aedc,
> argv=0x7ffcb8c7aed0) failed
> MPII_Init_thread(265)............:
> MPIR_init_comm_world(34).........:
> MPIR_Comm_commit(823)............:
> MPID_Comm_commit_post_hook(222)..:
> MPIDI_world_post_init(660).......:
> MPIDI_OFI_init_vcis(842).........:
> check_num_nics(891)..............:
> MPIR_Allreduce_allcomm_auto(4726):
> MPIC_Sendrecv(306)...............:
> MPIC_Wait(91)....................:
> MPIR_Wait(780)...................:
> MPIR_Wait_state(737).............:
> MPIDI_progress_test(134).........:
> MPIDI_OFI_handle_cq_error(791)...: OFI poll failed
> (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
> Abort(883025295) on node 0: Fatal error in internal_Init: Other MPI error,
> error stack:
> internal_Init(48306).............: MPI_Init(argc=0x7fff18beee1c,
> argv=0x7fff18beee10) failed
> MPII_Init_thread(265)............:
> MPIR_init_comm_world(34).........:
> MPIR_Comm_commit(823)............:
> MPID_Comm_commit_post_hook(222)..:
> MPIDI_world_post_init(660).......:
> MPIDI_OFI_init_vcis(842).........:
> check_num_nics(891)..............:
> MPIR_Allreduce_allcomm_auto(4726):
> MPIC_Sendrecv(306)...............:
> MPIC_Wait(91)....................:
> MPIR_Wait(780)...................:
> MPIR_Wait_state(737).............:
> MPIDI_progress_test(134).........:
> MPIDI_OFI_handle_cq_error(791)...: OFI poll failed
> (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
>
>
> On Fri, 26 Jul 2024 at 19:04, Zhou, Hui <zhouh at anl.gov> wrote:
>
>> Could you run cpi with MPIR_CVAR_DEBUG_SUMMARY=1 and post the output?
>>
>> Hui Zhou
>> ------------------------------
>> *From:* Stephen Wong via discuss <discuss at mpich.org>
>> *Sent:* Friday, July 26, 2024 4:46 AM
>> *To:* discuss at mpich.org <discuss at mpich.org>
>> *Cc:* Stephen Wong <stephen.photond at gmail.com>
>> *Subject:* [mpich-discuss] OFI poll failed error if using more than one
>> cluster node
>>
>> Hi, (I sent this previously without a subject line. ) I am using MPICH 4.
>> 2. 2 on Ubuntu 24. 04 testing with the small program cpi that calculates
>> the value of pi using MPI. I can start on host1 to run cpi on either host1
>> or host2 alone and I can
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>> ZjQcmQRYFpfptBannerEnd
>> Hi,
>>
>> (I sent this previously without a subject line.)
>>
>> I am using MPICH 4.2.2 on Ubuntu 24.04 testing with the small program cpi
>> that calculates the value of pi using MPI. I can start on host1 to run cpi
>> on either host1 or host2 alone and I can start on host2 to run cpi on
>> either host2 or host1 alone. The problem occurs only if I try to use both
>> host1 and host2 together.
>>
>> This is done using, for example, the command
>> *mpiexec -host host1,host2 -n 2 cpi*
>> then it ends with the error
>>
>> Abort(77718927) on node 1: Fatal error in internal_Init: Other MPI error,
>> error stack:
>> internal_Init(48306).............: MPI_Init(argc=0x7ffdb68e7fec,
>> argv=0x7ffdb68e7fe0) failed
>> MPII_Init_thread(265)............:
>> MPIR_init_comm_world(34).........:
>> MPIR_Comm_commit(823)............:
>> MPID_Comm_commit_post_hook(222)..:
>> MPIDI_world_post_init(660).......:
>> MPIDI_OFI_init_vcis(842).........:
>> check_num_nics(891)..............:
>> MPIR_Allreduce_allcomm_auto(4726):
>> MPIC_Sendrecv(306)...............:
>> MPIC_Wait(91)....................:
>> MPIR_Wait(780)...................:
>> MPIR_Wait_state(737).............:
>> MPIDI_progress_test(134).........:
>> MPIDI_OFI_handle_cq_error(791)...: OFI poll failed
>> (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
>> Abort(77718927) on node 0: Fatal error in internal_Init: Other MPI error,
>> error stack:
>> internal_Init(48306).............: MPI_Init(argc=0x7ffcb5b28adc,
>> argv=0x7ffcb5b28ad0) failed
>> MPII_Init_thread(265)............:
>> MPIR_init_comm_world(34).........:
>> MPIR_Comm_commit(823)............:
>> MPID_Comm_commit_post_hook(222)..:
>> MPIDI_world_post_init(660).......:
>> MPIDI_OFI_init_vcis(842).........:
>> check_num_nics(891)..............:
>> MPIR_Allreduce_allcomm_auto(4726):
>> MPIC_Sendrecv(306)...............:
>> MPIC_Wait(91)....................:
>> MPIR_Wait(780)...................:
>> MPIR_Wait_state(737).............:
>> MPIDI_progress_test(134).........:
>> MPIDI_OFI_handle_cq_error(791)...: OFI poll failed
>> (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
>> I searched through the archive of this mailing list and there is only one
>> thread that has this OFI poll failed error.
>> In the thread, it mentioned this has something to do with the device
>> configuration of ch4:ofi ?
>> I checked my configure log and it has
>> device : ch4:ofi (embedded libfabric)
>> in the configuration when I built the MPI. So I am wondering if I should
>> switch this option to something else? If this will fix it. I am not too
>> sure what other option I could substitute for ch4:ofi.
>>
>> *****************************************************
>>
>> Next I tried running configure for a build with the --enable-device =
>> ch3:nemesis option.
>> Now again I can run the cpi on any of host1 or host2 alone. If I run it
>> on host1 and host2 together, it just crashed with a core dump.
>>
>> Using the --enable-device = ch3:sock configure option resulted in more or
>> less the same problem but now it just quits silently when running on host1
>> and host2 together.
>>
>> Any ideas?
>> Thanks!
>> Stephen.
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20240805/cf640c50/attachment-0001.html>
More information about the discuss
mailing list