[mpich-discuss] Occasional hang with MPI_Intercomm_merge and OFI+provider verbs

[Iker Martín Álvarez]

Hi Hui,

Yes, it works for that provider, or at least I haven't had any issues with
it.

Kind regards,
Iker

Missatge de Zhou, Hui <zhouh at anl.gov> del dia dl., 28 d’oct. 2024 a les
20:01:

> Hi Iker,
>
> Does it work with `FI_PROVIDER="verbs;ofi_rxm"?
>
> Hui
> ------------------------------
> *From:* Iker Martín Álvarez via discuss <discuss at mpich.org>
> *Sent:* Monday, October 28, 2024 12:34 PM
> *To:* discuss at mpich.org <discuss at mpich.org>
> *Cc:* Iker Martín Álvarez <martini at uji.es>
> *Subject:* [mpich-discuss] Occasional hang with MPI_Intercomm_merge and
> OFI+provider verbs
>
> Hi, Lately I have been dealing with an unexpected problem when using
> MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my
> application hangs when two conditions are met. Specifically, the hang
> occurs when using the resulting Intracommunicator
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Hi,
>
> Lately I have been dealing with an unexpected problem when using
> MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my
> application hangs when two conditions are met.
>
> Specifically, the hang occurs when using the resulting Intracommunicator
> of MPI_Intercomm_merge in collective operations as MPI_Bcast. The
> conditions are
> - There is an oversubscription state. The number of processes is greater
> than the available number of physical cores.
> - Using CH4:ofi with FI_PROVIDER="verbs:ofi_rxd".
>
> I tested a minimal code with MPICH 4.2.0 and MPICH 4.2.3 configured as:
> ./configure --prefix=... --with-device=ch4:ofi --disable-psm3
>
> The minimal code
> <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_ofi_rxd_intracomm_hang/-/blob/main/BaseCode.c__;!!G_uCfscf7eWS!ZoB5c9APxvNk40SirehC83dWaIUE_w3yOQ2EEoJ4MaJlUaREOTMYR85Dd4SCEE0Exrr4U6ZETBg1sg$>
> to reproduce the problem is the following:
> ==========================
>
> #include <stdio.h>#include <stdlib.h>#include <mpi.h>int main(int argc, char* argv[]) {  int rank, numP, numO;  int rootBcast, order;  double test = 0;  int solution = 0;  MPI_Init(&argc, &argv);  MPI_Comm_rank(MPI_COMM_WORLD, &rank);  MPI_Comm_size(MPI_COMM_WORLD, &numP);  MPI_Comm intercomm, intracomm;  MPI_Comm_get_parent(&intercomm);  if(intercomm == MPI_COMM_NULL) {    numO = atoi(argv[1]);    MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numO, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &intercomm, MPI_ERRCODES_IGNORE);    order = 0;  } else { order = 1; }  MPI_Intercomm_merge(intercomm, order, &intracomm);  printf("TEST 1 P%02d/%d\n", rank, numP);  MPI_Bcast(&test, 1, MPI_DOUBLE, 0, intracomm); // Hangs here  if(solution) { MPI_Barrier(intercomm); }  printf("TEST 2 P%02d/%d\n", rank, numP);  MPI_Finalize();  return 0;}
>
> ==========================
> The code only hangs at the MPI_Bcast operation for some of the spawned
> processes. All my executions have been with a single node of 20 cores, with
> 10 initial processes and spawning 20 at the MPI_Comm_spawn function. If I
> change the variable "solution" to 1, I rarely get a hang of the
> application, but it still happens on some occasions.
>
> From my perspective, the code seems to follow the standard. Is this the
> case? I have been able to run the code with other providers for OFI, but I
> am confused as to why it does not work in this case.
>
> Thank you for your time.
> Best regards,
> Iker
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20241029/a0d7b17a/attachment.html>