<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Iker,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Could you try <code>FI_PROVIDER=verbs</code>?</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Iker Martín Álvarez <martini@uji.es><br>
<b>Sent:</b> Tuesday, March 26, 2024 2:35 AM<br>
<b>To:</b> Zhou, Hui <zhouh@anl.gov><br>
<b>Cc:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Subject:</b> Re: [mpich-discuss] Unexpected behaviour of MPI_Probe + MPI_Get_count</font>
<div> </div>
</div>
<style>
<!--
#x_pfptBannerc4ejmq7
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#D0D8DC!important;
max-width:none!important;
max-height:none!important}
-->
</style>
<div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
Hello Zhou, I just tried using both providers and in both cases the execution hangs at MPI_Init for those processes that are on a different node than the main one. This happens even for an MPI "Hello World" code for both providers. So
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="x_pfptBannerc4ejmq7" style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#D0D8DC!important; background-color:#D0D8DC; border-top:4px solid #90a4ae!important; border-top:4px solid #90a4ae">
<div id="x_pfptBannerc4ejmq7" style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBannerc4ejmq7" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBannerc4ejmq7" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">Hello Zhou,
<div><br>
</div>
<div>I just tried using both providers and in both cases the execution hangs at MPI_Init for those processes that are on a different node than the main one. This happens even for an MPI "Hello World" code for both providers.<br>
<br>
So I can't even check if it has the same problem.<br>
<br>
Thank you very much for your time.<br>
Best regards,<br>
</div>
<div></div>
<div>Iker</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">Missatge de Zhou, Hui <<a href="mailto:zhouh@anl.gov">zhouh@anl.gov</a>> del dia dl., 25 de març 2024 a les 23:05:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg3681080561595198016">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Iker,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Could you try setting <code>FI_PROVIDER=sockets</code> or <code>FI_PROVIDER=tcp</code> to see if the issue persists?</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
--</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hui</div>
<div id="x_m_3681080561595198016appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_3681080561595198016divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Iker Martín Álvarez <<a href="mailto:martini@uji.es" target="_blank">martini@uji.es</a>><br>
<b>Sent:</b> Monday, March 25, 2024 12:38 PM<br>
<b>To:</b> Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>><br>
<b>Cc:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Subject:</b> Re: [mpich-discuss] Unexpected behaviour of MPI_Probe + MPI_Get_count</font>
<div> </div>
</div>
<div>
<div style="font-size:1px; color:rgb(255,255,255); line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden; display:none">
Hello Zhou, Thanks for the quick reply. In the attached file you can see the result of running the code with the environment variable you gave. Kind regards, Iker Missatge de Zhou, Hui <zhouh@ anl. gov> del dia dl. , 25 de març 2024 a les
</div>
<div style="font-size:1px; color:rgb(255,255,255); line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden; display:none">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="x_m_3681080561595198016x_pfptBannerqyitdcn" style="display:block; text-align:left; margin:16px 0px; padding:8px 16px; border-radius:4px; min-width:200px; background-color:rgb(208,216,220); border-top:4px solid rgb(144,164,174)">
<div id="x_m_3681080561595198016x_pfptBannerqyitdcn" style="float:left; display:block; margin:0px 0px 1px; max-width:600px">
<div id="x_m_3681080561595198016x_pfptBannerqyitdcn" style="display:block; background-color:rgb(208,216,220); color:rgb(0,0,0); font-family:Arial,sans-serif; font-weight:bold; font-size:14px; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_m_3681080561595198016x_pfptBannerqyitdcn" style="font-weight:normal; display:block; background-color:rgb(208,216,220); color:rgb(0,0,0); font-family:Arial,sans-serif; font-size:12px; line-height:18px; margin-top:2px">
This message came from outside your organization. </div>
</div>
<div style="height:0px; clear:both; display:block; line-height:0; font-size:0.01px">
</div>
</div>
<div style="font-size:1px; color:rgb(255,255,255); line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden; display:none">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">
<div>Hello Zhou,<br>
</div>
<br>
<div>Thanks for the quick reply.<br>
In the attached file you can see the result of running the code with the environment variable you gave.<br>
</div>
<div><br>
</div>
<div>
<div>Kind regards,</div>
<div>Iker</div>
</div>
</div>
<br>
<div>
<div dir="ltr">Missatge de Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>> del dia dl., 25 de març 2024 a les 17:03:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Iker,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Could you try reproduce the issue by setting <code>MPIR_CVAR_DEBUG_SUMMARY=</code>1, and report the console output? The issue may be in specific provider. The log should show that.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
--</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hui</div>
<div id="x_m_3681080561595198016x_m_5460253305266554663appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_3681080561595198016x_m_5460253305266554663divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Iker Martín Álvarez via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Sent:</b> Monday, March 25, 2024 6:23 AM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Cc:</b> Iker Martín Álvarez <<a href="mailto:martini@uji.es" target="_blank">martini@uji.es</a>><br>
<b>Subject:</b> [mpich-discuss] Unexpected behaviour of MPI_Probe + MPI_Get_count</font>
<div> </div>
</div>
<div>
<div style="font-size:1px; color:rgb(255,255,255); line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden; display:none">
Hello, I recently encountered an unexpected behaviour of the MPI_Probe + MPI_Get_count functions under specific conditions. I was hoping that this forum could advise me on a solution. Specifically, the application performs an MPI_Send communication
</div>
<div style="font-size:1px; color:rgb(255,255,255); line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden; display:none">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="x_m_3681080561595198016x_m_5460253305266554663x_pfptBannergjiqy2i" style="display:block; text-align:left; margin:16px 0px; padding:8px 16px; border-radius:4px; min-width:200px; background-color:rgb(208,216,220); border-top:4px solid rgb(144,164,174)">
<div id="x_m_3681080561595198016x_m_5460253305266554663x_pfptBannergjiqy2i" style="float:left; display:block; margin:0px 0px 1px; max-width:600px">
<div id="x_m_3681080561595198016x_m_5460253305266554663x_pfptBannergjiqy2i" style="display:block; background-color:rgb(208,216,220); color:rgb(0,0,0); font-family:Arial,sans-serif; font-weight:bold; font-size:14px; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_m_3681080561595198016x_m_5460253305266554663x_pfptBannergjiqy2i" style="font-weight:normal; display:block; background-color:rgb(208,216,220); color:rgb(0,0,0); font-family:Arial,sans-serif; font-size:12px; line-height:18px; margin-top:2px">
This message came from outside your organization. </div>
</div>
<div style="height:0px; clear:both; display:block; line-height:0; font-size:0.01px">
</div>
</div>
<div style="font-size:1px; color:rgb(255,255,255); line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden; display:none">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">Hello,
<div><br>
</div>
<div>I recently encountered an unexpected behaviour of the MPI_Probe + MPI_Get_count functions under specific conditions. I was hoping that this forum could advise me on a solution.</div>
<div> <br>
Specifically, the application performs an MPI_Send communication from the root process to process B. Process B doesn't know the size of the message, so I use MPI_Probe + MPI_Get_count to discover it. However, as an example, if the size of the message is 1000
bytes, process B expects with MPI_Get_count function a total of 20 bytes.<br>
</div>
<div><br>
</div>
<div>The problem only occurs with a specific installation of MPICH and when the following conditions are met in my code:<br>
- The problem only occurs in internode communications.<br>
- The problem only appears if derived types are used in the communication. Specifically a derived type to communicate a vector of integers and a vector of reals, both with the same number of elements.<br>
- None of the MPI functions give an error code. They all return MPI_Sucess.<br>
- If instead of allocating the amount of bytes returned by MPI_Get_count(=20), I allocate the expected value (1000), the message is received correctly.<br>
- The size returned by MPI_Get_count seems to be variable depending on the total number of addresses with which the derived type is created.<br>
</div>
<div><br>
</div>
<div>I have attached the file to reproduce the problem. It can also be accessed via the GitLab link below:<br>
<a href="https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_ofi_mpi_probe_bug__;!!G_uCfscf7eWS!aVgghOB04ZFIQ9sus7BHy-d5is_qeaeC4HHkojD2AKAz4SjExQRNGSl8AAyhk85tIb_jsqY189JmMw$" target="_blank">https://lorca.act.uji.es/gitlab/martini/mpich_ofi_mpi_probe_bug</a><br>
It is designed to be run with 3 processes, two of them hosted on one node and the third on a different one.<br>
</div>
<div><br>
</div>
<div>As previously mentioned, this problem occurs when using MPICH with ch4:ofi without using the embedded option. Specifically, I have tested the following installations in which the error appears:<br>
</div>
<div>- MPICH 4.2.0 with config options: '--with-device=ch4:ofi' '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1'<br>
</div>
<div>- MPICH 4.0.3 with config options: '--with-device=ch4:ofi' '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1'</div>
<div>- MPICH 4.0.3 with config options: '--with-device=ch4:ofi' '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1' '--disable-psm3'<br>
</div>
<div>- MPICH 3.4.1 with config options: '--with-device=ch4:ofi' '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1'<br>
</div>
<div><br>
</div>
<div>However, it does work as expected for the following MPICH installations:</div>
<div>- MPICH 4.0.3 with config options: '--with-device=ch4:ofi' '--with-libfabric=embedded'<br>
</div>
<div>- MPICH 4.0.3 with config options: '--with-device=ch4:ucx' '--with-ucx=/soft/gnu/ucx-1.11'</div>
<div>- MPICH 3.4.1 with config options: '--with-device=ch4:ucx' '--with-ucx=/soft/gnu/ucx-1.11'</div>
<div><br>
</div>
<div>Although for these installations the code does work, we would like to use a different libfabric installation than the embedded one because we get better networking performance. In the case of UCX, it is because the application in question uses the MPI_Comm_spawn
call and MPICH does not currently support it with UCX.</div>
<div><br>
</div>
<div>Thank you for your help.<br>
Best regards,<br>
</div>
<div>Iker</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>