<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Kurt, The workers need to pass root=0. Mpi_proc_null indicates that the local process is not involved in the communication (like in p2p). This would be the appropriate argument for non-root arguments in the manager group, i. e. , if you start
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanneryod02qf" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanneryod02qf" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanneryod02qf" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanneryod02qf" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html>
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanneryod02qf { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonyod02qf:hover, .pfptPrimaryButtonyod02qf:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonyod02qf:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div dir="auto">Hi Kurt,</div>
<div dir="auto"><br>
</div>
<div dir="auto">The workers need to pass root=0. Mpi_proc_null indicates that the local process is not involved in the communication (like in p2p). This would be the appropriate argument for non-root arguments in the manager group, i.e., if you start with more
than one process. </div>
<div dir="auto">> all other processes in the same group as the root use MPI_PROC_NULL</div>
<div dir="auto"><br>
</div>
<div dir="auto"><a href="https://urldefense.us/v3/__https://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node91.htm__;!!G_uCfscf7eWS!aV1POdepzplFrCm-LLPMqZ1ET59oRoaqkMehvVh0QY88149KRsdfaIZBPCJrZmMTw4_MFNTmF9h4epvTkEoeSQ$">https://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node91.htm</a></div>
<div dir="auto"><br>
</div>
<div dir="auto">It also explains your observation of falling through the call without waiting.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Best Joachim </div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mccall, Kurt E. (MSFC-EV41) via discuss <discuss@mpich.org><br>
<b>Sent:</b> Sunday, April 21, 2024 11:52:48 PM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov><br>
<b>Subject:</b> [mpich-discuss] MPI_Reduce on an inter-communicator hangs</font>
<div> </div>
</div>
<div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="x_pfptBanner45qdy18" style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#D0D8DC!important; background-color:#D0D8DC; border-top:4px solid #90a4ae!important; border-top:4px solid #90a4ae">
<div id="x_pfptBanner45qdy18" style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBanner45qdy18" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBanner45qdy18" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<style>
<!--
#x_pfptBanner45qdy18
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#D0D8DC!important;
max-width:none!important;
max-height:none!important}
-->
</style>
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div class="x_WordSection1">
<p class="x_MsoNormal">I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one
</p>
<p class="x_MsoNormal">process in the local group (the single manager) and two processes in the remote group (the workers). The inter-</p>
<p class="x_MsoNormal">communicators are visited one at a time in the manager.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">All workers enter and exit MPI_Reduce without blocking, but the manager enters the first MPI_Reduce for the
</p>
<p class="x_MsoNormal">first inter-communicator and never returns. What am I doing wrong? I am using MPICH 4.1.2.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Here is my manager code:</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><b>#define N_PROC 4</b></p>
<p class="x_MsoNormal"><b>#define N_IN_GROUP 2</b></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><b>int main(int argc, char *argv[])</b></p>
<p class="x_MsoNormal"><b>{ </b></p>
<p class="x_MsoNormal"><b> int rank, world_size, error_codes[N_PROC];</b></p>
<p class="x_MsoNormal"><b> MPI_Comm intercoms[N_PROC];</b></p>
<p class="x_MsoNormal"><b> char hostname[64];</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> MPI_Init(&argc, &argv);</b></p>
<p class="x_MsoNormal"><b> MPI_Comm_rank(MPI_COMM_WORLD, &rank);</b></p>
<p class="x_MsoNormal"><b> MPI_Comm_size(MPI_COMM_WORLD, &world_size);</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> gethostname(hostname, sizeof(hostname));</b></p>
<p class="x_MsoNormal"><b> char *p = strstr(hostname, ".");</b></p>
<p class="x_MsoNormal"><b> *p = '\0';</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> MPI_Info info;</b></p>
<p class="x_MsoNormal"><b> MPI_Info_create(&info);</b></p>
<p class="x_MsoNormal"><b> MPI_Info_set(info, "host", hostname);</b></p>
<p class="x_MsoNormal"><b> MPI_Info_set(info, "bind_to", "core");</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> for (int i = 0; i < N_PROC; ++i)</b></p>
<p class="x_MsoNormal"><b> {</b></p>
<p class="x_MsoNormal"><b> MPI_Comm_spawn("test_reduce_work", argv, N_IN_GROUP, info,</b></p>
<p class="x_MsoNormal"><b> 0, MPI_COMM_SELF, &intercoms[i], &error_codes[i]);</b></p>
<p class="x_MsoNormal"><b> }</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> sleep(10);</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> unsigned array[100]{0};</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> for (int i = 0; i < N_PROC; ++i)</b></p>
<p class="x_MsoNormal"><b> {</b></p>
<p class="x_MsoNormal"><b> cout << "MANAGER: starting reduction " << i << "\n";</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> MPI_Reduce(NULL, array, 100, MPI_UNSIGNED, MPI_SUM, MPI_ROOT,</b></p>
<p class="x_MsoNormal"><b> intercoms[i]);</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> cout << "MANAGER: finished reduction " << i << "\n"; // we never reach this point</b></p>
<p class="x_MsoNormal"><b> }</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> for (int i = 0; i < 100; ++i) cout << array[i] << " ";</b></p>
<p class="x_MsoNormal"><b> cout << endl;</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> MPI_Finalize();</b></p>
<p class="x_MsoNormal"><b>}</b></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">And here is my worker code:</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><b>int main(int argc, char *argv[])</b></p>
<p class="x_MsoNormal"><b>{ </b></p>
<p class="x_MsoNormal"><b> int rank, world_size;</b></p>
<p class="x_MsoNormal"><b> MPI_Comm manager_intercom;</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> MPI_Init(&argc, &argv);</b></p>
<p class="x_MsoNormal"><b> MPI_Comm_rank(MPI_COMM_WORLD, &rank);</b></p>
<p class="x_MsoNormal"><b> MPI_Comm_size(MPI_COMM_WORLD, &world_size);</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> MPI_Comm_get_parent(&manager_intercom);</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> unsigned array[100]{1};</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> cout << "WORKER: starting reduction\n";</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> MPI_Reduce(array, NULL, 100, MPI_UNSIGNED, MPI_SUM, MPI_PROC_NULL,
</b></p>
<p class="x_MsoNormal"><b> manager_intercom);</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> cout << "WORKER: finishing reduction\n";</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> sleep(10);</b></p>
<p class="x_MsoNormal"><b> </b></p>
<p class="x_MsoNormal"><b> MPI_Finalize();</b></p>
<p class="x_MsoNormal"><b>}</b></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Finally, here it the invocation:</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><b>$ mpiexec -launcher ssh -print-all-exitcodes -wdir /home/kmccall/test_dir -np 1 -ppn 1 test_reduce_man</b></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Thanks,</p>
<p class="x_MsoNormal">Kurt</p>
<p class="x_MsoNormal"> </p>
</div>
</div>
</body>
</html>