<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Kent,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I just tried with PGI 20.1 on mpich v3.3.2. I think I hit a hang once when I was checking it manually, but then I couldn’t reproduce it even after 1000 times repeat. Anyway, we have made some changes to the ch3 header structures that potentially
makes the code more standard compliant. Could you try with latest development on github and see if the issue still occur on your end?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <br>
Hui Zhou<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Kent Cheung via discuss <discuss@mpich.org><br>
<b>Reply-To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Date: </b>Monday, September 21, 2020 at 4:52 AM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>Kent Cheung <Kent.Cheung@arm.com><br>
<b>Subject: </b>Re: [mpich-discuss] Intermittent hang in MPI_Finalize with PGI 20.1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Are there any updates on this issue? Thanks.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Kent<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Raffenetti, Kenneth J. <raffenet@mcs.anl.gov><br>
<b>Sent:</b> 24 June 2020 17:04<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Kent Cheung <Kent.Cheung@arm.com><br>
<b>Subject:</b> Re: [mpich-discuss] Intermittent hang in MPI_Finalize with PGI 20.1</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Kent,<br>
<br>
Thanks for your report. We have not seen this issue with any compiler/OS combination in our nightly tests. We are using PGI 19.4 at this time. I will request 20.1 be installed so we can investigate further.<br>
<br>
Ken <br>
<br>
On 6/23/20, 8:26 AM, "Kent Cheung via discuss" <discuss@mpich.org> wrote:<br>
<br>
I'm running into an issue where processes sometimes hang when calling MPI_Finalize. This happens with both versions 3.3.2 and 3.4a2 on a single node RedHat 7.5 x86-64 machine, when MPICH is compiled with PGI 20.1 with these configuration flags<br>
<br>
<br>
--enable-debug --enable-shared --enable-debuginfo --enable-sharedlib=gcc<br>
<br>
If I change the default optimization level (-O2) by configuring with<br>
<br>
--enable-fast=O1<br>
<br>
<br>
as well, the hang doesn't occur. Another data point is that the hang does not occur with PGI 19.5 with either optimization levels.<br>
<br>
I have been testing with the cpi.c code in the examples folder built with just<br>
<br>
mpicc cpi.c<br>
mpiexec -n 3 ./a.out<br>
<br>
<br>
Here is a the backtrace from one of the processes that is hanging<br>
<br>
(gdb) bt<br>
#0 MPID_nem_mpich_blocking_recv ()<br>
<br>
at /tmp/mpich-3.3.2/build/../src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:1038<br>
<br>
#1 MPIDI_CH3I_Progress () at ../src/mpid/ch3/channels/nemesis/src/ch3_progress.c:506<br>
<br>
#2 0x00000000004fc88d in MPIDI_CH3U_VC_WaitForClose ()<br>
<br>
at ../src/mpid/ch3/src/ch3u_handle_connection.c:383<br>
<br>
#3 0x0000000000442364 in MPID_Finalize () at ../src/mpid/ch3/src/mpid_finalize.c:110<br>
<br>
#4 0x0000000000408621 in PMPI_Finalize () at ../src/mpi/init/finalize.c:260<br>
<br>
#5 0x00000000004023e5 in main () at cpi.c:59<br>
<br>
<br>
<br>
Is there a potential fix to be made to MPICH to prevent processes hanging when MPICH is compiled with the default optimization level?<br>
<br>
Thanks,<br>
Kent<br>
<br>
<br>
<br>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any
purpose, or store or copy the information in any medium. Thank you. <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person,
use it for any purpose, or store or copy the information in any medium. Thank you.
<o:p></o:p></p>
</div>
</body>
</html>