<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Times New Roman""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi All,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m writing to get some advice and possibly report a bug. The circumstances are that we are currently working on updating HDF5 functionality and have run into an issue running a parallel test of a CFD code
(benchmark.hdf) from the <a href="https://github.com/CGNS/CGNS.git">CGNS code base</a>. I’ve debugged enough to see that our failure occurs during a call to MPI_File_set_view, with the failure signature as follows:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">[brtnfld@jelly] ~/scratch/CGNS/CGNS/src/ptests % mpirun -n 2 benchmark_hdf5</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Fatal error in PMPI_Barrier: Message truncated, error stack:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Barrier_impl(337)..............: Failure during collective</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Barrier_impl(330)..............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Barrier(294)...................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Barrier_intra(151).............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">barrier_smp_intra(111)..............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Bcast_impl(1462)...............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Bcast(1486)....................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Bcast_intra(1295)..............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Bcast_binomial(241)............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIC_Recv(352)......................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes received but buffer size is 1</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">[cli_1]: aborting job:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Fatal error in PMPI_Barrier: Message truncated, error stack:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Barrier_impl(337)..............: Failure during collective</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Barrier_impl(330)..............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Barrier(294)...................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Barrier_intra(151).............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">barrier_smp_intra(111)..............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Bcast_impl(1462)...............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Bcast(1486)....................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Bcast_intra(1295)..............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Bcast_binomial(241)............:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIC_Recv(352)......................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes received but buffer size is 1</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: H5F_close: Assertion `f->file_id > 0' failed.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Fatal error in PMPI_Allgather: Unknown error class, error stack:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">PMPI_Allgather(1002)......................: MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Allgather_impl(842)..................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Allgather(801).......................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Allgather_intra(216).................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIC_Sendrecv(475)........................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIC_Wait(243)............................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIDI_CH3i_Progress_wait(239).............: an error occurred while handling an event returned by MPIDU_Sock_Wait()</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIDI_CH3I_Progress_handle_sock_event(451):</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIDU_Socki_handle_read(649)..............: connection failure (set=0,sock=1,errno=104:Connection reset by peer)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">[cli_0]: aborting job:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Fatal error in PMPI_Allgather: Unknown error class, error stack:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">PMPI_Allgather(1002)......................: MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Allgather_impl(842)..................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Allgather(801).......................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIR_Allgather_intra(216).................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIC_Sendrecv(475)........................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIC_Wait(243)............................:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIDI_CH3i_Progress_wait(239).............: an error occurred while handling an event returned by MPIDU_Sock_Wait()</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIDI_CH3I_Progress_handle_sock_event(451):</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">MPIDU_Socki_handle_read(649)..............: connection failure (set=0,sock=1,errno=104:Connection reset by peer)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: H5F_close: Assertion `f->file_id > 0’ failed.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Please note that the above trace was the original stacktrace which appears to utilize sockets, though I’ve reproduced the same problem by running on an SMP with shared memory.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">While it’s not definitive that the issue has anything to do with the above stack trace, the very same benchmark runs perfectly well utilizing PHDF5 built with OpenMPI. My own testing is with MPICH version
3.2 available from your download site and with OpenMPI 2.0.1 (also their latest download). Both MPI releases were built from source on my Fedora 25 Linux distribution using GCC 6.2.1 20160916 (Red Hat 6.2.1-2).</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Given that the synchronous calls into MPI_File_set_view appear to be coded correctly AND that there isn’t much in the way of input parameters that would cause problems (other than incorrect coding), we tend
to believe that the internal message queues between processes may somehow be corrupted. This impression is strengthened by the fact that our recent codebase changes (which are unrelated to the actual calls to MPI_File_set_view) may have introduced this issue.
Note too, that the code paths to MPI_File_set_view have been taken many times previously and those function calls have all succeeded.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Are there any suggestions out there as to how to further debug this potential corruption issue?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Many thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Richard A. Warren</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
</body>
</html>