<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The backtrace simply shows the process is in the progress loop, i.e. waiting for a message from another process but never come. The reason could due to a bug in mpich, or a bug in the application library, or in the test.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-- <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Orion Poplawski via discuss <discuss@mpich.org><br>
<b>Sent:</b> Thursday, January 13, 2022 9:32 PM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Orion Poplawski <orion@nwra.com><br>
<b>Subject:</b> [mpich-discuss] netcdf-fortran 4.5.4 test hang on s390x</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hello -<br>
<br>
I'm working on updating the Fedora netcdf-fortran package to version <br>
4.5.4. However, a test is hanging when run on s390x (other arches are <br>
fine). This is with:<br>
<br>
mpich-3.4.1-3.fc35.s390x<br>
hdf5-mpich-1.12.1-1.fc36.s390x<br>
netcdf-mpich-4.8.1-2.fc36.s390x<br>
<br>
The test program is nf03_test4/f90tst_parallel_compressed and is run with:<br>
<br>
mpiexec -np 4 ./f90tst_parallel_compressed<br>
<br>
Here's a backtrace where it seems to be stuck:<br>
<br>
#0 0x000003ffb14fbc94 in MPID_nem_tcp_connpoll <br>
(in_blocking_poll=<optimized out>) at <br>
src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c:1822<br>
#1 0x000003ffb14ef78c in MPID_nem_network_poll (in_blocking_progress=1) <br>
at src/mpid/ch3/channels/nemesis/src/mpid_nem_network_poll.c:11<br>
#2 MPID_nem_mpich_blocking_recv (completions=615, in_fbox=<synthetic <br>
pointer>, cell=<synthetic pointer>) at <br>
./src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:964<br>
#3 MPIDI_CH3I_Progress (progress_state=0x3ffdf66c168, <br>
is_blocking=<optimized out>) at <br>
src/mpid/ch3/channels/nemesis/src/ch3_progress.c:363<br>
#4 0x000003ffb1b2ecc0 in MPIR_Wait_state (status=<optimized out>, <br>
state=0x3ffdf66c24c, request_ptr=0x3ffb1d17510 <MPIR_Request_direct>) at <br>
src/mpi/request/wait.c:32<br>
#5 MPIR_Wait_impl.isra.0 (request_ptr=request_ptr@entry=0x3ffb1d17510 <br>
<MPIR_Request_direct>, status=<optimized out>) at src/mpi/request/wait.c:55<br>
#6 0x000003ffb1484ca0 in MPID_Wait (status=0x1, <br>
request_ptr=0x3ffb1d17510 <MPIR_Request_direct>) at <br>
./src/mpid/ch3/include/mpidpost.h:264<br>
#7 MPIC_Wait (request_ptr=0x3ffb1d17510 <MPIR_Request_direct>, <br>
errflag=errflag@entry=0x3ffdf66c8ec) at src/mpi/coll/helper_fns.c:64<br>
#8 0x000003ffb1485a04 in MPIC_Sendrecv (sendbuf=<optimized out>, <br>
sendcount=1, sendtype=<optimized out>, dest=<optimized out>, <br>
sendtag=<optimized out>, recvbuf=0x2aa1bf8f8b0, recvcount=1, <br>
recvtype=1275069445, source=1, recvtag=14,<br>
comm_ptr=0x3ffae42f690, status=<optimized out>, <br>
errflag=0x3ffdf66c8ec) at src/mpi/coll/helper_fns.c:333<br>
#9 0x000003ffb141fe4e in MPIR_Allreduce_intra_recursive_doubling <br>
(sendbuf=<optimized out>, recvbuf=0x3ffdf66c9e0, count=<optimized out>, <br>
datatype=<optimized out>, op=<optimized out>, comm_ptr=0x3ffae42f690, <br>
errflag=0x3ffdf66c8ec)<br>
at src/mpi/coll/allreduce/allreduce_intra_recursive_doubling.c:127<br>
#10 0x000003ffb1322a68 in MPIR_Allreduce_allcomm_auto <br>
(sendbuf=0x3ffdf66c9d0, sendbuf@entry=<error reading variable: value has <br>
been optimized out>, recvbuf=0x3ffdf66c9e0,<br>
recvbuf@entry=<error reading variable: value has been optimized <br>
out>, count=count@entry=1, datatype=datatype@entry=1275069445, <br>
op=op@entry=1476395011, comm_ptr=0x3ffae42f690, errflag=0x3ffdf66c8ec)<br>
at src/mpi/coll/allreduce/allreduce.c:145<br>
#11 0x000003ffb1322c28 in MPIR_Allreduce_impl (sendbuf=<error reading <br>
variable: value has been optimized out>, recvbuf=<error reading <br>
variable: value has been optimized out>, count=<optimized out>, <br>
datatype=<optimized out>,<br>
op=op@entry=1476395011, comm_ptr=0x3ffae42f690, <br>
errflag=0x3ffdf66c8ec) at src/mpi/coll/allreduce/allreduce.c:210<br>
#12 0x000003ffb1322d6a in MPID_Allreduce (errflag=0x3ffdf66c8ec, <br>
comm=0x3ffae42f690, op=op@entry=1476395011, datatype=<optimized out>, <br>
datatype@entry=1275069445, count=<optimized out>, count@entry=1,<br>
recvbuf=<error reading variable: value has been optimized out>, <br>
sendbuf=<error reading variable: value has been optimized out>) at <br>
./src/mpid/ch3/include/mpid_coll.h:41<br>
#13 MPIR_Allreduce (sendbuf=<error reading variable: value has been <br>
optimized out>, recvbuf=<error reading variable: value has been <br>
optimized out>, count=count@entry=1, datatype=datatype@entry=1275069445, <br>
op=op@entry=1476395011,<br>
comm_ptr=0x3ffae42f690, errflag=0x3ffdf66c8ec) at <br>
src/mpi/coll/allreduce/allreduce.c:262<br>
#14 0x000003ffb1323742 in PMPI_Allreduce (sendbuf=<optimized out>, <br>
sendbuf@entry=<error reading variable: value has been optimized out>, <br>
recvbuf=<optimized out>, recvbuf@entry=0x3ffdf66c9e0, count=count@entry=1,<br>
datatype=datatype@entry=1275069445, op=op@entry=1476395011, <br>
comm=-1006632958) at src/mpi/coll/allreduce/allreduce.c:387<br>
#15 0x000003ffb0e11644 in H5D__mpio_array_gatherv.constprop.0 <br>
(local_array=<optimized out>, local_array@entry=0x2aa1bfac440, <br>
local_array_num_entries=<optimized out>, _gathered_array=<optimized out>,<br>
_gathered_array@entry=0x3ffdf66cb78, <br>
_gathered_array_num_entries=_gathered_array_num_entries@entry=0x3ffdf66cb88, <br>
allgather=allgather@entry=true, comm=-1006632958, sort_func=0x0, root=0, <br>
array_entry_size=368)<br>
at ../../src/H5Dmpio.c:552<br>
#16 0x000003ffb0e086cc in H5D__link_chunk_filtered_collective_io <br>
(fm=0x2aa1bef9c60, type_info=0x3ffdf66ce00, io_info=0x3ffdf66ce80) at <br>
../../src/H5Dmpio.c:1409<br>
#17 H5D__chunk_collective_io (io_info=0x3ffdf66ce80, <br>
type_info=0x3ffdf66ce00, fm=fm@entry=0x2aa1bef9c60) at <br>
../../src/H5Dmpio.c:886<br>
#18 0x000003ffb0e0abb4 in H5D__chunk_collective_write <br>
(io_info=<optimized out>, type_info=<optimized out>, nelmts=<optimized <br>
out>, file_space=<optimized out>, mem_space=0x2aa1bf8fac0, <br>
fm=0x2aa1bef9c60) at ../../src/H5Dmpio.c:967<br>
#19 0x000003ffb0b78702 in H5D__write (dataset=0x2aa1bf0cda0, <br>
mem_type_id=<optimized out>, mem_space=<optimized out>, <br>
file_space=0x2aa1beee270, buf=<optimized out>) at ../../src/H5Dio.c:761<br>
#20 0x000003ffb0dd9598 in H5VL__native_dataset_write (obj=0x2aa1bf0cda0, <br>
mem_type_id=216172782113784251, mem_space_id=<optimized out>, <br>
file_space_id=288230376151711833, dxpl_id=<optimized out>, <br>
buf=0x3ffa41b9010, req=0x0)<br>
at ../../src/H5VLnative_dataset.c:206<br>
#21 0x000003ffb0db7b72 in H5VL__dataset_write (obj=<optimized out>, <br>
cls=<optimized out>, mem_type_id=mem_type_id@entry=216172782113784251, <br>
mem_space_id=mem_space_id@entry=288230376151711834,<br>
file_space_id=file_space_id@entry=288230376151711833, <br>
dxpl_id=792633534417207358, buf=0x3ffa41b9010, req=0x0) at <br>
../../src/H5VLcallback.c:2079<br>
#22 0x000003ffb0dc1b9a in H5VL_dataset_write (vol_obj=0x2aa1bf0cd00, <br>
mem_type_id=216172782113784251, mem_space_id=288230376151711834, <br>
file_space_id=288230376151711833, dxpl_id=<optimized out>, <br>
buf=0x3ffa41b9010, req=0x0)<br>
at ../../src/H5VLcallback.c:2111<br>
#23 0x000003ffb0b7506c in H5Dwrite (dset_id=<optimized out>, <br>
mem_type_id=216172782113784251, mem_space_id=288230376151711834, <br>
file_space_id=288230376151711833, dxpl_id=792633534417207358, <br>
buf=0x3ffa41b9010) at ../../src/H5Dio.c:289<br>
#24 0x000003ffb1f8fdc8 in NC4_put_vars (ncid=<optimized out>, varid=0, <br>
startp=0x0, countp=0x2aa1bf0e720, stridep=0x3ffdf66f388, data=<optimized <br>
out>, mem_nc_type=5) at ../libhdf5/../../libhdf5/hdf5var.c:1594<br>
#25 0x000003ffb1f90148 in NC4_put_vara (ncid=<optimized out>, <br>
varid=<optimized out>, startp=<optimized out>, countp=<optimized out>, <br>
op=<optimized out>, memtype=5) at ../libhdf5/../../libhdf5/hdf5var.c:1178<br>
#26 0x000003ffb1f3b96c in NC_put_vara (ncid=<optimized out>, <br>
varid=<optimized out>, start=0x2aa1bf8f620, edges=0x2aa1bf8f910, <br>
value=value@entry=0x3ffa41b9010, memtype=5) at <br>
../libdispatch/../../libdispatch/dvarput.c:97<br>
#27 0x000003ffb1f3be08 in nc_put_vara_float (ncid=<optimized out>, <br>
varid=<optimized out>, startp=<optimized out>, countp=<optimized out>, <br>
op=0x3ffa41b9010) at ../libdispatch/../../libdispatch/dvarput.c:695<br>
#28 0x000003ffb269464e in nf_put_vara_real_ (ncid=<optimized out>, <br>
varid=<optimized out>, start=<error reading variable: value has been <br>
optimized out>, counts=<error reading variable: value has been optimized <br>
out>, rvals=...)<br>
at ../../fortran/nf_varaio.F90:372<br>
#29 0x000003ffb26ede9e in netcdf::nf90_put_var_4d_fourbytereal <br>
(ncid=<error reading variable: value has been optimized out>, <br>
varid=<error reading variable: value has been optimized out>,<br>
values=<error reading variable: value has been optimized out>, <br>
start=<error reading variable: value has been optimized out>,<br>
count=<error reading variable: value requires 27021597768417284 <br>
bytes, which is more than max-value-size>, stride=<error reading <br>
variable: value has been optimized out>, map=<error reading variable: <br>
value has been optimized out>)<br>
at netcdf_expanded.f90:1035<br>
#30 0x000002aa1b1840ba in f90tst_parallel_compressed () at <br>
../../nf03_test4/f90tst_parallel_compressed.F90:255<br>
#31 0x000002aa1b182588 in main (argc=<optimized out>, argv=<optimized <br>
out>) at ../../nf03_test4/f90tst_parallel_compressed.F90:14<br>
<br>
It seems to be stuck in MPID_nem_mpich_blocking_recv.<br>
<br>
I have no idea how to debug this further. Any suggestions would be <br>
greatly appreciated.<br>
<br>
It does appear to have been introduced between netcdf-fortran 4.5.2 and <br>
4.5.4. I can still build and run the tests for 4.5.2.<br>
<br>
-- <br>
Orion Poplawski<br>
he/him/his - surely the least important thing about me<br>
Manager of NWRA Technical Systems 720-772-5637<br>
NWRA, Boulder/CoRA Office FAX: 303-415-9702<br>
3380 Mitchell Lane orion@nwra.com<br>
Boulder, CO 80301 <a href="https://www.nwra.com/">https://www.nwra.com/</a><br>
</div>
</span></font></div>
</body>
</html>