[mpich-discuss] BUG in ADIOI_NFS_WriteStrided
Rob Latham
robl at mcs.anl.gov
Fri Dec 19 20:52:14 CST 2014
On 12/19/2014 02:33 PM, Eric Chamberland wrote:
> Hi Howard,
>
> the bug is present also with MPICH-3.1.3...
>
> So, for discuss at mpich.org list readers, here is the valgrind output for
> a bug revealed with valgrind (sorry, I didn't compiled MPICH in debug
> mode), reported to OpenMPI earlier today:
> http://www.open-mpi.org/community/lists/devel/2014/12/16691.php
>
> sorry for the "duplicated" report again.
>
> I encountered a new bug while testing our collective MPI I/O
> functionnalities over NFS. This is not a big issue for us, but I think
> someone should have a look at it.
Please don't use NFS for MPI-IO. ROMIO makes a best effort but there's
no way to guarantee you won't corrupt a block of data (NFS clients are
allowed to cache... arbitrarily, it seems). There are so many good
parallel file systems with saner consistency semantics .
This looks like maybe a calloc would clean it right up.
==rob
>
> While running at 3 processes, we have this error on rank #0 and rank #2,
> knowing that rank #1 have nothing to write (0 length size) on this
> particular PMPI_File_write_all_begin call:
>
> ==3434== Syscall param write(buf) points to uninitialised byte(s)
> ==3434== at 0x108D0380: __write_nocancel (in /lib64/libpthread-2.17.so)
> ==3434== by 0x11DB9D46: ADIOI_NFS_WriteStrided (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DD264F: ADIOI_GEN_WriteStridedColl (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DB5F8F: MPIOI_File_write_all_begin (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DB60F3: PMPI_File_write_all_begin (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x4CCAE6: SYEnveloppeMessage<std::string>
> PAIO::ecritureIndexeParBlocMPI<PAIOType<double>, PtrPorteurConst<Arete,
> Arete>, FunctorCopieInfosSurDansVectPAType<PAIOType<double>,
> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
> Arete>*> > const>, FunctorAccesseurPorteurLocal<PtrPorteurConst<Arete,
> Arete> > >(PAGroupeProcessus&, ADIOI_FileD*, long long,
> PtrPorteurConst<Arete, Arete>, PtrPorteurConst<Arete, Arete>,
> FunctorCopieInfosSurDansVectPAType<PAIOType<double>,
> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
> Arete>*> > const>&, FunctorAccesseurPorteurLocal<PtrPorteurConst<Arete,
> Arete> >&, long, DistributionComposantes&, long, unsigned long, unsigned
> long, std::string const&) (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== by 0x4DDBFE:
> GISLectureEcriture<double>::visiteMaillage(Maillage const&) (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== by 0x4BCB22:
> GISLectureEcriture<double>::ecritGISMPI(std::string,
> GroupeInfoSur<double> const&, std::string const&) (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== by 0x48E213: main (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== Address 0x1a12cd10 is 224 bytes inside a block of size 524,448
> alloc'd
> ==3434== at 0x4C2C27B: malloc (in
> /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==3434== by 0x11DADA96: MPL_trmalloc (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DD9285: ADIOI_Malloc_fn (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DB9AC8: ADIOI_NFS_WriteStrided (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DD264F: ADIOI_GEN_WriteStridedColl (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DB5F8F: MPIOI_File_write_all_begin (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DB60F3: PMPI_File_write_all_begin (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x4CCAE6: SYEnveloppeMessage<std::string>
> PAIO::ecritureIndexeParBlocMPI<PAIOType<double>, PtrPorteurConst<Arete,
> Arete>, FunctorCopieInfosSurDansVectPAType<PAIOType<double>,
> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
> Arete>*> > const>, FunctorAccesseurPorteurLocal<PtrPorteurConst<Arete,
> Arete> > >(PAGroupeProcessus&, ADIOI_FileD*, long long,
> PtrPorteurConst<Arete, Arete>, PtrPorteurConst<Arete, Arete>,
> FunctorCopieInfosSurDansVectPAType<PAIOType<double>,
> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
> Arete>*> > const>&, FunctorAccesseurPorteurLocal<PtrPorteurConst<Arete,
> Arete> >&, long, DistributionComposantes&, long, unsigned long, unsigned
> long, std::string const&) (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== by 0x4DDBFE:
> GISLectureEcriture<double>::visiteMaillage(Maillage const&) (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== by 0x4BCB22:
> GISLectureEcriture<double>::ecritGISMPI(std::string,
> GroupeInfoSur<double> const&, std::string const&) (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== by 0x48E213: main (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== Uninitialised value was created by a client request
> ==3434== at 0x11DADEE5: MPL_trmalloc (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DD9285: ADIOI_Malloc_fn (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DB9AC8: ADIOI_NFS_WriteStrided (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DD264F: ADIOI_GEN_WriteStridedColl (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DB5F8F: MPIOI_File_write_all_begin (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x11DB60F3: PMPI_File_write_all_begin (in
> /opt/mpich-3.1.3/lib64/libmpi.so.12.0.4)
> ==3434== by 0x4CCAE6: SYEnveloppeMessage<std::string>
> PAIO::ecritureIndexeParBlocMPI<PAIOType<double>, PtrPorteurConst<Arete,
> Arete>, FunctorCopieInfosSurDansVectPAType<PAIOType<double>,
> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
> Arete>*> > const>, FunctorAccesseurPorteurLocal<PtrPorteurConst<Arete,
> Arete> > >(PAGroupeProcessus&, ADIOI_FileD*, long long,
> PtrPorteurConst<Arete, Arete>, PtrPorteurConst<Arete, Arete>,
> FunctorCopieInfosSurDansVectPAType<PAIOType<double>,
> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
> Arete>*> > const>&, FunctorAccesseurPorteurLocal<PtrPorteurConst<Arete,
> Arete> >&, long, DistributionComposantes&, long, unsigned long, unsigned
> long, std::string const&) (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== by 0x4DDBFE:
> GISLectureEcriture<double>::visiteMaillage(Maillage const&) (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== by 0x4BCB22:
> GISLectureEcriture<double>::ecritGISMPI(std::string,
> GroupeInfoSur<double> const&, std::string const&) (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
> ==3434== by 0x48E213: main (in
> /home/mefpp_ericc/GIREF/bin/Test.LectureEcritureGISMPI.opt)
>
> Can't tell if it is a big issue or not, but I thought I should mention
> it to the list....
>
> We run without this valgrind error when I use my local disk partition
> instead of an nfs parition or if I run with only 1 process (which
> always have something to write for each PMPI_File_write_all_begin) and
> write to an nfs partition.
>
> Have you guys thinked about unifying this part of code? Making it a
> sub-library? (please don't flame me... ;-) )
>
> Anyway,
>
> thanks,
>
> Eric
>
>
> On 12/19/2014 02:16 PM, Howard Pritchard wrote:
>> HI Eric,
>>
>> Does your app also work with MPICH? The romio in Open MPI is getting a
>> bit old, so it would be useful to know if you see the same valgrind
>> error using a recent MPICH.
>>
>> Howard
>>
>>
>> 2014-12-19 9:50 GMT-07:00 Eric Chamberland
>> <Eric.Chamberland at giref.ulaval.ca
>> <mailto:Eric.Chamberland at giref.ulaval.ca>>:
>>
>> Hi,
>>
>> I encountered a new bug while testing our collective MPI I/O
>> functionnalities over NFS. This is not a big issue for us, but I
>> think someone should have a look at it.
>>
>> While running at 3 processes, we have this error on rank #0 and rank
>> #2, knowing that rank #1 have nothing to write (0 length size) on
>> this particular PMPI_File_write_all_begin call:
>>
>>
>> ==19211== Syscall param write(buf) points to uninitialised byte(s)
>> ==19211== at 0x10CB739D: ??? (in /lib64/libpthread-2.17.so
>> <http://libpthread-2.17.so>)
>> ==19211== by 0x27438431: ADIOI_NFS_WriteStrided
>> (ad_nfs_write.c:645)
>> ==19211== by 0x27451963: ADIOI_GEN_WriteStridedColl
>> (ad_write_coll.c:159)
>> ==19211== by 0x274321BD: MPIOI_File_write_all_begin
>> (write_allb.c:114)
>> ==19211== by 0x27431DBF:
>> mca_io_romio_dist_MPI_File___write_all_begin (write_allb.c:44)
>> ==19211== by 0x2742A367: mca_io_romio_file_write_all___begin
>> (io_romio_file_write.c:264)
>> ==19211== by 0x12126520: PMPI_File_write_all_begin
>> (pfile_write_all_begin.c:74)
>> ==19211== by 0x4D7CFB: SYEnveloppeMessage<std::__string>
>> PAIO::__ecritureIndexeParBlocMPI<__PAIOType<double>,
>> PtrPorteurConst<Arete, Arete>,
>> FunctorCopieInfosSurDansVectPA__Type<PAIOType<double>,
>> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
>> Arete>*> > const>,
>> FunctorAccesseurPorteurLocal<__PtrPorteurConst<Arete, Arete> >
>> >(PAGroupeProcessus&, ompi_file_t*, long long,
>> PtrPorteurConst<Arete, Arete>, PtrPorteurConst<Arete, Arete>,
>> FunctorCopieInfosSurDansVectPA__Type<PAIOType<double>,
>> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
>> Arete>*> > const>&,
>> FunctorAccesseurPorteurLocal<__PtrPorteurConst<Arete, Arete> >&,
>> long, DistributionComposantes&, long, unsigned long, unsigned long,
>> std::string const&) (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== by 0x4E9A67:
>> GISLectureEcriture<double>::__visiteMaillage(Maillage const&) (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== by 0x4C79A2:
>> GISLectureEcriture<double>::__ecritGISMPI(std::string,
>> GroupeInfoSur<double> const&, std::string const&) (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== by 0x4961AD: main (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== Address 0x295af060 is 144 bytes inside a block of size
>> 524,288 alloc'd
>> ==19211== at 0x4C2C27B: malloc (in
>> /usr/lib64/valgrind/vgpreload___memcheck-amd64-linux.so)
>> ==19211== by 0x2745E78E: ADIOI_Malloc_fn (malloc.c:50)
>> ==19211== by 0x2743757C: ADIOI_NFS_WriteStrided
>> (ad_nfs_write.c:497)
>> ==19211== by 0x27451963: ADIOI_GEN_WriteStridedColl
>> (ad_write_coll.c:159)
>> ==19211== by 0x274321BD: MPIOI_File_write_all_begin
>> (write_allb.c:114)
>> ==19211== by 0x27431DBF:
>> mca_io_romio_dist_MPI_File___write_all_begin (write_allb.c:44)
>> ==19211== by 0x2742A367: mca_io_romio_file_write_all___begin
>> (io_romio_file_write.c:264)
>> ==19211== by 0x12126520: PMPI_File_write_all_begin
>> (pfile_write_all_begin.c:74)
>> ==19211== by 0x4D7CFB: SYEnveloppeMessage<std::__string>
>> PAIO::__ecritureIndexeParBlocMPI<__PAIOType<double>,
>> PtrPorteurConst<Arete, Arete>,
>> FunctorCopieInfosSurDansVectPA__Type<PAIOType<double>,
>> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
>> Arete>*> > const>,
>> FunctorAccesseurPorteurLocal<__PtrPorteurConst<Arete, Arete> >
>> >(PAGroupeProcessus&, ompi_file_t*, long long,
>> PtrPorteurConst<Arete, Arete>, PtrPorteurConst<Arete, Arete>,
>> FunctorCopieInfosSurDansVectPA__Type<PAIOType<double>,
>> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
>> Arete>*> > const>&,
>> FunctorAccesseurPorteurLocal<__PtrPorteurConst<Arete, Arete> >&,
>> long, DistributionComposantes&, long, unsigned long, unsigned long,
>> std::string const&) (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== by 0x4E9A67:
>> GISLectureEcriture<double>::__visiteMaillage(Maillage const&) (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== by 0x4C79A2:
>> GISLectureEcriture<double>::__ecritGISMPI(std::string,
>> GroupeInfoSur<double> const&, std::string const&) (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== by 0x4961AD: main (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== Uninitialised value was created by a heap allocation
>> ==19211== at 0x4C2C27B: malloc (in
>> /usr/lib64/valgrind/vgpreload___memcheck-amd64-linux.so)
>> ==19211== by 0x2745E78E: ADIOI_Malloc_fn (malloc.c:50)
>> ==19211== by 0x2743757C: ADIOI_NFS_WriteStrided
>> (ad_nfs_write.c:497)
>> ==19211== by 0x27451963: ADIOI_GEN_WriteStridedColl
>> (ad_write_coll.c:159)
>> ==19211== by 0x274321BD: MPIOI_File_write_all_begin
>> (write_allb.c:114)
>> ==19211== by 0x27431DBF:
>> mca_io_romio_dist_MPI_File___write_all_begin (write_allb.c:44)
>> ==19211== by 0x2742A367: mca_io_romio_file_write_all___begin
>> (io_romio_file_write.c:264)
>> ==19211== by 0x12126520: PMPI_File_write_all_begin
>> (pfile_write_all_begin.c:74)
>> ==19211== by 0x4D7CFB: SYEnveloppeMessage<std::__string>
>> PAIO::__ecritureIndexeParBlocMPI<__PAIOType<double>,
>> PtrPorteurConst<Arete, Arete>,
>> FunctorCopieInfosSurDansVectPA__Type<PAIOType<double>,
>> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
>> Arete>*> > const>,
>> FunctorAccesseurPorteurLocal<__PtrPorteurConst<Arete, Arete> >
>> >(PAGroupeProcessus&, ompi_file_t*, long long,
>> PtrPorteurConst<Arete, Arete>, PtrPorteurConst<Arete, Arete>,
>> FunctorCopieInfosSurDansVectPA__Type<PAIOType<double>,
>> std::vector<InfoSur<double, Arete>*, std::allocator<InfoSur<double,
>> Arete>*> > const>&,
>> FunctorAccesseurPorteurLocal<__PtrPorteurConst<Arete, Arete> >&,
>> long, DistributionComposantes&, long, unsigned long, unsigned long,
>> std::string const&) (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== by 0x4E9A67:
>> GISLectureEcriture<double>::__visiteMaillage(Maillage const&) (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== by 0x4C79A2:
>> GISLectureEcriture<double>::__ecritGISMPI(std::string,
>> GroupeInfoSur<double> const&, std::string const&) (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211== by 0x4961AD: main (in
>> /home/mefpp_ericc/GIREF/bin/__Test.LectureEcritureGISMPI.__opt)
>> ==19211==
>>
>> Can't tell if it is a big issue or not, but I thought I should
>> mention it to the list....
>>
>> We run without this valgrind error when I use my local disk
>> partition instead of an nfs parition or if I run with only 1
>> process (which always have something to write for each
>> PMPI_File_write_all_begin) and write to an nfs partition.
>>
>> Using openmpi-1.8.4rc3 compiled in "debug" mode:
>>
>> ompi_info -all :
>>
>> http://www.giref.ulaval.ca/~__ericc/ompi_bug/ompi_info.all.__184rc3.txt.gz
>>
>>
>> <http://www.giref.ulaval.ca/~ericc/ompi_bug/ompi_info.all.184rc3.txt.gz>
>> config.log:
>> http://www.giref.ulaval.ca/~__ericc/ompi_bug/config.184rc3.__log.gz
>> <http://www.giref.ulaval.ca/~ericc/ompi_bug/config.184rc3.log.gz>
>>
>> Thanks,
>>
>> Eric
>> _________________________________________________
>> devel mailing list
>> devel at open-mpi.org <mailto:devel at open-mpi.org>
>> Subscription: http://www.open-mpi.org/__mailman/listinfo.cgi/devel
>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
>> Link to this post:
>> http://www.open-mpi.org/__community/lists/devel/2014/12/__16691.php
>> <http://www.open-mpi.org/community/lists/devel/2014/12/16691.php>
>>
>
> _______________________________________________
> discuss mailing list discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
_______________________________________________
discuss mailing list discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss
More information about the discuss
mailing list