[mpich-devel] external32 (was: Re: binding a new MPI type with fortran)

[Rob Latham]

On Wed, Apr 03, 2013 at 09:19:59AM -0500, Bob Cernohous wrote:
> I've been out sick and haven't followed this conversation too closely, but 
> I've gotten a support note that seems somewhat related -- at least they're 
> asking about external32.  Any comments from the experts before I try to 
> dig in?  ROMIO supports external32?  A quick search shows external32 code 
> is common, not ad_bg specific.

external32 is indeed a part of the standard.  for 15 years ROMIO had
no external32 support, and no one seemed to care too much.  Mostly,
the role of external32 -- a portable file format -- is better played
by pnetcdf and hdf5, which in addition to being portable are also
self-describing.


Intel contributed external32 support to ROMIO, but Dave and I are
pretty sure it's not going to work when type-on-platform is different
from type-in-external32

See the end of  https://trac.mpich.org/projects/mpich/ticket/1754

> The MPI standard defines a portable data representation for MPI I/O, 
> see e.g. [1]. On BGQ, attempting to write data via MPI I/O with 
> the data representation set to 'external32', MPI does not report any 
> errors but the data written is not in the correct format. Furthermore 
> trying to read the data back with the same data representation fails 
> and does not reproduce the original data. 

Right.  I think the Intel-contributed external32 work, warts and all,
went into MPICH 3.0

sorry we did not return an error in older versions.   that's a bug.

> The example Fortran code does the following: 
> 1) write a few numbers (integer, float and double) to file via MPI I/O 
> in native format 
> 2) read the numbers back in native format 
> 3) write the same numbers to file in external32 format 
> 4) output MPI errors/success when using MPI I/O 
> 5) read the numbers back from file 
>  
> 1+2 work flawlessly 
> 3+4 show no errors and instead indicate MPI SUCCESS 
> 5 fails to reproduce correct numbers 

Once you re-sync with MPICH, this use case will work -- integer,
float, and double have the same size types in native and external32
format.  

> The data written in 1) appears as big endian data as expected. The 
> data written in 3) seems to be little endian (for integer and float) 
> and strangely byte-swapped for doubles. 
>  
> [1] http://www.mpi-forum.org/docs/mpi21-report/node285.htm 
>  
> 
> Bob Cernohous:  (T/L 553) 507-253-6093
> 
> BobC at us.ibm.com
> IBM Rochester, Building 030-2(C335), Department 61L
> 3605 Hwy 52 North, Rochester,  MN 55901-7829
> 
> > Chaos reigns within.
> > Reflect, repent, and reboot.
> > Order shall return.
> 
> 
> 
> 
> From:   Dave Goodell <goodell at mcs.anl.gov>
> To:     Larry Baker <baker at usgs.gov>, 
> Cc:     devel at mpich.org
> Date:   04/02/2013 04:48 PM
> Subject:        Re: [mpich-devel] binding a new MPI type with fortran
> Sent by:        devel-bounces at mpich.org
> 
> 
> 
> On Apr 2, 2013, at 4:34 PM CDT, Larry Baker <baker at usgs.gov> wrote:
> 
> > Jeff referred me to the MPI 3.0 spec (
> http://mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf) for data sizes.  But, 
> even it is contradictory.  For example, in section 3.2.2, it says an 
> MPI_INTEGER is a Fortran (default KIND) INTEGER.  Fortran does not specify 
> the size of a (default KIND) INTEGER.  However, Table 13.2 says it is 4 
> bytes.  It could just as easily be 8 bytes in a 64-bit execution 
> environment.  In that case, Fortran requires that the size of all the 
> default KIND numeric types in Table 13.2 would double in size.  But, not 
> necessarily pointers; pointers in Fortran do not have a specified size 
> (storage unit). 
> 
> Table 13.2 lists sizes for a specific kind of encoding of MPI datatypes 
> known as "external32".  You should not use it as a reference for the 
> actual size of the MPI datatype, which must instead always match the 
> actual language type in the corresponding language binding.
> 
> > The MPI 3.0 spec says if you send an MPI_INTEGER, you have to receive an 
> MPI_INTEGER.  I didn't exhaustively search the MPI 3.0 spec, but you might 
> also have to make sure the default KIND INTEGER is the same for the sender 
> and receiver.  This might be beyond the scope of MPI, and I do not know 
> how exhaustive the run-time tests can be to detect violations of the 
> rules.  (A message would have to carry type metadata for the payload data 
> in addition to the payload data itself.)  Section 17 had lots of 
> information about Fortran and C data sizes, including automatically 
> conversions.  I didn't read enough to know if the spec covers the case of 
> exchanging default KIND INTEGERs between two executables with different 
> sizes.  Maybe it is true that MPI mandates something like lexical scoping 
> rules, such that, for example, all members of an MPI application must 
> assign the same semantics to their fundamental data types -- the default 
> KIND INTEGER, etc., in the case of Fortran.
> 
> Most users should ignore the rules regarding conversions on heterogeneous 
> systems.  They don't make a lot of sense even if you're an MPI expert, 
> and:
> 
> 1) almost no MPI implementations properly support this behavior
> 
> 2) almost no user codes are written correctly to deal with it, even if #1 
> were true
> 
> -Dave
> 

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA