[mpich-discuss] 1. Single file reading by all processors in MPI. 2.Difference between blocking and non-blocking call of MPI_neighbor_alltoallw

[Joseph Schuchart]

On 5/10/20 7:30 PM, Jeff Hammond via discuss wrote:
> 
> 
>> On May 9, 2020, at 8:34 PM, hritikesh semwal 
>> <hritikesh.semwal at gmail.com> wrote:
>>
>> 
>>
>>
>> On Sun, May 10, 2020 at 4:18 AM Jeff Hammond <jeff.science at gmail.com 
>> <mailto:jeff.science at gmail.com>> wrote:
>>
>>
>>
>>     On Sat, May 9, 2020 at 10:10 AM hritikesh semwal via discuss
>>     <discuss at mpich.org <mailto:discuss at mpich.org>> wrote:
>>
>>         Hello,
>>
>>         I have following two questions with MPI,
>>
>>         1. Do I need to give a separate input file to each processor
>>         in a group or all of them can read the input from a single
>>         file? All data inside the file should be read by every processor.
>>
>>
>>     You can read the input file from every process but it isn't a good
>>     idea.  Read the input file from rank 0 and broadcast the
>>     contents.  This assumes that your input file is small.  If your
>>     input file is huge, then you should consider designing it for
>>     parallel reading with MPI_File_* functions.
>>
>>
>> Thanks for your response. I am reading two input files in my parallel 
>> CFD solver at different places of the code. The first input file has 7 
>> integer values and second input file has 15 integer values. Is it 
>> large to read it through conventional C functions for file reading 
>> i.e. fscanf() and is broadcast good for this amount of data?
> 
> It depends on your filesystem details but I’d expect the cutoff where 
> parallel reading of input to be necessary is greater than 1 MB and 
> potentially much more than that. Anything less than 4K (or whatever the 
> filesystem page size is if not 4K) is going to be dominated by block 
> read latency, which can’t be reduced with parallelism.

The latency/bandwidth for reading is only one aspect though, the 
filesystem overhead is another. Your site administrators will be 
grateful for you *not* trying to open the same file from a couple 
thousand processes at once... Reading it once in a single process and 
distributing it in a broadcast is imo the best option in this case.

Cheers
Joseph, who brought down Lustre installations in the past...

> 
> Jeff
> 
>>         2. Can you please tell me what is the difference between
>>         MPI_Neighbor_alltoallw and MPI_Ineighbor_alltoallw? As I read
>>         in MPI 3.0 document that MPI_Neighbor_alltoallw also used
>>         non-blocking send and receive inside its execution.
>>
>>
>>     MPI_Neighbor_alltoallw is a blocking function while
>>     MPI_Ineighbor_alltoallw is not.  Using non-blocking send and
>>     receive inside its execution does not mean
>>     that MPI_Neighbor_alltoallw doesn't block - if that is how it's
>>     implemented, then there will be the equivalent of a MPI_Waitall
>>     before the function returns.  But in any case, just because the
>>     MPI document suggests a possible implementation does not mean that
>>     is how all implementations work.
>>
>>
>> Yes, there is MPI_Waitall inside the MPI_Neighbor_alltoallw before it 
>> returns.
>>
>>
>>     Jeff
>>
>>         Thank you.
>>         _______________________________________________
>>         discuss mailing list discuss at mpich.org <mailto:discuss at mpich.org>
>>         To manage subscription options or unsubscribe:
>>         https://lists.mpich.org/mailman/listinfo/discuss
>>
>>
>>
>>     -- 
>>     Jeff Hammond
>>     jeff.science at gmail.com <mailto:jeff.science at gmail.com>
>>     http://jeffhammond.github.io/
>>
> 
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>