[mpich-discuss] stdin redirection

[Balaji, Pavan]

If you want to read large files, you should really use MPI-I/O, not stdin.  We support reading from stdin for users who migrate from sequential programs to parallel programming, but that's not meant to be the real mechanism to read files.  I'm not supportive of trying to modify mpich for such use cases.

  -- Pavan

> On Jun 5, 2016, at 11:18 AM, Seo, Sangmin <sseo at anl.gov> wrote:
> 
> Hello Salvatore,
> 
> The current MPICH uses pipe to redirect stdin to processes, but pipe has a limited capacity (e.g., 64KB in Linux. Please refer to ​http://man7.org/linux/man-pages/man7/pipe.7.html). Therefore, if we redirect a file, which is larger than the pipe capacity, to stdin, the current MPICH implementation is not handling it properly because write() does not succeed to write a large data (> the pipe capacity) to the pipe for stdin.
> 
> I created a ticket for this issue (https://trac.mpich.org/projects/mpich/ticket/2336), but I'd recommend you to use file I/O in order to handle a large file instead of using stdin redirection.
> 
> — Sangmin
> 
> 
>> On Jun 1, 2016, at 4:37 AM, Filippone, Salvatore <Salvatore.Filippone at cranfield.ac.uk> wrote:
>> 
>> Hello,
>> I have found something I do not understand with mpich 3.2.0. The
>> problem started with an application, but to demonstrate it I have
>> written a very simple-minded program which I am including here. 
>> The program simply reads lines from standard input  and reprints them
>> on standard output from a second process. 
>> 
>> Now if I redirect stdin from a small(ish) file it works as expected:
>> ======================================================================
>> [sfilippo at localhost MPI_cat]$ mpicc -v
>> mpicc for MPICH version 3.2
>> Using built-in specs.
>> COLLECT_GCC=gcc
>> COLLECT_LTO_WRAPPER=/opt/gnu/5.3.0/libexec/gcc/x86_64-unknown-linux-
>> gnu/5.3.0/lto-wrapper
>> Target: x86_64-unknown-linux-gnu
>> Configured with: ../gcc-5.3.0/configure --prefix=/opt/gnu/5.3.0 --
>> enable-languages=c,c++,fortran --with-
>> gmp=/usr/local/travel/GCC/BUILDS/gmp --with-
>> mpfr=/usr/local/travel/GCC/BUILDS/mpfr --with-
>> mpc=/usr/local/travel/GCC/BUILDS/mpc --with-
>> isl=/usr/local/travel/GCC/BUILDS/isl
>> Thread model: posix
>> gcc version 5.3.0 (GCC) 
>> [sfilippo at localhost MPI_cat]$ mpicc -o mpicat mpicat.c
>> [sfilippo at localhost MPI_cat]$ head mpicat.c | mpirun -np 2 ./mpicat
>> #include <stdlib.h>
>> #include <stdio.h>
>> #include "mpi.h"
>> #define LNSZ  129
>> #define TAGI  234
>> #define TAGC  567
>> int main(int argc, char *argv[] )
>> {
>>  char linebuf[LNSZ];
>>  int ierr, iam, np, itmp ;
>> [sfilippo at localhost MPI_cat]$ 
>> ======================================================================
>> However if I use a long(ish) file (such as a configure script) I get
>> either no output or an error. On another machine with GNU 4.9.3 I
>> always get the silent failure.
>> ======================================================================
>> [sfilippo at localhost MPI_cat]$ mpirun -np 2 ./mpicat < configure 
>> [sfilippo at localhost MPI_cat]$ mpirun -np 2 ./mpicat < configure 
>> [proxy:0:0 at localhost.localdomain] HYD_pmcd_pmip_control_cmd_cb
>> (../../../../mpich-3.2/src/pm/hydra/pm/pmiserv/pmip_cb.c:937): process
>> reading stdin too slowly; can't keep up
>> [proxy:0:0 at localhost.localdomain] HYDT_dmxu_poll_wait_for_event
>> (../../../../mpich-3.2/src/pm/hydra/tools/demux/demux_poll.c:76):
>> callback returned error status
>> [proxy:0:0 at localhost.localdomain] main (../../../../mpich-
>> 3.2/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for
>> event
>> [sfilippo at localhost MPI_cat]$ 
>> =======================================================================
>> 
>> Am I missing something? 
>> Is this supposed to work? I could find nothing relevant in the FAQ
>> 
>> Thanks a lot 
>> Salvatore
>> -- 
>> Salvatore Filippone, PhD
>> Centre for Computational Engineering Sciences,
>> School of Aerospace, Transport and Manufacturing, Bldg 52
>> Cranfield University
>> Cranfield
>> MK43 0AL
>> United Kingdom
>> tel: +44(0) 1234 758361
>> 
>> <mpicat.c>_______________________________________________
>> discuss mailing list     discuss at mpich.org
>> To manage subscription options or unsubscribe:
>> https://lists.mpich.org/mailman/listinfo/discuss
> 
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss