[mpich-discuss] stdin redirection

[Seo, Sangmin]

Hello Salvatore,

The current MPICH uses pipe to redirect stdin to processes, but pipe has a limited capacity (e.g., 64KB in Linux. Please refer to ​http://man7.org/linux/man-pages/man7/pipe.7.html). Therefore, if we redirect a file, which is larger than the pipe capacity, to stdin, the current MPICH implementation is not handling it properly because write() does not succeed to write a large data (> the pipe capacity) to the pipe for stdin.

I created a ticket for this issue (https://trac.mpich.org/projects/mpich/ticket/2336), but I'd recommend you to use file I/O in order to handle a large file instead of using stdin redirection.

— Sangmin


> On Jun 1, 2016, at 4:37 AM, Filippone, Salvatore <Salvatore.Filippone at cranfield.ac.uk> wrote:
> 
> Hello,
> I have found something I do not understand with mpich 3.2.0. The
> problem started with an application, but to demonstrate it I have
> written a very simple-minded program which I am including here. 
> The program simply reads lines from standard input  and reprints them
> on standard output from a second process. 
> 
> Now if I redirect stdin from a small(ish) file it works as expected:
> ======================================================================
> [sfilippo at localhost MPI_cat]$ mpicc -v
> mpicc for MPICH version 3.2
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/opt/gnu/5.3.0/libexec/gcc/x86_64-unknown-linux-
> gnu/5.3.0/lto-wrapper
> Target: x86_64-unknown-linux-gnu
> Configured with: ../gcc-5.3.0/configure --prefix=/opt/gnu/5.3.0 --
> enable-languages=c,c++,fortran --with-
> gmp=/usr/local/travel/GCC/BUILDS/gmp --with-
> mpfr=/usr/local/travel/GCC/BUILDS/mpfr --with-
> mpc=/usr/local/travel/GCC/BUILDS/mpc --with-
> isl=/usr/local/travel/GCC/BUILDS/isl
> Thread model: posix
> gcc version 5.3.0 (GCC) 
> [sfilippo at localhost MPI_cat]$ mpicc -o mpicat mpicat.c
> [sfilippo at localhost MPI_cat]$ head mpicat.c | mpirun -np 2 ./mpicat
> #include <stdlib.h>
> #include <stdio.h>
> #include "mpi.h"
> #define LNSZ  129
> #define TAGI  234
> #define TAGC  567
> int main(int argc, char *argv[] )
> {
>   char linebuf[LNSZ];
>   int ierr, iam, np, itmp ;
> [sfilippo at localhost MPI_cat]$ 
> ======================================================================
> However if I use a long(ish) file (such as a configure script) I get
> either no output or an error. On another machine with GNU 4.9.3 I
> always get the silent failure.
> ======================================================================
> [sfilippo at localhost MPI_cat]$ mpirun -np 2 ./mpicat < configure 
> [sfilippo at localhost MPI_cat]$ mpirun -np 2 ./mpicat < configure 
> [proxy:0:0 at localhost.localdomain] HYD_pmcd_pmip_control_cmd_cb
> (../../../../mpich-3.2/src/pm/hydra/pm/pmiserv/pmip_cb.c:937): process
> reading stdin too slowly; can't keep up
> [proxy:0:0 at localhost.localdomain] HYDT_dmxu_poll_wait_for_event
> (../../../../mpich-3.2/src/pm/hydra/tools/demux/demux_poll.c:76):
> callback returned error status
> [proxy:0:0 at localhost.localdomain] main (../../../../mpich-
> 3.2/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for
> event
> [sfilippo at localhost MPI_cat]$ 
> =======================================================================
> 
> Am I missing something? 
> Is this supposed to work? I could find nothing relevant in the FAQ
> 
> Thanks a lot 
> Salvatore
> -- 
> Salvatore Filippone, PhD
> Centre for Computational Engineering Sciences,
> School of Aerospace, Transport and Manufacturing, Bldg 52
> Cranfield University
> Cranfield
> MK43 0AL
> United Kingdom
> tel: +44(0) 1234 758361
> 
> <mpicat.c>_______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss