<html dir="ltr"><head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Dear mpich discussion group,<br>
<br>
the following issue appeared when running some benchmarks with MPI Gather:<br>
<br>
Gathering data (calling MPI_Gather(...) ) involing 2048 processes and 2 MB of data (4096 MB total) that each process sends fails with the following output:<br>
____________________________<br>
<br>
Rank 1024 [Thu Nov 26 09:43:16 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack:<br>
PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=524288, MPI_INT, rbuf=(nil), rcount=524288, MPI_INT, root=0, MPI_COMM_WORLD) failed<br>
MPIR_Gather_impl(775).: <br>
MPIR_Gather(735)......: <br>
MPIR_Gather_intra(347): <br>
MPIC_Send(360)........: Negative count, value is -2147483648<br>
_pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:43:16 2015] PE RANK 1024 exit signal Aborted<br>
[NID 00307] 2015-11-26 09:43:16 Apid 949450: initiated application termination<br>
Application 949450 exit codes: 134<br>
Application 949450 exit signals: Killed<br>
Application 949450 resources: utime ~1s, stime ~137s, Rss ~2110448, inblocks ~617782, outblocks ~1659320<br>
____________________________<br>
<br>
The following are some tests that I ran to better understand the problem:<br>
<br>
2047 processes - 2 MB (4094 MB total) -> works!<br>
<br>
2048 processes - 2047.5 KB (~1.999512 MB) (4095 MB total) -> works!<br>
<br>
2048 processes - 3 MB (6144 MB total) -> fails:<br>
____________________________<br>
<br>
Rank 1024 [Thu Nov 26 09:41:15 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack:<br>
PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=786432, MPI_INT, rbuf=(nil), rcount=786432, MPI_INT, root=0, MPI_COMM_WORLD) failed<br>
MPIR_Gather_impl(775).: <br>
MPIR_Gather(735)......: <br>
MPIR_Gather_intra(347): <br>
MPIC_Send(360)........: Negative count, value is -1073741824<br>
_pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:41:15 2015] PE RANK 1024 exit signal Aborted<br>
[NID 00307] 2015-11-26 09:41:15 Apid 949448: initiated application termination<br>
Application 949448 exit codes: 134<br>
Application 949448 exit signals: Killed<br>
Application 949448 resources: utime ~1s, stime ~139s, Rss ~3159984, inblocks ~617782, outblocks ~1659351<br>
____________________________<br>
<br>
2047 processes - 3 MB (6141 MB total) -> fails:<br>
____________________________<br>
<br>
Rank 1024 [Thu Nov 26 09:40:31 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack:<br>
PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=786432, MPI_INT, rbuf=(nil), rcount=786432, MPI_INT, root=0, MPI_COMM_WORLD) failed<br>
MPIR_Gather_impl(775).: <br>
MPIR_Gather(735)......: <br>
MPIR_Gather_intra(347): <br>
MPIC_Send(360)........: Negative count, value is -1076887552<br>
_pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:40:32 2015] PE RANK 1024 exit signal Aborted<br>
[NID 00307] 2015-11-26 09:40:32 Apid 949446: initiated application termination<br>
Application 949446 exit codes: 134<br>
Application 949446 exit signals: Killed<br>
Application 949446 resources: utime ~1s, stime ~134s, Rss ~3157072, inblocks ~617780, outblocks ~1659351<br>
____________________________<br>
<br>
8 processes - 625 MB (5000 MB total) -> works!<br>
<br>
I can think of some pitfalls that might cause this issue but I do not have the knowledge of the internally called routines to check them. Is someone familier with the implementation of MPI_Gather(...) and willing to help me?<br>
<br>
Best regards<br>
<br>
Florian<br>
<div><br>
<div style="font-family:Tahoma; font-size:13px">
<div style="margin:0px"><font size="2"><span style="font-size:11pt"><font face="Arial"><font color="dimgray"><b>Deutsches Zentrum für Luft-
</b></font><font color="#666666"><b>und Raumfah</b></font><font color="dimgray"><b>rt</b></font><font color="dimgray"> e. V. (DLR)</font></font></span></font></div>
<div style="margin:0px"><font face="Arial" size="3" color="#696969"><span style="font-size:11pt">German Aerospace Center</span></font></div>
<div style="margin:0px"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Arial,sans-serif" size="2" color="dimgray"><span style="font-size:8.5pt">Institute of</span></font></span></font><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Arial,sans-serif" size="2" color="dimgray"><span style="font-size:8.5pt"> Planetary
Research | Planetary Physics | Rutherfordstraße 2 | 12489 Berlin</span></font></span></font></div>
<div style="margin:0px"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Arial,sans-serif" size="2" color="dimgray"><span style="font-size:8.5pt"></span></font></span></font> </div>
<div style="margin:0px"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"></span></font></div>
<div style="margin:0px"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font color="dimgray">Florian Willich| Intern - Software Developer (Parallel Applications)</font></span></font></div>
<div style="margin:0px"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><a href="mailto:florian.willlich@dlr.de" target="_blank"><font face="Arial,sans-serif" size="2"><span style="font-size:8.5pt">florian.willlich@dlr.de</span></font></a></span></font></div>
<div style="margin:0px"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><a href="http://www.dlr.de/" target="_blank"><font face="Arial,sans-serif" size="2"><span style="font-size:8.5pt" lang="en-US">DLR.de</span></font></a></span></font></div>
</div>
</div>
</div>
</body>
</html>