<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">I believe that creating 2046 communicators will also fail, so this doesn't have anything to do with windows themselves, just their apparent need to create an associated window.<div><br></div><div>The limitation in MPICH to create more than ~2000 communicators is known.</div><div><br></div><div>Why do you need to create ~2000 dynamic windows? I can't think of a reasonable application that would do this.</div><div><br></div><div>Jeff</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 19, 2016 at 10:49 AM, Marvin Smith <span dir="ltr"><<a href="mailto:Marvin.Smith@sncorp.com" target="_blank">Marvin.Smith@sncorp.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="2" face="sans-serif">Good morning, </font>
<br>
<br><font size="2" face="sans-serif"> I wanted to present an
issue I am having with MPICH and validate whether this is a configuration
problem, a limitation with MPICH, or a bug. </font>
<br>
<br><font size="2" face="sans-serif">I am writing an application which uses
a large number of MPI windows, each window is given a relatively large
amount of memory. This has never been a problem before, however
we discovered if you allocate more than 2045 windows, you get an exception
thrown. </font>
<br>
<br><font size="2" face="sans-serif">Notes:</font>
<ul>
<li><font size="2" face="sans-serif">I am compiling using g++, version 4.8.5
on Red Hat Enterprise Linux version 7.2. </font>
</li><li><font size="2" face="sans-serif">My MPICH version is listed at the bottom
of this email. It was installed via yum and is the RHEL default.</font>
</li><li><font size="2" face="sans-serif">I have attached sample output, to include
the stdout/stderr. Also included is a Makefile and a simple example.</font>
</li><li><font size="2" face="sans-serif">The boundary of failure is between 2045
and 2046 windows.</font>
</li><li><font size="2" face="sans-serif">I have verified on my system this problem
repeats even if I distribute windows between multiple communicators.</font>
</li><li><font size="2" face="sans-serif">I have not tested yet against ompi or
mvapich. </font></li></ul>
<br>
<br><font size="2" face="sans-serif">#-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------#</font>
<br><font size="2" face="sans-serif">#-
Here is my output
-#</font>
<br><font size="2" face="sans-serif">#-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------#</font>
<br>
<br><font size="2" face="sans-serif">mpirun -np 2 -hosts localhost ./mpi-win-test
2046 1</font>
<br><font size="2" face="sans-serif">Initialized Rank: 0, Number Processors:
2, Hostname: test-machine</font>
<br><font size="2" face="sans-serif">Initialized Rank: 1, Number Processors:
2, Hostname: test-machine</font>
<br><font size="2" face="sans-serif">Fatal error in MPI_Win_create_dynamic:
Other MPI error, error stack:</font>
<br><font size="2" face="sans-serif">MPI_Win_create_dynamic(154)...<wbr>.......:
MPI_Win_create_dynamic(MPI_<wbr>INFO_NULL, MPI_COMM_WORLD, win=0x10c1464) failed</font>
<br><font size="2" face="sans-serif">MPID_Win_create_dynamic(139)..<wbr>.......:
</font>
<br><font size="2" face="sans-serif">win_init(254).................<wbr>.......:
</font>
<br><font size="2" face="sans-serif">MPIR_Comm_dup_impl(55)........<wbr>.......:
</font>
<br><font size="2" face="sans-serif">MPIR_Comm_copy(1552)..........<wbr>.......:
</font>
<br><font size="2" face="sans-serif">MPIR_Get_contextid(799).......<wbr>.......:
</font>
<br><font size="2" face="sans-serif">MPIR_Get_contextid_sparse_<wbr>group(1146):
Cannot allocate context ID because of fragmentation (0/2048 free
on this process; ignore_id=0)</font>
<br><font size="2" face="sans-serif">Fatal error in MPI_Win_create_dynamic:
Other MPI error, error stack:</font>
<br><font size="2" face="sans-serif">MPI_Win_create_dynamic(154)...<wbr>.......:
MPI_Win_create_dynamic(MPI_<wbr>INFO_NULL, MPI_COMM_WORLD, win=0x19ef444) failed</font>
<br><font size="2" face="sans-serif">MPID_Win_create_dynamic(139)..<wbr>.......:
</font>
<br><font size="2" face="sans-serif">win_init(254).................<wbr>.......:
</font>
<br><font size="2" face="sans-serif">MPIR_Comm_dup_impl(55)........<wbr>.......:
</font>
<br><font size="2" face="sans-serif">MPIR_Comm_copy(1552)..........<wbr>.......:
</font>
<br><font size="2" face="sans-serif">MPIR_Get_contextid(799).......<wbr>.......:
</font>
<br><font size="2" face="sans-serif">MPIR_Get_contextid_sparse_<wbr>group(1146):
Cannot allocate context ID because of fragmentation (0/2048 free
on this process; ignore_id=0)</font>
<br>
<br><font size="2" face="sans-serif">==============================<wbr>==============================<wbr>=======================</font>
<br><font size="2" face="sans-serif">= BAD TERMINATION OF ONE OF YOUR
APPLICATION PROCESSES</font>
<br><font size="2" face="sans-serif">= EXIT CODE: 1</font>
<br><font size="2" face="sans-serif">= CLEANING UP REMAINING PROCESSES</font>
<br><font size="2" face="sans-serif">= YOU CAN IGNORE THE BELOW CLEANUP
MESSAGES</font>
<br><font size="2" face="sans-serif">==============================<wbr>==============================<wbr>=======================</font>
<br><font size="2" face="sans-serif">make: *** [run] Error 1</font>
<br>
<br>
<br>
<br><font size="2" face="sans-serif"><b>#-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-#</b></font>
<br><font size="2" face="sans-serif"><b>#-
Here is my Sample Makefile
-#</b></font>
<br><font size="2" face="sans-serif"><b>#-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-#</b></font>
<br>
<br><font size="2" face="sans-serif"># Path to mpich on RHEL7</font>
<br><font size="2" face="sans-serif">MPI_INCL=-I/usr/include/mpich-<wbr>x86_64</font>
<br><font size="2" face="sans-serif">MPI_LIBS=-L/usr/lib64/mpich/<wbr>lib -lmpich</font>
<br>
<br><font size="2" face="sans-serif"># C++11 Bindings (Being lazy with
string)</font>
<br><font size="2" face="sans-serif">CXX_ARGS=-std=c++11</font>
<br>
<br><font size="2" face="sans-serif"># Make th test</font>
<br><font size="2" face="sans-serif">all: mpi-win-test</font>
<br>
<br><font size="2" face="sans-serif">mpi-win-test: mpi-win-test.cpp</font>
<br><font size="2" face="sans-serif"> g++
$< -o $@ $(MPI_INCL) $(MPI_LIBS) $(CXX_ARGS)</font>
<br>
<br>
<br><font size="2" face="sans-serif"># Args for application</font>
<br><font size="2" face="sans-serif">NUM_WINDOWS=2046</font>
<br><font size="2" face="sans-serif">USE_DYNAMIC=1</font>
<br>
<br><font size="2" face="sans-serif"># Sample run usage</font>
<br><font size="2" face="sans-serif">#</font>
<br><font size="2" face="sans-serif"># Args:</font>
<br><font size="2" face="sans-serif"># -
Number of Windows</font>
<br><font size="2" face="sans-serif"># - Type of windows (1 dynamic,
0 static)</font>
<br><font size="2" face="sans-serif">run:</font>
<br><font size="2" face="sans-serif"> mpirun
-np 2 -hosts localhost ./mpi-win-test $(NUM_WINDOWS) $(USE_DYNAMIC)</font>
<br>
<br>
<br>
<br><font size="2" face="sans-serif"><b>#-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-#</b></font>
<br><font size="2" face="sans-serif"><b>#-
Here is my Sample Application
-#</b></font>
<br><font size="2" face="sans-serif"><b>#-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-#</b></font>
<br>
<br>
<br><font size="2" face="sans-serif">#include <mpi.h></font>
<br>
<br><font size="2" face="sans-serif">#include <iostream></font>
<br><font size="2" face="sans-serif">#include <string></font>
<br><font size="2" face="sans-serif">#include <vector></font>
<br>
<br><font size="2" face="sans-serif">using namespace std;</font>
<br>
<br><font size="2" face="sans-serif">int main( int argc, char* argv[] )</font>
<br><font size="2" face="sans-serif">{</font>
<br><font size="2" face="sans-serif"> // Number of MPI Windows</font>
<br><font size="2" face="sans-serif"> int num_windows = std::stoi(argv[1]);</font>
<br>
<br><font size="2" face="sans-serif"> bool use_dynamic = std::stoi(argv[2]);</font>
<br>
<br><font size="2" face="sans-serif"> // Initialize MPI</font>
<br><font size="2" face="sans-serif"> MPI_Init( &argc, &argv
);</font>
<br>
<br><font size="2" face="sans-serif"> // Get the rank and size</font>
<br><font size="2" face="sans-serif"> int rank, nprocs;</font>
<br><font size="2" face="sans-serif"> MPI_Comm_size( MPI_COMM_WORLD,
&nprocs );</font>
<br><font size="2" face="sans-serif"> MPI_Comm_rank( MPI_COMM_WORLD,
&rank );</font>
<br>
<br><font size="2" face="sans-serif"> // Get the processor name</font>
<br><font size="2" face="sans-serif"> char hostname[MPI_MAX_PROCESSOR_<wbr>NAME];</font>
<br><font size="2" face="sans-serif"> int hostname_len;</font>
<br><font size="2" face="sans-serif"> MPI_Get_processor_name(
hostname, &hostname_len);</font>
<br>
<br><font size="2" face="sans-serif"> // Print Message</font>
<br><font size="2" face="sans-serif"> for( int i=0; i<nprocs;
i++ ){</font>
<br><font size="2" face="sans-serif"> MPI_Barrier(MPI_COMM_WORLD);</font>
<br><font size="2" face="sans-serif"> if( i ==
rank ){</font>
<br><font size="2" face="sans-serif">
std::cout << "Initialized Rank: " << rank <<
", Number Processors: " << nprocs << ", Hostname:
" << hostname << std::endl;</font>
<br><font size="2" face="sans-serif"> }</font>
<br><font size="2" face="sans-serif"> }</font>
<br><font size="2" face="sans-serif"> </font>
<br><font size="2" face="sans-serif"> </font>
<br><font size="2" face="sans-serif"> // MPI Variables</font>
<br><font size="2" face="sans-serif"> vector<MPI_Aint>
sdisp_remotes(num_windows);</font>
<br><font size="2" face="sans-serif"> vector<MPI_Aint>
sdisp_locals(num_windows);</font>
<br>
<br><font size="2" face="sans-serif"> // Create MPI Windows</font>
<br><font size="2" face="sans-serif"> vector<MPI_Win>
windows(num_windows);</font>
<br><font size="2" face="sans-serif">
</font>
<br><font size="2" face="sans-serif"> int64_t buffer_size =
1000;</font>
<br><font size="2" face="sans-serif"> char* buffer =
new char[buffer_size];</font>
<br>
<br><font size="2" face="sans-serif"> for( int i=0; i<num_windows;
i++ )</font>
<br><font size="2" face="sans-serif"> { </font>
<br><font size="2" face="sans-serif"> if( use_dynamic
)</font>
<br><font size="2" face="sans-serif"> {</font>
<br><font size="2" face="sans-serif">
MPI_Win_create_dynamic( MPI_INFO_NULL, MPI_COMM_WORLD, &windows[i]
);</font>
<br><font size="2" face="sans-serif"> }</font>
<br>
<br><font size="2" face="sans-serif"> else</font>
<br><font size="2" face="sans-serif"> {</font>
<br><font size="2" face="sans-serif">
MPI_Win_create( &buffer,</font>
<br><font size="2" face="sans-serif">
buffer_size, </font>
<br><font size="2" face="sans-serif">
1, </font>
<br><font size="2" face="sans-serif">
MPI_INFO_NULL,
</font>
<br><font size="2" face="sans-serif">
MPI_COMM_WORLD,</font>
<br><font size="2" face="sans-serif">
&windows[i]
);</font>
<br><font size="2" face="sans-serif"> }
</font>
<br><font size="2" face="sans-serif"> }</font>
<br><font size="2" face="sans-serif"> </font>
<br><font size="2" face="sans-serif"> </font>
<br><font size="2" face="sans-serif"> // Exception always occurs
prior to reaching this point.</font>
<br><font size="2" face="sans-serif"> </font>
<br><font size="2" face="sans-serif"> </font>
<br><font size="2" face="sans-serif"> // More Code Here that
I am removing for brevity</font>
<br>
<br><font size="2" face="sans-serif"> // Wait at the barrier</font>
<br><font size="2" face="sans-serif"> MPI_Barrier( MPI_COMM_WORLD
);</font>
<br>
<br><font size="2" face="sans-serif"> // Remove all windows</font>
<br><font size="2" face="sans-serif"> for( int i=0; i<num_windows;
i++)</font>
<br><font size="2" face="sans-serif"> {</font>
<br><font size="2" face="sans-serif"> // Destroy
the MPI Window</font>
<br><font size="2" face="sans-serif"> MPI_Win_free(
&windows[i] );</font>
<br><font size="2" face="sans-serif"> }</font>
<br><font size="2" face="sans-serif"> windows.clear();</font>
<br>
<br><font size="2" face="sans-serif"> // Clear buffer </font>
<br><font size="2" face="sans-serif"> delete [] buffer;</font>
<br><font size="2" face="sans-serif"> buffer = nullptr;</font>
<br><font size="2" face="sans-serif"> </font>
<br><font size="2" face="sans-serif"> // Close MPI</font>
<br><font size="2" face="sans-serif"> MPI_Finalize();</font>
<br>
<br><font size="2" face="sans-serif"> return 0;</font>
<br><font size="2" face="sans-serif">}</font>
<br>
<br><font size="2" face="sans-serif"><b>#-----------------------------<wbr>------------------------------<wbr>---------------#</b></font>
<br><font size="2" face="sans-serif"><b>#-
MPICH Version Output
-#</b></font>
<br><font size="2" face="sans-serif"><b>#-----------------------------<wbr>------------------------------<wbr>---------------#</b></font>
<br><font size="2" face="sans-serif">MPICH Version:
3.0.4</font>
<br><font size="2" face="sans-serif">MPICH Release date:
Wed Apr 24 10:08:10 CDT 2013</font>
<br><font size="2" face="sans-serif">MPICH Device:
ch3:nemesis</font>
<br><font size="2" face="sans-serif">MPICH configure:
--build=x86_64-redhat-linux-<wbr>gnu --host=x86_64-redhat-linux-gnu
--program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
--bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share
--includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec
--localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man
--infodir=/usr/share/info --enable-sharedlibs=gcc --enable-shared --enable-lib-depend
--disable-rpath --enable-fc --with-device=ch3:nemesis --with-pm=hydra:gforker
--sysconfdir=/etc/mpich-x86_64 --includedir=/usr/include/<wbr>mpich-x86_64 --bindir=/usr/lib64/mpich/bin
--libdir=/usr/lib64/mpich/lib --datadir=/usr/share/mpich --mandir=/usr/share/man/mpich
--docdir=/usr/share/mpich/doc --htmldir=/usr/share/mpich/doc --with-hwloc-prefix=system
FC=gfortran F77=gfortran CFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic -fPIC CXXFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic -fPIC FCFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic -fPIC FFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic -fPIC LDFLAGS=-Wl,-z,noexecstack MPICH2LIB_CFLAGS=-O2
-g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic MPICH2LIB_CXXFLAGS=-O2
-g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic MPICH2LIB_FCFLAGS=-O2
-g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic MPICH2LIB_FFLAGS=-O2
-g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic</font>
<br><font size="2" face="sans-serif">MPICH CC:
cc -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic -fPIC -O2</font>
<br><font size="2" face="sans-serif">MPICH CXX:
c++ -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic -fPIC -O2</font>
<br><font size="2" face="sans-serif">MPICH F77:
gfortran -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic -fPIC -O2</font>
<br><font size="2" face="sans-serif">MPICH FC:
gfortran -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic -fPIC -O2</font>
<br>
<div>
CONFIDENTIALITY NOTICE - SNC EMAIL: This email and any attachments are confidential, may contain proprietary, protected, or export controlled information, and are intended for the use of the intended recipients only. Any review, reliance, distribution, disclosure, or forwarding of this email and/or attachments outside of Sierra Nevada Corporation (SNC) without express written approval of the sender, except to the extent required to further properly approved SNC business purposes, is strictly prohibited. If you are not the intended recipient of this email, please notify the sender immediately, and delete all copies without reading, printing, or saving in any manner. --- Thank You.<br>
</div>
<br>______________________________<wbr>_________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/<wbr>mailman/listinfo/discuss</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div>