<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Marvin,
<div class=""><br class="">
</div>
<div class="">currently, this is a limitation inside MPICH.</div>
<div class="">For every new window, MPICH internally creates a new communicator - for every communicator a new (unique) context ID is required -</div>
<div class="">and the number of different context IDs is limited to 2048.</div>
<div class=""><br class="">
</div>
<div class="">This context-id/ communicator is required for internal synchronization (e.g. barriers) . We have to ensure, that his communication is </div>
<div class="">not interfering with other communication on other windows or communicators.</div>
<div class=""><br class="">
</div>
<div class="">If you use more communicators, you should run into this problem earlier, because you already create other communicators </div>
<div class="">(the limit is per process) </div>
<div class=""><br class="">
</div>
<div class="">Lena </div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 19, 2016, at 12:49 PM, Marvin Smith <<a href="mailto:Marvin.Smith@sncorp.com" class="">Marvin.Smith@sncorp.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><font size="2" face="sans-serif" class="">Good morning, </font><br class="">
<br class="">
<font size="2" face="sans-serif" class=""> I wanted to present an issue I am having with MPICH and validate whether this is a configuration problem, a limitation with MPICH, or a bug.
</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">I am writing an application which uses a large number of MPI windows, each window is given a relatively large amount of memory. This has never been a problem before, however we discovered if you allocate more than
2045 windows, you get an exception thrown. </font> <br class="">
<br class="">
<font size="2" face="sans-serif" class="">Notes:</font>
<ul class="">
<li class=""><font size="2" face="sans-serif" class="">I am compiling using g++, version 4.8.5 on Red Hat Enterprise Linux version 7.2. </font>
</li><li class=""><font size="2" face="sans-serif" class="">My MPICH version is listed at the bottom of this email. It was installed via yum and is the RHEL default.</font>
</li><li class=""><font size="2" face="sans-serif" class="">I have attached sample output, to include the stdout/stderr. Also included is a Makefile and a simple example.</font>
</li><li class=""><font size="2" face="sans-serif" class="">The boundary of failure is between 2045 and 2046 windows.</font>
</li><li class=""><font size="2" face="sans-serif" class="">I have verified on my system this problem repeats even if I distribute windows between multiple communicators.</font>
</li><li class=""><font size="2" face="sans-serif" class="">I have not tested yet against ompi or mvapich. </font></li></ul>
<br class="">
<br class="">
<font size="2" face="sans-serif" class="">#-----------------------------------------------------------------------------------------------#</font>
<br class="">
<font size="2" face="sans-serif" class="">#- Here is my output -#</font>
<br class="">
<font size="2" face="sans-serif" class="">#-----------------------------------------------------------------------------------------------#</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class="">mpirun -np 2 -hosts localhost ./mpi-win-test 2046 1</font>
<br class="">
<font size="2" face="sans-serif" class="">Initialized Rank: 0, Number Processors: 2, Hostname: test-machine</font>
<br class="">
<font size="2" face="sans-serif" class="">Initialized Rank: 1, Number Processors: 2, Hostname: test-machine</font>
<br class="">
<font size="2" face="sans-serif" class="">Fatal error in MPI_Win_create_dynamic: Other MPI error, error stack:</font>
<br class="">
<font size="2" face="sans-serif" class="">MPI_Win_create_dynamic(154)..........: MPI_Win_create_dynamic(MPI_INFO_NULL, MPI_COMM_WORLD, win=0x10c1464) failed</font>
<br class="">
<font size="2" face="sans-serif" class="">MPID_Win_create_dynamic(139).........: </font>
<br class="">
<font size="2" face="sans-serif" class="">win_init(254)........................: </font>
<br class="">
<font size="2" face="sans-serif" class="">MPIR_Comm_dup_impl(55)...............: </font>
<br class="">
<font size="2" face="sans-serif" class="">MPIR_Comm_copy(1552).................: </font>
<br class="">
<font size="2" face="sans-serif" class="">MPIR_Get_contextid(799)..............: </font>
<br class="">
<font size="2" face="sans-serif" class="">MPIR_Get_contextid_sparse_group(1146): Cannot allocate context ID because of fragmentation (0/2048 free on this process; ignore_id=0)</font>
<br class="">
<font size="2" face="sans-serif" class="">Fatal error in MPI_Win_create_dynamic: Other MPI error, error stack:</font>
<br class="">
<font size="2" face="sans-serif" class="">MPI_Win_create_dynamic(154)..........: MPI_Win_create_dynamic(MPI_INFO_NULL, MPI_COMM_WORLD, win=0x19ef444) failed</font>
<br class="">
<font size="2" face="sans-serif" class="">MPID_Win_create_dynamic(139).........: </font>
<br class="">
<font size="2" face="sans-serif" class="">win_init(254)........................: </font>
<br class="">
<font size="2" face="sans-serif" class="">MPIR_Comm_dup_impl(55)...............: </font>
<br class="">
<font size="2" face="sans-serif" class="">MPIR_Comm_copy(1552).................: </font>
<br class="">
<font size="2" face="sans-serif" class="">MPIR_Get_contextid(799)..............: </font>
<br class="">
<font size="2" face="sans-serif" class="">MPIR_Get_contextid_sparse_group(1146): Cannot allocate context ID because of fragmentation (0/2048 free on this process; ignore_id=0)</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class="">===================================================================================</font>
<br class="">
<font size="2" face="sans-serif" class="">= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</font>
<br class="">
<font size="2" face="sans-serif" class="">= EXIT CODE: 1</font> <br class="">
<font size="2" face="sans-serif" class="">= CLEANING UP REMAINING PROCESSES</font>
<br class="">
<font size="2" face="sans-serif" class="">= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</font>
<br class="">
<font size="2" face="sans-serif" class="">===================================================================================</font>
<br class="">
<font size="2" face="sans-serif" class="">make: *** [run] Error 1</font> <br class="">
<br class="">
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""><b class="">#------------------------------------------------------------------------------------------#</b></font>
<br class="">
<font size="2" face="sans-serif" class=""><b class="">#- Here is my Sample Makefile -#</b></font>
<br class="">
<font size="2" face="sans-serif" class=""><b class="">#------------------------------------------------------------------------------------------#</b></font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""># Path to mpich on RHEL7</font> <br class="">
<font size="2" face="sans-serif" class="">MPI_INCL=-I/usr/include/mpich-x86_64</font>
<br class="">
<font size="2" face="sans-serif" class="">MPI_LIBS=-L/usr/lib64/mpich/lib -lmpich</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""># C++11 Bindings (Being lazy with string)</font>
<br class="">
<font size="2" face="sans-serif" class="">CXX_ARGS=-std=c++11</font> <br class="">
<br class="">
<font size="2" face="sans-serif" class=""># Make th test</font> <br class="">
<font size="2" face="sans-serif" class="">all: mpi-win-test</font> <br class="">
<br class="">
<font size="2" face="sans-serif" class="">mpi-win-test: mpi-win-test.cpp</font> <br class="">
<font size="2" face="sans-serif" class=""> g++ $< -o $@ $(MPI_INCL) $(MPI_LIBS) $(CXX_ARGS)</font>
<br class="">
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""># Args for application</font> <br class="">
<font size="2" face="sans-serif" class="">NUM_WINDOWS=2046</font> <br class="">
<font size="2" face="sans-serif" class="">USE_DYNAMIC=1</font> <br class="">
<br class="">
<font size="2" face="sans-serif" class=""># Sample run usage</font> <br class="">
<font size="2" face="sans-serif" class="">#</font> <br class="">
<font size="2" face="sans-serif" class=""># Args:</font> <br class="">
<font size="2" face="sans-serif" class=""># - Number of Windows</font> <br class="">
<font size="2" face="sans-serif" class=""># - Type of windows (1 dynamic, 0 static)</font>
<br class="">
<font size="2" face="sans-serif" class="">run:</font> <br class="">
<font size="2" face="sans-serif" class=""> mpirun -np 2 -hosts localhost ./mpi-win-test $(NUM_WINDOWS) $(USE_DYNAMIC)</font>
<br class="">
<br class="">
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""><b class="">#------------------------------------------------------------------------------------------#</b></font>
<br class="">
<font size="2" face="sans-serif" class=""><b class="">#- Here is my Sample Application -#</b></font>
<br class="">
<font size="2" face="sans-serif" class=""><b class="">#------------------------------------------------------------------------------------------#</b></font>
<br class="">
<br class="">
<br class="">
<font size="2" face="sans-serif" class="">#include <mpi.h></font> <br class="">
<br class="">
<font size="2" face="sans-serif" class="">#include <iostream></font> <br class="">
<font size="2" face="sans-serif" class="">#include <string></font> <br class="">
<font size="2" face="sans-serif" class="">#include <vector></font> <br class="">
<br class="">
<font size="2" face="sans-serif" class="">using namespace std;</font> <br class="">
<br class="">
<font size="2" face="sans-serif" class="">int main( int argc, char* argv[] )</font>
<br class="">
<font size="2" face="sans-serif" class="">{</font> <br class="">
<font size="2" face="sans-serif" class=""> // Number of MPI Windows</font> <br class="">
<font size="2" face="sans-serif" class=""> int num_windows = std::stoi(argv[1]);</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""> bool use_dynamic = std::stoi(argv[2]);</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""> // Initialize MPI</font> <br class="">
<font size="2" face="sans-serif" class=""> MPI_Init( &argc, &argv );</font> <br class="">
<br class="">
<font size="2" face="sans-serif" class=""> // Get the rank and size</font> <br class="">
<font size="2" face="sans-serif" class=""> int rank, nprocs;</font> <br class="">
<font size="2" face="sans-serif" class=""> MPI_Comm_size( MPI_COMM_WORLD, &nprocs );</font>
<br class="">
<font size="2" face="sans-serif" class=""> MPI_Comm_rank( MPI_COMM_WORLD, &rank );</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""> // Get the processor name</font> <br class="">
<font size="2" face="sans-serif" class=""> char hostname[MPI_MAX_PROCESSOR_NAME];</font>
<br class="">
<font size="2" face="sans-serif" class=""> int hostname_len;</font> <br class="">
<font size="2" face="sans-serif" class=""> MPI_Get_processor_name( hostname, &hostname_len);</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""> // Print Message</font> <br class="">
<font size="2" face="sans-serif" class=""> for( int i=0; i<nprocs; i++ ){</font>
<br class="">
<font size="2" face="sans-serif" class=""> MPI_Barrier(MPI_COMM_WORLD);</font>
<br class="">
<font size="2" face="sans-serif" class=""> if( i == rank ){</font> <br class="">
<font size="2" face="sans-serif" class=""> std::cout << "Initialized Rank: " << rank << ", Number Processors: " << nprocs << ", Hostname: " << hostname << std::endl;</font>
<br class="">
<font size="2" face="sans-serif" class=""> }</font> <br class="">
<font size="2" face="sans-serif" class=""> }</font> <br class="">
<font size="2" face="sans-serif" class=""> </font><br class="">
<font size="2" face="sans-serif" class=""> </font><br class="">
<font size="2" face="sans-serif" class=""> // MPI Variables</font> <br class="">
<font size="2" face="sans-serif" class=""> vector<MPI_Aint> sdisp_remotes(num_windows);</font>
<br class="">
<font size="2" face="sans-serif" class=""> vector<MPI_Aint> sdisp_locals(num_windows);</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""> // Create MPI Windows</font> <br class="">
<font size="2" face="sans-serif" class=""> vector<MPI_Win> windows(num_windows);</font>
<br class="">
<font size="2" face="sans-serif" class=""> </font><br class="">
<font size="2" face="sans-serif" class=""> int64_t buffer_size = 1000;</font> <br class="">
<font size="2" face="sans-serif" class=""> char* buffer = new char[buffer_size];</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""> for( int i=0; i<num_windows; i++ )</font>
<br class="">
<font size="2" face="sans-serif" class=""> { </font><br class="">
<font size="2" face="sans-serif" class=""> if( use_dynamic )</font> <br class="">
<font size="2" face="sans-serif" class=""> {</font> <br class="">
<font size="2" face="sans-serif" class=""> MPI_Win_create_dynamic( MPI_INFO_NULL, MPI_COMM_WORLD, &windows[i] );</font>
<br class="">
<font size="2" face="sans-serif" class=""> }</font> <br class="">
<br class="">
<font size="2" face="sans-serif" class=""> else</font> <br class="">
<font size="2" face="sans-serif" class=""> {</font> <br class="">
<font size="2" face="sans-serif" class=""> MPI_Win_create( &buffer,</font>
<br class="">
<font size="2" face="sans-serif" class=""> buffer_size,
</font><br class="">
<font size="2" face="sans-serif" class=""> 1, </font><br class="">
<font size="2" face="sans-serif" class=""> MPI_INFO_NULL,
</font><br class="">
<font size="2" face="sans-serif" class=""> MPI_COMM_WORLD,</font>
<br class="">
<font size="2" face="sans-serif" class=""> &windows[i] );</font>
<br class="">
<font size="2" face="sans-serif" class=""> } </font> <br class="">
<font size="2" face="sans-serif" class=""> }</font> <br class="">
<font size="2" face="sans-serif" class=""> </font><br class="">
<font size="2" face="sans-serif" class=""> </font><br class="">
<font size="2" face="sans-serif" class=""> // Exception always occurs prior to reaching this point.</font>
<br class="">
<font size="2" face="sans-serif" class=""> </font><br class="">
<font size="2" face="sans-serif" class=""> </font><br class="">
<font size="2" face="sans-serif" class=""> // More Code Here that I am removing for brevity</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""> // Wait at the barrier</font> <br class="">
<font size="2" face="sans-serif" class=""> MPI_Barrier( MPI_COMM_WORLD );</font>
<br class="">
<br class="">
<font size="2" face="sans-serif" class=""> // Remove all windows</font> <br class="">
<font size="2" face="sans-serif" class=""> for( int i=0; i<num_windows; i++)</font>
<br class="">
<font size="2" face="sans-serif" class=""> {</font> <br class="">
<font size="2" face="sans-serif" class=""> // Destroy the MPI Window</font>
<br class="">
<font size="2" face="sans-serif" class=""> MPI_Win_free( &windows[i] );</font>
<br class="">
<font size="2" face="sans-serif" class=""> }</font> <br class="">
<font size="2" face="sans-serif" class=""> windows.clear();</font> <br class="">
<br class="">
<font size="2" face="sans-serif" class=""> // Clear buffer </font><br class="">
<font size="2" face="sans-serif" class=""> delete [] buffer;</font> <br class="">
<font size="2" face="sans-serif" class=""> buffer = nullptr;</font> <br class="">
<font size="2" face="sans-serif" class=""> </font><br class="">
<font size="2" face="sans-serif" class=""> // Close MPI</font> <br class="">
<font size="2" face="sans-serif" class=""> MPI_Finalize();</font> <br class="">
<br class="">
<font size="2" face="sans-serif" class=""> return 0;</font> <br class="">
<font size="2" face="sans-serif" class="">}</font> <br class="">
<br class="">
<font size="2" face="sans-serif" class=""><b class="">#--------------------------------------------------------------------------#</b></font>
<br class="">
<font size="2" face="sans-serif" class=""><b class="">#- MPICH Version Output -#</b></font>
<br class="">
<font size="2" face="sans-serif" class=""><b class="">#--------------------------------------------------------------------------#</b></font>
<br class="">
<font size="2" face="sans-serif" class="">MPICH Version: 3.0.4</font> <br class="">
<font size="2" face="sans-serif" class="">MPICH Release date: Wed Apr 24 10:08:10 CDT 2013</font>
<br class="">
<font size="2" face="sans-serif" class="">MPICH Device: ch3:nemesis</font>
<br class="">
<font size="2" face="sans-serif" class="">MPICH configure: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
--datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-sharedlibs=gcc --enable-shared --enable-lib-depend --disable-rpath
--enable-fc --with-device=ch3:nemesis --with-pm=hydra:gforker --sysconfdir=/etc/mpich-x86_64 --includedir=/usr/include/mpich-x86_64 --bindir=/usr/lib64/mpich/bin --libdir=/usr/lib64/mpich/lib --datadir=/usr/share/mpich --mandir=/usr/share/man/mpich --docdir=/usr/share/mpich/doc
--htmldir=/usr/share/mpich/doc --with-hwloc-prefix=system FC=gfortran F77=gfortran CFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC CXXFLAGS=-m64
-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC FCFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic -fPIC FFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC LDFLAGS=-Wl,-z,noexecstack MPICH2LIB_CFLAGS=-O2
-g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic MPICH2LIB_CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic MPICH2LIB_FCFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic MPICH2LIB_FFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic</font>
<br class="">
<font size="2" face="sans-serif" class="">MPICH CC: cc -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -O2</font>
<br class="">
<font size="2" face="sans-serif" class="">MPICH CXX: c++ -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -O2</font>
<br class="">
<font size="2" face="sans-serif" class="">MPICH F77: gfortran -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -O2</font>
<br class="">
<font size="2" face="sans-serif" class="">MPICH FC: gfortran -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -O2</font>
<br class="">
<div class="">CONFIDENTIALITY NOTICE - SNC EMAIL: This email and any attachments are confidential, may contain proprietary, protected, or export controlled information, and are intended for the use of the intended recipients only. Any review, reliance, distribution,
disclosure, or forwarding of this email and/or attachments outside of Sierra Nevada Corporation (SNC) without express written approval of the sender, except to the extent required to further properly approved SNC business purposes, is strictly prohibited.
If you are not the intended recipient of this email, please notify the sender immediately, and delete all copies without reading, printing, or saving in any manner. --- Thank You.<br class="">
</div>
_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" class="">https://lists.mpich.org/mailman/listinfo/discuss</a></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>