<meta http-equiv="Content-Type" content="text/html; charset=us-ascii"><font size="2" face="sans-serif">Thanks for the quick reply. Do you
see this value increasing in the future? In the meantime, I have
a solution with less windows. </font>
<br>
<br><font size="2" face="sans-serif">Thanks,</font>
<br><font size="2" face="sans-serif">Marvin</font>
<br>
<br>
<br>
<br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">"Oden, Lena"
<loden@anl.gov></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">"discuss@mpich.org"
<discuss@mpich.org></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">09/19/2016 11:11 AM</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: [mpich-discuss]
MPI Window limit on mpich</font>
<br>
<hr noshade="">
<br>
<br>
<br><font size="3">Hi Marvin, </font>
<br>
<br><font size="3">currently, this is a limitation inside MPICH.</font>
<br><font size="3">For every new window, MPICH internally creates a new communicator
- for every communicator a new (unique) context ID is required -</font>
<br><font size="3">and the number of different context IDs is limited to
2048.</font>
<br>
<br><font size="3">This context-id/ communicator is required for internal
synchronization (e.g. barriers) . We have to ensure, that his communication
is </font>
<br><font size="3">not interfering with other communication on other windows
or communicators.</font>
<br>
<br><font size="3">If you use more communicators, you should run into this
problem earlier, because you already create other communicators </font>
<br><font size="3">(the limit is per process) </font>
<br>
<br><font size="3">Lena </font>
<br>
<br><font size="3">On Sep 19, 2016, at 12:49 PM, Marvin Smith <</font><a href="mailto:Marvin.Smith@sncorp.com"><font size="3" color="blue"><u>Marvin.Smith@sncorp.com</u></font></a><font size="3">>
wrote:</font>
<br>
<br><font size="2" face="sans-serif">Good morning, </font><font size="3"><br>
</font><font size="2" face="sans-serif"><br>
I wanted to present an issue I am having with MPICH and validate
whether this is a configuration problem, a limitation with MPICH, or a
bug. </font><font size="3"><br>
</font><font size="2" face="sans-serif"><br>
I am writing an application which uses a large number of MPI windows, each
window is given a relatively large amount of memory. This has never
been a problem before, however we discovered if you allocate more than
2045 windows, you get an exception thrown. </font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
Notes:</font><font size="3"> </font>
<ul>
<li><font size="2" face="sans-serif">I am compiling using g++, version 4.8.5
on Red Hat Enterprise Linux version 7.2. </font><font size="3">
</font>
<li><font size="2" face="sans-serif">My MPICH version is listed at the bottom
of this email. It was installed via yum and is the RHEL default.</font><font size="3">
</font>
<li><font size="2" face="sans-serif">I have attached sample output, to include
the stdout/stderr. Also included is a Makefile and a simple example.</font><font size="3">
</font>
<li><font size="2" face="sans-serif">The boundary of failure is between 2045
and 2046 windows.</font><font size="3"> </font>
<li><font size="2" face="sans-serif">I have verified on my system this problem
repeats even if I distribute windows between multiple communicators.</font><font size="3">
</font>
<li><font size="2" face="sans-serif">I have not tested yet against ompi or
mvapich. </font></ul><font size="3"><br>
</font><font size="2" face="sans-serif"><br>
#-----------------------------------------------------------------------------------------------#</font><font size="3">
</font><font size="2" face="sans-serif"><br>
#-
Here is my
output
-#</font><font size="3"> </font><font size="2" face="sans-serif"><br>
#-----------------------------------------------------------------------------------------------#</font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
mpirun -np 2 -hosts localhost ./mpi-win-test 2046 1</font><font size="3">
</font><font size="2" face="sans-serif"><br>
Initialized Rank: 0, Number Processors: 2, Hostname: test-machine</font><font size="3">
</font><font size="2" face="sans-serif"><br>
Initialized Rank: 1, Number Processors: 2, Hostname: test-machine</font><font size="3">
</font><font size="2" face="sans-serif"><br>
Fatal error in MPI_Win_create_dynamic: Other MPI error, error stack:</font><font size="3">
</font><font size="2" face="sans-serif"><br>
MPI_Win_create_dynamic(154)..........: MPI_Win_create_dynamic(MPI_INFO_NULL,
MPI_COMM_WORLD, win=0x10c1464) failed</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPID_Win_create_dynamic(139).........: <br>
win_init(254)........................: <br>
MPIR_Comm_dup_impl(55)...............: <br>
MPIR_Comm_copy(1552).................: <br>
MPIR_Get_contextid(799)..............: <br>
MPIR_Get_contextid_sparse_group(1146): Cannot allocate context ID
because of fragmentation (0/2048 free on this process; ignore_id=0)</font><font size="3">
</font><font size="2" face="sans-serif"><br>
Fatal error in MPI_Win_create_dynamic: Other MPI error, error stack:</font><font size="3">
</font><font size="2" face="sans-serif"><br>
MPI_Win_create_dynamic(154)..........: MPI_Win_create_dynamic(MPI_INFO_NULL,
MPI_COMM_WORLD, win=0x19ef444) failed</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPID_Win_create_dynamic(139).........: <br>
win_init(254)........................: <br>
MPIR_Comm_dup_impl(55)...............: <br>
MPIR_Comm_copy(1552).................: <br>
MPIR_Get_contextid(799)..............: <br>
MPIR_Get_contextid_sparse_group(1146): Cannot allocate context ID
because of fragmentation (0/2048 free on this process; ignore_id=0)</font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
===================================================================================</font><font size="3">
</font><font size="2" face="sans-serif"><br>
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</font><font size="3">
</font><font size="2" face="sans-serif"><br>
= EXIT CODE: 1</font><font size="3"> </font><font size="2" face="sans-serif"><br>
= CLEANING UP REMAINING PROCESSES</font><font size="3"> </font><font size="2" face="sans-serif"><br>
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</font><font size="3">
</font><font size="2" face="sans-serif"><br>
===================================================================================</font><font size="3">
</font><font size="2" face="sans-serif"><br>
make: *** [run] Error 1</font><font size="3"> <br>
<br>
<br>
</font><font size="2" face="sans-serif"><b><br>
#------------------------------------------------------------------------------------------#</b></font><font size="3">
</font><font size="2" face="sans-serif"><b><br>
#- Here
is my Sample Makefile
-#</b></font><font size="3">
</font><font size="2" face="sans-serif"><b><br>
#------------------------------------------------------------------------------------------#</b></font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
# Path to mpich on RHEL7</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPI_INCL=-I/usr/include/mpich-x86_64</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPI_LIBS=-L/usr/lib64/mpich/lib -lmpich</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
# C++11 Bindings (Being lazy with string)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
CXX_ARGS=-std=c++11</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
# Make th test</font><font size="3"> </font><font size="2" face="sans-serif"><br>
all: mpi-win-test</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
mpi-win-test: mpi-win-test.cpp</font><font size="3"> </font><font size="2" face="sans-serif"><br>
g++ $< -o $@ $(MPI_INCL) $(MPI_LIBS) $(CXX_ARGS)</font><font size="3">
<br>
<br>
</font><font size="2" face="sans-serif"><br>
# Args for application</font><font size="3"> </font><font size="2" face="sans-serif"><br>
NUM_WINDOWS=2046</font><font size="3"> </font><font size="2" face="sans-serif"><br>
USE_DYNAMIC=1</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
# Sample run usage</font><font size="3"> </font><font size="2" face="sans-serif"><br>
#</font><font size="3"> </font><font size="2" face="sans-serif"><br>
# Args:</font><font size="3"> </font><font size="2" face="sans-serif"><br>
# - Number of Windows</font><font size="3">
</font><font size="2" face="sans-serif"><br>
# - Type of windows (1 dynamic, 0 static)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
run:</font><font size="3"> </font><font size="2" face="sans-serif"><br>
mpirun -np 2 -hosts localhost ./mpi-win-test
$(NUM_WINDOWS) $(USE_DYNAMIC)</font><font size="3"> <br>
<br>
<br>
</font><font size="2" face="sans-serif"><b><br>
#------------------------------------------------------------------------------------------#</b></font><font size="3">
</font><font size="2" face="sans-serif"><b><br>
#- Here
is my Sample Application
-#</b></font><font size="3"> </font><font size="2" face="sans-serif"><b><br>
#------------------------------------------------------------------------------------------#</b></font><font size="3">
<br>
<br>
</font><font size="2" face="sans-serif"><br>
#include <mpi.h></font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
#include <iostream></font><font size="3"> </font><font size="2" face="sans-serif"><br>
#include <string></font><font size="3"> </font><font size="2" face="sans-serif"><br>
#include <vector></font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
using namespace std;</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
int main( int argc, char* argv[] )</font><font size="3"> </font><font size="2" face="sans-serif"><br>
{</font><font size="3"> </font><font size="2" face="sans-serif"><br>
// Number of MPI Windows</font><font size="3"> </font><font size="2" face="sans-serif"><br>
int num_windows = std::stoi(argv[1]);</font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
bool use_dynamic = std::stoi(argv[2]);</font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
// Initialize MPI</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPI_Init( &argc, &argv );</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
// Get the rank and size</font><font size="3"> </font><font size="2" face="sans-serif"><br>
int rank, nprocs;</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPI_Comm_size( MPI_COMM_WORLD, &nprocs );</font><font size="3">
</font><font size="2" face="sans-serif"><br>
MPI_Comm_rank( MPI_COMM_WORLD, &rank );</font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
// Get the processor name</font><font size="3"> </font><font size="2" face="sans-serif"><br>
char hostname[MPI_MAX_PROCESSOR_NAME];</font><font size="3">
</font><font size="2" face="sans-serif"><br>
int hostname_len;</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPI_Get_processor_name( hostname, &hostname_len);</font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
// Print Message</font><font size="3"> </font><font size="2" face="sans-serif"><br>
for( int i=0; i<nprocs; i++ ){</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPI_Barrier(MPI_COMM_WORLD);</font><font size="3">
</font><font size="2" face="sans-serif"><br>
if( i == rank ){</font><font size="3"> </font><font size="2" face="sans-serif"><br>
std::cout << "Initialized
Rank: " << rank << ", Number Processors: " <<
nprocs << ", Hostname: " << hostname << std::endl;</font><font size="3">
</font><font size="2" face="sans-serif"><br>
}</font><font size="3"> </font><font size="2" face="sans-serif"><br>
}</font><font size="3"> </font><font size="2" face="sans-serif"><br>
<br>
<br>
// MPI Variables</font><font size="3"> </font><font size="2" face="sans-serif"><br>
vector<MPI_Aint> sdisp_remotes(num_windows);</font><font size="3">
</font><font size="2" face="sans-serif"><br>
vector<MPI_Aint> sdisp_locals(num_windows);</font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
// Create MPI Windows</font><font size="3"> </font><font size="2" face="sans-serif"><br>
vector<MPI_Win> windows(num_windows);</font><font size="3">
</font><font size="2" face="sans-serif"><br>
<br>
int64_t buffer_size = 1000;</font><font size="3"> </font><font size="2" face="sans-serif"><br>
char* buffer = new char[buffer_size];</font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
for( int i=0; i<num_windows; i++ )</font><font size="3">
</font><font size="2" face="sans-serif"><br>
{ <br>
if( use_dynamic )</font><font size="3"> </font><font size="2" face="sans-serif"><br>
{</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPI_Win_create_dynamic( MPI_INFO_NULL,
MPI_COMM_WORLD, &windows[i] );</font><font size="3"> </font><font size="2" face="sans-serif"><br>
}</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
else</font><font size="3"> </font><font size="2" face="sans-serif"><br>
{</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPI_Win_create( &buffer,</font><font size="3">
</font><font size="2" face="sans-serif"><br>
buffer_size, <br>
1, <br>
MPI_INFO_NULL, <br>
MPI_COMM_WORLD,</font><font size="3"> </font><font size="2" face="sans-serif"><br>
&windows[i] );</font><font size="3"> </font><font size="2" face="sans-serif"><br>
} </font><font size="3"> </font><font size="2" face="sans-serif"><br>
}</font><font size="3"> </font><font size="2" face="sans-serif"><br>
<br>
<br>
// Exception always occurs prior to reaching this point.</font><font size="3">
</font><font size="2" face="sans-serif"><br>
<br>
<br>
// More Code Here that I am removing for brevity</font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
// Wait at the barrier</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPI_Barrier( MPI_COMM_WORLD );</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
// Remove all windows</font><font size="3"> </font><font size="2" face="sans-serif"><br>
for( int i=0; i<num_windows; i++)</font><font size="3">
</font><font size="2" face="sans-serif"><br>
{</font><font size="3"> </font><font size="2" face="sans-serif"><br>
// Destroy the MPI Window</font><font size="3">
</font><font size="2" face="sans-serif"><br>
MPI_Win_free( &windows[i] );</font><font size="3">
</font><font size="2" face="sans-serif"><br>
}</font><font size="3"> </font><font size="2" face="sans-serif"><br>
windows.clear();</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
// Clear buffer <br>
delete [] buffer;</font><font size="3"> </font><font size="2" face="sans-serif"><br>
buffer = nullptr;</font><font size="3"> </font><font size="2" face="sans-serif"><br>
<br>
// Close MPI</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPI_Finalize();</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
return 0;</font><font size="3"> </font><font size="2" face="sans-serif"><br>
}</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><b><br>
#--------------------------------------------------------------------------#</b></font><font size="3">
</font><font size="2" face="sans-serif"><b><br>
#- MPICH Version
Output -#</b></font><font size="3">
</font><font size="2" face="sans-serif"><b><br>
#--------------------------------------------------------------------------#</b></font><font size="3">
</font><font size="2" face="sans-serif"><br>
MPICH Version: 3.0.4</font><font size="3">
</font><font size="2" face="sans-serif"><br>
MPICH Release date: Wed Apr 24 10:08:10 CDT
2013</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPICH Device: ch3:nemesis</font><font size="3">
</font><font size="2" face="sans-serif"><br>
MPICH configure: --build=x86_64-redhat-linux-gnu
--host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking
--prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin
--sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
--libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib
--mandir=/usr/share/man --infodir=/usr/share/info --enable-sharedlibs=gcc
--enable-shared --enable-lib-depend --disable-rpath --enable-fc --with-device=ch3:nemesis
--with-pm=hydra:gforker --sysconfdir=/etc/mpich-x86_64 --includedir=/usr/include/mpich-x86_64
--bindir=/usr/lib64/mpich/bin --libdir=/usr/lib64/mpich/lib --datadir=/usr/share/mpich
--mandir=/usr/share/man/mpich --docdir=/usr/share/mpich/doc --htmldir=/usr/share/mpich/doc
--with-hwloc-prefix=system FC=gfortran F77=gfortran CFLAGS=-m64 -O2 -g
-pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC
CXXFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC
FCFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC
FFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC
LDFLAGS=-Wl,-z,noexecstack MPICH2LIB_CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic MPICH2LIB_CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic MPICH2LIB_FCFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic MPICH2LIB_FFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPICH CC: cc -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic -fPIC -O2</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPICH CXX: c++ -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
-m64 -mtune=generic -fPIC -O2</font><font size="3"> </font><font size="2" face="sans-serif"><br>
MPICH F77: gfortran -m64 -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic -fPIC -O2</font><font size="3">
</font><font size="2" face="sans-serif"><br>
MPICH FC: gfortran -m64 -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic -fPIC -O2</font><font size="3">
</font>
<br><font size="3">CONFIDENTIALITY NOTICE - SNC EMAIL: This email and any
attachments are confidential, may contain proprietary, protected, or export
controlled information, and are intended for the use of the intended recipients
only. Any review, reliance, distribution, disclosure, or forwarding of
this email and/or attachments outside of Sierra Nevada Corporation (SNC)
without express written approval of the sender, except to the extent required
to further properly approved SNC business purposes, is strictly prohibited.
If you are not the intended recipient of this email, please notify the
sender immediately, and delete all copies without reading, printing, or
saving in any manner. --- Thank You.</font>
<br><font size="3">_______________________________________________<br>
discuss mailing list </font><a href="mailto:discuss@mpich.org"><font size="3" color="blue"><u>discuss@mpich.org</u></font></a><font size="3"><br>
To manage subscription options or unsubscribe:</font><font size="3" color="blue"><u><br>
</u></font><a href="https://lists.mpich.org/mailman/listinfo/discuss"><font size="3" color="blue"><u>https://lists.mpich.org/mailman/listinfo/discuss</u></font></a>
<br><tt><font size="2">_______________________________________________<br>
discuss mailing list discuss@mpich.org<br>
To manage subscription options or unsubscribe:<br>
</font></tt><a href="https://lists.mpich.org/mailman/listinfo/discuss"><tt><font size="2">https://lists.mpich.org/mailman/listinfo/discuss</font></tt></a>
<br>
<div>
CONFIDENTIALITY NOTICE - SNC EMAIL: This email and any attachments are confidential, may contain proprietary, protected, or export controlled information, and are intended for the use of the intended recipients only. Any review, reliance, distribution, disclosure, or forwarding of this email and/or attachments outside of Sierra Nevada Corporation (SNC) without express written approval of the sender, except to the extent required to further properly approved SNC business purposes, is strictly prohibited. If you are not the intended recipient of this email, please notify the sender immediately, and delete all copies without reading, printing, or saving in any manner. --- Thank You.<br>
</div>