<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">Hi Martin,<div>Thanks for the response.</div><div>Unfortunately, I don't have direct access to the Dockerfile used to create this image, but I have at times built my own when necessary.  Here is an example:</div><div><a href="https://github.com/LSSTDESC/dockerfiles/blob/master/lsst_sims/Dockerfile-buildAllFromSource" target="_blank">https://github.com/LSSTDESC/dockerfiles/blob/master/lsst_sims/Dockerfile-buildAllFromSource</a><br></div><div>There's a bit going on there with the newinstall.sh script that includes installing miniconda plus a bunch of python packages including mpich 3.2.1, then downloading and building some C/C++ code.</div><div>In my search for clues, I see you are absolutely right, there is a hope to reuse the same (or similar) version of mpi in the container as is available on the native system: <a href="https://www.alcf.anl.gov/user-guides/singularity" target="_blank">https://www.alcf.anl.gov/user-guides/singularity</a>.  That is not what is happening here, and I'm not certain how easily I can change that. I might be able to modify this one package that includes the python code I want to run.</div><div><br></div><div>I've tried to simplify what I'm up to, so I connect to one of the compute nodes and there I am able to start up shifter or singularity (depending on what center I'm working on - the error is the same in both cases). Here I'm using shifter on a compute node, and I completely skip using srun, but the result is the same:</div><div><br></div><div><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">shifter --image=<imageName> ./run_shifter_smp.sh $PWD/drp-test $PWD/filesToIngest.txt |& tee smp.log</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">root INFO: Loading config overrride file '/global/cscratch1/sd/desc/DC2/Run1.2iTest-20180830/obs_lsstCam/config/ingest.py'</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">LsstCamMapper WARN: Unable to find calib root directory</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">CameraMapper INFO: Loading Posix exposure registry from /global/cscratch1/sd/desc/DC2/Run1.2i-20180830/drp-test</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">[mpiexec@nid00011] HYDU_create_process (utils/launch/launch.c:75): execvp error on file srun (No such file or directory)</span><br></div><div><br></div><div>that run_shifter_smp.sh script just sets up my env and calls the routine where I can ask that it run in SMP mode:</div><div>ingestDriver.py $1 @$2 --cores 20 --mode link --batch-type smp<br></div><div><br></div><div>I can double check where mpiexec is pointing and see:</div><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">(lsst-scipipe-10a4fa6) sh-4.2$ type mpiexec</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><div><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">mpiexec is /opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6/Linux64/mpich/3.2.1/bin/mpiexec</span></div><div>which is in our software, rather than the system's mpiexec</div><div><br></div><div>I also took a look at the slew of environment variables containing "MPI", it's at the end of this email in case it's of any use.</div><div><br></div><div>Here's a look at the modules loaded in this env:</div><div><div>module list</div><div>Currently Loaded Modulefiles:</div><div>  1) modules/<a href="http://3.2.10.6/" target="_blank">3.2.10.6</a>                               13) dvs/2.7_2.2.65-6.0.5.2_16.2__gbec2cb0</div><div>  2) intel/<a href="http://18.0.1.163/" target="_blank">18.0.1.163</a>                               14) alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari</div><div>  3) craype-network-aries                           15) rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari</div><div>  4) craype/2.5.14                                  16) atp/2.1.1</div><div>  5) cray-libsci/18.03.1                            17) PrgEnv-intel/6.0.4</div><div>  6) udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari        18) craype-haswell</div><div>  7) ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari         19) cray-mpich-abi/7.7.0</div><div>  8) pmi/5.0.13                                     20) nano/2.2.6</div><div>  9) dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari         21) altd/2.0</div><div> 10) gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari  22) darshan/3.1.4</div><div> 11) xpmem/2.2.4-6.0.5.1_8.18__g35d5e73.ari         23) Base-opts/2.4.123-6.0.5.0_11.2__g6460790.ari</div><div> 12) job/2.2.2-6.0.5.0_8.47__g3c644b5.ari</div></div><div><br></div><div><br></div><div>Reading the FAQ, I was curious about this statement when MPICH is set up with Slurm to use srun:</div><div>"<font color="#000000" face="sans-serif"><span style="font-size:12.800000190734863px">Once configured with slurm, no internal process manager is built for MPICH; the user is expected to use SLURM's launch models (such as srun)."</span></font></div><div><a href="https://wiki.mpich.org/mpich/index.php/FAQ#Q:_How_do_I_use_MPICH_with_slurm.3F" target="_blank">https://wiki.mpich.org/mpich/index.php/FAQ#Q:_How_do_I_use_MPICH_with_slurm.3F</a><br></div><div>Does that mean the host system's mpi likely only knows about SLURM's launch models and wouldn't even be able to use SMP?</div><div><br></div><div>Take care,<br></div><div>Heather</div><div><br></div><div><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">(lsst-scipipe-10a4fa6) sh-4.2$ set|grep MPI  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">CRAY_MPICH2_DIR=/opt/cray/pe/mpt/7.7.0/gni/mpich-intel/16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">CRAY_MPICH2_VER=7.7.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">CRAY_MPICH_BASEDIR=/opt/cray/pe/mpt/7.7.0/gni  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">CRAY_MPICH_DIR=/opt/cray/pe/mpt/7.7.0/gni/mpich-intel/16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">CRAY_MPICH_ROOTDIR=/opt/cray/pe/mpt/7.7.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">CRAY_PRE_COMPILE_OPTS=-hnetwork=aries  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">MPI4PY_DIR=/opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6/Linux64/mpi4py/2.0.0+6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">MPICH_ABORT_ON_ERROR=1  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">MPICH_DIR=/opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6/Linux64/mpich/3.2.1  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">MPICH_MPIIO_DVS_MAXNODES=32  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">MPI_DIR=/opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6/Linux64/mpi/0.0.1+3  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_FFTW2_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_FFTW_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_GA_DEFAULT_GENCOMPILERS_GNU='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_HDF5_DEFAULT_GENCOMPILERS_GNU='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_HDF5_PARALLEL_DEFAULT_GENCOMPILERS_GNU='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_HDF5_PARALLEL_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_LIBSCI_DEFAULT_GENCOMPILERS_CRAY_x86_64=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_LIBSCI_DEFAULT_GENCOMPILERS_GNU_x86_64='7.1 6.1 5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_LIBSCI_DEFAULT_GENCOMPILERS_INTEL_x86_64=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_LIBSCI_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_LIBSCI_GENCOMPILERS_CRAY_x86_64=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_LIBSCI_GENCOMPILERS_GNU_x86_64='7.1 6.1 5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_LIBSCI_GENCOMPILERS_INTEL_x86_64=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_LIBSCI_REQUIRED_PRODUCTS=PE_MPICH  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_ALTERNATE_LIBS_dpm=_dpm  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_ALTERNATE_LIBS_multithreaded=_mt  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_CXX_PKGCONFIG_LIBS=mpichcxx  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_DEFAULT_DIR_CRAY_DEFAULT64=64  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_DEFAULT_FIXED_PRGENV=INTEL  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_DEFAULT_GENCOMPILERS_CRAY=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_DEFAULT_GENCOMPILERS_GNU='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_DEFAULT_GENCOMPS_CRAY=86  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_DEFAULT_GENCOMPS_GNU='51 49'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/pe/mpt/7.7.0/gni/mpich-@PRGENV@@PE_MPICH_DEFAULT_DIR_DEFAULT64@/@PE_MPICH_DEFAULT_GENCOMPS@/lib/pkgconfig  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_DEFAULT_VOLATILE_PRGENV='CRAY GNU'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_DIR_CRAY_DEFAULT64=64  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_FIXED_PRGENV=INTEL  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_FORTRAN_PKGCONFIG_LIBS=mpichf90  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_GENCOMPILERS_CRAY=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_GENCOMPILERS_GNU='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_GENCOMPS_CRAY=86  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_GENCOMPS_GNU='51 49'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_MODULE_NAME=cray-mpich  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_NV_LIBS=  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_NV_LIBS_nvidia20=-lcudart  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_NV_LIBS_nvidia35=-lcudart  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_NV_LIBS_nvidia60=-lcudart  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_PKGCONFIG_LIBS=mpich  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_PKGCONFIG_VARIABLES=PE_MPICH_NV_LIBS_@accelerator@:PE_MPICH_ALTERNATE_LIBS_@multithreaded@:PE_MPICH_ALTERNATE_LIBS_@dpm@  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_TARGET_VAR_nvidia20=-lcudart  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_TARGET_VAR_nvidia35=-lcudart  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_VOLATILE_PKGCONFIG_PATH=/opt/cray/pe/mpt/7.7.0/gni/mpich-@PRGENV@@PE_MPICH_DIR_DEFAULT64@/@PE_MPICH_GENCOMPS@/lib/pkgconfig  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_MPICH_VOLATILE_PRGENV='CRAY GNU'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_NETCDF_DEFAULT_GENCOMPILERS_GNU='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_NETCDF_HDF5PARALLEL_DEFAULT_GENCOMPILERS_GNU='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_NETCDF_HDF5PARALLEL_DEFAULT_REQUIRED_PRODUCTS=PE_HDF5_PARALLEL:PE_MPICH  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PARALLEL_NETCDF_DEFAULT_GENCOMPILERS_GNU='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_haswell=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_interlagos=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_mic_knl=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_sandybridge=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_skylake=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_x86_64=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_GNU_haswell='5.3 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_GNU_interlagos='5.3 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_GNU_mic_knl=5.3  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_GNU_sandybridge='5.3 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_GNU_skylake=6.1  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_GNU_x86_64='5.3 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_haswell=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_interlagos=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_mic_knl=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_sandybridge=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_skylake=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_x86_64=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PETSC_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI:PE_HDF5_PARALLEL:PE_TPSL  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PKGCONFIG_DEFAULT_PRODUCTS=PE_TRILINOS:PE_TPSL_64:PE_TPSL:PE_PETSC:PE_PARALLEL_NETCDF:PE_NETCDF_HDF5PARALLEL:PE_NETCDF:PE_MPICH:PE_LIBSCI:PE_HDF5_PARALLEL:PE_HDF5:PE_GA:PE_FFTW2:PE_FFTW  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_PKGCONFIG_PRODUCTS=PE_MPICH:PE_LIBSCI  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_haswell=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_interlagos=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_mic_knl=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_sandybridge=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_x86_64=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_x86_skylake=8.6  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_haswell='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_interlagos='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_mic_knl=5.1  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_sandybridge='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_x86_64='5.1 4.9'  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_x86_skylake=6.1  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_haswell=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_interlagos=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_mic_knl=16.0  </span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_sandybridge=16.0</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_x86_64=16.0</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_x86_skylake=16.0</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_64_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_haswell=8.6</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_interlagos=8.6</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_mic_knl=8.6</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_sandybridge=8.6</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_x86_64=8.6</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_x86_skylake=8.6</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_GNU_haswell='5.1 4.9'</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_GNU_interlagos='5.1 4.9'</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_GNU_mic_knl=5.1</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_GNU_sandybridge='5.1 4.9'</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_GNU_x86_64='5.1 4.9'</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_GNU_x86_skylake=6.1</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_haswell=16.0</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_interlagos=16.0</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_mic_knl=16.0</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_sandybridge=16.0</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_x86_64=16.0</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_x86_skylake=16.0</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TPSL_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TRILINOS_DEFAULT_GENCOMPILERS_CRAY_x86_64=8.6</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TRILINOS_DEFAULT_GENCOMPILERS_GNU_x86_64='5.1 4.9'</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TRILINOS_DEFAULT_GENCOMPILERS_INTEL_x86_64=16.0</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">PE_TRILINOS_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_HDF5_PARALLEL:PE_NETCDF_HDF5PARALLEL:PE_LIBSCI:PE_TPSL</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">SETUP_MPI='mpi 0.0.1+3 -f Linux64 -Z /opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6'</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">SETUP_MPI4PY='mpi4py 2.0.0+6 -f Linux64 -Z /opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6'</span><br style="box-sizing:border-box;color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px"><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">SETUP_MPICH='mpich 3.2.1 -f Linux64 -Z /opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6'</span></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Aug 31, 2018 at 3:47 PM Kandes, Martin <<a href="mailto:mkandes@sdsc.edu">mkandes@sdsc.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">




<div dir="ltr">
<div id="m_-3587972344519091708divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hi Heather,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Can you copy and paste your Slurm batch job script here to give us an overview of what the job looks like? It'd also be helpful if you could provided the definition (or recipe) file for this container and a list of the
 software modules available at the site you are running the container.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Marty</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">P.S. In general, you want to have the same MPI implementation and version installed within the Singularity container as-is the one available on the host systems where the container will run.
<br>
</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-3587972344519091708divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Heather Kelly <<a href="mailto:heather999kelly@gmail.com" target="_blank">heather999kelly@gmail.com</a>><br>
<b>Sent:</b> Friday, August 31, 2018 12:34:25 PM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
<b>Subject:</b> [mpich-discuss] using mpi4py in a Singluarity container run at a large computing center with Slurm installed</font>
<div> </div>
</div>

<div>
<div dir="ltr">Hi,
<div>Complete newbie here.</div>
<div>I have a Singularity container (created by someone else), that includes a python script that uses mpi4py which is installed in the image, as is mpi.  I'm trying to run this at a large computing center where SLURM is installed.  The code in the container,
 wants to utilize its own installation of mpi4py and mpi. The code provides flags that allow a user to specify if they want to allow the code to use Slurm, SMP, or nothing at all.  The default is SMP.</div>
<div><br>
</div>
<div>When I attempt to run this code in the image, on a compute node of this computing center, I receive an error:</div>
<div><span style="color:rgb(69,84,100);font-family:SourceSansPro,"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px">HYDU_create_process (utils/launch/launch.c:75): execvp error on file srun (No such file or directory</span> </div>
<div>even though I have specified to the program that I want to use SMP, yet, it appears mpiexec is trying to submit a job to Slurm.  Checking the environment, it appears mpiexec is pointing to the version installed in the container, and not the one available
 at the computing center.</div>
<div><br>
</div>
<div>Is there an env variable or some way to set the process management system to avoid using Slurm?</div>
<div><br>
</div>
<div>In other contexts this code works just fine, my problem seems specific to running this in a container at these large computing centers where Slurm is available.  It's as if the local computing center's mpi install is taking precedence..and perhaps that's
 just how it works, but I'd like to find a way around that.<br>
</div>
<div><br>
</div>
<div>Take care,</div>
<div>Heather</div>
</div>
</div>
</div>

_______________________________________________<br>
discuss mailing list     <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
</blockquote></div>