<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>Thank you for the hint! I finally got my job running on my own
MPI build. Just out of curious: Actually, I searched this
libmpiprofilesupport library before and discovered that it is
specific to Spectrum MPI so I didn't go further. But MPICH just
ran magically after I load Spectrum MPI and have the Spectrum MPI
lib path in $LD_LIBRARY_PATH. Why does MPICH have to use Spectrum
MPI's libraries?</p>
<p><br>
</p>
<p>Thanks!</p>
<p>Feimi<br>
</p>
<div class="moz-cite-prefix">On 9/19/2021 1:57 PM, Zhou, Hui wrote:<br>
</div>
<blockquote type="cite"
cite="mid:SA0PR09MB74178E9B4D186CDADC2F3E02A9DF9@SA0PR09MB7417.namprd09.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
> ./step-17.release: error while loading shared libraries:
libmpiprofilesupport.so.3: cannot open shared object file: No
such file or directory<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
This is complaining about cannot finding a dynamic library (that
is linked in your binary) on the compute node. Make sure the
path to that library is in the LD_LIBRARY_PATH.<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
Feimi Yu <a class="moz-txt-link-rfc2396E" href="mailto:yuf2@rpi.edu"><yuf2@rpi.edu></a><br>
<b>Sent:</b> Saturday, September 18, 2021 9:13 PM<br>
<b>To:</b> Zhou, Hui <a class="moz-txt-link-rfc2396E" href="mailto:zhouh@anl.gov"><zhouh@anl.gov></a>;
<a class="moz-txt-link-abbreviated" href="mailto:discuss@mpich.org">discuss@mpich.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:discuss@mpich.org"><discuss@mpich.org></a><br>
<b>Subject:</b> Re: [mpich-discuss] Installing MPICH on
clusters</font>
<div> </div>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;">Hi Hui,</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">Thank you for
the response! Here is the Slurm batch file I used to run a
program with MPICH configured with Hydra:</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><i>#!/bin/bash</i><i><br>
</i><i>#SBATCH --job-name=5e-7</i><i><br>
</i><i>#SBATCH --partition=el8</i><i><br>
</i><i>#SBATCH --time 6:00:00</i><i><br>
</i><i>#SBATCH --ntasks 40</i><i><br>
</i><i>#SBATCH --nodes 1</i><i><br>
</i><i>#SBATCH --gres=gpu:4</i><i><br>
</i><i><br>
</i><i>date</i><i><br>
</i><i>export
LD_LIBRARY_PATH=/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/lib:$MPI_ROOT:$LD_LIBRARY_PATH</i><i><br>
</i><i>srun --mpi=mpichmx hostname -s | sort -u >
/tmp/hosts.$SLURM_JOB_ID</i><i><br>
</i><i>awk "{ print \$0 \":40\"; }" /tmp/hosts.$SLURM_JOB_ID
>/tmp/tmp.$SLURM_JOB_ID</i><i><br>
</i><i>mv /tmp/tmp.$SLURM_JOB_ID ./hosts.$SLURM_JOB_ID</i><i><br>
</i><i><br>
</i><i>/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/bin/mpiexec
-f ./hosts.$SLURM_JOB_ID -np $SLURM_NPROCS
./step-17.release</i><i><br>
</i><i><br>
</i><i>date</i><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">And the error
message is:</p>
<p style="margin-top: 0px; margin-bottom: 0px;">./step-17.release:
error while loading shared libraries:
libmpiprofilesupport.so.3: cannot open shared object file:
No such file or directory</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">I was not sure
if this is related to a network problem because the clusters
use Infiniband. Running "/sbin/ifconfig" gives ib0, ib1, ib2
and ib3. I tried the option "-iface ib0" and the error
message became:</p>
<p style="margin-top: 0px; margin-bottom: 0px;">[mpiexec@dcs176]
HYDU_sock_get_iface_ip (utils/sock/sock.c:451): unable to
find interface ib0
<br>
[mpiexec@dcs176] HYDU_sock_create_and_listen_portstr
(utils/sock/sock.c:496): unable to get network interface IP<br>
[mpiexec@dcs176] HYD_pmci_launch_procs
(pm/pmiserv/pmiserv_pmci.c:79): unable to create PMI port<br>
[mpiexec@dcs176] main (ui/mpich/mpiexec.c:322): process
manager returned error launching processes</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">Specifying
ib1-ib3 gives similar results.<br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">Thanks!</p>
<p style="margin-top: 0px; margin-bottom: 0px;">Feimi<br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<div class="x_moz-cite-prefix">On 9/17/21 7:57 PM, Zhou, Hui
wrote:<br>
</div>
<blockquote type="cite">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Hi Feimi,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Hydra should be able to work with slurm. How are you
launching the job and what is the failure message?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
-- <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Hui Zhou<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" face="Calibri, sans-serif"
color="#000000"><b>From:</b> Feimi Yu via discuss
<a class="x_moz-txt-link-rfc2396E"
href="mailto:discuss@mpich.org" moz-do-not-send="true"><discuss@mpich.org></a><br>
<b>Sent:</b> Friday, September 17, 2021 10:55 AM<br>
<b>To:</b> <a class="x_moz-txt-link-abbreviated"
href="mailto:discuss@mpich.org" moz-do-not-send="true">
discuss@mpich.org</a> <a
class="x_moz-txt-link-rfc2396E"
href="mailto:discuss@mpich.org" moz-do-not-send="true">
<discuss@mpich.org></a><br>
<b>Cc:</b> Feimi Yu <a class="x_moz-txt-link-rfc2396E"
href="mailto:yuf2@rpi.edu" moz-do-not-send="true">
<yuf2@rpi.edu></a><br>
<b>Subject:</b> [mpich-discuss] Installing MPICH on
clusters</font>
<div> </div>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;">Hi,</p>
<p style="margin-top: 0px; margin-bottom: 0px;">I'm
working on a supercomputer which only provides Spectrum
MPI implementation in modules. Since our code does not
perform well with Spectrum MPI I decided to install an
MPICH build on our own partition (I'm not an
administrator.) The supercomputer has a rhel8 system on
ppc64le architecture with Slurm as the process manager.
I tried several building options according to the user
guide but could not run a job so I have a few questions.
Here are things I tried:</p>
<p style="margin-top: 0px; margin-bottom: 0px;">1. Build
with Hydra PM. I could not launch a job with Hydra at
all.</p>
<p style="margin-top: 0px; margin-bottom: 0px;">2. Then I
decided to use ``--with-pm=none`` option to build and
use srun + ``mpiexec -f hostfile`` to launch my job. But
what confuses me is the PMI setting:</p>
<p style="margin-top: 0px; margin-bottom: 0px;">srun
--mpi=list gives following:</p>
<p style="margin-top: 0px; margin-bottom: 0px;">srun:
mpi/mpichgm<br>
srun: mpi/mpichmx<br>
srun: mpi/none<br>
srun: mpi/mvapich<br>
srun: mpi/openmpi<br>
srun: mpi/pmi2<br>
srun: mpi/lam<br>
srun: mpi/mpich1_p4<br>
srun: mpi/mpich1_shmem</p>
<p style="margin-top: 0px; margin-bottom: 0px;">At first I
tried use pmix since I found pmix libraries. But it
didn't do the trick. It segfaults on PMPI_Init_thread().
The error message is:</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><i>[dcs135:2312190]
PMIX ERROR: NOT-FOUND in file client/pmix_client.c at
line 562</i></p>
<p style="margin-top: 0px; margin-bottom: 0px;"><i>Abort(1090831)
on node 0 (rank 0 in comm 0): Fatal error in
PMPI_Init_thread: Other MPI error, error stack:</i><i><br>
</i><i>MPIR_Init_thread(159): </i><i><br>
</i><i>MPID_Init(509).......: </i><i><br>
</i><i>MPIR_pmi_init(92)....: PMIX_Init returned -46 </i><i><br>
</i><i>[dcs135:2312190:0:2312190] Caught signal 11
(Segmentation fault: address not mapped to object at
address (nil))</i><i><br>
</i></p>
<p style="margin-top: 0px; margin-bottom: 0px;">Then I
switched to pmi2 but make keeps telling me undefined
reference to PMI2 library. (actually I couldn't find the
pmi2 libraries either.)</p>
<p style="margin-top: 0px; margin-bottom: 0px;">Then I
used ``--with-pmi=slurm``, and it turned out that I
couldn't locate the Slurm header files. I guess I don't
have the permission to access them.</p>
<p style="margin-top: 0px; margin-bottom: 0px;">I was
wondering if it is still possible for me to build a
usable MPICH as a user? If yes, how can I do to have the
PMI work?<br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">Thanks!</p>
Feimi </div>
</blockquote>
</div>
</div>
</blockquote>
</body>
</html>