<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 22, 2018 at 3:52 PM, Martin Cuma <span dir="ltr"><<a href="mailto:martin.cuma@utah.edu" target="_blank">martin.cuma@utah.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I finally discovered that MPICH 3.2.1 includes ofi netmod which I am trying to build to provide unified library for tcp and InfiniBand. That's awesome since I am looking forward to ofi as a replacement for the old mxm netmod.<br>
<br>
So far it's working great, though it seems like the performance is lagging a bit behind the old mxm netmod on a few IB clusters I tried but I figure that's due to the ofi infancy.<br>
<br>
However, I am having trouble building 3.2.1 with ofi using PGI compilers (18.1), where they complain:<br>
libtool: compile: pgcc -DHAVE_CONFIG_H -I. -I../../../srcdir/mpich/3.2.1 -I./src/include -I../../../srcdir/mpich/3.2.1/<wbr>src/include -I../../../srcdir/mpich/3.2.1/<wbr>src/mpi/datatype -I../../../srcdir/mpich/3.2.1/<wbr>src/mpi/spawn -I../../../srcdir/mpich/3.2.1/<wbr>src/util/dbg -Isrc/util/logging/common -I../../../srcdir/mpich/3.2.1/<wbr>src/util/mem -I../../../srcdir/mpich/3.2.1/<wbr>src/util/thread -I../../../srcdir/mpich/3.2.1/<wbr>src/util/type -I../../../srcdir/mpich/3.2.1/<wbr>src/util/wrappers -I./src/util/wrappers -I../../../srcdir/mpich/3.2.1/<wbr>src/util/assert -I./src/util/timers -I../../../srcdir/mpich/3.2.1/<wbr>src/util/pointer -I./src/binding/cxx -I/uufs/<a href="http://chpc.utah.edu/sys/srcdir/mpich/3.2.1/src/binding/fortran/mpif_h" rel="noreferrer" target="_blank">chpc.utah.edu/sys/srcd<wbr>ir/mpich/3.2.1/src/binding/<wbr>fortran/mpif_h</a> -Isrc/binding/fortran/use_mpi -I../../../srcdir/mpich/3.2.1/<wbr>src/mpid/ch3/include -I./src/mpid/ch3/include -I../../../srcdir/mpich/3.2.1/<wbr>src/mpid/ch3/channels/nemesis/<wbr>include -I./src/mpid/ch3/channels/neme<wbr>sis/include -I../../../srcdir/mpich/3.2.1/<wbr>src/mpid/ch3/channels/nemesis/<wbr>utils/monitor -I../../../srcdir/mpich/3.2.1/<wbr>src/mpid/common/datatype -I../../../srcdir/mpich/3.2.1/<wbr>src/mpid/common/datatype -I../../../srcdir/mpich/3.2.1/<wbr>src/mpid/common/sched -I../../../srcdir/mpich/3.2.1/<wbr>src/mpid/common/thread -I../../../srcdir/mpich/3.2.1/<wbr>src/mpid/common/timers -I../../../srcdir/mpich/3.2.1/<wbr>src/pmi/simple -I/uufs/<a href="http://chpc.utah.edu/sys/builddir/mpich/3.2.1p/src/mpl/include" rel="noreferrer" target="_blank">chpc.utah.edu/sys/buil<wbr>ddir/mpich/3.2.1p/src/mpl/incl<wbr>ude</a> -I/uufs/<a href="http://chpc.utah.edu/sys/srcdir/mpich/3.2.1/src/mpl/include" rel="noreferrer" target="_blank">chpc.utah.edu/sys/srcd<wbr>ir/mpich/3.2.1/src/mpl/include</a> -I/uufs/<a href="http://chpc.utah.edu/sys/srcdir/mpich/3.2.1/src/openpa/src" rel="noreferrer" target="_blank">chpc.utah.edu/sys/srcd<wbr>ir/mpich/3.2.1/src/openpa/src</a> -I/uufs/<a href="http://chpc.utah.edu/sys/builddir/mpich/3.2.1p/src/openpa/src" rel="noreferrer" target="_blank">chpc.utah.edu/sys/buil<wbr>ddir/mpich/3.2.1p/src/openpa/<wbr>src</a> -D_REENTRANT -I/uufs/<a href="http://chpc.utah.edu/sys/builddir/mpich/3.2.1p/src/mpi/romio/include" rel="noreferrer" target="_blank">chpc.utah.edu/sys/buil<wbr>ddir/mpich/3.2.1p/src/mpi/romi<wbr>o/include</a> -I/uufs/<a href="http://chpc.utah.edu/sys/installdir/libfabric/1.6.0/include" rel="noreferrer" target="_blank">chpc.utah.edu/sys/inst<wbr>alldir/libfabric/1.6.0/include</a> -tp=nehalem -fPIC -DNDEBUG -DNVALGRIND -O2 -c ../../../srcdir/mpich/3.2.1/sr<wbr>c/mpid/ch3/channels/nemesis/ne<wbr>tmod/ofi/ofi_tagged.c -MD -fpic -DPIC -o src/mpid/ch3/channels/nemesis/<wbr>netmod/ofi/.libs/lib_libmpi_la<wbr>-ofi_tagged.o<br>
PGC-S-0209-EOF in macro call to FI_RC_RETRY (../../../srcdir/mpich/3.2.1/s<wbr>rc/mpid/ch3/channels/nemesis/n<wbr>etmod/ofi/ofi_tagged.c: 143)<br>
PGC-S-0218-Missing #endif (../../../srcdir/mpich/3.2.1/s<wbr>rc/mpid/ch3/channels/nemesis/n<wbr>etmod/ofi/ofi_tagged.c: 143)<br>
PGC/x86-64 Linux 18.1-1: compilation completed with severe errors<br>
<br>
Any thought on what can be the problem? The same build went fine with GNU and Intel compilers.<br></blockquote><div><br></div><div>If GCC and ICC are fine, my guess is that this is a bug in the PGI C compiler. This wouldn't be the first time there were issues related to PGI in MPICH...</div><div><br></div><div>Given that the C ABI is defined by the system rather than the compiler, why do you need to use PGI? You need to use PGI Fortran to get the Fortran stuff to work, but GCC, Clang, or ICC should be fine (except for a possible issue with GCC+PGF builds related to memset changes I made years ago, which I assert is a PGI bug).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
The suggestion concerns verbose make output. The user's guide says to run "make VERBOSE=1" to see what I pasted above. However, that did not work for me. What did work is "make V=1". If that is indeed the current magic word to make make verbose, can you please update the user's guide to save us (including me again) time to search through the Makefile to figure this out? <br></blockquote></div><div class="gmail_extra"><br></div><div class="gmail_extra">The V=1 option is documented at line 125 of README.vin, which seems like a reasonable place to document it.</div><div class="gmail_extra"><br></div>I don't see "verbose" anywhere in <a href="https://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1-userguide.pdf">https://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1-userguide.pdf</a>. Frankly, I've never read that document anyways. I rely on README* or "git grep" for everything.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Jeff<br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div></div>