[mpich-discuss] Configure/make fails when given flag --with-shared-memory=sysv

[Åsmund Ervik]

Hello all,

I'm compiling mvapich2 on an old quirky cluster where I'm not root. After having problems with shared memory using mvapich2 with plain configure/make (I had to disable shared memory for any jobs doing allreduce), I decided to try to configure/make using --with-shared-memory=sysv . This failed at the "make" step, with the following error:
   CC       src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.lo
src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c: In function 'mv2_allocate_shm_local':
src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:182: error: 'O_CREAT' undeclared (first use in this function)
src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:182: error: (Each undeclared identifier is reported only once
src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:182: error: for each function it appears in.)
src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:182: error: 'O_RDWR' undeclared (first use in this function)
src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:182: error: 'O_EXCL' undeclared (first use in this function)
src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c: In function 'mv2_rma_allocate_shm':
src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:375: error: 'O_CREAT' undeclared (first use in this function)
src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:375: error: 'O_RDWR' undeclared (first use in this function)
make[2]: *** [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.lo] Error 1

The full configure command was
./configure --prefix=/home/asmunder/mvapich2-2.0 --with-ib-libpath=/cvos/shared/apps/ofed/1.3.1/lib64/ --with-ib-include=/cvos/shared/apps/ofed/1.3.1/include --with-shared-memory=sysv --enable-libpci


In the end I traced this down to lines 22 and 26 in rdma_iba_1sc.c , which is an #ifdef that only includes fcntl.h (containing O_CREAT et al) when the flag _SMP_LIMIC_ is defined. Obviously this flag was not defined in my case. I removed these two lines, and now configure/make works fine. 

Also, in the end, I found that this flag was not necessary, and that the correct and much simpler solution was to set the environment variable MV2_SHMEM_DIR to the correct path.

Regards,
Åsmund