<html dir="ltr"><head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">
<div style="direction:ltr; font-family:Tahoma; color:#000000; font-size:10pt">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment">Dear Pavan,
<div><br>
</div>
<div>thank you for hints! Ok, I was able to install MXM locally without root privileges. Then I have configured and compiled MPICH3 with MXM. Everything went smoothly and finally I got a working binaries and libs. Unfortunately, root is still needed because
a simple MPI HELLO WORLD code now crashes with the following error message:</div>
<div><br>
</div>
<div><span style="font-family:Menlo; font-size:11px">%mpirun -np 2 ./a.out </span></div>
<div>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">[1415719636.245549] [n1:37817:0] shm.c:65 MXM WARN Could not open the KNEM device file at /dev/knem : No such file or directory. Won't use knem.</p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">[1415719636.245549] [n1:37818:0] shm.c:65 MXM WARN Could not open the KNEM device file at /dev/knem : No such file or directory. Won't use knem.</p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">[1415719636.263914] [n1:37817:0] ib_dev.c:443 MXM ERROR ibv_query_device() returned 38: No such file or directory</p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">[1415719636.264134] [n1:37818:0] ib_dev.c:443 MXM ERROR ibv_query_device() returned 38: No such file or directory</p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">Fatal error in MPI_Init: Other MPI error, error stack:</p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPIR_Init_thread(498).........: </p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPID_Init(187)................: channel initialization failed</p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPIDI_CH3_Init(89)............: </p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPID_nem_init(320)............: </p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPID_nem_mxm_init(157)........: </p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPID_nem_mxm_vc_terminate(451): mxm_init failed (Input/output error)</p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">Fatal error in MPI_Init: Other MPI error, error stack:</p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPIR_Init_thread(498).........: </p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPID_Init(187)................: channel initialization failed</p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPIDI_CH3_Init(89)............: </p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPID_nem_init(320)............: </p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPID_nem_mxm_init(157)........: </p>
<p style="margin-right:0px; margin-left:0px; font-size:11px; font-family:Menlo">MPID_nem_mxm_vc_terminate(451): mxm_init failed (Input/output error)</p>
</div>
<div>%ldd a.out</div>
<div>
<div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>linux-vdso.so.1 => (0x00007ffff29ff000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libmpi.so.12 => /nobackup/global/x_vicvy/mpich3-dev-bin.mxm/lib/libmpi.so.12 (0x00007f9fd4fa1000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libm.so.6 => /lib64/libm.so.6 (0x00007f9fd4cfe000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f9fd4ae8000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libc.so.6 => /lib64/libc.so.6 (0x00007f9fd4754000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libdl.so.2 => /lib64/libdl.so.2 (0x00007f9fd454f000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libmxm.so.2 => /nobackup/global/x_vicvy/hpcx-v1.2.0-255-icc-MLNX_OFED_LINUX-2.3-1.5.0-redhat6.5/mxm/lib/libmxm.so.2 (0x00007f9fd41f5000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libz.so.1 => /lib64/libz.so.1 (0x00007f9fd3fdf000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00007f9fd3dd1000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>librt.so.1 => /lib64/librt.so.1 (0x00007f9fd3bc9000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libgpfs.so => /usr/lib64/libgpfs.so (0x00007f9fd39ba000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9fd379c000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libifport.so.5 => /software/apps/intel/composer_xe_2015.0.090/compiler/lib/intel64/libifport.so.5 (0x00007f9fd356f000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libifcore.so.5 => /software/apps/intel/composer_xe_2015.0.090/compiler/lib/intel64/libifcore.so.5 (0x00007f9fd3239000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libimf.so => /software/apps/intel/composer_xe_2015.0.090/compiler/lib/intel64/libimf.so (0x00007f9fd2d7e000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libsvml.so => /software/apps/intel/composer_xe_2015.0.090/compiler/lib/intel64/libsvml.so (0x00007f9fd212f000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libintlc.so.5 => /software/apps/intel/composer_xe_2015.0.090/compiler/lib/intel64/libintlc.so.5 (0x00007f9fd1ed5000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>/lib64/ld-linux-x86-64.so.2 (0x00007f9fd5499000)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><span class="Apple-tab-span" style="white-space:pre"></span>libirng.so => /software/apps/intel/composer_xe_2015.0.090/compiler/lib/intel64/libirng.so (0x00007f9fd1ccd000)</div>
</div>
<div></div>
</div>
<div><br>
</div>
<div>Since ’nemesis:ch3:ib’ is fixed, I will check it out. </div>
<div><br>
</div>
<div>With best regards,</div>
<div>Victor.</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>