Unfortunately, this is a known problem at the moment. BLCR checkpointing hasn't worked for a few versions now. It's something we're working to fix in a future version. <div><br></div><div>Thanks,</div><div>Wesley<br>
<br>On Monday, April 7, 2014, Marcelo Paiva Ramos <<a href="mailto:marcelo.paiva@cptec.inpe.br">marcelo.paiva@cptec.inpe.br</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
Hi,</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
Can you help me to solve
this problem?<br>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<br>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div>cat /etc/issue</div>
<div>CentOS release 6.5 (Final)</div>
<div>Kernel \r on an \m</div>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<br>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div>uname -a</div>
<div>Linux server 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25
19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux</div>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<br>
</div>
<b style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">INSTALL: blcr-0.8.5</b><br style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div>tar xzvf blcr-0.8.5.tar.gz</div>
<div>cd blcr-0.8.5</div>
<div>mkdir builddir</div>
<div>cd builddir</div>
<div>../configure --prefix=/opt/blcr</div>
<div>make</div>
<div>make install</div>
<div>/sbin/insmod /opt/blcr/lib/blcr/2.6.32-431.<u></u>11.2.el6.x86_64/blcr_imports.<u></u>ko</div>
<div>/sbin/insmod /opt/blcr/lib/blcr/2.6.32-431.<u></u>11.2.el6.x86_64/blcr.ko</div>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div>uname -r</div>
<div>2.6.32-431.11.2.el6.x86_64</div>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div>lsmod | grep blcr</div>
<div>blcr 115465 0 </div>
<div>blcr_imports 10715 1 blcr</div>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div>ldconfig -p | grep blcr</div>
<div><span style="white-space:pre-wrap"> </span>libcr_run.so.0
(libc6,x86-64) => /opt/blcr/lib/libcr_run.so.0</div>
<div><span style="white-space:pre-wrap"> </span>libcr_run.so
(libc6,x86-64) => /opt/blcr/lib/libcr_run.so</div>
<div><span style="white-space:pre-wrap"> </span>libcr_omit.so.0
(libc6,x86-64) => /opt/blcr/lib/libcr_omit.so.0</div>
<div><span style="white-space:pre-wrap"> </span>libcr_omit.so
(libc6,x86-64) => /opt/blcr/lib/libcr_omit.so</div>
<div><span style="white-space:pre-wrap"> </span>libcr.so.0
(libc6,x86-64) => /opt/blcr/lib/libcr.so.0</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<span style="white-space:pre-wrap"> </span>libcr.so (libc6,x86-64) =>
/opt/blcr/lib/libcr.so<br>
chkconfig --list | grep blcr<br>
blcr 0:off 1:off 2:on 3:on 4:on
5:on 6:off<br>
<br>
</div>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<b>INSTALL: mpich-3.1</b></div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div>tar xzvf mpich-3.1.tar.gz</div>
<div>cd mpich-3.1</div>
<div>./configure --disable-fast CFLAGS=-O2 FFLAGS=-O2 CXXFLAGS=-O2
FCFLAGS=-O2 --prefix=/opt/mpich2/ CC=/opt/intel/bin/icc
FC=/opt/intel/bin/ifort F77=/opt/intel/bin/ifort
--enable-checkpointing --with-hydra-ckpointlib=blcr
--with-blcr=/opt/blcr --with-blcr-include=/opt/blcr/<u></u>include
--with-blcr-lib=/opt/blcr/lib</div>
<div>make</div>
<div>make install</div>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<br>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<b>.bashrc</b></div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div>export PATH=$PATH:/usr/local/sbin:/<u></u>usr/local/bin:/usr/sbin:/usr/<u></u>bin:/sbin:/bin:/opt/blcr/bin:/<u></u>opt/mpich2/bin</div>
<div>export LD_LIBRARY_PATH=/usr/local/<u></u>lib:/usr/local/lib64:/opt/<u></u>intel/lib/intel64:/opt/blcr/<u></u>lib:/opt/mpich2/lib</div>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<br>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div><b>mpiexec -info</b></div>
<div>HYDRA build details:</div>
<div> Version: 3.1</div>
<div> Release Date: Thu Feb 20
11:41:13 CST 2014</div>
<div> CC: /opt/intel/bin/icc -O2
</div>
<div> CXX: g++ -O2 </div>
<div> F77: /opt/intel/bin/ifort -O2
</div>
<div> F90: /opt/intel/bin/ifort -O2
</div>
<div> Configure options:
'--disable-option-checking' '--prefix=/opt/mpich2'
'--disable-fast' 'CFLAGS=-O2 -O0' 'FFLAGS=-O2 -O0' 'CXXFLAGS=-O2
' 'FCFLAGS=-O2 ' 'CC=/opt/intel/bin/icc'
'FC=/opt/intel/bin/ifort' 'F77=/opt/intel/bin/ifort'
'--enable-checkpointing' '--with-hydra-ckpointlib=blcr'
'--with-blcr=/opt/blcr' '--with-blcr-include=/opt/<u></u>blcr/include'
'--with-blcr-lib=/opt/blcr/<u></u>lib' '--cache-file=/dev/null'
'--srcdir=.' 'LDFLAGS= -L/opt/blcr/lib' 'LIBS=-lrt -lcr
-lpthread ' 'CPPFLAGS= -I/root/mpich-3.1/src/mpl/<u></u>include
-I/root/mpich-3.1/src/mpl/<u></u>include
-I/root/mpich-3.1/src/openpa/<u></u>src
-I/root/mpich-3.1/src/openpa/<u></u>src -I/root/mpich-3.1/src/mpi/<u></u>romio/include
-I/opt/blcr/include'</div>
<div> Process Manager: pmi</div>
<div> Launchers available: ssh rsh fork
slurm ll lsf sge pbs manual persist</div>
<div> Topology libraries available: hwloc</div>
<div> Resource management kernels available: user slurm ll
lsf sge pbs cobalt</div>
<div> Checkpointing libraries available: blcr</div>
<div> Demux engines available: poll select</div>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<br>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<br>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<b>ERROR</b></div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<div>mpiexec -n 1 -ckpointlib blcr -ckpoint-interval 20
-ckpoint-prefix /home/marcelo/TESTE/ ./teste</div>
<div>[proxy:0:0@server] requesting checkpoint</div>
<div>[proxy:0:0@server] checkpoint completed</div>
<div>[proxy:0:0@server] HYDT_ckpoint_blcr_checkpoint
(tools/ckpoint/blcr/ckpoint_<u></u>blcr.c:241): Checkpointing
failed. Make sure BLCR kernel module is loaded. Unknown error
2356</div>
<div>[proxy:0:0@server] ckpoint_thread
(tools/ckpoint/ckpoint.c:76): blcr checkpoint returned error</div>
<div>[proxy:0:0@server] requesting checkpoint</div>
<div>[proxy:0:0@server] checkpoint completed</div>
<div>[proxy:0:0@server] HYDT_ckpoint_blcr_checkpoint
(tools/ckpoint/blcr/ckpoint_<u></u>blcr.c:241): Checkpointing
failed. Make sure BLCR kernel module is loaded. Unknown error
2356</div>
<div>[proxy:0:0@server] ckpoint_thread
(tools/ckpoint/ckpoint.c:76): blcr checkpoint returned error</div>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
<br>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
Best regards,<br>
</div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;color:rgb(34,34,34);text-transform:none;font-size:13px;white-space:normal;font-family:arial,sans-serif;word-spacing:0px">
Marcelo.</div>
</div>
</blockquote></div>