<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoPlainText"><span style="font-family:"Courier New"">I downloaded the nightly tarball and recompiled/installed mpich (used mpich-master-v3.0.4-259-gf322ce79)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">I still get this (output below) with a simple hello world program.<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">Now you must understand that I have no idea what I’m doing (really). I wanted to test some debugging features under slurm so installed slurm myself on a workstation with just 2 cores and have the
bare minimum setup. I’m doing the following<o:p></o:p></span></p>
<p class="MsoPlainText" style="margin-left:36.0pt"><span style="font-family:"Courier New"">sudo munged &<o:p></o:p></span></p>
<p class="MsoPlainText" style="margin-left:36.0pt"><span style="font-family:"Courier New"">sudo slurmd &<o:p></o:p></span></p>
<p class="MsoPlainText" style="margin-left:36.0pt"><span style="font-family:"Courier New"">sudo slurmctld -D<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">and then I can run jobs on the local machine and it seems to be ok, except that mpi jobs always give the double free error as below when run under slurm, but are just fine when run from the command
line.<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">My suspicion is that slurm is not actually using the hydra pm that I just compiled. I installed slurm from rpms. Should I recompile slurm myself and somehow tell it which mpi to use?
<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">My job script looks as follows<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">######################<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#!/bin/bash<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""># Create the job script from the supplied parameters<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#SBATCH --job-name=pvserver<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#SBATCH --time=00:04:00<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#SBATCH --nodes=1<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#SBATCH --partition=normal<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#SBATCH --output=/home/biddisco/slurm.out<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#SBATCH --error=/home/biddisco/slurm.err<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#SBATCH --mem=2048MB<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#export<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""># echo "Path is $PATH"<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""># echo "LD_LIBRARY_PATH is " $LD_LIBRARY_PATH<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""># cd /home/biddisco/build/pv-38/bin/<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#export PMI_DEBUG=9<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#ulimit -s unlimited<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">#ulimit -c 0<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">/home/biddisco/apps/mpich-3.0.4/bin/mpiexec -rmk slurm -n 2 /home/biddisco/build/hello/hello<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">######################<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">It gives the same result with or without the –rmk slurm and the #ulimit settings.<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">Apologies for wasting your time, I’m certain I’m doing something wrong – I just don’t know what.<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">JB<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">biddisco@breno2 ~ $ more ~/slurm.err<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">*** glibc detected *** /home/biddisco/build/hello/hello: double free or corruption (fasttop): 0x0000000001896340 ***<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">======= Backtrace: =========<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f9a1695cb96]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">/home/biddisco/build/hello/hello(MPIDI_Populate_vc_node_ids+0x3f9)[0x427c89]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">/home/biddisco/build/hello/hello(MPID_Init+0x136)[0x4253f6]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">/home/biddisco/build/hello/hello(MPIR_Init_thread+0x22f)[0x414cbf]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">/home/biddisco/build/hello/hello(MPI_Init+0xae)[0x4146ee]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">/home/biddisco/build/hello/hello(main+0x22)[0x413f2e]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f9a168ff76d]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">/home/biddisco/build/hello/hello[0x413e31]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">======= Memory map: ========<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">00400000-0051a000 r-xp 00000000 08:01 8661191 /home/biddisco/build/hello/hello<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">0071a000-00727000 r--p 0011a000 08:01 8661191 /home/biddisco/build/hello/hello<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">00727000-00729000 rw-p 00127000 08:01 8661191 /home/biddisco/build/hello/hello<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">00729000-00751000 rw-p 00000000 00:00 0<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">01895000-018b6000 rw-p 00000000 00:00 0 [heap]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a166c8000-7f9a166dd000 r-xp 00000000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a166dd000-7f9a168dc000 ---p 00015000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a168dc000-7f9a168dd000 r--p 00014000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a168dd000-7f9a168de000 rw-p 00015000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a168de000-7f9a16a93000 r-xp 00000000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a16a93000-7f9a16c92000 ---p 001b5000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a16c92000-7f9a16c96000 r--p 001b4000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a16c96000-7f9a16c98000 rw-p 001b8000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a16c98000-7f9a16c9d000 rw-p 00000000 00:00 0<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a16c9d000-7f9a16cb5000 r-xp 00000000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a16cb5000-7f9a16eb4000 ---p 00018000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a16eb4000-7f9a16eb5000 r--p 00017000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a16eb5000-7f9a16eb6000 rw-p 00018000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a16eb6000-7f9a16eba000 rw-p 00000000 00:00 0<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a16eba000-7f9a16edc000 r-xp 00000000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a170c1000-7f9a170c4000 rw-p 00000000 00:00 0<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a170d9000-7f9a170dc000 rw-p 00000000 00:00 0<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a170dc000-7f9a170dd000 r--p 00022000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7f9a170dd000-7f9a170df000 rw-p 00023000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7fff52f27000-7fff52f48000 rw-p 00000000 00:00 0 [stack]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">7fff52fff000-7fff53000000 r-xp 00000000 00:00 0 [vdso]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:"Courier New"">ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]<o:p></o:p></span></p>
</div>
</body>
</html>