<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Martin,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">PMIx with MPICH is not well tested with large number of nodes and we are not aware of the issue you described. We’ll look into it. Meanwhile, if you could file a github issue, that will help us better tracking it.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for reporting it.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <br>
Hui Zhou<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Audet, Martin via discuss <discuss@mpich.org><br>
<b>Date: </b>Tuesday, December 22, 2020 at 5:16 PM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>Audet, Martin <Martin.Audet@cnrc-nrc.gc.ca>, Raymond, Stephane <Stephane.Raymond@cnrc-nrc.gc.ca><br>
<b>Subject: </b>[mpich-discuss] Is PMIx working well for ch4:ucx ? Intermitent Seg Fault in MPIR_pmi_init()<o:p></o:p></span></p>
</div>
<p class="MsoNormal">Hello MPICH_Users && MPICH_Developers,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Is the new mpich version 3.4rc1 is supposed to work well with PMIx when using the ch4:ucx device or is it still considered experimental ?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Yesterday when trying it with a my usual “hellompihost2.cpp” test program on our cluster it turns out that it created intermittent Segmentation fault in the MPIR_pmi_init() initialization function which is indirectly called by MPI_Init().
It seems that the higher number of “processors” (MPI ranks) involved, the higher the chances of having this problem. With 24 nodes each with 24 or 48 “processors” (MPI ranks) the probability seems to be 100%. When keeping 24 “processors” per node, the probability
of having this problem increase rapidly when increasing the node count from 9 to 10 (i.e. at 9 nodes it works most of the time and with 10 nodes it fails most of the time).<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Note that my “hellompihost2.cpp” program works very well with OpenMPI and with the same mpich 3.4rc1 but with the ch3:sock or ch3:nemesis channel (using the PMI2 startup mechanism).<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">We use Slurm version 20.02.6 and PMIx 3.1.5. Our OS is CentOS 7.9 (latest kernel and packages) and we use MOFED 4.9.2.2.4.0. As you can see our environment is absolutely not exotic and should be very common among mpich users.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Below is the output from uname, mpichversion, the source code of hellompihost2.cpp and a sample of the output when it crashes.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Martin Audet<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[audetm@hn audetm]$ uname -a</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Linux hn.galerkin.res.nrc.gc.ca 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[audetm@hn audetm]$ mpichversion
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">MPICH Version: 3.4rc1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">MPICH Release date: Thu Dec 10 14:41:59 CST 2020</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">MPICH Device: ch4:ucx</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">MPICH configure: --with-device=ch4:ucx --with-hcoll=/opt/mellanox/hcoll --with-pmix=/usr --prefix=/work/software/x86_64/mpi/mpich-ch4_ucx-3.4rc1 --enable-fast=all --enable-romio --with-file-system=ufs+nfs+lustre
--enable-shared --enable-sharedlibs=gcc</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">MPICH CC: gcc -std=gnu99 -std=gnu99 -DNDEBUG -DNVALGRIND -O2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">MPICH CXX: g++ -DNDEBUG -DNVALGRIND -O2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">MPICH F77: gfortran -O2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">MPICH FC: gfortran -O2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">MPICH Custom Information:
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[audetm@hn audetm]$</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">#include <mpi.h></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">#include <unistd.h></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">#include <sched.h></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">#include <iostream></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">#include <vector></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">#include <sstream></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">#include <string></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">int main(int argc, char **argv)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">{</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Init(&argc, &argv);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> std::ostringstream ostr;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> const MPI_Comm cur_comm = MPI_COMM_WORLD;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> int comm_rank, comm_size;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Comm_rank(cur_comm, &comm_rank);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Comm_size(cur_comm, &comm_size);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> char name_buf[MPI_MAX_PROCESSOR_NAME+1];</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> int name_len;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Get_processor_name(name_buf, &name_len);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> name_buf[name_len] = '\0';</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> ostr << "rank " << comm_rank << " running on " << name_buf << " PID " << getpid() << " pinned to CPUs ";</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> cpu_set_t cpuset;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> const int ret = sched_getaffinity(0, sizeof(cpuset), &cpuset);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> if (ret == 0) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> bool prev = false;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> for (int i_cpu=0; i_cpu < CPU_SETSIZE; i_cpu++) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> if (CPU_ISSET(i_cpu, &cpuset)) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> if (prev) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> ostr << ',';</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> ostr << i_cpu;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> prev = true;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> else {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> ostr << "(unknown)";</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> const std::string msg(ostr.str());</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> const int msg_len = int(msg.size());</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> const int msg_len2 = msg_len;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> enum { STR_LEN_TAG, STR_VAL_TAG };</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> const int ROOT_RANK = 0;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Request req_tbl[2];</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Isend(&msg_len2, 1, MPI_INT, ROOT_RANK, STR_LEN_TAG, cur_comm, &req_tbl[0]);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Isend(msg.data(), msg_len, MPI_CHAR, ROOT_RANK, STR_VAL_TAG, cur_comm, &req_tbl[1]);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> if (comm_rank == ROOT_RANK) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> std::cout << "Running with " << comm_size << " process\n";</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> std::vector<char> recv_msg;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> for (int i_rank=0; i_rank < comm_size; i_rank++) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> int recv_len;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Status stat;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Recv(&recv_len, 1, MPI_INT, i_rank, STR_LEN_TAG, cur_comm, &stat);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> recv_msg.resize(recv_len+2);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Recv(&recv_msg[0], recv_len, MPI_CHAR, i_rank, STR_VAL_TAG, cur_comm, &stat);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> recv_msg[recv_len ] = '\n';</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> recv_msg[recv_len+1] = '\0';</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> std::cout << &recv_msg[0];</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> std::cout << std::flush;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Status stat_tbl[2];</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Waitall(2, req_tbl, stat_tbl);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Finalize();</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> return 0;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">}</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">audetm@hn audetm]$ time srun --nodes=16 --ntasks-per-node=48 --ntasks-per-core=2 --exclusive ./hellompihost2_ch4_ucx_34rc1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn12:29033:0:29033] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x905aea0)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn12:29032:0:29032] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x9726690)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn12:29049:0:29049] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x423fea0)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn12:29034:0:29034] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x3308690)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn9:29497:0:29497] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x986e460)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn9:29523:0:29523] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x9a3f460)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn9:29491:0:29491] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x31d0460)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn9:29521:0:29521] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x8a86460)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn12:29037:0:29037] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x55e5690)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn9:29493:0:29493] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x96de460)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">==== backtrace (tid: 29032) ====</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">0 0x000000000004d455 ucs_debug_print_backtrace() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">1 0x0000000000305a6c MPIR_pmi_init() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2 0x000000000031303f MPID_Init() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">3 0x0000000000202fa6 MPIR_Init_thread() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">4 0x0000000000202df2 PMPI_Init() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">5 0x0000000000401805 main() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">6 0x0000000000022555 __libc_start_main() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">7 0x0000000000401709 _start() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">=================================</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn9:29485:0:29485] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x4381c50)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">==== backtrace (tid: 29033) ====</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">0 0x000000000004d455 ucs_debug_print_backtrace() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">1 0x0000000000305a6c MPIR_pmi_init() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2 0x000000000031303f MPID_Init() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">3 0x0000000000202fa6 MPIR_Init_thread() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">4 0x0000000000202df2 PMPI_Init() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">5 0x0000000000401805 main() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">6 0x0000000000022555 __libc_start_main() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">7 0x0000000000401709 _start() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">=================================</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">==== backtrace (tid: 29049) ====</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">0 0x000000000004d455 ucs_debug_print_backtrace() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">1 0x0000000000305a6c MPIR_pmi_init() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2 0x000000000031303f MPID_Init() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">3 0x0000000000202fa6 MPIR_Init_thread() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">4 0x0000000000202df2 PMPI_Init() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">5 0x0000000000401805 main() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">6 0x0000000000022555 __libc_start_main() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">7 0x0000000000401709 _start() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">=================================</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn7:34315:0:34315] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xbbcd450)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn7:34310:0:34310] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x47b4c40)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn7:34311:0:34311] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x86f2c40)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn7:34320:0:34320] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x2775c40)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn7:34322:0:34322] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x8a5e450)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn2:34371:0:34371] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x9e8fc70)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn2:34394:0:34394] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x3036480)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn2:34370:0:34370] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x2951480)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn12:29030:0:29030] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x5b4e690)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn11:29124:0:29124] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x843c470)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">==== backtrace (tid: 29034) ====</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">0 0x000000000004d455 ucs_debug_print_backtrace() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">1 0x0000000000305a6c MPIR_pmi_init() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2 0x000000000031303f MPID_Init() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">3 0x0000000000202fa6 MPIR_Init_thread() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">4 0x0000000000202df2 PMPI_Init() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">5 0x0000000000401805 main() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">6 0x0000000000022555 __libc_start_main() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">7 0x0000000000401709 _start() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">=================================</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn12:29059:0:29059] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xa1f3690)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn7:34331:0:34331] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x2630c40)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn12:29044:0:29044] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x639b690)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn11:29159:0:29159] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xaa1fc60)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[cn11:29126:0:29126] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x735bc60)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">==== backtrace (tid: 29037) ====</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">0 0x000000000004d455 ucs_debug_print_backtrace() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">1 0x0000000000305a6c MPIR_pmi_init() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2 0x000000000031303f MPID_Init() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">3 0x0000000000202fa6 MPIR_Init_thread() :0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">4 0x0000000000202df2 PMPI_Init() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">5 0x0000000000401805 main() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">6 0x0000000000022555 __libc_start_main() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">7 0x0000000000401709 _start() ???:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">=================================</span><o:p></o:p></p>
</div>
</body>
</html>