<div dir="auto">Unless MPICH configure has changed recently, —enable-g=debug enables debug symbols, and is all you need. Your choice is sufficient but overkill. It may introduce nontrivial performance overheads. </div><div dir="auto"><br></div><div dir="auto">Jeff</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Nov 13, 2021 at 10:57 PM Mccall, Kurt E. (MSFC-EV41) via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="m_1986873516637256965WordSection1">
<div>
<div style="border-style:solid none none;border-top-width:1pt;padding:3pt 0in 0in;border-top-color:rgb(225,225,225)">
<p class="MsoNormal">Hui,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I built MPICH 4.0a2 with gcc 4.8.5, and included the -enable-g=all flag to “configure” so that debugging symbols would be present. The code is crashing my call to MPI_Type_commit, in libpthreads.so. gdb give this stack trace below.
Since MPICH 3.3.2, has there been changes in how custom types are created (the code worked in 3.3.2)? I included my type-creating code after the stack trace.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Program received signal SIGSEGV, Segmentation fault.<u></u><u></u></p>
<p class="MsoNormal">MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0,
<u></u><u></u></p>
<p class="MsoNormal"> array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0,
<u></u><u></u></p>
<p class="MsoNormal"> array_of_types=array_of_types@entry=0x7fffcaa24410, <u></u>
<u></u></p>
<p class="MsoNormal"> newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>)<u></u><u></u></p>
<p class="MsoNormal"> at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659<u></u><u></u></p>
<p class="MsoNormal">659 MPIR_Ensure_Aint_fits_in_int(old_dtp->builtin_element_size);<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">(gdb) where<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">#0 MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0,
<u></u><u></u></p>
<p class="MsoNormal"> array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0,
<u></u><u></u></p>
<p class="MsoNormal"> array_of_types=array_of_types@entry=0x7fffcaa24410, <u></u>
<u></u></p>
<p class="MsoNormal"> newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>)<u></u><u></u></p>
<p class="MsoNormal"> at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659<u></u><u></u></p>
<p class="MsoNormal">#1 0x00007fec8b9b2608 in type_struct (count=count@entry=8, blocklength_array=blocklength_array@entry=0x128b6b0,
<u></u><u></u></p>
<p class="MsoNormal"> displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410,
<u></u><u></u></p>
<p class="MsoNormal"> newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:206<u></u><u></u></p>
<p class="MsoNormal">#2 0x00007fec8b9b4b9e in type_struct (newtype=0x7fffcaa242dc, oldtype_array=0x7fffcaa24410,
<u></u><u></u></p>
<p class="MsoNormal"> displacement_array=0x7fffcaa243c0, blocklength_array=0x128b6b0, count=8)<u></u><u></u></p>
<p class="MsoNormal"> at ../mpich-4.0a2/src/mpi/datatype/type_create.c:227<u></u><u></u></p>
<p class="MsoNormal">#3 MPIR_Type_struct (count=count@entry=8, blocklength_array=0x128b6b0,
<u></u><u></u></p>
<p class="MsoNormal"> displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410,
<u></u><u></u></p>
<p class="MsoNormal"> newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:235<u></u><u></u></p>
<p class="MsoNormal">#4 0x00007fec8b9b7b08 in MPIR_Type_create_struct_impl (count=count@entry=8,
<u></u><u></u></p>
<p class="MsoNormal"> array_of_blocklengths=array_of_blocklengths@entry=0x7fffcaa24440,
<u></u><u></u></p>
<p class="MsoNormal"> array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0,
<u></u><u></u></p>
<p class="MsoNormal"> array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x12853fc)<u></u><u></u></p>
<p class="MsoNormal"> at ../mpich-4.0a2/src/mpi/datatype/type_create.c:908<u></u><u></u></p>
<p class="MsoNormal">#5 0x00007fec8b85ad26 in internal_Type_create_struct (newtype=0x12853fc, array_of_types=0x7fffcaa24410,
<u></u><u></u></p>
<p class="MsoNormal"> array_of_displacements=<optimized out>, array_of_blocklengths=0x7fffcaa24440, count=8)<u></u><u></u></p>
<p class="MsoNormal"> at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:79<u></u><u></u></p>
<p class="MsoNormal">#6 PMPI_Type_create_struct (count=8, array_of_blocklengths=0x7fffcaa24440, array_of_displacements=0x7fffcaa243c0,
<u></u><u></u></p>
<p class="MsoNormal"> array_of_types=0x7fffcaa24410, newtype=0x12853fc)<u></u><u></u></p>
<p class="MsoNormal"> at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:164<u></u><u></u></p>
<p class="MsoNormal">#7 0x0000000000438dfb in needles::MpiMsgBasic::createMsgDataType (this=0x12853fc) at src/MsgBasic.cpp:97<u></u><u></u></p>
<p class="MsoNormal">#8 0x0000000000412b77 in needles::NeedlesMpiManager::init (this=0x12853a0, argc=23, argv=0x7fffcaa24e08, rank=20,
<u></u><u></u></p>
<p class="MsoNormal"> world_size=21) at src/NeedlesMpiManager.cpp:204<u></u><u></u></p>
<p class="MsoNormal">#9 0x000000000040605f in main (argc=23, argv=0x7fffcaa24e08) at src/NeedlesMpiManagerMain.cpp:142<u></u><u></u></p>
<p class="MsoNormal">(gdb) <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Here is my code that creates the custom type and then calls MPI_Type_commit:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> MsgBasic obj;<u></u><u></u></p>
<p class="MsoNormal"> int struct_len = 8, i;<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> int block_len[struct_len];<u></u><u></u></p>
<p class="MsoNormal"> MPI_Datatype types[struct_len];<u></u><u></u></p>
<p class="MsoNormal"> MPI_Aint displacements[struct_len];<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> i = 0;<u></u><u></u></p>
<p class="MsoNormal"> block_len[i] = 1;<u></u><u></u></p>
<p class="MsoNormal"> types[i] = MPI_LOGICAL;<u></u><u></u></p>
<p class="MsoNormal"> displacements[i] = (size_t) &obj.tuple_valid_ - (size_t) &obj;<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> ++i;<u></u><u></u></p>
<p class="MsoNormal"> block_len[i] = 1;<u></u><u></u></p>
<p class="MsoNormal"> types[i] = MPI_LOGICAL;<u></u><u></u></p>
<p class="MsoNormal"> displacements[i] = (size_t) &obj.tuple_seq_valid_ - (size_t) &obj;<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> // the int array "start_" member<u></u><u></u></p>
<p class="MsoNormal"> ++i;<u></u><u></u></p>
<p class="MsoNormal"> block_len[i] = Tuple::N_INDICES_MAX_;<u></u><u></u></p>
<p class="MsoNormal"> types[i] = MPI_SHORT;<u></u><u></u></p>
<p class="MsoNormal"> displacements[i] = (size_t) &obj.start_ - (size_t) &obj;<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> // the int array "end_" member<u></u><u></u></p>
<p class="MsoNormal"> ++i;<u></u><u></u></p>
<p class="MsoNormal"> block_len[i] = Tuple::N_INDICES_MAX_;<u></u><u></u></p>
<p class="MsoNormal"> types[i] = MPI_SHORT;<u></u><u></u></p>
<p class="MsoNormal"> displacements[i] = (size_t) &obj.end_ - (size_t) &obj;<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> // the integer "opcode_" member<u></u><u></u></p>
<p class="MsoNormal"> ++i;<u></u><u></u></p>
<p class="MsoNormal"> block_len[i] = 1;<u></u><u></u></p>
<p class="MsoNormal"> types[i] = MPI_INT;<u></u><u></u></p>
<p class="MsoNormal"> displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj;<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> // the boolean "success_" member<u></u><u></u></p>
<p class="MsoNormal"> ++i;<u></u><u></u></p>
<p class="MsoNormal"> block_len[i] = 1;<u></u><u></u></p>
<p class="MsoNormal"> types[i] = MPI_LOGICAL; // NOTE: might be MPI_BOOLEAN in later version<u></u><u></u></p>
<p class="MsoNormal"> displacements[i] = (size_t) &obj.success_ - (size_t) &obj;<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> // the double "run_time_sec_" member<u></u><u></u></p>
<p class="MsoNormal"> ++i;<u></u><u></u></p>
<p class="MsoNormal"> block_len[i] = 1;<u></u><u></u></p>
<p class="MsoNormal"> types[i] = MPI_DOUBLE;<u></u><u></u></p>
<p class="MsoNormal"> displacements[i] = (size_t) &obj.run_time_sec_ - (size_t) &obj;<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> // the char array "error_msg_" member<u></u><u></u></p>
<p class="MsoNormal"> ++i;<u></u><u></u></p>
<p class="MsoNormal"> block_len[i] = NeedlesMpi::ERROR_MSG_LEN_ + 1;<u></u><u></u></p>
<p class="MsoNormal"> types[i] = MPI_CHAR;<u></u><u></u></p>
<p class="MsoNormal"> displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj;<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> MPI_Type_create_struct(struct_len, block_len, displacements,<u></u><u></u></p>
<p class="MsoNormal"> types, &msg_data_type_);<u></u><u></u></p>
<p class="MsoNormal"> MPI_Type_commit(&msg_data_type_);<u></u><u></u></p></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"><div class="m_1986873516637256965WordSection1"><div><div style="border-style:solid none none;border-top-width:1pt;padding:3pt 0in 0in;border-top-color:rgb(225,225,225)">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Kurt<u></u><u></u></p>
<p class="MsoNormal"><b><u></u> <u></u></b></p>
<p class="MsoNormal"><b>From:</b> Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>>
<br>
<b>Sent:</b> Sunday, October 24, 2021 6:46 PM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
<b>Cc:</b> Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov" target="_blank">kurt.e.mccall@nasa.gov</a>><br>
<b>Subject:</b> [EXTERNAL] Re: Maximum number of inter-communicators?<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black">Hi Kurt,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black">There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In
older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release --
</span><a href="https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.org%2Fstatic%2Fdownloads%2F4.0a2%2Fmpich-4.0a2.tar.gz&data=04%7C01%7Ckurt.e.mccall%40nasa.gov%7Cf784f0a87c7245e8a5f808d99855e822%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637708316997034873%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=P2D25y9EReF3fLdicTKBU5N1k5tzRtAH2a9ZbOLf3cs%3D&reserved=0" target="_blank"><span style="font-size:12pt">http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz</span></a><span style="font-size:12pt;color:black">,
and see if the issue persist?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black">-- <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black">Hui<u></u><u></u></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="m_1986873516637256965divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Mccall, Kurt E. (MSFC-EV41) via discuss <</span><a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><span style="color:black">><br>
<b>Sent:</b> Sunday, October 24, 2021 2:37 PM<br>
<b>To:</b> </span><a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><span style="color:black"> <</span><a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><span style="color:black">><br>
<b>Cc:</b> Mccall, Kurt E. (MSFC-EV41) <</span><a href="mailto:kurt.e.mccall@nasa.gov" target="_blank">kurt.e.mccall@nasa.gov</a><span style="color:black">><br>
<b>Subject:</b> [mpich-discuss] Maximum number of inter-communicators?</span> <u></u>
<u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="m_1986873516637256965xmsonormal">Hi,<u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal"> <u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal">Based on a paper I read about giving an MPI job some fault tolerance, I’m exclusively connecting my processes with inter-communicators.<u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal">I’ve found that if I increase the number of processes beyond a certain point, many processes don’t get created at all and the whole job
<u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal">crashes. Am I running up against an operating system limit (like the number of open file descriptors – it is set at 1024), or some sort of
<u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal">MPICH limit?<u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal"> <u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal">If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes,
<u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal">and each manager connected to 8 worker processes on the manager’s own node. This is the largest job I’ve been able to create
<u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal">without it crashing. Attempting to increase the number of workers beyond 8 results in a crash.<u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal"> <u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal">I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0.<u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal"> <u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal">Thanks,<u></u><u></u></p>
<p class="m_1986873516637256965xmsonormal">Kurt<u></u><u></u></p>
</div>
</div>
</div>
</div>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
</blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>