<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><style>body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }div.foxdiv20170420231755884667 { }body { font-size: 10.5pt; font-family: 微软雅黑; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>
<div><span></span>I notice that after sending that email. </div><div>Sorry for that. </div>
<div><br></div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><div>jt.meng@siat.ac.cn</div></div></span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; PADDING-TOP: 8px"><div><b>From:</b> <a href="mailto:jeff.science@gmail.com">Jeff Hammond</a></div><div><b>Date:</b> 2017-04-20 23:05</div><div><b>To:</b> <a href="mailto:discuss@mpich.org">MPICH</a></div><div><b>Subject:</b> Re: [mpich-discuss] Limitation of open files on Mira</div></div></div><div><div class="FoxDiv20170420231755884667"><div dir="ltr">This is the wrong email list for such a question. Please contact ALCF support regarding Mira.<div><br></div><div>Jeff</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 20, 2017 at 1:35 AM, <a href="mailto:jt.meng@siat.ac.cn">jt.meng@siat.ac.cn</a> <span dir="ltr"><<a href="mailto:jt.meng@siat.ac.cn" target="_blank">jt.meng@siat.ac.cn</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div>
<div><span></span>Hi Rob, </div><div> Can we increase the number of open files on Mira to 1 million? <span style="font-size:10.5pt;line-height:1.5;background-color:window">Currently the number of open files is set to be 65536, and users can increase it manually to its hard limitation of 81920. </span></div><div> Can we increase this hard limitation to 1 million for Mira for a one week test on MPI IO in Mira? Thanks. </div><div><br></div><div>---------------<span style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)"> /etc/security/<wbr>limits.conf</span>-------------------<wbr>--------------</div><div><span style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)">root soft <wbr>nofile 65536<br>root hard <wbr>nofile 81920 <--</span></div>
<div><br></div><div>Best, </div><hr style="width:210px;height:1px" color="#b5c4df" size="1" align="left">
<div><span><div style="MARGIN:10px;FONT-FAMILY:verdana;FONT-SIZE:10pt"><div><a href="mailto:jt.meng@siat.ac.cn" target="_blank">jt.meng@siat.ac.cn</a></div></div></span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div style="PADDING-RIGHT:8px;PADDING-LEFT:8px;FONT-SIZE:12px;FONT-FAMILY:tahoma;COLOR:#000000;BACKGROUND:#efefef;PADDING-BOTTOM:8px;PADDING-TOP:8px"><div><b>From:</b> <a href="mailto:robl@mcs.anl.gov" target="_blank">Latham, Robert J.</a></div><div><b>Date:</b> 2017-04-19 22:35</div><div><b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a></div><div><b>Subject:</b> Re: [mpich-discuss] Installation on Blue Gene/Q</div></div></div><div><div>On Wed, 2017-04-19 at 13:58 +0200, pramod kumbhar wrote:</div>
<div>> Hi Rob,</div>
<div>> </div>
<div>> Is this already installed/avaialble on MIRA? I would like to try this</div>
<div>> to check some mpi i/o hints.</div>
<div> </div>
<div>Yes, there are several versions available:</div>
<div> </div>
<div>manual progress optimized:</div>
<div>/projects/aurora_app/mpich3-<wbr>ch4-ofi/install/gnu/bin</div>
<div> </div>
<div>auto progress optimized:</div>
<div>/projects/aurora_app/mpich3-<wbr>ch4-ofi/install-auto/gnu/bin</div>
<div> </div>
<div>manual progress debug:</div>
<div>/projects/aurora_app/mpich3-<wbr>ch4-ofi/install/gnu.debug/bin</div>
<div> </div>
<div>auto progress debug:</div>
<div>/projects/aurora_app/mpich3-<wbr>ch4-ofi/install-auto/gnu.<wbr>debug/bin</div>
<div> </div>
<div>There are some performance considerations (both positive and negative)</div>
<div>to consider:</div>
<div> </div>
<div>Current performance improvements versus [IBM's vendor-supplied MPICH]:</div>
<div> </div>
<div>1.) Pt2pt latency - better than 2x speedup for small messages, large</div>
<div>ones with few ranks-per-node less so due to detriment #2 listed below.</div>
<div> </div>
<div>2.) RMA latency - exponentially better for small messages, then becomes</div>
<div>gradually even for large messages in the case of put and get.</div>
<div> </div>
<div>Current performance detriments versus [IBM's vendor-supplied MPICH]:</div>
<div> </div>
<div>1.) No optimized collectives -- all collectives are pt2pt algorithms in</div>
<div>MPICH CH4. There is a plan to develop optimized collectives via OFI</div>
<div>triggered operations but that work wouldn't begin until the April-May</div>
<div>timeframe at the earliest. There is also a plan to make optimized</div>
<div>pt2pt collectives available sooner.</div>
<div> </div>
<div>2.) Only 1 rget injection fifo utilized per rank to pt2pt communication</div>
<div>--- pami can utilize up all 11 to maximize internode bandwidth, you</div>
<div>would mainly see this have an impact if you are running < 8 ranks per</div>
<div>node and passing large messages.</div>
<div> </div>
<div>==rob</div>
<div>______________________________<wbr>_________________</div>
<div>discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a></div>
<div>To manage subscription options or unsubscribe:</div>
<div><a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/<wbr>mailman/listinfo/discuss</a></div>
</div></blockquote>
</div><br>______________________________<wbr>_________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/<wbr>mailman/listinfo/discuss</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div>
</div></div></blockquote>
</body></html>