<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
span.gmailsignatureprefix
{mso-style-name:gmail_signature_prefix;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Thanks for your answer Jeff.<br>
<br>
I’m, going to double check the ROCM5 build later on and get back with the results, but for now the build commands used for the cuda version are:<br>
<br>
<i><span style="font-family:"Courier New"">./configure --prefix=${install_dir}/ucx_1.14.1 --without-bfd --with-cuda=</span></i><span style="font-family:"Courier New"">
</span><code><span style="font-size:10.0pt">nvidia/Linux_x86_64/21.2/cuda/11.2/</span></code><i><span style="font-family:"Courier New""> --without-knem --without-rocm --enable-gtest --enable-examples<br>
</span></i><span style="font-family:"Courier New"">make -j16<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">make install<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I see your comment on github (<a href="https://github.com/pmodels/mpich/issues/6675">https://github.com/pmodels/mpich/issues/6675</a>) as well, I’ll change the value of <i>if-yaksa-depth
</i>and check the results. Thank you for you insight on that.<br>
<br>
Best,<br>
Lucas.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Jeff Hammond <jeff.science@gmail.com> <br>
<b>Sent:</b> Monday, 18 September 2023 13:26<br>
<b>To:</b> discuss@mpich.org<br>
<b>Cc:</b> Lucas Zacchi de Medeiros <lucasd@cadence.com><br>
<b>Subject:</b> Re: [mpich-discuss] libmpi size unusually large when building with CUDA support<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:red">EXTERNAL MAIL</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">If the CPU-only and ROCm builds are the same size, I wonder if ROCm support was compiled in at all. How did you verify that?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Can you provide the full build commands for each so I can reproduce these?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">You might see if -yaksa-depth=1 (or 0 perhaps) changes the results. There is a performance tradeoff, but if your applications don't use interesting noncontiguous datatypes, it shouldn't matter.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Jeff<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Sep 15, 2023 at 5:53 PM Lucas Zacchi de Medeiros via discuss <<a href="mailto:discuss@mpich.org">discuss@mpich.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292F">I work on a project that supports different architectures, so we've built 3 separate versions of MPICH 4.1.1 (cuda11.2, rocm5.4.3 and cpu-only with ucx1.14.1)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292F">The cpu-only and the rocm5 libraries are around 50MB each. The CUDA version on the other hand, is close to 1.8GB! There doesn’t seem to be anything out of the ordinary with the builds
and all versions are working as expected. </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292F">This is the output of </span><span style="font-size:9.0pt;font-family:Consolas;color:#24292F">du -sh</span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292F"> performed
on the install directories:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292F">First the rocm5 build:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:9.0pt;font-family:Consolas;color:#24292F;border:none windowtext 1.0pt;padding:0cm">$ du -sh LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/*</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:9.0pt;font-family:Consolas;color:#24292F;border:none windowtext 1.0pt;padding:0cm">(...)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:9.0pt;font-family:Consolas;color:#24292F;border:none windowtext 1.0pt;padding:0cm">53M LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/libmpi.a</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:9.0pt;font-family:Consolas;color:#24292F;border:none windowtext 1.0pt;padding:0cm">40M LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/libmpi.so.12.3.0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292F">And then the CUDA build:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:9.0pt;font-family:Consolas;color:#24292F;border:none windowtext 1.0pt;padding:0cm">$ du -sh LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/*</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:9.0pt;font-family:Consolas;color:#24292F;border:none windowtext 1.0pt;padding:0cm">(...)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:9.0pt;font-family:Consolas;color:#24292F;border:none windowtext 1.0pt;padding:0cm">1.8G LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/libmpi.a</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;background:white">
<span style="font-size:9.0pt;font-family:Consolas;color:#24292F;border:none windowtext 1.0pt;padding:0cm">1.7G LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/libmpi.so.12.3.0<br>
<br>
</span><span style="font-size:9.0pt;font-family:Consolas;color:#24292F"><br>
</span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292F">This issue really complicates packaging and distribution since we provide both the archive and the shared libraries, I am looking at more than 3GB for MPICH only.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292F">After some investigation, it doesn’t seem the problem is coming from our end. Is it possible that something in mpich’s build process is causing this excessive file size?<br>
<br>
Thanks for the help.<br>
<br>
Kind regards,<br>
Lucas</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:9.0pt;font-family:Consolas;color:#24292F"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://urldefense.com/v3/__https:/lists.mpich.org/mailman/listinfo/discuss__;!!EHscmS1ygiU1lA!EaOkIWaGMsFGOtnvVJqLcS20A0toTn_gIEHsyespF0xr8WerBJidCx6LZnB9gSbIy-LWgxCcNMimtEgH9E1y$" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><span class="gmailsignatureprefix">-- </span><o:p></o:p></p>
<div>
<p class="MsoNormal">Jeff Hammond<br>
<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br>
<a href="https://urldefense.com/v3/__http:/jeffhammond.github.io/__;!!EHscmS1ygiU1lA!EaOkIWaGMsFGOtnvVJqLcS20A0toTn_gIEHsyespF0xr8WerBJidCx6LZnB9gSbIy-LWgxCcNMimtJQmD4Bd$" target="_blank">http://jeffhammond.github.io/</a><o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>