<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannervukfxpy" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannervukfxpy" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannervukfxpy" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannervukfxpy" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannervukfxpy { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonvukfxpy:hover, .pfptPrimaryButtonvukfxpy:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonvukfxpy:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Abadi;
panose-1:2 11 6 4 2 1 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Aptos",sans-serif;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Zhou, Hui <zhouh@anl.gov><br>
<b>Date: </b>Wednesday, June 12, 2024 at 1:15</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">PM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>Niyaz Murshed <Niyaz.Murshed@arm.com>, nd <nd@arm.com><br>
<b>Subject: </b>Re: Mpich over RDMA sample<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you
can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-- <br>
Hui Zhou</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Niyaz Murshed via discuss <discuss@mpich.org><br>
<b>Date: </b>Wednesday, June 12, 2024 at 9:03</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>discuss@mpich.org <discuss@mpich.org><br>
<b>Cc: </b>Niyaz Murshed <Niyaz.Murshed@arm.com>, nd <nd@arm.com><br>
<b>Subject: </b>[mpich-discuss] Mpich over RDMA sample</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:1.0pt;color:white">Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/
--with-ofi=/opt/libfabric/ When I run any applications between 2 directly </span>
<span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div style="border:none;border-top:solid #90A4AE 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#D0D8DC!important;border-top:#90a4ae!important" id="pfptBannerou1nuhm">
<div id="pfptBannerou1nuhm">
<div id="pfptBannerou1nuhm">
<p class="MsoNormal" style="margin-left:.5in;line-height:13.5pt;background:#D0D8DC">
<b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div id="pfptBannerou1nuhm">
<p class="MsoNormal" style="margin-left:.5in;line-height:13.5pt;background:#D0D8DC">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#D0D8DC"><span style="font-size:12.0pt;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">Hello, </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">I am trying to learn about MPICH and its performance over RDMA.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">I am using libfabric and installed mpich using the below configure.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">./</span><b><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:#CA3323">configure</span></b><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black"> --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and
not over RoCE.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">Is there any way to test commination over RoCE ?</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter?</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">Server : </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1 </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">Client :</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:14.0pt;font-family:"Abadi",sans-serif;color:black">fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1 </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;color:black">Regards,</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;color:black">Niyaz</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>