<div dir="ltr">Thanks again, Hui. It looks like I hadn't understood the structure or delegation of responsibility between mpich and libfabric, nor the extent to which libfabric itself comprised part of the runtime interface. I'll give this a try, and I expect this knowledge will be useful to me for other MPI that use libfabric as well!<div><br></div><div>~jonathon</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 1, 2022 at 7:46 AM Zhou, Hui <<a href="mailto:zhouh@anl.gov">zhouh@anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
For libfabric, which used by ch4:ofi, you can these settings from libfabric manpages. It depends on your provider. For example,
<a href="https://ofiwg.github.io/libfabric/v1.11.1/man/fi_tcp.7.html" id="gmail-m_8176009850770590655LPlnkOWALinkPreview" target="_blank">
https://ofiwg.github.io/libfabric/v1.11.1/man/fi_tcp.7.html</a><br>
</div>
<div>
<div id="gmail-m_8176009850770590655LPBorder_GTaHR0cHM6Ly9vZml3Zy5naXRodWIuaW8vbGliZmFicmljL3YxLjExLjEvbWFuL2ZpX3RjcC43Lmh0bWw." style="width:100%;margin-top:16px;margin-bottom:16px;max-width:800px;min-width:424px">
<table id="gmail-m_8176009850770590655LPContainer591305" role="presentation" style="padding:12px 36px 12px 12px;width:100%;border:1px solid rgb(200,200,200);border-radius:2px">
<tbody>
<tr style="border-spacing:0px" valign="top">
<td style="width:100%">
<div id="gmail-m_8176009850770590655LPTitle591305" style="font-size:21px;font-weight:300;margin-right:8px;font-family:wf_segoe-ui_light,"Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;margin-bottom:12px">
<a id="gmail-m_8176009850770590655LPUrlAnchor591305" href="https://ofiwg.github.io/libfabric/v1.11.1/man/fi_tcp.7.html" style="text-decoration:none" target="_blank">fi_tcp(7) - GitHub Pages</a></div>
<div id="gmail-m_8176009850770590655LPDescription591305" style="font-size:14px;max-height:100px;color:rgb(102,102,102);font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;margin-bottom:12px;margin-right:8px;overflow:hidden">
NAME. fi_tcp - The msg sockets Fabric Provider. OVERVIEW. The tcp provider can be used on any system that supports TCP sockets. The provider is not intended to provide performance improvements over regular TCP sockets, but rather to allow developers to write,
test,and debug application code even on platforms that do not have high-performance fabric hardware.</div>
<div id="gmail-m_8176009850770590655LPMetadata591305" style="font-size:14px;font-weight:400;color:rgb(166,166,166);font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif">
<a href="http://ofiwg.github.io" target="_blank">ofiwg.github.io</a></div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<br>
<div id="gmail-m_8176009850770590655appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_8176009850770590655divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Jonathon Anderson <<a href="mailto:janderson@ciq.co" target="_blank">janderson@ciq.co</a>><br>
<b>Sent:</b> Friday, July 1, 2022 8:40 AM<br>
<b>To:</b> Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>><br>
<b>Cc:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Subject:</b> Re: [mpich-discuss] How to specify which Ethernet / IP interface to use?</font>
<div> </div>
</div>
<div>
<div dir="auto">Thank you very much for this information, Hi! Can you point me to where these environment variables are documented for future reference? I tried to find something like this and was unable.
<div dir="auto"><br>
</div>
<div dir="auto">~jonathon</div>
<div dir="auto"><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Fri, Jul 1, 2022, 7:35 AM Zhou, Hui <<a href="mailto:zhouh@anl.gov" target="_blank">zhouh@anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Jonathon,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Since the binary name is "<font size="2"><span style="font-size:11pt">mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64</span></font>", I assume it is mpich -3.4.2 configured with ch4:ofi. We no longer use mpich2 to refer to the project, just mpich.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
With ch4:ofi, try setting the following environment variables:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
FI_PROVIDER=tcp</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
FI_TCP_IFACE=eth0</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
That assumes you want to use the tcp provider and the private interface is eth0. There are other providers that you may choose and the iface selection variables will be different. You can set MPIR_CVAR_DEBUG_SUMMARY=1 to show the list of providers mpich probes
during init.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Best wishes,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-- <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hui<br>
</div>
<div id="gmail-m_8176009850770590655x_m_-5821644481285228186appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_8176009850770590655x_m_-5821644481285228186divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Jonathon Anderson via discuss <<a href="mailto:discuss@mpich.org" rel="noreferrer" target="_blank">discuss@mpich.org</a>><br>
<b>Sent:</b> Thursday, June 30, 2022 10:49 PM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" rel="noreferrer" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" rel="noreferrer" target="_blank">discuss@mpich.org</a>><br>
<b>Cc:</b> Jonathon Anderson <<a href="mailto:janderson@ciq.co" rel="noreferrer" target="_blank">janderson@ciq.co</a>><br>
<b>Subject:</b> Re: [mpich-discuss] How to specify which Ethernet / IP interface to use?</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>I'm trying to use mpich2 [1] on a cluster of compute nodes each of<br>
which has two Ethernet interfaces: one, an encumbered public /<br>
Internet interface, and another a private / intranet interface. mpich<br>
is defaulting to trying to use the public interface, which then hangs<br>
indefinitely with both nodes in my job stuck listening via UDP.<br>
<br>
I experience the same behavior whether I use mpiexec or srun --mpi=pmi2.<br>
<br>
How do I tell mpich which interface to use?<br>
<br>
Thanks!<br>
<br>
~jonathon<br>
<br>
<br>
[1]: mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64<br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org" rel="noreferrer" target="_blank">
discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
</div>
</span></font></div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>