<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Guilherme,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
With mpich-4.0b1, you can try building mpich using <code>--with-ch4-vci-method=implicit</code>. In addition, try pass "mpi_assert_no_any_tag" info hint during comm creation. That will allow mpich to distribute operations with different tags even within the
same communicator to different VCIs, again, in a hashing fashion.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This feature is in the experimental stage, please file GitHub issues if you encounter problems.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
It is also possible to explicitly specify which VCI you would like to use for arbitrary individual pt2pt communication by embedding the vci information in the tag. This feature is not exactly standard-conforming, thus we are only using it for experimental purposes.
You'll need to edit <code>src/mpid/ch4/src/ch4_vci.h</code> and remove the line "#error MPICH_VCI__TAG not implemented." to enable it. With the tag method, tags are restricted to 15bits, the highest 5 bits designate source vci, the middle 5 bits designate
destination vci, and the last 5 bits for user-defined purposes. The drawback, of course, is you will have much-reduced tag space for application logic.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Both the communicator, implicit, or tag vci-methods are MPI standard conformant, so you can use it with any MPI implementations without breaking your applications. But the actual performance of course will depend on the implementation.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-- <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui Zhou<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Guilherme Valarini <guilherme.a.valarini@gmail.com><br>
<b>Sent:</b> Monday, December 6, 2021 12:16 PM<br>
<b>To:</b> Zhou, Hui <zhouh@anl.gov><br>
<b>Cc:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Subject:</b> Re: [mpich-discuss] How locking on multi-VCI works</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hello Zhou,
<div><br>
</div>
<div>First of all, thanks for the help!</div>
<div><br>
</div>
<div>Let me explain a little bit my use case: my team and I have a distributed and multithreaded event system implemented on top of MPI, where multiple non-blocking MPI messages are exchanged between multiple nodes. Checking our internal performance traces,
we saw that there was some contention happening at the MPI layer, especially when many threads were being used.</div>
<div><br>
</div>
<div>Digging a little bit deeper we found some studies explaining the current multithread support of many MPI implementations and even limitations regarding the standard itself, which might explain the problems we encountered. Since each event is mapped to
a unique TAG, that would be the preferred mechanism of extracting network parallelism. But since we also want to support other MPI implementations (e.g. openmpi), we think that using multiple communicators might be a better option.</div>
<div><br>
</div>
<div>I was hoping that messages sent to two different processes but from the same process through the same VCI would use two different locks at their origin. But now I see that a VCI is directly mapped to a hardware context of some sort. So it makes sense that
the same lock would be shared between the two previously described messages.</div>
<div><br>
</div>
<div>If you have any other general hints on how to better extract network parallelism at the MPI level, I would be grateful. 😉 <br>
</div>
<div><br>
</div>
<div>Note: Sorry for the duplicate. I forgot to reply to the mailing list as well.</div>
<div><br>
</div>
<div>Thaks again for the help,</div>
<div>Guilherme Valarini</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">Em seg., 6 de dez. de 2021 às 14:34, Zhou, Hui <<a href="mailto:zhouh@anl.gov">zhouh@anl.gov</a>> escreveu:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The total number of VCIs are configured with <code>--with-ch4-max-vcis=#</code>. The maximum is 64. The default used to be 1, but it is changed to 64 in 4.0b1 release. There is also an option to control the vci assignment method:
<code>--enable-ch4-vci-method={communicator,tag,implicit}</code>. The default is
<code>communicator</code>, with which we assign vci to communicators in a round-robin fashion. If you create communicators consecutively, they are expected to have different VCIs. The other
<code>vci-method</code>s are at the experimental stage. If the communicator method is insufficient for your application, it may be worth a try. We'd like to understand your use case better before pointing you that way.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am not exactly understanding your question. The vci locks are local process locks, so if you have N VCIs, you will have N channels for each process. With
<code>vci-method=communicator</code>, the vcis are one-to-one matched, i.e. rank 1 vci 1 only communicates to rank 2 (any ranks with the same communicator) vci 1.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
-- <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hui Zhou<br>
</div>
<div id="x_gmail-m_-4176695593114987252appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-4176695593114987252divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Guilherme Valarini via discuss <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Sent:</b> Monday, December 6, 2021 10:10 AM<br>
<b>To:</b> <a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a> <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>><br>
<b>Cc:</b> Guilherme Valarini <<a href="mailto:guilherme.a.valarini@gmail.com" target="_blank">guilherme.a.valarini@gmail.com</a>><br>
<b>Subject:</b> [mpich-discuss] How locking on multi-VCI works</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Hello everyone,</div>
<div><br>
</div>
<div>I got one question regarding the multi-VCI support and possible locking contentions of MPICH on multi-threaded environments.</div>
<div><br>
</div>
<div>I understand that there is a direct mapping between a VCI and a communicator, so global locking is avoided on a multi-threaded application. But I wanted to know: how do these VCIs work? When I have N VCIs, do I have N virtual channels per rank (thus, one
global lock per VCI-rank pair) or only 2 channels at all (one lock per VCI)? I was wondering if, for example, two MPI_Sends targeting different ranks on the same comm might need to be synchronized using such a global lock.<br>
</div>
<div><br>
</div>
<div>Thanks for the help!</div>
<div>Guilherme Valarini</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>