<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-size:12pt"><span style="font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">Thanks Rajeev, is this going to be using rendezvous protocol, in which case the receiver has to receive an envelope and ping back. In that case, it may
make sense to me? Does mpich have a way of controlling message sizes for eager/rendezvous as intel MPI does? I havn't found any environment variables.<br>
</span></span>
<div style="font-size:12pt"><span style="font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)"><br>
</span></div>
<div style="font-size:12pt">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">Thanks,</div>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">Steve</span></div>
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Thakur, Rajeev <thakur@anl.gov><br>
<b>Sent:</b> Monday, June 28, 2021 9:13 PM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Hudson, Stephen Tobias P <shudson@anl.gov><br>
<b>Subject:</b> Re: [mpich-discuss] Iprobe does not find all messages.</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:"Lucida Grande"}
@font-face
{}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle19
{font-family:"Lucida Grande",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="font-family:"Lucida Grande",sans-serif">I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress.</span></p>
<p class="x_MsoNormal"><span style="font-family:"Lucida Grande",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-family:"Lucida Grande",sans-serif">Rajeev</span></p>
<p class="x_MsoNormal"><span style="font-family:"Lucida Grande",sans-serif"> </span></p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b><span style="font-size:12.0pt; color:black">From: </span>
</b><span style="font-size:12.0pt; color:black">"Hudson, Stephen Tobias P via discuss" <discuss@mpich.org><br>
<b>Reply-To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Date: </b>Monday, June 28, 2021 at 4:30 PM<br>
<b>To: </b>"discuss@mpich.org" <discuss@mpich.org><br>
<b>Cc: </b>"Hudson, Stephen Tobias P" <shudson@anl.gov><br>
<b>Subject: </b>[mpich-discuss] Iprobe does not find all messages.</span></p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Hi, </span></p>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">TLDR: Why does iprobe not pick up messages that should be there?</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers.
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent
several messages.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">loop:</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> 1. receive_func (loop over all workers doing iprobe - until none have anything)</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> 2. do some other stuff.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">(actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py#L346)</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call
to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does).</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put
a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe?</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages).</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake?</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated,
then okay.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">I am running on a Ubuntu laptop with $ mpirun --version</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">HYDRA build details:</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> Version: 3.2.1</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Thanks,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Steve</span></p>
</div>
</div>
</div>
</body>
</html>