<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Sajid,
<div class=""><br class="">
</div>
<div class="">You are correct, by default MPICH 3.3 uses ch3:nemesis:tcp. If you have Mellanox IB and you want to use that you have to specify `—with-device=ch4:ucx` at MPICH configuration time. As for why you see that drop in bandwidth for some large message
sizes it’s difficult to pin-point the reason by just looking at the config.log. For large messages (you can probably look up the threshold from the CH3 code, I think it’s either 16 or 32 KBs) MPICH will switch from EAGER to RENDEZVOUS protocol, which requires
some additional handshaking between sender and receiver before data can be actually transferred; This might impact performance. </div>
<div class=""><br class="">
</div>
<div class="">One thing that surprises me more however is that you have much better latency and bandwidth for small messages in MPICH compared with other MPI libraries. I would make sure you are running on separate nodes when using MPICH and not having your
processes on the same node instead. There was a bug in MPICH hydra with slurm process manager for some time that would silently fail to detect separate nodes depending on whether they had some hostname that was not recognized properly.</div>
<div class=""><br class="">
</div>
<div class="">Hope this helps, </div>
<div class=""><br class="">
<div class="">
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;">
Giuseppe Congiu</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;">
Postdoctoral Appointee</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;">
MCS Division</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;">
Argonne National Laboratory</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;">
9700 South Cass Ave., Lemont, IL 60439</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Nov 26, 2019, at 12:55 PM, Sajid Ali via discuss <<a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<p style="margin:0px 0px 1.2em" class="">Hi MPICH-developers, </p>
<p style="margin:0px 0px 1.2em" class="">I’ve been trying to benchmark mpi software currently available on my univ. cluster with OSU micro benchmarks. I used this script to build and run the benchmark :
<code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline" class="">
<a href="https://github.com/s-sajid-ali/nu-quest-mpi/blob/master/curr_mpi_bench/submit.sh" target="_blank" class="">https://github.com/s-sajid-ali/nu-quest-mpi/blob/master/curr_mpi_bench/submit.sh</a></code></p>
<p style="margin:0px 0px 1.2em" class="">And here is a plot of the results I obtained :<br class="">
<span id="cid:ii_k3dgygab0"><curr_mpi.png></span></p>
<p style="margin:0px 0px 1.2em" class="">Could someone give me a clue as to why the behaviour of mpich bandwidth vs latency is so odd ? Why does the bandwidth increase first and then fall off at high message sizes?
</p>
<p style="margin:0px 0px 1.2em" class="">I’m also attaching the config.log for this particular mpich build which seems to indicate that MPICH was built with ch3 and nemesis:tcp. Does this mean that it didn’t pick up the infiniband libraries or that it is somehow
using IPoIB? Is there a way to get this information ?</p>
<p style="margin:0px 0px 1.2em" class="">Thanks in advance for the help! <br class="">
<br class="">
<br class="">
--<br class="">
Sajid Ali | PhD Candidate<br class="">
Applied Physics<br class="">
Northwestern University<br class="">
<a href="http://s-sajid-ali.github.io/" target="_blank" class="">s-sajid-ali.github.io</a></p>
</div>
<span id="cid:f_k3g7yuzc0"><config.log></span>_______________________________________________<br class="">
discuss mailing list <a href="mailto:discuss@mpich.org" class="">discuss@mpich.org</a><br class="">
To manage subscription options or unsubscribe:<br class="">
<a href="https://lists.mpich.org/mailman/listinfo/discuss" class="">https://lists.mpich.org/mailman/listinfo/discuss</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>