<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Michka,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Which libfabric version are you using? Could you try setting `MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG=` to see if there is more debug messages?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-- <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hui Zhou<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Michka Popoff via discuss <discuss@mpich.org><br>
<b>Sent:</b> Saturday, May 1, 2021 4:33 PM<br>
<b>To:</b> discuss@mpich.org <discuss@mpich.org><br>
<b>Cc:</b> Michka Popoff <michkapopoff@gmail.com><br>
<b>Subject:</b> [mpich-discuss] Mpich failure with external libfabric on macOS: OFI resource bind failed</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">Hi
<div class=""><br class="">
</div>
<div class="">Homebrew maintainer here (<a href="https://github.com/Homebrew" class="">https://github.com/Homebrew</a>).</div>
<div class="">Homebrew ships mpich as a package on both MacOS and Linux.</div>
<div class=""><br class="">
</div>
<div class="">The issue below was found with version 3.4.1 but might have been there for longer.</div>
<div class=""><br class="">
</div>
<span class="">We noticed that mpich was building it’s own internal libfabric dependency.<br class="">
</span><span class="">After reading <a href="https://lists.mpich.org/pipermail/discuss/2021-January/006092.html" class="">https://lists.mpich.org/pipermail/discuss/2021-January/006092.html</a>,<br class="">
</span><span class="">we added `--with-device=ch4:ofi` and set the libfabric path with the --with-libfabric= flag,<br class="">
</span><span class="">to use our own version.<br class="">
</span><span class=""><br class="">
</span>
<div class=""><font color="#000000" class="">The build is fine. We have a small test to check if mpich is still working fine;</font></div>
<div class=""><font color="#000000" class=""><br class="">
</font></div>
<div class=""><font color="#000000" class="">
<div class="">#include <mpi.h></div>
<div class="">#include <stdio.h></div>
<div class="">int main()</div>
<div class="">{</div>
<div class=""> int size, rank, nameLen;</div>
<div class=""> char name[MPI_MAX_PROCESSOR_NAME];</div>
<div class=""> MPI_Init(NULL, NULL);</div>
<div class=""> MPI_Comm_size(MPI_COMM_WORLD, &size);</div>
<div class=""> MPI_Comm_rank(MPI_COMM_WORLD, &rank);</div>
<div class=""> MPI_Get_processor_name(name, &nameLen);</div>
<div class=""> printf("[%d/%d] Hello, world! My name is %s.\\n", rank, size, name);</div>
<div class=""> MPI_Finalize();</div>
<div class=""> return 0;</div>
<div class="">}</div>
<div class=""><br class="">
</div>
<div class="">Executing the test fails with a weird error:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">/usr/local/Cellar/mpich/3.4.1_2/bin/mpicc hello.c -o hello</div>
<div class="">./hello</div>
<div class="">Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack:</div>
<div class="">MPIR_Init_thread(152).......: </div>
<div class="">MPID_Init(597)..............: </div>
<div class="">MPIDI_OFI_mpi_init_hook(674): </div>
<div class="">create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM)</div>
<div class="">[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247</div>
<div class="">:</div>
<div class="">system msg for write_line failure : Bad file descriptor</div>
<div class="">Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack:</div>
<div class="">MPIR_Init_thread(152).......: </div>
<div class="">MPID_Init(597)..............: </div>
<div class="">MPIDI_OFI_mpi_init_hook(674): </div>
<div class="">create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM)</div>
<div class="">[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247</div>
<div class="">:</div>
<div class="">system msg for write_line failure : Bad file descriptor</div>
</div>
<div class=""><br class="">
</div>
<div class="">This test passes fine on Linux, and fails only on MacOS. Using the internal libfabric is fine on both platforms.</div>
<div class=""><br class="">
</div>
<div class="">Here is the related discussion:</div>
<div class=""><a href="https://github.com/Homebrew/homebrew-core/pull/73062" class="">https://github.com/Homebrew/homebrew-core/pull/73062</a></div>
<div class=""><br class="">
</div>
<div class="">Maybe you could help us debug this issue?</div>
<div class=""><br class="">
</div>
<div class="">Regards</div>
<div class=""><br class="">
</div>
<div class="">Michka</div>
<div class=""><br class="">
</div>
</font></div>
</div>
</body>
</html>