<div dir="ltr">Corrected error message as below: <br><table border="1" cellspacing="0" cellpadding="0" style="border-collapse:collapse;border:none"><tbody><tr><td width="208" valign="top" style="width:155.8pt;border:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal"> <u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;padding:0in 5.4pt"><p class="MsoNormal">mt<u></u><u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;padding:0in 5.4pt"><p class="MsoNormal">Dp<u></u><u></u></p></td></tr><tr><td width="208" valign="top" style="width:155.8pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt"><p class="MsoNormal">Mpich 3.2.1<u></u><u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal">pass<u></u><u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal">Pass<u></u><u></u></p></td></tr><tr><td width="208" valign="top" style="width:155.8pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt"><p class="MsoNormal">Mpich 3.3.2<u></u><u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal">Fail</p><p class="MsoNormal">error: executing task of job 490150 failed: execution daemon on host "host1" didn't accept task<br></p></td><td width="208" valign="top" style="width:155.85pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal">Fail<u></u><u></u></p><p class="MsoNormal"></p><p class="MsoNormal">[proxy:0:0@host1] HYDU_sock_write (utils/sock/sock.c:289): write error ( Bad file descriptor)<br></p><p class="MsoNormal"><u></u></p><p class="MsoNormal"> 2 [proxy:0:0@host1] main (pm/pmiserv/pmip.c:189): unable to send the proxyID to the server</p></td></tr></tbody></table></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 28, 2020 at 9:40 PM Shuwei Zhao <<a href="mailto:shuweizhao1991@gmail.com">shuweizhao1991@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><p class="MsoNormal">Hi<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I was trying to upgrade our mpich version from 3.2.1 to 3.3.2 to consume the latest stable version of mpich, however the new mpich version run get failed. <u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">We have 2 parallel environment configuration as below:<u></u><u></u></p><table border="1" cellspacing="0" cellpadding="0" style="border-collapse:collapse;border:none"><tbody><tr><td width="312" valign="top" style="width:233.75pt;border:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal">mt<u></u><u></u></p></td><td width="312" valign="top" style="width:233.75pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;padding:0in 5.4pt"><p class="MsoNormal">dp<u></u><u></u></p></td></tr><tr><td width="312" valign="top" style="width:233.75pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt"><p class="MsoNormal">pe_name mt<u></u><u></u></p><p class="MsoNormal">slots 2000000<u></u><u></u></p><p class="MsoNormal">used_slots 1181<u></u><u></u></p><p class="MsoNormal">bound_slots 0<u></u><u></u></p><p class="MsoNormal">user_lists NONE <u></u><u></u></p><p class="MsoNormal">xuser_lists NONE <u></u><u></u></p><p class="MsoNormal">start_proc_args /bin/true<u></u><u></u></p><p class="MsoNormal">stop_proc_args /bin/true<u></u><u></u></p><p class="MsoNormal">per_pe_task_prolog NONE<u></u><u></u></p><p class="MsoNormal">per_pe_task_epilog NONE<u></u><u></u></p><p class="MsoNormal">allocation_rule $pe_slots<u></u><u></u></p><p class="MsoNormal"><span style="background:yellow">control_slaves FALSE</span><u></u><u></u></p><p class="MsoNormal">job_is_first_task FALSE<u></u><u></u></p><p class="MsoNormal">urgency_slots min<u></u><u></u></p><p class="MsoNormal">accounting_summary FALSE<u></u><u></u></p><p class="MsoNormal">daemon_forks_slaves FALSE<u></u><u></u></p><p class="MsoNormal"><span style="background:yellow">master_forks_slaves TRUE</span><u></u><u></u></p></td><td width="312" valign="top" style="width:233.75pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal">pe_name dp<u></u><u></u></p><p class="MsoNormal">slots 10000<u></u><u></u></p><p class="MsoNormal">used_slots 0<u></u><u></u></p><p class="MsoNormal">bound_slots 0<u></u><u></u></p><p class="MsoNormal">user_lists NONE <u></u><u></u></p><p class="MsoNormal">xuser_lists NONE <u></u><u></u></p><p class="MsoNormal">start_proc_args /bin/true<u></u><u></u></p><p class="MsoNormal">stop_proc_args /bin/true<u></u><u></u></p><p class="MsoNormal">per_pe_task_prolog NONE<u></u><u></u></p><p class="MsoNormal">per_pe_task_epilog NONE<u></u><u></u></p><p class="MsoNormal">allocation_rule $round_robin<u></u><u></u></p><p class="MsoNormal"><span style="background:yellow">control_slaves TRUE</span><u></u><u></u></p><p class="MsoNormal">job_is_first_task FALSE<u></u><u></u></p><p class="MsoNormal">urgency_slots min<u></u><u></u></p><p class="MsoNormal">accounting_summary FALSE<u></u><u></u></p><p class="MsoNormal">daemon_forks_slaves FALSE<u></u><u></u></p><p class="MsoNormal"><span style="background:yellow">master_forks_slaves FALSE</span><u></u><u></u></p></td></tr></tbody></table><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Running command:<u></u><u></u></p><p class="MsoNormal">We are using qsub to submit workers and master-worker connection will be established using MPI_COMM_ACCEPT and MPI_COMM_CONNECT<u></u><u></u></p><p class="MsoNormal">qsub -P bnormal -pe mt 1 -e sge_err -o sge_out mpiexec -n 1 /path/to/my/binary binary_arguments<u></u><u></u></p><p class="MsoNormal">qsub -P bnormal -pe dp 1 -e sge_err -o sge_out mpiexec -n 1 /path/to/my/binary binary_arguments<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Running result:<u></u><u></u></p><table border="1" cellspacing="0" cellpadding="0" style="border-collapse:collapse;border:none"><tbody><tr><td width="208" valign="top" style="width:155.8pt;border:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal"><u></u> <u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;padding:0in 5.4pt"><p class="MsoNormal">mt<u></u><u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;padding:0in 5.4pt"><p class="MsoNormal">Dp<u></u><u></u></p></td></tr><tr><td width="208" valign="top" style="width:155.8pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt"><p class="MsoNormal">Mpich 3.2.1<u></u><u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal">pass<u></u><u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal">Pass<u></u><u></u></p></td></tr><tr><td width="208" valign="top" style="width:155.8pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt"><p class="MsoNormal">Mpich 3.3.2<u></u><u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal">Fail</p><p class="MsoNormal">[proxy:0:0@host1] HYDU_sock_write (utils/sock/sock.c:289): write error ( Bad file descriptor)<u></u><u></u></p><p class="MsoNormal"> 2 [proxy:0:0@host1] main (pm/pmiserv/pmip.c:189): unable to send the proxyID to the server<u></u><u></u></p></td><td width="208" valign="top" style="width:155.85pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt"><p class="MsoNormal">Fail<u></u><u></u></p><p class="MsoNormal">error: executing task of job 490150 failed: execution daemon on host "host1" didn't accept task<u></u><u></u></p></td></tr></tbody></table><p class="MsoNormal"><u></u> <u></u></p>Could you please explain why the run get failed in mpich-3.3 and any solution we can use to get the run pass with mpich-3.3.2? <div><br><p class="MsoNormal">Appreciate any help.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Thanks</p></div></div>
</blockquote></div>