<div dir="ltr">Sorry for the late reply,<div><br></div><div>I tried using that option, but the outcome was the same.</div><div>However, with the info key "hostfile" and creating a file with the the following text, it works as expected. In this example, creating 10 and 5 processes respectively in n00 and n01.</div><div><br></div><div>n00:10</div><div>n01:5</div><div><br></div><div>When trying this key, I noticed that if the parent processes try to finish their execution while their sons do some work, the parents are blocked in MPI_Finalize until the sons call this function too. </div><div>This happens even if for both groups all their processes call MPI_Comm_disconnect. I think this is happening because both groups are still connected and therefore the parents wait until the sons call this function.</div><div>Could it be that I am missing something?</div><div><br></div><div>I based my conclusión on what is told here:</div><div><a href="https://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node226.htm">https://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node226.htm</a><br></div><div><br></div><div>Thanks, Iker</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El jue, 4 mar 2021 a las 18:31, Raffenetti, Kenneth J. (<<a href="mailto:raffenet@mcs.anl.gov">raffenet@mcs.anl.gov</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I notice in the output that the sbatch version uses --launcher slurm while the interactive version uses --launcher ssh. Can you try adding --launcher ssh to the mpiexec command of you sbatch script and see if it makes a difference?<br>
<br>
Ken<br>
<br>
On 3/1/21, 1:38 PM, "Iker Martín Álvarez" <<a href="mailto:martini@uji.es" target="_blank">martini@uji.es</a>> wrote:<br>
<br>
Hi Ken,<br>
Thanks for your reply.<br>
Here are attached two files with the arg "-v". BatchOutput.txt has the output which is giving the error message when the code is executed with the command sbatch, while the file InteractiveOutput.txt is the one executed interactively in the node and work as expected. Both of them has been compiled and executed with mpich 3.4.1.<br>
<br>
Thanks, Iker<br>
<br>
<br>
El vie, 26 feb 2021 a las 16:15, Raffenetti, Kenneth J. (<<a href="mailto:raffenet@mcs.anl.gov" target="_blank">raffenet@mcs.anl.gov</a>>) escribió:<br>
<br>
<br>
Hi,<br>
<br>
Could you add "-v" to your mpiexec command and provide the output? The "host" info key is handled by the process manager when executing the spawn command.<br>
<br>
Ken<br>
<br>
On 2/24/21, 11:55 AM, "Iker Martín Álvarez via discuss" <<a href="mailto:discuss@mpich.org" target="_blank">discuss@mpich.org</a>> wrote:<br>
<br>
Hi,<br>
I have been working around the MPI_Comm_spawn function with the key "host" along a value in a system with the SLURM resource manager (slurm-wlm 17.11.2). The function works as expected, but when I send to execute the code with the sbatch command, an error arises. This does not happen when I execute directly the code in the same machine SLURM decided to execute it when it was sended with sbatch. In both cases with the key "host", as when I do not use the key, it works just fine.<br>
<br>
The same code has been tested with MPICH 3.3.2 and 3.4.1, which gives different errors. Also, I tried it with other implementations (OpenMPI and Intel MPI), which works as expected creating the processes in the indicated host.<br>
<br>
I would like to create processes by MPI_Comm_spawn in an assigned host, so if there are other key values for the Info argument, I could try them, but I have not found any looking through MPICH documentation.<br>
<br>
Here is the code I have been using:<br>
<br>
int main(int argc, char ** argv) {<br>
<br>
int myId, numP;<br>
MPI_Info info;<br>
MPI_Comm comm;<br>
<br>
MPI_Init(&argc, &argv);<br>
MPI_Comm_rank(MPI_COMM_WORLD, &myId);<br>
MPI_Comm_size(MPI_COMM_WORLD, &numP); <br>
<br>
MPI_Comm comm_par;<br>
MPI_Comm_get_parent(&comm_par);<br>
if(comm_par != MPI_COMM_NULL ) {<br>
if(myId == ROOT) {<br>
printf("SONS\n"); fflush(stdout);<br>
}<br>
} else {<br>
if(myId == ROOT) {<br>
printf("FATHERS\n"); fflush(stdout);<br>
}<br>
MPI_Info_create(&info);<br>
MPI_Info_set(info, "host", "n00");<br>
MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numP, info, ROOT, MPI_COMM_WORLD, &comm, MPI_ERRCODES_IGNORE);<br>
}<br>
MPI_Finalize();<br>
return 0;<br>
}<br>
<br>
<br>
For MPICH 3.4.1 there is sometimes no error and the code stops working at MPI_Comm_spawn function, other times this error is shown:<br>
Abort(1615120) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Internal MPI error!, error stack:<br>
MPIR_Init_thread(152)...:<br>
MPID_Init(562)..........:<br>
MPIDU_Init_shm_init(195):<br>
Init_shm_barrier(94)....: Internal MPI error! barrier not initialized<br>
<br>
<br>
Also, the error code for MPICH 3.3.2:<br>
<br>
Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size<br>
Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size<br>
Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
libbacktrace: no debug info in ELF executable<br>
internal ABORT - process 0<br>
<br>
Thanks, Iker<br>
<br>
<br>
<br>
</blockquote></div>