<div style="line-height:1.7;color:#000000;font-size:14px;font-family:arial">Thanks. I found on our 16 nodes cluster, when I run another application which has more initialization parameters, hydra can only spawn 49 process (3 per node). And the segment fault happened in <span style="color: rgb(68, 68, 68); font-family: 'Open Sans', Helvetica, Arial, sans-serif; line-height: 24px;">PMI_Spawn_multiple. So I tried to use smpd, and it seems to be able to spawn more. However, when child processes increase to 150+ (almost 10 per node), it seems to run out of resources. My question is whether there is a upper limit of child processes one parent process can spawn and how can I determine the upper limit? If I want to use one parent process to spawn more processes, using its child processes to spawn descendant is the only method ( Suppose we want to spawn 160 process on 160 nodes )? </span><br><div></div><div id="divNeteaseMailCard"></div><br>At 2013-09-27 21:43:43,"Wesley&nbs
p;Bland" <wbland@mcs.anl.gov> wrote:<br> <blockquote id="isReplyContent" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">You're probably just running out of resources. 90 processes (not threads) on one node is a lot. A common use of MPI is to have one (or a few) processes per node and if you need more local instances, you investigate using additional threads (such as OpenMP).<div><br></div><div><div>Thanks,</div><div>Wesley</div><div><br></div><div><div>On Sep 26, 2013, at 9:45 PM, myself <<a href="mailto:chcdlf@126.com">chcdlf@126.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="line-height: 1.7; font-size: 14px; font-family: arial; "><div style="color: rgb(34, 34, 34); font-size: small; line-height: normal;">I'm trying to spawn several processes on two node. When the number of child processes is small, it works fine, such as less than 80. However, when I tried to spawn 90+ proc
esses, mpirun told me "Segmentation fault (core dumped)". Is something I did wrong or I didn't do?</div><div style="color: rgb(34, 34, 34); font-size: small; line-height: normal;"><br></div><div style="color: rgb(34, 34, 34); font-size: small; line-height: normal;">My MPICH compile command: </div><div style="color: rgb(34, 34, 34); font-size: small; line-height: normal;"><br></div><div style="color: rgb(34, 34, 34); font-size: small; line-height: normal;"><pre style="font-family: Menlo, Monaco, 'Courier New', monospace; padding: 8.5px; font-size: 12px; color: rgb(68, 68, 68); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right
-radius: 4px; border-bottom-left-radius: 4px; margin-top: 0px; margin-bottom: 9px; line-height: 18px; background-color: rgb(245, 245, 245); border: 1px solid rgba(0, 0, 0, 0.14902); word-break: break-all;"><code style="font-family: Menlo, Monaco, 'Courier New', monospace; padding: 0px; color: inherit; border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; background-color: transparent; border: 0px;">$ ./configure --prefix=/opt/mpich3 --with-device=ch3:nemesis --with-pm=hydra --enable-fast=none --enable-g=dbg CFLAGS=-fPIC --disable-f77 --disable-fc</code></pre><div style="color: rgb(34, 34, 34); font-size: small; line-height: normal;"><br></div>Here is my source code:</div><div style="color: rgb(34, 34, 34); font-size: small; line-height: normal;"><br></div><div style="color: rgb(34, 34, 34); font-size: small; line-height: normal;"><pre style="font-family: Menlo, Monaco, 'Courier New', monospace; padding: 8.
5px; font-size: 12px; color: rgb(68, 68, 68); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; margin-top: 0px; margin-bottom: 9px; line-height: 18px; background-color: rgb(245, 245, 245); border: 1px solid rgba(0, 0, 0, 0.14902); word-break: break-all;"><code style="font-family: Menlo, Monaco, 'Courier New', monospace; padding: 0px; color: inherit; border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; background-color: transparent; border: 0px;"> #include "mpi.h"
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main( int argc, char *argv[] ) {
int HLEN=2;
char *host[]={"node1","node2"};
char name[65];
int count;
if(argc==2)
count=atoi(argv[1]);
else
count=2;
int root=0;
char* array_of_commands[count];
int array_of_maxprocs[count];
int errcodes[count];
MPI_Info array_of_info[count];
MPI_Comm parentcomm, intercomm;
int i;
MPI_Init( &argc, &argv );
for(i=0; i< count; i++){
array_of_commands[i]="/home/mpitest/spawn";
array_of_maxprocs[i]=1;
MPI_Info_create(&array_of_info[i]);
MPI_Info_set(array_of_info[i],"host",host[i % HLEN]);
}
MPI_Comm_get_parent( &parentcomm );
if (parentcomm == MPI_COMM_NULL) {
MPI_Comm_spawn_multiple( count, array_of_commands, MPI_ARGVS_NULL /*array_of_argv*/,
array_of_maxprocs, array_of_info, root, MPI_COMM_WORLD, &intercomm, errcodes );
printf("I'm the parent.\n");
} else {
gethostname(name, sizeof(name));
printf("I'm the spawned at %s.\n",name);
}
fflush(stdout);
MPI_Finalize();
return 0;
} </code></pre><div><code style="font-family: Menlo, Monaco, 'Courier New', monospace; padding: 0px; color: inherit; border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; background-color: transparent; border: 0px;"><br></code></div></div></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span>_______________________________________________<br>discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>To manage subscription options or unsubscribe:<br><a href="https://lists.mpich.org/mailman/listinfo/discuss">https://lists.mpich.org/mailman/listinfo/discuss</a></blockquote></div><br></div></blockquote></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span>