[mpich-discuss] Process Group Collision for multiple clients from different host machines having same pid with MPI_Comm_accept

Min Si msi at anl.gov
Wed May 3 16:30:00 CDT 2017


Hi Hirak,

I created a ticket for this fix. Thanks for pointing out this problem.
https://github.com/pmodels/mpich/issues/2619

Min
On 5/3/17 11:02 AM, Balaji, Pavan wrote:
> Yup.  +1.
>
> Note that this needs to be fixed in Hydra as well, which essentially does the same thing.
>
>    -- Pavan
>
>> On May 1, 2017, at 11:52 PM, Roy, Hirak <Hirak_Roy at mentor.com> wrote:
>>
>> Hi Min,
>>   
>> I found the following piece of code in MPICH source : src/pm/util/pmiserv.c :
>>   
>>      /* We include the pid of the PMI server as a way to allow multiple
>>         PMI servers to coexist.  This is needed to support connect/accept
>>         operations when multiple mpiexec's are used, and the KVS space
>>         is served directly by mpiexec (it should really have the
>>         hostname as well, just to avoid getting the same pid on two
>>         different hosts, but this is probably good enough for most
>>         uses) */
>>     
>>      MPIU_Snprintf( (char *)(kvs->kvsname), MAXNAMELEN, "kvs_%d_%d",
>>                   (int)getpid(), kvsnum++ );
>>      kvs->pairs     = 0;
>>      kvs->lastByIdx = 0;
>>      kvs->lastIdx   = -1;
>>   
>>   
>> and in src/pm/hydra/pm/pmiserv/common.c
>>   
>>   
>> HYD_status HYD_pmcd_pmi_allocate_kvs(struct HYD_pmcd_pmi_kvs ** kvs, int pgid)
>> {
>>      HYD_status status = HYD_SUCCESS;
>>   
>>      HYDU_FUNC_ENTER();
>>      HYDU_MALLOC(*kvs, struct HYD_pmcd_pmi_kvs *, sizeof(struct HYD_pmcd_pmi_kvs), status);
>>      HYDU_snprintf((*kvs)->kvsname, PMI_MAXKVSLEN, "kvs_%d_%d", (int) getpid(), pgid);
>>      (*kvs)->key_pair = NULL;
>>   
>>    fn_exit:
>>      HYDU_FUNC_EXIT();
>>      return status;
>>   
>>    fn_fail:
>>      goto fn_exit;
>> }
>>   
>>   
>> I think hostname should be added to kvsname to make it unique when accept/connect are being done from multiple processes which are not invoked by the same mpiexec.
>>   
>>   
>> Thanks,
>> Hirak
>>   
>>   
>>   
>>   
>>   
>> Hi Hirak,
>>   
>> Before look into PMI, it would be good to first make sure if this is a
>> problem in your server-client code, or in the dynamic process part of
>> MPICH code. Could you please reproduce this issue with a simple program
>> and give it to us ?
>>   
>> One thing I noticed is that the server program is multithreaded. Are you
>> using multiple threads to accept client connection ? Anyway, a
>> reproducer program will be great.
>>   
>> Please also try to use the latest MPICH release and see if it happens.
>>   
>> In summary, it would be great if you can send us the following files.
>> - A reproducer program
>> - MPICH's config.log (you can find it in the directory where you build
>> MPICH)
>>   
>> Thanks,
>> Min
>>   
>> On 4/14/17 1:15 AM, Roy, Hirak wrote:
>>>
>>> Dear MPICH team,
>>>
>>> We use MPICH for a server-client application. We use MPICH-3.0.4 with
>>> sock channel.
>>> In this application there is one server and 100 clients.
>>> Each client is launched independently in different host-machines using
>>> individual-wrapper scripts. (we explicitly use : mpiexec -n 1 )
>>>
>>> The server is multithreaded and it uses MPI_Comm_accept (on
>>> MPI_COMM_SELF) and clients use MPI_Comm_connect to connect.
>>> We have observed the following issue after all the clients connect to
>>> server :
>>>   if we send message to a client (lets say 'm'), it reaches
>>> unexpectedly to some other client (lets say 'n'). { server sends the
>>> message using the communicator returned by accept call }. This happens
>>> randomly in one out of 5-6 runs.
>>>
>>> On further looking into MPICH code, we found that
>>> 1) There is a collsion of pg (process-group) of two processes (m and
>>> n) after mpi-comm-accept
>>> 2) As a result of (1), comm->vc are same (for m and n, although comm
>>> are different). It seems that the unique string (something like
>>> kva_<int>_int) is not unique for such two processes. 'm' and 'n'
>>> processes are running in different host-machine and they have the same
>>> pid. The kva string looked like kva_pid_rank.
>>>
>>>
>>> We have the following questions :
>>> 1) Have we built MPICH with some kind of incorrect
>>> configuration (hydra configuration at the end of the email) ?
>>> 2) Are we using incorrect process-manager or configuration and that is
>>> why there is a possible collision of process-groups?
>>> 3) What is the purpose of process group sharing/uniquifying? If there
>>> is no real reason for this, could it be disabled or will something
>>> else rely on the id string being unique?
>>> 4) If there are no other work-around, what could be done to make the
>>> id string unique? Add the host-name? Will everything else be ok with this?
>>>
>>>
>>> It would be good if you can let us know if there is any workaround for
>>> this issue or not.
>>>
>>>
>>> Thanks,
>>> Hirak Roy
>>>
>>> HYDRA build details:
>>>      CXX:                             no  -O3 -fPIC
>>>      F77:                             no
>>>      F90:                             no
>>>      Configure options: '--disable-option-checking'
>>> '--prefix=/home/hroy/local/mpich-3.0.4/linux_x86_64' '--disable-f77'
>>> '--disable-fc' '--disable-f90modules' '--disable-cxx'
>>> '--enable-fast=nochkmsg' '--enable-fast=notiming'
>>> '--enable-fast=ndebug' '--enable-fast=O3' '--with-device=ch3:sock'
>>> 'CFLAGS=-O3 -fPIC -O3' 'CXXFLAGS=-O3 -fPIC'
>>> 'CC=/u/prod/gnu/gcc/20121129/gcc-4.5.0-linux_x86_64/bin/gcc' 'LDFLAGS=
>>> ' '--cache-file=/dev/null' '--srcdir=.' 'LIBS=-lrt -lpthread'
>>> 'CPPFLAGS= -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpl/include
>>> -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpl/include
>>> -I/home/hroy/tools/mpich/mpich-3.0.4/src/openpa/src
>>> -I/home/hroy/tools/mpich/mpich-3.0.4/src/openpa/src
>>> -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpi/romio/include'
>>>      Process Manager:                         pmi
>>>      Launchers available:                     ssh rsh fork slurm ll lsf
>>> sge manual persist
>>>      Topology libraries available:            hwloc
>>>      Resource management kernels available:   user slurm ll lsf sge pbs
>>> cobalt
>>>      Checkpointing libraries available:
>>>      Demux engines available:                 poll select
>>>
>>>
>>>
>>   
>> _______________________________________________
>> discuss mailing list     discuss at mpich.org
>> To manage subscription options or unsubscribe:
>> https://lists.mpich.org/mailman/listinfo/discuss
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss


More information about the discuss mailing list