[mpich-discuss] mpich hangs in MPI_Send() - multi threaded environment

Sat Jul 18 10:17:37 CDT 2015

Jeff,

That's not needed if the receive is being called from a different thread, like the case described below.

 -- Pavan

From: Jeff Hammond
Reply-To: "discuss at mpich.org<mailto:discuss at mpich.org>"
Date: Friday, July 17, 2015 at 11:03 PM
To: "discuss at mpich.org<mailto:discuss at mpich.org>"
Cc: Gary Funck
Subject: Re: [mpich-discuss] mpich hangs in MPI_Send() - multi threaded environment

Blocking Send to self will hang if Irecv not pre-posted. You doing that?

Jeff

On Friday, July 17, 2015, Nenad Vukicevic <nenad at intrepid.com<mailto:nenad at intrepid.com>> wrote:
I will via separate mail.  But, I see that I made a mistake in my description as pthread does MPI_Send () to itself and not to another rank.
I'll try reversing the order (send to others then to yourself).

On 7/17/2015 4:26 PM, Balaji, Pavan wrote:
You should not need to pass any additional configuration options.  MPICH is thread-safe by default.

Can you send us a simple program that reproduces the error?

   -- Pavan

On 7/17/15, 6:16 PM, "Nenad Vukicevic" <nenad at intrepid.com> wrote:

I am having a problem where the system locks up inside the MPI_Send()
routine.  In my test, each MPI rank has an additional pthread and system
locks up when:

- main thread does MPI_Recv from ANY rank
- pthread does MPI_Send to another rank

I verified with MPI_Init_thread() that I can run MPI_THREAD_MULTIPLE
environment.

The same thing happened on MPICH with Fedora 20 (3.0.4) and the one
build from 3.2b3 sources.  When building from sources I provided
'--enable-threads=multiple' option.  I also tried to play with
'--enable-thread-cs' option but got build failure when 'per-object' was
selected.

Is this supposed to work?

Thanks,
Nenad

I am attaching traces form GDB for the rank that locks up.

(gdb) info thread
   Id   Target Id         Frame
   2    Thread 0x7ffff6a5a700 (LWP 29570) "barrier_test"
0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from
/usr/lib64/libpthread.so.0
* 1    Thread 0x7ffff7a64b80 (LWP 29568) "barrier_test"
0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ (
     lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>,
kind=MPIU_Nest_global_mutex) at ../src/src/include/mpiimplthreadpost.h:190
(gdb) where
#0  0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_
(lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>,
     kind=MPIU_Nest_global_mutex) at
../src/src/include/mpiimplthreadpost.h:190
#1  0x00007ffff7c5db42 in MPIDI_CH3I_Progress
(progress_state=progress_state at entry=0x7fffffffd2c0,
is_blocking=is_blocking at entry=1)
     at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:507
#2  0x00007ffff7b5e795 in PMPI_Recv (buf=0x7fffffffd61c, count=1,
datatype=1275069445, source=-2, tag=299, comm=1140850688,
status=0x7fffffffd620)
     at ../src/src/mpi/pt2pt/recv.c:157
#3  0x0000000000401732 in receive_int () at comm.c:52
#4  0x0000000000400bf2 in main (argc=1, argv=0x7fffffffd758) at
barrier_test.c:39
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff6a5a700 (LWP 29570))]
#0  0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from
/usr/lib64/libpthread.so.0
(gdb) where
#0  0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from
/usr/lib64/libpthread.so.0
#1  0x00007ffff7c5d614 in MPIDI_CH3I_Progress_delay
(completion_count=<optimized out>)
     at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:566
#2  MPIDI_CH3I_Progress
(progress_state=progress_state at entry=0x7ffff6a59710,
is_blocking=is_blocking at entry=1)
     at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:347
#3  0x00007ffff7b632ec in PMPI_Send (buf=0x7ffff6a5985c, count=1,
datatype=1275069445, dest=0, tag=199, comm=1140850688)
     at ../src/src/mpi/pt2pt/send.c:145
#4  0x0000000000400e42 in barrier_thread_release (id=0) at barrier.c:115
#5  0x0000000000401098 in barrier_helper (arg=0x0) at barrier.c:186
#6  0x0000003ef0407ee5 in start_thread () from /usr/lib64/libpthread.so.0
#7  0x0000003eef8f4d1d in clone () from /usr/lib64/libc.so.6
_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss

--
Jeff Hammond
jeff.science at gmail.com<mailto:jeff.science at gmail.com>
http://jeffhammond.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20150718/dada7ad2/attachment.html>
-------------- next part --------------
_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss