<div dir="ltr">On Wed, Jan 2, 2013 at 9:49 AM, Jeff Hammond <span dir="ltr"><<a href="mailto:jhammond@alcf.anl.gov" target="_blank">jhammond@alcf.anl.gov</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I wrote a program that calls MPI from a pthread. MPI_THREAD_MULTIPLE<br>
is used. I cancel and then join the pthread back into the main<br>
program. After I do this, any calls to MPI, including MPI_Abort (see<br>
first gdb+source below) and MPI_Barrier (see second gdb+source below)<br>
deadlock on pthread_mutex_lock.<br>
<br>
I can understand that canceling the thread while it is holding the MPI<br>
pthread mutex is a bad idea, but I'm not sure this is an incorrect<br>
program. Is pthread_cancel using signals? I recall that MPICH<br>
functions are not guaranteed to be signal-safe.<br></blockquote><div><br></div><div style>On Linux, pthread_cancel is implemented using tgkill ("man tgkill").</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Is it possible to implement MPI_Abort such that acquisition of the MPI<br>
mutex is not required in order to allow deadlocked programs to<br>
terminate? It seems rather unfortunate that MPI_Abort does not<br>
perform the desired action here. Obviously, I can use abort(), but on<br>
some implementations, this will leave the system resources in a dirty<br>
state.<br>
<br>
I know a bunch of ways to work around this, but it would be helpful to<br>
understand exactly what the problem is. I wonder if there is a way to<br>
modify MPICH such that the mutex is released when a thread holding it<br>
is cancelled.<br>
<br>
Is it possible for MPICH to provide an MPIX call or otherwise document<br>
an internal, MPICH-specific function to clear the MPI mutex? I could<br>
resolve this problem if I could merely call MPIX_Mutex_force_unlock()<br>
and let the program terminate as desired.<br>
<br>
Thanks,<br>
<br>
Jeff<br>
<br>
$ make<br>
mpicc -g -O0 -Wall -std=gnu99 -c bug.c -o bug.o<br>
mpicc -g -O0 -Wall -std=gnu99 safemalloc.o bug.o -lm -o bug.x<br>
rm bug.o<br>
<br>
======== stuck in abort/finalize ========<br>
<br>
$ gdb ./bug.x<br>
GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC 2012)<br>
Copyright 2004 Free Software Foundation, Inc.<br>
GDB is free software, covered by the GNU General Public License, and you are<br>
welcome to change it and/or distribute copies of it under certain conditions.<br>
Type "show copying" to see the conditions.<br>
There is absolutely no warranty for GDB. Type "show warranty" for details.<br>
This GDB was configured as "x86_64-apple-darwin"...Reading symbols for<br>
shared libraries ...... done<br>
<br>
warning: Could not find object file<br>
"/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no<br>
debug information available for "bug.c".<br>
<br>
<br>
(gdb) run<br>
Starting program:<br>
/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x<br>
Reading symbols for shared libraries +++++........................ done<br>
test done except for pthread shutdown<br>
0: Progress<br>
all done<br>
<br>
^C<br>
Program received signal SIGINT, Interrupt.<br>
0x00007fff912fdbf2 in __psynch_mutexwait ()<br>
(gdb) bt<br>
#0 0x00007fff912fdbf2 in __psynch_mutexwait ()<br>
#1 0x00007fff8c80a1a1 in pthread_mutex_lock ()<br>
#2 0x000000010004912b in MPIU_Thread_CS_enter_lockname_recursive_impl_ ()<br>
#3 0x0000000100048d64 in MPI_Abort ()<br>
#4 0x0000000100000cc0 in main ()<br>
(gdb) quit<br>
The program is running. Exit anyway? (y or n) y<br>
<br>
$ cat bug.c<br>
#include <stdio.h><br>
#include <stdlib.h><br>
#include <unistd.h><br>
#include <string.h><br>
#include <assert.h><br>
#include <limits.h><br>
#include <pthread.h><br>
<br>
#include <mpi.h><br>
<br>
#define DEBUG<br>
<br>
pthread_t Progress_thread;<br>
<br>
static void * Progress_function(void * dummy)<br>
{<br>
int rank;<br>
MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br>
<br>
while (1)<br>
{<br>
#ifdef DEBUG<br>
printf("%d: Progress \n", rank);<br>
#endif<br>
MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD,<br>
MPI_STATUS_IGNORE);<br>
//usleep(500);<br>
}<br>
<br>
return NULL;<br>
}<br>
<br>
int main(int argc, char * argv[])<br>
{<br>
int rc;<br>
int rank, size;<br>
int provided;<br>
<br>
MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);<br>
if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1);<br>
<br>
MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br>
MPI_Comm_size(MPI_COMM_WORLD, &size);<br>
<br>
rc = pthread_create(&Progress_thread, NULL, &Progress_function, NULL);<br>
if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc);<br>
<br>
MPI_Barrier(MPI_COMM_WORLD);<br>
<br>
printf("test done except for pthread shutdown \n");<br>
fflush(stdout);<br>
<br>
MPI_Barrier(MPI_COMM_WORLD);<br>
<br>
rc = pthread_cancel(Progress_thread);<br>
if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc);<br>
<br>
void * rv;<br>
rc = pthread_join(Progress_thread, &rv);<br>
if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc);<br>
<br>
printf("all done \n");<br>
fflush(stdout);<br>
<br>
MPI_Abort(MPI_COMM_SELF, 2);<br>
MPI_Finalize();<br>
<br>
return 0;<br>
}<br>
<br>
======== stuck in barrier ========<br>
<br>
$ gdb ./bug.x<br>
GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC 2012)<br>
Copyright 2004 Free Software Foundation, Inc.<br>
GDB is free software, covered by the GNU General Public License, and you are<br>
welcome to change it and/or distribute copies of it under certain conditions.<br>
Type "show copying" to see the conditions.<br>
There is absolutely no warranty for GDB. Type "show warranty" for details.<br>
This GDB was configured as "x86_64-apple-darwin"...Reading symbols for<br>
shared libraries ...... done<br>
<br>
warning: Could not find object file<br>
"/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no<br>
debug information available for "bug.c".<br>
<br>
<br>
(gdb) run<br>
Starting program:<br>
/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x<br>
Reading symbols for shared libraries +++++........................ done<br>
before pthread shutdown<br>
0: Progress<br>
after pthread shutdown<br>
^C<br>
Program received signal SIGINT, Interrupt.<br>
0x00007fff912fdbf2 in __psynch_mutexwait ()<br>
(gdb) bt<br>
#0 0x00007fff912fdbf2 in __psynch_mutexwait ()<br>
#1 0x00007fff8c80a1a1 in pthread_mutex_lock ()<br>
#2 0x000000010000f7eb in MPIU_Thread_CS_enter_lockname_recursive_impl_ ()<br>
#3 0x000000010000f4f7 in MPI_Barrier ()<br>
#4 0x0000000100000c4d in main ()<br>
(gdb) quit<br>
The program is running. Exit anyway? (y or n) y<br>
Jeffs-MacBook-Pro:mpi-pt jhammond$ cat bug.c<br>
#include <stdio.h><br>
#include <stdlib.h><br>
#include <unistd.h><br>
#include <string.h><br>
#include <assert.h><br>
#include <limits.h><br>
#include <pthread.h><br>
<br>
#include <mpi.h><br>
<br>
#define DEBUG<br>
<br>
pthread_t Progress_thread;<br>
<br>
static void * Progress_function(void * dummy)<br>
{<br>
int rank;<br>
MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br>
<br>
while (1)<br>
{<br>
#ifdef DEBUG<br>
printf("%d: Progress \n", rank);<br>
#endif<br>
MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD,<br>
MPI_STATUS_IGNORE);<br>
//usleep(500);<br>
}<br>
<br>
return NULL;<br>
}<br>
<br>
int main(int argc, char * argv[])<br>
{<br>
int rc;<br>
int rank, size;<br>
int provided;<br>
<br>
MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);<br>
if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1);<br>
<br>
MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br>
MPI_Comm_size(MPI_COMM_WORLD, &size);<br>
<br>
rc = pthread_create(&Progress_thread, NULL, &Progress_function, NULL);<br>
if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc);<br>
<br>
MPI_Barrier(MPI_COMM_WORLD);<br>
<br>
printf("before pthread shutdown \n");<br>
fflush(stdout);<br>
<br>
MPI_Barrier(MPI_COMM_WORLD);<br>
<br>
rc = pthread_cancel(Progress_thread);<br>
if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc);<br>
<br>
void * rv;<br>
rc = pthread_join(Progress_thread, &rv);<br>
if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc);<br>
<br>
printf("after pthread shutdown \n");<br>
fflush(stdout);<br>
<br>
MPI_Barrier(MPI_COMM_WORLD);<br>
<br>
printf("before MPI Abort/Finalize \n");<br>
fflush(stdout);<br>
<br>
MPI_Abort(MPI_COMM_SELF, 2);<br>
MPI_Finalize();<br>
<br>
printf("after MPI Abort/Finalize \n");<br>
fflush(stdout);<br>
<br>
return 0;<br>
}<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Jeff Hammond<br>
Argonne Leadership Computing Facility<br>
University of Chicago Computation Institute<br>
<a href="mailto:jhammond@alcf.anl.gov">jhammond@alcf.anl.gov</a> / <a href="tel:%28630%29%20252-5381" value="+16302525381">(630) 252-5381</a><br>
<a href="http://www.linkedin.com/in/jeffhammond" target="_blank">http://www.linkedin.com/in/jeffhammond</a><br>
<a href="https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond" target="_blank">https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond</a><br>
_______________________________________________<br>
discuss mailing list <a href="mailto:discuss@mpich.org">discuss@mpich.org</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mpich.org/mailman/listinfo/discuss" target="_blank">https://lists.mpich.org/mailman/listinfo/discuss</a><br>
</font></span></blockquote></div><br></div></div>