[Ace-users] [ace-users] Thread_Timer_Queue_Adapter deadlock

Caleb Epstein caleb.epstein at gmail.com
Thu Feb 14 11:18:06 CST 2008 

    ACE VERSION: 5.6.2

    HOST MACHINE and OPERATING SYSTEM:
Linux tux2132 2.6.20-16-generic #2 SMP Fri Feb 1 02:59:08 UTC 2008
i686 GNU/Linux

    TARGET MACHINE and OPERATING SYSTEM, if different from HOST:
Same

    COMPILER NAME AND VERSION (AND PATCHLEVEL):
gcc (GCC) 4.1.2


    THE $ACE_ROOT/ace/config.h FILE [if you use a link to a platform-
    specific file, simply state which one]:
config-linux.h

    THE $ACE_ROOT/include/makeinclude/platform_macros.GNU FILE [if you
    use a link to a platform-specific file, simply state which one
    (unless this isn't used in this case, e.g., with Microsoft Visual
    C++)]:
platform_linux.GNU

    CONTENTS OF $ACE_ROOT/bin/MakeProjectCreator/config/default.features
    (used by MPC when you generate your own makefiles):
Not used.

    AREA/CLASS/EXAMPLE AFFECTED:
[What example failed?  What module failed to compile?]

    DOES THE PROBLEM AFFECT:
        COMPILATION?
        LINKING?
        EXECUTION?
        OTHER (please specify)?
Execution.

    SYNOPSIS:
Applications that hold a mutex lock while performing operations on an
ACE_Thread_Timer_Queue_Adapter can cause a deadlock.  A test program
is attached.

    DESCRIPTION:
Because the svc routine in ACE_Thread_Timer_Queue_Adapter calls the
internal Timer_Queue's expire method with its own mutex locked, an
application that locks some other mutex in a timeout_handler can
deadlock with some other thread that is holding the same application
mutex and trying to schedule or cancel timers.

Thread A: lock mutex A, schedule a timer (which tries to lock TTQA's
internal mutex)
Thread B (Timer_Queue_Adapter's svc thread): called handle_timeout
with TTQA's internal mutex locked, attempts to schedule/cancel timers
deadlock with thread A.

Stack trace:
Thread 2 (Thread -1212687472 (LWP 32559)):
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7cd9dae in __lll_mutex_lock_wait () from
/lib/tls/i686/cmov/libpthread.so.0
#2  0xb7cd5aac in _L_mutex_lock_51 () from /lib/tls/i686/cmov/libpthread.so.0
#3  0xb7cd549d in pthread_mutex_lock () from /lib/tls/i686/cmov/libpthread.so.0
#4  0xb7ee2582 in ACE_OS::mutex_lock (m=0x80516c0) at OS_NS_Thread.cpp:2024
#5  0x0804aa53 in ACE_OS::thread_mutex_lock (m=0x80516c0) at
/pub/apps/linux-i686/include/ace/OS_NS_Thread.inl:3405
#6  0x0804aad9 in ACE_Thread_Mutex::acquire (this=0x80516c0) at
/pub/apps/linux-i686/include/ace/Thread_Mutex.inl:53
#7  0x0804aaef in ACE_Guard<ACE_Thread_Mutex>::acquire (this=0xb7b7d17c)
    at /pub/apps/linux-i686/include/ace/Guard_T.inl:12
#8  0x0804ab23 in ACE_Guard (this=0xb7b7d17c, l=@0x80516c0) at
/pub/apps/linux-i686/include/ace/Guard_T.inl:38
#9  0x0804e332 in Handler::handle_timeout (this=0xbff19548) at
ace_ttqa_deadlock.cpp:14
#10 0x0804d1eb in
ACE_Event_Handler_Handle_Timeout_Upcall<ACE_Recursive_Thread_Mutex>::timeout
(this=0x8055750,
    timer_queue=@0x80556b0, event_handler=0xbff19548, act=0x0,
recurring_timer=0, cur_time=@0xb7b7d270)
    at /pub/apps/linux-i686/include/ace/Timer_Queue_T.cpp:412
#11 0x0804d2d1 in ACE_Timer_Queue_T<ACE_Event_Handler*,
ACE_Event_Handler_Handle_Timeout_Upcall<ACE_Recursive_Thread_Mutex>,
ACE_Recursive_Thread_Mutex>::upcall (this=0x80556b0, info=@0xb7b7d218,
cur_time=@0xb7b7d270)
    at /pub/apps/linux-i686/include/ace/Timer_Queue_T.inl:170
#12 0x0804d38a in ACE_Timer_Queue_T<ACE_Event_Handler*,
ACE_Event_Handler_Handle_Timeout_Upcall<ACE_Recursive_Thread_Mutex>,
ACE_Recursive_Thread_Mutex>::expire (this=0x80556b0,
cur_time=@0xb7b7d270)
    at /pub/apps/linux-i686/include/ace/Timer_Queue_T.cpp:274
#13 0x0804b617 in ACE_Timer_Queue_T<ACE_Event_Handler*,
ACE_Event_Handler_Handle_Timeout_Upcall<ACE_Recursive_Thread_Mutex>,
ACE_Recursive_Thread_Mutex>::expire (this=0x80556b0) at
/pub/apps/linux-i686/include/ace/Timer_Queue_T.inl:151
#14 0x0804ba2e in
ACE_Thread_Timer_Queue_Adapter<ACE_Timer_Heap_T<ACE_Event_Handler*,
ACE_Event_Handler_Handle_Timeout_Upcall<ACE_Recursive_Thread_Mutex>,
ACE_Recursive_Thread_Mutex> >::svc (this=0x80516e0)
    at /pub/apps/linux-i686/include/ace/Timer_Queue_Adapters.cpp:282
#15 0xb7f125c0 in ACE_Task_Base::svc_run (args=0x80516e0) at Task.cpp:275
#16 0xb7f12828 in ACE_Thread_Adapter::invoke_i (this=0x8055920) at
Thread_Adapter.cpp:146
#17 0xb7f129f6 in ACE_Thread_Adapter::invoke (this=0x8055920) at
Thread_Adapter.cpp:95
#18 0xb7eac061 in ace_thread_adapter (args=0x8055920) at
Base_Thread_Adapter.cpp:116
#19 0xb7cd331b in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#20 0xb7c5c57e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 1 (Thread -1212684592 (LWP 32556)):
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7cd9dae in __lll_mutex_lock_wait () from
/lib/tls/i686/cmov/libpthread.so.0
#2  0xb7cd5ac4 in _L_mutex_lock_241 () from /lib/tls/i686/cmov/libpthread.so.0
#3  0xb7cd5622 in pthread_mutex_lock () from /lib/tls/i686/cmov/libpthread.so.0
#4  0xb7ee2582 in ACE_OS::mutex_lock (m=0x8051734) at OS_NS_Thread.cpp:2024
#5  0x0804aa53 in ACE_OS::thread_mutex_lock (m=0x8051734) at
/pub/apps/linux-i686/include/ace/OS_NS_Thread.inl:3405
#6  0x0804aa67 in ACE_OS::recursive_mutex_lock (m=0x8051734) at
/pub/apps/linux-i686/include/ace/OS_NS_Thread.inl:691
#7  0x0804aa7b in ACE_Recursive_Thread_Mutex::acquire (this=0x8051734)
    at /pub/apps/linux-i686/include/ace/Recursive_Thread_Mutex.inl:37
#8  0x0804aa91 in ACE_Guard<ACE_Recursive_Thread_Mutex>::acquire
(this=0xbff19498)
    at /pub/apps/linux-i686/include/ace/Guard_T.inl:12
#9  0x0804aac5 in ACE_Guard (this=0xbff19498, l=@0x8051734) at
/pub/apps/linux-i686/include/ace/Guard_T.inl:38
#10 0x0804ae42 in
ACE_Thread_Timer_Queue_Adapter<ACE_Timer_Heap_T<ACE_Event_Handler*,
ACE_Event_Handler_Handle_Timeout_Upcall<ACE_Recursive_Thread_Mutex>,
ACE_Recursive_Thread_Mutex> >::schedule (this=0x80516e0,
handler=0xbff19548, act=0x0,
    future_time=@0xbff194e0, interval=@0x8051678) at
/pub/apps/linux-i686/include/ace/Timer_Queue_Adapters.cpp:199
#11 0x0804b829 in Handler::schedule (this=0xbff19548) at
ace_ttqa_deadlock.cpp:22
#12 0x08049e71 in main () at ace_ttqa_deadlock.cpp:43


    REPEAT BY:
See attached test program.

    SAMPLE FIX/WORKAROUND:
ACE_Thread_Timer_Queue_Adapter should unlock its internal mutex before
calling expire.

-- 
Caleb Epstein
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ace_ttqa_deadlock.cpp
Type: text/x-c++src
Size: 853 bytes
Desc: not available
Url : http://list.isis.vanderbilt.edu/pipermail/ace-users/attachments/20080214/0f337567/attachment.bin

More information about the Ace-users mailing list