[Ace-users] [tao-users] memory leak with thread-per-connection model

Wed Nov 7 13:03:27 CST 2007

Hi,

Do you join all the threads? 

Johnny 

> -----Original Message-----
> From: Friedhelm Wolf [mailto:fwolf at dre.vanderbilt.edu] 
> Sent: Wednesday, November 07, 2007 5:46 PM
> To: Johnny Willemsen
> Cc: tao-users at cse.wustl.edu
> Subject: Re: [tao-users] memory leak with thread-per-connection model
> 
> Hi Johnny,
> 
> I don't know, whether I understood you correctly, but here is 
> what I did
> to get more information: I ran my application with one, two and three
> open connections. The valgrind results are interesting:
> 
> One connection:
> 
> ==17289== LEAK SUMMARY:
> ==17289==    definitely lost: 185 bytes in 3 blocks.
> ==17289==      possibly lost: 72 bytes in 1 blocks.
> ==17289==    still reachable: 149 bytes in 7 blocks.
> ==17289==         suppressed: 0 bytes in 0 blocks.
> 
> Two connections:
> 
> ==17508== LEAK SUMMARY:
> ==17508==    definitely lost: 185 bytes in 3 blocks.
> ==17508==      possibly lost: 144 bytes in 2 blocks.
> ==17508==    still reachable: 149 bytes in 7 blocks.
> ==17508==         suppressed: 0 bytes in 0 blocks.
> 
> Three connections:
> 
> ==17601== LEAK SUMMARY:
> ==17601==    definitely lost: 185 bytes in 3 blocks.
> ==17601==      possibly lost: 216 bytes in 3 blocks.
> ==17601==    still reachable: 149 bytes in 7 blocks.
> ==17601==         suppressed: 0 bytes in 0 blocks.
> 
> While the "definitely lost" parts (which are definitely memory leaks)
> stay the same, there seems to be a "possibly lost" portion of 72 bytes
> for each new thread.
> 
> According to valgrind this happens in:
> 
> ==17601== 216 bytes in 3 blocks are possibly lost in loss 
> record 10 of 10
> ==17601==    at 0x1B907F75: calloc (vg_replace_malloc.c:175)
> ==17601==    by 0x1B8F2188: (within /lib/ld-2.3.6.so)
> ==17601==    by 0x1B8F224B: _dl_allocate_tls (in /lib/ld-2.3.6.so)
> ==17601==    by 0x1BD47681: pthread_create@@GLIBC_2.1 (in
> /lib/tls/libpthread-2.3.6.so)
> ==17601==    by 0x1BCA0ACA: ACE_OS::thr_create(void* 
> (*)(void*), void*,
> long, unsigned long*, unsigned long*, long, void*, unsigned,
> ACE_Base_Thread_Adapter*) (OS_NS_Thread.cpp:4267)
> ==17601==    by 0x1BCD3890: ACE_Thread_Manager::spawn_i(void*
> (*)(void*), void*, long, unsigned long*, unsigned long*, long, int,
> void*, unsigned, ACE_Task_Base*) (Thread.inl:97)
> ==17601==    by 0x1BCD717F: 
> ACE_Thread_Manager::spawn_n(unsigned, void*
> (*)(void*), void*, long, long, int, ACE_Task_Base*, unsigned long*,
> void**, unsigned*) (Thread_Manager.cpp:734)
> ==17601==    by 0x1BCD10E5: ACE_Task_Base::activate(long, int, int,
> long, int, ACE_Task_Base*, unsigned long*, void**, unsigned*, unsigned
> long*) (Task.cpp:174)
> ==17601==    by 0x1BB79862:
> TAO_Thread_Per_Connection_Handler::activate(long, int, int, long, int,
> ACE_Task_Base*, unsigned long*, void**, unsigned*, unsigned long*)
> (Thread_Per_Connection_Handler.cpp:60)
> ==17601==    by 0x1BB10B83:
> TAO_Concurrency_Strategy<TAO_IIOP_Connection_Handler>::activat
> e_svc_handler(TAO_IIOP_Connection_Handler*,
> void*) (Acceptor_Impl.cpp:122)
> ==17601==    by 0x1BB0EF6D:
> ACE_Strategy_Acceptor<TAO_IIOP_Connection_Handler,
> ACE_SOCK_Acceptor>::activate_svc_handler(TAO_IIOP_Connection_Handler*)
> (Acceptor.cpp:779)
> ==17601==    by 0x1BB0FB1E: ACE_Acceptor<TAO_IIOP_Connection_Handler,
> ACE_SOCK_Acceptor>::handle_input(int) (Acceptor.cpp:406)
> 
> I hope this helps.
> 
> Thanks,
> Friedhelm
> 
> Johnny Willemsen wrote:
> > Hi,
> > 
> > We do have some leaks at exit, the most important thing to check is
> > that whether you have leaks per thread. Assuming your test 
> proram has
> > created X threads I would expect to have X blocks lost at the moment
> > we still have a leak per thread. I only see some leaks which seem to
> > are not related to a thread, some are related to the ORB.
> > 
> > Johnny
> > 
> >> -----Original Message----- From: Friedhelm Wolf
> >> [mailto:fwolf at dre.vanderbilt.edu] Sent: Tuesday, November 06, 2007
> >> 10:35 PM To: Johnny Willemsen Cc: tao-users at cse.wustl.edu Subject:
> >> Re: [tao-users] memory leak with thread-per-connection model
> >> 
> >> Hi Johnny,
> >> 
> >> after updating to the current svn status (11/6/07), I get 
> >> different, but not better valgrind results, as you can see from the
> >> attached log:
> >> 
> >> ==31988== 1 bytes in 1 blocks are definitely lost in loss record 1
> >> of 10 ==31988==    at 0x1B9077FB: operator new(unsigned,
> >> std::nothrow_t const&) (vg_replace_malloc.c:135) ==31988==    by
> >> 0x1BC7FD2F: ACE_TSS<ACE_Dynamic>::make_TSS_TYPE() const 
> >> (TSS_T.cpp:60) ==31988==    by 0x1BC8007B:
> >> ACE_TSS<ACE_Dynamic>::ts_get() const (TSS_T.cpp:219) ==31988==
> >> by 0x1BC80222: ACE_TSS_Singleton<ACE_Dynamic, 
> >> ACE_Null_Mutex>::instance() (TSS_T.cpp:53) ==31988==    by
> >> 0x1BC7FCB6: ACE_Dynamic::instance() (Dynamic.cpp:26) ==31988==
> >> by 0x1BB15A74: 
> >> TAO_Connect_Creation_Strategy<TAO_IIOP_Connection_Handler>::ma 
> >> ke_svc_handler(TAO_IIOP_Connection_Handler*&) (Svc_Handler.cpp:72) 
> >> ==31988==    by 0x1BB1429A: 
> >> ACE_Strategy_Connector<TAO_IIOP_Connection_Handler, 
> >> 
> ACE_SOCK_Connector>::make_svc_handler(TAO_IIOP_Connection_Handler*&)
> >>  (Connector.cpp:895) ==31988==    by 0x1BB15138: 
> >> ACE_Connector<TAO_IIOP_Connection_Handler, 
> >> ACE_SOCK_Connector>::connect_i(TAO_IIOP_Connection_Handler*&, 
> >> TAO_IIOP_Connection_Handler**, ACE_INET_Addr const&, 
> >> ACE_Synch_Options const&, ACE_INET_Addr const&, int, int, int)
> >> (Connector.cpp:369) ==31988==    by 0x1BB12EBA: 
> >> TAO_IIOP_Connector::begin_connection(TAO_IIOP_Connection_Handler*&,
> >>  TAO::Profile_Transport_Resolver*, TAO_IIOP_Endpoint*, 
> >> ACE_Time_Value*) (Connector.cpp:328) ==31988==    by 0x1BB13C0E: 
> >> 
> TAO_IIOP_Connector::make_connection(TAO::Profile_Transport_Resolver*,
> >>  TAO_Transport_Descriptor_Interface&, ACE_Time_Value*) 
> >> (IIOP_Connector.cpp:195) ==31988==    by 0x1BB833F0: 
> >> TAO_Connector::connect(TAO::Profile_Transport_Resolver*, 
> >> TAO_Transport_Descriptor_Interface*, ACE_Time_Value*) 
> >> (Transport_Connector.cpp:375) ==31988==    by 0x1BB578F0: 
> >> TAO::Profile_Transport_Resolver::try_connect_i(TAO_Transport_D 
> >> escriptor_Interface*, ACE_Time_Value*, bool)
> >> (Profile_Transport_Resolver.cpp:171) ==31988== ==31988== ==31988==
> >> 36 bytes in 1 blocks are definitely lost in loss record 6 of 10 
> >> ==31988==    at 0x1B9077FB: operator new(unsigned, std::nothrow_t 
> >> const&) (vg_replace_malloc.c:135) ==31988==    by 0x1BB48F7F: 
> >> ACE_TSS<TAO_ORB_Core_TSS_Resources>::make_TSS_TYPE() const 
> >> (TSS_T.cpp:60) ==31988==    by 0x1BB26CFB: 
> >> ACE_TSS<TAO_ORB_Core_TSS_Resources>::ts_get() const (TSS_T.cpp:219)
> >>  ==31988==    by 0x1BB25C38:
> >> TAO_Leader_Follower::set_client_thread() (TSS_T.cpp:53) ==31988==
> >> by 0x1BB25DD6: TAO_Leader_Follower::wait_for_event(TAO_LF_Event*,
> >> TAO_Transport*, ACE_Time_Value*) (Leader_Follower.inl:205) 
> >> ==31988==    by 0x1BB271D0: 
> >> TAO_LF_Connect_Strategy::wait_i(TAO_LF_Event*, TAO_Transport*, 
> >> ACE_Time_Value*) (LF_Connect_Strategy.cpp:54) ==31988==    by
> >> 0x1BAF05FE: TAO_Connect_Strategy::wait(TAO_Transport*, 
> >> ACE_Time_Value*) (Connect_Strategy.cpp:41) ==31988==    by
> >> 0x1BB82693: 
> >> TAO_Connector::wait_for_connection_completion(TAO::Profile_Tra 
> >> nsport_Resolver*, TAO_Transport*&, ACE_Time_Value*)
> >> (Transport_Connector.cpp:528) ==31988==    by 0x1BB13533: 
> >> TAO_IIOP_Connector::complete_connection(int, 
> >> TAO_Transport_Descriptor_Interface&,
> >> TAO_IIOP_Connection_Handler**&, TAO_IIOP_Endpoint**, unsigned,
> >> TAO::Profile_Transport_Resolver*, TAO_LF_Multi_Event*,
> >> ACE_Time_Value*) (IIOP_Connector.cpp:418) ==31988==    by
> >> 0x1BB13C80: 
> >> 
> TAO_IIOP_Connector::make_connection(TAO::Profile_Transport_Resolver*,
> >>  TAO_Transport_Descriptor_Interface&, ACE_Time_Value*) 
> >> (IIOP_Connector.cpp:225) ==31988==    by 0x1BB833F0: 
> >> TAO_Connector::connect(TAO::Profile_Transport_Resolver*, 
> >> TAO_Transport_Descriptor_Interface*, ACE_Time_Value*) 
> >> (Transport_Connector.cpp:375) ==31988==    by 0x1BB578F0: 
> >> TAO::Profile_Transport_Resolver::try_connect_i(TAO_Transport_D 
> >> escriptor_Interface*, ACE_Time_Value*, bool)
> >> (Profile_Transport_Resolver.cpp:171) ==31988== ==31988== ==31988==
> >> 72 bytes in 1 blocks are possibly lost in loss record 9 of 10 
> >> ==31988==    at 0x1B907F75: calloc (vg_replace_malloc.c:175) 
> >> ==31988==    by 0x1B8F2188: (within /lib/ld-2.3.6.so) ==31988==
> >> by 0x1B8F224B: _dl_allocate_tls (in /lib/ld-2.3.6.so) ==31988==
> >> by 0x1BD47681: pthread_create@@GLIBC_2.1 (in 
> >> /lib/tls/libpthread-2.3.6.so) ==31988==    by 0x1BCA0ACA:
> >> ACE_OS::thr_create(void* (*)(void*), void*, long, unsigned long*,
> >> unsigned long*, long, void*, unsigned, ACE_Base_Thread_Adapter*)
> >> (OS_NS_Thread.cpp:4267) ==31988==    by 0x1BCD3890:
> >> ACE_Thread_Manager::spawn_i(void* (*)(void*), void*, long, unsigned
> >> long*, unsigned long*, long, int, void*, unsigned, ACE_Task_Base*)
> >> (Thread.inl:97) ==31988==    by 0x1BCD717F: 
> >> ACE_Thread_Manager::spawn_n(unsigned, void* (*)(void*), void*,
> >> long, long, int, ACE_Task_Base*, unsigned long*, void**, unsigned*)
> >> (Thread_Manager.cpp:734) ==31988==    by 0x1BCD10E5:
> >> ACE_Task_Base::activate(long, int, int, long, int, ACE_Task_Base*,
> >> unsigned long*, void**, unsigned*, unsigned long*) (Task.cpp:174) 
> >> ==31988==    by 0x1BB79862: 
> >> TAO_Thread_Per_Connection_Handler::activate(long, int, int, long,
> >> int, ACE_Task_Base*, unsigned long*, void**, unsigned*, unsigned
> >> long*) (Thread_Per_Connection_Handler.cpp:60) ==31988==    by
> >> 0x1BB10B83: 
> >> TAO_Concurrency_Strategy<TAO_IIOP_Connection_Handler>::activat 
> >> e_svc_handler(TAO_IIOP_Connection_Handler*, void*)
> >> (Acceptor_Impl.cpp:122) ==31988==    by 0x1BB0EF6D: 
> >> ACE_Strategy_Acceptor<TAO_IIOP_Connection_Handler, 
> >> ACE_SOCK_Acceptor>::activate_svc_handler(TAO_IIOP_Connection_H 
> >> andler*) (Acceptor.cpp:779) ==31988==    by 0x1BB0FB1E:
> >> ACE_Acceptor<TAO_IIOP_Connection_Handler, 
> >> ACE_SOCK_Acceptor>::handle_input(int) (Acceptor.cpp:406) ==31988== 
> >> ==31988== ==31988== 148 bytes in 1 blocks are definitely lost in
> >> loss record 10 of 10 ==31988==    at 0x1B9077FB: operator
> >> new(unsigned, std::nothrow_t const&) (vg_replace_malloc.c:135) 
> >> ==31988==    by 0x1BB8432F: 
> >> ACE_TSS<TAO_TSS_Resources>::make_TSS_TYPE() const (TSS_T.cpp:60) 
> >> ==31988==    by 0x1BB8447B: ACE_TSS<TAO_TSS_Resources>::ts_get()
> >> const (TSS_T.cpp:219) ==31988==    by 0x1BB84622:
> >> TAO_TSS_Singleton<TAO_TSS_Resources, ACE_Thread_Mutex>::instance()
> >> (TSS_T.cpp:53) ==31988==    by 0x1BB840A6:
> >> TAO_TSS_Resources::instance() (TSS_Resources.cpp:44) ==31988==
> >> by 0x1BB3D526: TAO_ORB_Core::gui_resource_factory() 
> >> (ORB_Core.cpp:1508) ==31988==    by 0x1BB25D10:
> >> TAO_Leader_Follower::reactor() (Leader_Follower.cpp:135) ==31988==
> >> by 0x1BB3DDF4: TAO_ORB_Core::reactor() (ORB_Core.cpp:2826) 
> >> ==31988==    by 0x1BB47E10: TAO_ORB_Core::init(int&, char**) 
> >> (ORB_Core.cpp:1144) ==31988==    by 0x1BB3AF72:
> >> CORBA::ORB_init(int&, char**, char const*) (ORB.cpp:1350) ==31988==
> >> by 0x805EB68: main (lpgame_server.cpp:22) ==31988== ==31988== LEAK
> >> SUMMARY: ==31988==    definitely lost: 185 bytes in 3 blocks. 
> >> ==31988==      possibly lost: 72 bytes in 1 blocks. ==31988==
> >> still reachable: 149 bytes in 7 blocks. ==31988==
> >> suppressed: 0 bytes in 0 blocks. ==31988== Reachable blocks (those
> >> to which a pointer was found) are not shown. ==31988== To see them,
> >> rerun with: --show-reachable=yes
> >> 
> >> 
> >> Thanks, Friedhelm
> >> 
> >> Johnny Willemsen wrote:
> >>> Hi,
> >>> 
> >>> What is the exact svn number that you are using. This
> >> should be fixed on the
> >>> head version of today, see bugzilla 3108. This memory leak
> >> does exist in TAO
> >>> 1.6.1 and 1.6.0.
> >>> 
> >>> Regards,
> >>> 
> >>> Johnny Willemsen Remedy IT Postbus 101 2650 AC  Berkel en
> >>> Rodenrijs The Netherlands www.theaceorb.nl / www.remedy.nl
> >>> 
> >>> *** Integrated compile and test statistics see 
> >>> http://scoreboard.theaceorb.nl *** *** Commercial service and
> >>> support for ACE/TAO/CIAO             *** *** See
> >>> http://www.theaceorb.nl/en/support.html                 ***
> >>> 
> >>>> TAO VERSION : svn head ACE VERSION : svn head
> >>>> 
> >>>> HOST/TARGET MACHINE and OPERATING SYSTEM: Linux
> >>>> tango.dre.vanderbilt.edu 2.6.10 #1 SMP Sat Jan 1 17:55:41 CST
> >>>> 2005 i686 GNU/Linux
> >>>> 
> >>>> COMPILER NAME AND VERSION (AND PATCHLEVEL): gcc (GCC) 4.1.2
> >>>> 20061115 (prerelease) (Debian 4.1.1-21)
> >>>> 
> >>>> CONTENTS OF $ACE_ROOT/ace/config.h #include
> >>>> <ace/config-linux.h>
> >>>> 
> >>>> CONTENTS OF $ACE_ROOT/include/makeinclude/platform_macros.GNU 
> >>>> include $(ACE_ROOT)/include/makeinclude/platform_linux.GNU
> >>>> 
> >>>> CONTENTS OF 
> >>>> $ACE_ROOT/bin/MakeProjectCreator/config/default.features: n/a
> >>>> 
> >>>> LEVEL OF URGENCY (LOW, MEDIUM, or HIGH): LOW
> >>>> 
> >>>> AREA/CLASS/EXAMPLE AFFECTED: thread-per-connection model
> >>>> 
> >>>> DOES THE PROBLEM AFFECT: execution / memory management
> >>>> 
> >>>> SYNOPSIS: memory leak with thread-per-connection model
> >>>> 
> >>>> DESCRIPTION:
> >>>> 
> >>>> Hi there,
> >>>> 
> >>>> I implemented a standard CORBA application, consisting of a 
> >>>> client and a server. When I check the server with valgrind in
> >>>> the default configuration there is no memory leak. As soon as I
> >>>> add a svc.conf file with the following content:
> >>>> 
> >>>> static Server_Strategy_Factory "-ORBConcurrency
> >> thread-per-connection"
> >>>> valgrind indicates, that there are memory leaks. (see
> >> attached file)
> >>>> Is this a bug in TAO?
> >>>> 
> >>>> Thanks, Friedhelm
> >>>> 
> 