<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">HI Daniel,<div class=""><br class=""></div><div class="">In addition to what Johnny said, do you have log output from the server during the period from 11:38:25 to 15:49:37? I ask because that last client connection is assigned handle #1542 which is quite a large number for a supposedly idle server. Do you have other I/O happening on a regular basis, maybe writing to a file, or connecting to a database or something?</div><div class=""><br class=""></div><div class="">I'm guessing you have a resource leak somewhere, not closing a file or socket. </div><div class=""><br class=""></div><div class="">By default, TAO's reactor uses select() which typically has a limit of 1024 handles. Since the handle number is used as an index to a bit array, a high number such as #1542 will be out of bounds even if you only have a handful of sockets you are interested in. So the server is probably just in its normal run state, but unable to select on the client's connection.</div><div class=""><br class=""></div><div class="">You can try using lsof -p <server pid> which if I'm guessing right will show you 1500 or so open files or sockets.</div><div class=""><br class=""></div><div class="">In fact, I noticed that the first client connection you noted uses handle #28, while previously the server had opened connection to the naming service using handle #9. So 19 handles were consumed in 2.25 minutes. Later you have a 4 hour 11 minute gap with 1514 handles consumed, so in the first case roughly 8 handles per minute were leaked, and roughly 6 per minute in the latter case.</div><div class=""><br class=""></div><div class="">Now if you need all these open resources, perhaps you can switch to the dev_poll reactor rather than the default reactor. See docs/Options.html for information on setting reactor type via the advanced resource factory.</div><div class=""><br class=""></div><div class="">Best regards,</div><div class="">Phil</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 13, 2017, at 4:32 AM, Johnny Willemsen <<a href="mailto:jwillemsen@remedy.nl" class="">jwillemsen@remedy.nl</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="margin-top: 0px; margin-bottom: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">Hi,</div><div style="margin-top: 0px; margin-bottom: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">Thanks for using the PRF form. Can you attach a debugger and see where the server is looping exactly?</div><div style="margin-top: 0px; margin-bottom: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><br class=""></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">Best regards,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><pre class="moz-signature" cols="72" style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">Johnny Willemsen
Remedy IT
Postbus 81 | 6930 AB Westervoort | The Netherlands
<a class="moz-txt-link-freetext" href="http://www.remedy.nl/">http://www.remedy.nl</a>
</pre><div class="moz-cite-prefix" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">On 01/12/2017 02:32 PM, Daniel Suchodolski wrote:<br class=""></div><blockquote cite="mid:D7A28D1AB9B4CA418082377DBC86651B0A8E92ED@SDBE0041.dbe.local" type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><div style="direction: ltr; font-family: Tahoma; font-size: 10pt;" class="">Hi TAO,<br class=""><br class=""><br class=""> <span class="Apple-converted-space"> </span>TAO VERSION: 2.4.1<br class=""> <span class="Apple-converted-space"> </span>ACE VERSION: 6.4.1<br class=""><br class=""> <span class="Apple-converted-space"> </span>HOST MACHINE and OPERATING SYSTEM:<br class=""> <span class="Apple-converted-space"> </span>Debian 8 (Jessie)<br class=""><br class=""> <span class="Apple-converted-space"> </span>COMPILER: g++ (Debian 4.9.2-10) 4.9.2<span class="Apple-converted-space"> </span><br class=""><br class=""><br class=""> <span class="Apple-converted-space"> </span>DOES THE PROBLEM AFFECT:<br class=""> <span class="Apple-converted-space"> </span>EXECUTION? YES<br class=""><br class=""><br class=""> <span class="Apple-converted-space"> </span>SYNOPSIS:<br class="">CORBA service after some time working idle (1 night) can't process requests. (migration result)<br class=""><br class=""> <span class="Apple-converted-space"> </span>DESCRIPTION:<br class="">At the beginning I want to highlight that we use version of ACE-TAO compiled with option "threads 0". The problem started to happen with many services after migration to newest version of ACE TAO and operating system. Up to now the system worked stable on version 1.2.1/5.2.1 (linux lenny).<br class=""><br class="">How it works:<br class="">Server application registers CORBA service in NamingService.<br class="">If a client connects shortly after start of the server<br class="">then the server works fine: the server process request properly.<br class=""><br class="">After some time (for example 1 night) a client connects to the Server, but<br class="">when the client tries to use service then the server hangs up and use 100% of a processor.<br class="">Client is blocked by the server until the server is killed. Debugging, we found out that the problem is somewhere inside CORBA invocation. Very unclear are debug information seen during loading adatp3-services.svc, but we are not able fully interpret this issue.<br class=""><br class="">The services is run with the following ORBParameters:<br class=""> <span class="Apple-converted-space"> </span>-ORBDottedDecimalAddresses 1<br class=""> <span class="Apple-converted-space"> </span>-ORBDebug -ORBDebugLevel 10 -ORBVerboseLogging 2 -ORBInitRef NameService=corbaloc::server:30033/NameService<br class=""> <span class="Apple-converted-space"> </span>-ORBSvcConf adatp3-services.svc<br class=""><br class="">and adatp3-services.svc:<br class="">static Advanced_Resource_Factory "-ORBReactorMaskSignals 0 -ORBInputCDRAllocator null -ORBReactorType select_st -ORBConnec<br class="">tionCacheLock null"<br class="">static Server_Strategy_Factory "-ORBAllowReactivationOfSystemids 0"<br class="">static Client_Strategy_Factory "-ORBTransportMuxStrategy EXCLUSIVE -ORBClientConnectionHandler RW"<br class=""><br class=""><br class=""> <span class="Apple-converted-space"> </span>REPEAT BY:<br class="">Every Time<br class=""><br class=""> <span class="Apple-converted-space"> </span>TAO LOG:<br class=""><br class="">The Log of the server is showed below. The log is divided into parts:<br class=""><br class="">[Start Server]<br class="">[Client connect to Server after short time]<br class="">[Client is connecting after some time]<br class=""><br class=""></div></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">tao-bugs mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class=""><a href="mailto:tao-bugs@list.isis.vanderbilt.edu" class="">tao-bugs@list.isis.vanderbilt.edu</a></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class=""><a href="http://list.isis.vanderbilt.edu/cgi-bin/mailman/listinfo/tao-bugs" class="">http://list.isis.vanderbilt.edu/cgi-bin/mailman/listinfo/tao-bugs</a></span></div></blockquote></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div style="font-size: 12.8px;" class=""><div style="margin: 0px; padding: 0px; border: 0px;" class=""><span style="font-family: arial, helvetica, sans-serif; font-size: large;" class="">--</span></div><div style="margin: 0px; padding: 0px; border: 0px;" class=""><span style="font-family: arial, helvetica, sans-serif; font-size: large;" class="">Phil Mesnier</span><br class=""></div></div><div style="font-size: 12.8px;" class=""><div style="font-size: 12.8px;" class=""><font face="arial, helvetica, sans-serif" size="2" class="">Principal Engineer & Partner</font></div><div style="font-size: 12.8px;" class=""><font face="arial, helvetica, sans-serif" class=""><br class=""></font></div><div style="font-size: 12.8px;" class=""><span style="font-family: arial, helvetica, sans-serif;" class=""><font size="2" color="#000000" class="">OCI | WE ARE SOFTWARE ENGINEERS.</font></span></div><div style="font-size: 12.8px;" class=""><span style="font-family: arial, helvetica, sans-serif; font-size: x-small;" class="">tel +1.314.579.0066 x225</span><br class=""></div><div style="font-size: 12.8px;" class=""><a href="http://ociweb.com/" target="_blank" class="">ociweb.com</a></div><div class=""><br class=""></div></div></div></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br class=""></div></div></div></body></html>