<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hah, I knew it! <div class=""><br class=""></div><div class="">Thank you for the update, Good luck with your project. </div><div class=""><br class=""></div><div class="">Best regards,</div><div class="">Phil</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Jan 16, 2017, at 4:48 AM, Daniel Suchodolski <<a href="mailto:Daniel.Suchodolski@pitradwar.com" class="">Daniel.Suchodolski@pitradwar.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; direction: ltr; font-family: Tahoma; font-size: 10pt;" class="">Hi Phil and Johny,<br class=""><br class=""><br class="">Thank you for your response and advice.<br class=""><br class=""><br class="">The problem is resolved.<br class=""><br class="">A good idea was using command lsof. My application was regularly creating file descriptors.<br class=""><br class="">Now application is working stable.<br class=""><br class="">Thank you again.<br class=""><br class=""><br class="">Best regards,<br class="">Daniel<br class=""><br class=""><div style="font-family: 'Times New Roman'; font-size: 16px;" class=""><hr tabindex="-1" class=""><div id="divRpF115889" style="direction: ltr;" class=""><font size="2" face="Tahoma" class=""><b class="">Od:</b><span class="Apple-converted-space"> </span>Phil Mesnier [<a href="mailto:mesnierp@ociweb.com" class="">mesnierp@ociweb.com</a>]<br class=""><b class="">Wysłano:</b><span class="Apple-converted-space"> </span>13 stycznia 2017 12:58<br class=""><b class="">Do:</b><span class="Apple-converted-space"> </span>Daniel Suchodolski<br class=""><b class="">DW:</b><span class="Apple-converted-space"> </span><a href="mailto:tao-bugs@list.isis.vanderbilt.edu" class="">tao-bugs@list.isis.vanderbilt.edu</a><br class=""><b class="">Temat:</b><span class="Apple-converted-space"> </span>Re: [tao-bugs] Service after some time working idle (1 night) can't process requests<br class=""></font><br class=""></div><div class=""></div><div class=""><div dir="auto" class="" style="word-wrap: break-word;"><div dir="auto" class="" style="word-wrap: break-word;">HI Daniel,<div class=""><br class=""></div><div class="">In addition to what Johnny said, do you have log output from the server during the period from 11:38:25 to 15:49:37? I ask because that last client connection is assigned handle #1542 which is quite a large number for a supposedly idle server. Do you have other I/O happening on a regular basis, maybe writing to a file, or connecting to a database or something?</div><div class=""><br class=""></div><div class="">I'm guessing you have a resource leak somewhere, not closing a file or socket. </div><div class=""><br class=""></div><div class="">By default, TAO's reactor uses select() which typically has a limit of 1024 handles. Since the handle number is used as an index to a bit array, a high number such as #1542 will be out of bounds even if you only have a handful of sockets you are interested in. So the server is probably just in its normal run state, but unable to select on the client's connection.</div><div class=""><br class=""></div><div class="">You can try using lsof -p <server pid> which if I'm guessing right will show you 1500 or so open files or sockets.</div><div class=""><br class=""></div><div class="">In fact, I noticed that the first client connection you noted uses handle #28, while previously the server had opened connection to the naming service using handle #9. So 19 handles were consumed in 2.25 minutes. Later you have a 4 hour 11 minute gap with 1514 handles consumed, so in the first case roughly 8 handles per minute were leaked, and roughly 6 per minute in the latter case.</div><div class=""><br class=""></div><div class="">Now if you need all these open resources, perhaps you can switch to the dev_poll reactor rather than the default reactor. See docs/Options.html for information on setting reactor type via the advanced resource factory.</div><div class=""><br class=""></div><div class="">Best regards,</div><div class="">Phil</div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Jan 13, 2017, at 4:32 AM, Johnny Willemsen <<a href="mailto:jwillemsen@remedy.nl" class="" target="_blank">jwillemsen@remedy.nl</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="" style="margin-top: 0px; margin-bottom: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);">Hi,</div><div class="" style="margin-top: 0px; margin-bottom: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);"><br class=""></div><div class="" style="margin-top: 0px; margin-bottom: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);">Thanks for using the PRF form. Can you attach a debugger and see where the server is looping exactly?</div><div class="" style="margin-top: 0px; margin-bottom: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);"><br class=""></div><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;">Best regards,</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);"><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);"><pre class="moz-signature" cols="72" style="font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);">Johnny Willemsen
Remedy IT
Postbus 81 | 6930 AB Westervoort | The Netherlands
<a class="moz-txt-link-freetext" href="http://www.remedy.nl/" target="_blank">http://www.remedy.nl</a>
</pre><div class="moz-cite-prefix" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);">On 01/12/2017 02:32 PM, Daniel Suchodolski wrote:<br class=""></div><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);"><div class="" style="direction: ltr; font-family: Tahoma; font-size: 10pt;">Hi TAO,<br class=""><br class=""><br class=""> <span class="Apple-converted-space"> </span>TAO VERSION: 2.4.1<br class=""> <span class="Apple-converted-space"> </span>ACE VERSION: 6.4.1<br class=""><br class=""> <span class="Apple-converted-space"> </span>HOST MACHINE and OPERATING SYSTEM:<br class=""> <span class="Apple-converted-space"> </span>Debian 8 (Jessie)<br class=""><br class=""> <span class="Apple-converted-space"> </span>COMPILER: g++ (Debian 4.9.2-10) 4.9.2<span class="Apple-converted-space"> </span><br class=""><br class=""><br class=""> <span class="Apple-converted-space"> </span>DOES THE PROBLEM AFFECT:<br class=""> <span class="Apple-converted-space"> </span>EXECUTION? YES<br class=""><br class=""><br class=""> <span class="Apple-converted-space"> </span>SYNOPSIS:<br class="">CORBA service after some time working idle (1 night) can't process requests. (migration result)<br class=""><br class=""> <span class="Apple-converted-space"> </span>DESCRIPTION:<br class="">At the beginning I want to highlight that we use version of ACE-TAO compiled with option "threads 0". The problem started to happen with many services after migration to newest version of ACE TAO and operating system. Up to now the system worked stable on version 1.2.1/5.2.1 (linux lenny).<br class=""><br class="">How it works:<br class="">Server application registers CORBA service in NamingService.<br class="">If a client connects shortly after start of the server<br class="">then the server works fine: the server process request properly.<br class=""><br class="">After some time (for example 1 night) a client connects to the Server, but<br class="">when the client tries to use service then the server hangs up and use 100% of a processor.<br class="">Client is blocked by the server until the server is killed. Debugging, we found out that the problem is somewhere inside CORBA invocation. Very unclear are debug information seen during loading adatp3-services.svc, but we are not able fully interpret this issue.<br class=""><br class="">The services is run with the following ORBParameters:<br class=""> <span class="Apple-converted-space"> </span>-ORBDottedDecimalAddresses 1<br class=""> <span class="Apple-converted-space"> </span>-ORBDebug -ORBDebugLevel 10 -ORBVerboseLogging 2 -ORBInitRef NameService=corbaloc::server:30033/NameService<br class=""> <span class="Apple-converted-space"> </span>-ORBSvcConf adatp3-services.svc<br class=""><br class="">and adatp3-services.svc:<br class="">static Advanced_Resource_Factory "-ORBReactorMaskSignals 0 -ORBInputCDRAllocator null -ORBReactorType select_st -ORBConnec<br class="">tionCacheLock null"<br class="">static Server_Strategy_Factory "-ORBAllowReactivationOfSystemids 0"<br class="">static Client_Strategy_Factory "-ORBTransportMuxStrategy EXCLUSIVE -ORBClientConnectionHandler RW"<br class=""><br class=""><br class=""> <span class="Apple-converted-space"> </span>REPEAT BY:<br class="">Every Time<br class=""><br class=""> <span class="Apple-converted-space"> </span>TAO LOG:<br class=""><br class="">The Log of the server is showed below. The log is divided into parts:<br class=""><br class="">[Start Server]<br class="">[Client connect to Server after short time]<br class="">[Client is connecting after some time]<br class=""><br class=""></div></blockquote><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);"><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;">_______________________________________________</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);"><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;">tao-bugs mailing list</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);"><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;"><a href="mailto:tao-bugs@list.isis.vanderbilt.edu" class="" target="_blank">tao-bugs@list.isis.vanderbilt.edu</a></span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255);"><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;"><a href="http://list.isis.vanderbilt.edu/cgi-bin/mailman/listinfo/tao-bugs" class="" target="_blank">http://list.isis.vanderbilt.edu/cgi-bin/mailman/listinfo/tao-bugs</a></span></div></blockquote></div><br class=""><div class=""><div class="" style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; word-wrap: break-word;"><div class="" style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; word-wrap: break-word;"><div class=""><div class="" style="font-size: 12.8px;"><div class="" style="margin: 0px; padding: 0px; border: 0px;"><span class="" style="font-family: arial, helvetica, sans-serif; font-size: large;">--</span></div><div class="" style="margin: 0px; padding: 0px; border: 0px;"><span class="" style="font-family: arial, helvetica, sans-serif; font-size: large;">Phil Mesnier</span><br class=""></div></div><div class="" style="font-size: 12.8px;"><div class="" style="font-size: 12.8px;"><font class="" size="2" face="arial, helvetica, sans-serif">Principal Engineer & Partner</font></div><div class="" style="font-size: 12.8px;"><font class="" face="arial, helvetica, sans-serif"><br class=""></font></div><div class="" style="font-size: 12.8px;"><span class="" style="font-family: arial, helvetica, sans-serif;"><font class="" size="2">OCI | WE ARE SOFTWARE ENGINEERS.</font></span></div><div class="" style="font-size: 12.8px;"><span class="" style="font-family: arial, helvetica, sans-serif; font-size: x-small;">tel +1.314.579.0066 x225</span><br class=""></div><div class="" style="font-size: 12.8px;"><a href="http://ociweb.com/" class="" target="_blank">ociweb.com</a></div><div class=""><br class=""></div></div></div></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline"></div><br class=""></div></div></div></div></div></div><span style="font-size: 9pt; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-family: Calibri; color: rgb(89, 89, 89);" class="">PIT-RADWAR S.A.<br class="">z siedzibą w Warszawie<br class="">ul.Poligonowa 30<br class=""><br class="">Wpisana do Rejestru Przedsiębiorców Krajowego Rejestru Sądowego<br class="">przez Sąd Rejonowy dla<span class="Apple-converted-space"> </span><a href="http://m.st/" class="">m.st</a>. Warszawy w Warszawie, XIII Wydział Gospodarczy.<br class="">NR KRS 0000297470<br class="">NIP: 525 000 9298<br class="">Kapitał zakładowy 421 542 770,00 PLN w całości wpłacony.<span style="font-size: 6pt;" class=""><span class="Apple-converted-space"> </span><br class=""><br class=""></span><span style="font-size: 9pt; font-family: Calibri; color: rgb(89, 89, 89);" class=""><a href="http://www.pitradwar.com/" class="">www.pitradwar.com</a></span><span style="font-size: 6pt;" class=""><br class=""><br class=""></span><span style="font-size: 8pt; font-family: Calibri; color: rgb(89, 89, 89);" class="">Niniejsza wiadomość jest przeznaczona wyłącznie dla wskazanego w niej adresata i stanowi własność PIT-RADWAR S.A. Jeżeli nie jesteście Państwo adresatem tej wiadomości, bądź otrzymaliście ją przez pomyłkę, informujemy, że jej rozpowszechnianie lub kopiowanie oraz jakiekolwiek czynienie z niej użytku jest zakazane i może naruszać prawo. Prosimy o powiadomienie o powyższym nadawcy i trwałe usunięcie wiadomości wraz z załącznikami.<br class=""><font color="#009900" face="Webdings" size="4" class="">P</font><font color="#009900" face="verdana,arial,helvetica" size="1" class=""><span class="Apple-converted-space"> </span><strong class="">Zanim wydrukujesz, pomyśl o środowisku</strong></font><br class=""><br class="">This message is intended only for use of the named addressee and is the property of PIT-RADWAR S.A. If you are not the receiver of this e-mail or you have received it accidentally we inform that publishing or copying this message or any other using it is forbidden and may break the law. We kindly ask to inform the sender of the message about what is mentioned above and to remove the message permanently from your computer together with attached files.<br class=""><font color="#009900" face="Webdings" size="4" class="">P</font><font color="#009900" face="verdana,arial,helvetica" size="1" class=""><span class="Apple-converted-space"> </span><strong class="">Please consider the environment before printing this email</strong></font></span></span></div></blockquote></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div style="font-size: 12.8px;" class=""><div style="margin: 0px; padding: 0px; border: 0px;" class=""><span style="font-family: arial, helvetica, sans-serif; font-size: large;" class="">--</span></div><div style="margin: 0px; padding: 0px; border: 0px;" class=""><span style="font-family: arial, helvetica, sans-serif; font-size: large;" class="">Phil Mesnier</span><br class=""></div></div><div style="font-size: 12.8px;" class=""><div style="font-size: 12.8px;" class=""><font face="arial, helvetica, sans-serif" size="2" class="">Principal Engineer & Partner</font></div><div style="font-size: 12.8px;" class=""><font face="arial, helvetica, sans-serif" class=""><br class=""></font></div><div style="font-size: 12.8px;" class=""><span style="font-family: arial, helvetica, sans-serif;" class=""><font size="2" color="#000000" class="">OCI | WE ARE SOFTWARE ENGINEERS.</font></span></div><div style="font-size: 12.8px;" class=""><span style="font-family: arial, helvetica, sans-serif; font-size: x-small;" class="">tel +1.314.579.0066 x225</span><br class=""></div><div style="font-size: 12.8px;" class=""><a href="http://ociweb.com/" target="_blank" class="">ociweb.com</a></div><div class=""><br class=""></div></div></div></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br class=""></div></body></html>