[ace-bugs] ACE_Svc_Handler handle_input not called in some scenarios randomly when using ACE_SSL_Sock_Stream and ACE_Acceptor

Wed Apr 22 02:12:46 CDT 2020

Hi,

Thanks for the reproducer, I currently don't have time to review your
test and analyze it in detail for free. See
http://www.dre.vanderbilt.edu/~schmidt/TAO-support.html for an overview
of the ACE/TAO support model,
http://www.dre.vanderbilt.edu/~schmidt/commercial-support.html has an
overview of the support providers, including Remedy IT, the company I
work for.

Maybe someone else on the list has time to analyze this in detail, if
not, consider hiring someone, through the commercial services we can
assist you.

Best regards,

Johnny Willemsen
Remedy IT
http://www.remedy.nl

On 4/21/20 5:04 PM, Somisetty, Nagesh wrote:
> Hi,
> 
>  
> 
> I wrote a simple server and client example and built it with the latest
> ACE 6.5.8. I could still recreate the issue that was earlier reported.
> 
>  
> 
> Please find attached the tar file containing the server and client
> example applications written.
> 
> *Linux OS version of our development environment: Red Hat Enterprise
> Linux Server release 6.4*
> 
> *GCC version: gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-3)*
> 
> * *
> 
> Steps to run the server and client example applications are as follows.
> 
>  1. *Server*: tls_server_app <server_ip:port> <debug_level>
>     <path_to_SSL_key_file> <path_to_SSL_certificate_file>
>     <size_of_client_msg>
>     For example: tls_server_app 172.21.41.55:4443 info server.key
>     server.cert 494
>     The current client example sends a message of 494 bytes. This needs
>     to be changed accordingly for a different sized client message.
>  2. *Client*: tls_client_app <server_ip> <port>
>     <path_to_SSL_certificate_file>
>     For example: tls_client_app 172.21.41.55 4443 server.pem
>  3. Once we run the server and the client for about 20-30 minutes (or
>     sometimes even earlier), we see that the following log on the server
>     indicating that there is no handle_input received for the inactivity
>     time duration.
>     2020-04-21 14:08:54.333208 [19984] TCP Connection will be closed as
>     connection is inactive for long time.
> 
>  
> 
> Regards,
> 
> Nagesh
> 
> *From:*Johnny Willemsen <jwillemsen at remedy.nl>
> *Sent:* Tuesday, April 14, 2020 2:26 PM
> *To:* Somisetty, Nagesh <NSomisetty at tnsi.com>;
> ace-bugs at list.isis.vanderbilt.edu
> *Cc:* Zhao, Changchun <czhao at tnsi.com>; Santhappanavara, Sharath
> <SSanthappanavara at tnsi.com>; Gupta, Ajay <agupta at tnsi.com>
> *Subject:* Re: [ace-bugs] ACE_Svc_Handler handle_input not called in
> some scenarios randomly when using ACE_SSL_Sock_Stream and ACE_Acceptor
> 
>  
> 
> Hi,
> 
> Maybe also try a newer RHEL release with newer SSL version, maybe it is
> an issue in the underlying OS.
> 
> Johnny Willemsen
> Remedy IT
> http://www.remedy.nl
> 
> On 4/14/20 10:52 AM, Johnny Willemsen wrote:
>> Hi,
>>
>> 6.5.0 is the latest minor, 6.5.8 is the latest micro but not much has
>> very likely changed in this area.
>>
>> Please try to extend/add a unit test under ACE_wrappers/tests to
>> reproduce this issue.
>>
>> Regards,
>>
>> Johnny Willemsen
>> Remedy IT
>> http://www.remedy.nl
>>
>> On 4/13/20 2:59 PM, Somisetty, Nagesh wrote:
>>> Thanks Johnny.
>>>
>>>  
>>>
>>> I’ve also tried with the ACE 6.5.0, but it did not help. I was still
>>> observing the same issues as reported in the PRF.
>>>
>>>  
>>>
>>> Regards,
>>>
>>> Nagesh
>>>
>>>  
>>>
>>> *From:*Johnny Willemsen <jwillemsen at remedy.nl
> <mailto:jwillemsen at remedy.nl>>
>>> *Sent:* Monday, April 13, 2020 6:00 PM
>>> *To:* Somisetty, Nagesh <NSomisetty at tnsi.com
> <mailto:NSomisetty at tnsi.com>>;
>>> ace-bugs at list.isis.vanderbilt.edu
> <mailto:ace-bugs at list.isis.vanderbilt.edu>
>>> *Cc:* Zhao, Changchun <czhao at tnsi.com <mailto:czhao at tnsi.com>>;
> Santhappanavara, Sharath
>>> <SSanthappanavara at tnsi.com <mailto:SSanthappanavara at tnsi.com>>;
> Gupta, Ajay <agupta at tnsi.com <mailto:agupta at tnsi.com>>
>>> *Subject:* Re: [ace-bugs] ACE_Svc_Handler handle_input not called in
>>> some scenarios randomly when using ACE_SSL_Sock_Stream and ACE_Acceptor
>>>
>>>  
>>>
>>> Hi,
>>>
>>> Thanks for using the PRF form. 6.2.0 is already pretty old, can you trie
>>> ACE 6.5.8 which you can download from
>>> http://download.dre.vanderbilt.edu, that is the most recent micro
> release.
>>>
>>> Regards,
>>>
>>> Johnny Willemsen
>>>
>>> Remedy IT
>>>
>>> http://www.remedy.nl
>>>
>>> On 4/13/20 11:28 AM, Somisetty, Nagesh wrote:
>>>
>>> ACE VERSION: 6.2.0
>>>
>>> HOST MACHINE and OPERATING SYSTEM: RHEL Server 6.8
>>> If on Windows based OS's, which version of WINSOCK do you
>>> use?:
>>>
>>> TARGET MACHINE and OPERATING SYSTEM, if different from HOST: RHEL
>>> Server 6.8
>>> COMPILER NAME AND VERSION (AND PATCHLEVEL): g++ (GCC) 4.4.7 20120313
>>> (Red Hat 4.4.7-3)
>>>
>>> THE $ACE_ROOT/ace/config.h FILE [if you use a link to a platform-
>>> specific file, simply state which one]:
>>>
>>> THE $ACE_ROOT/include/makeinclude/platform_macros.GNU FILE [if you
>>> use a link to a platform-specific file, simply state which one
>>> (unless this isn't used in this case, e.g., with Microsoft Visual
>>> C++)]:
>>>
>>> CONTENTS OF $ACE_ROOT/bin/MakeProjectCreator/config/default.features
>>> (used by MPC when you generate your own makefiles):
>>>
>>> AREA/CLASS/EXAMPLE AFFECTED:
>>> ACE_Svc_Handler handle_input not called (in some cases) when using
>>> ACE_SSL_Sock_Stream and ACE_Acceptor
>>>
>>> DOES THE PROBLEM AFFECT:
>>> COMPILATION?
>>> LINKING?
>>> On Unix systems, did you run make realclean first?
>>> EXECUTION?
>>> OTHER (please specify)?
>>> Described problem affects the EXECUTION of the application
>>>
>>> SYNOPSIS:
>>> ACE_Svc_Handler handle_input not called in some scenarios when using
>>> ACE_Acceptor and ACE_SSL_Sock_Stream
>>>
>>> DESCRIPTION:
>>> 1) I have an ACE_Acceptor based server application that works either
>>> as a TCP or TLS server.
>>> 2) The TCP and TLS services are defined as below,
>>> typedef ACE_Acceptor<TCPServer, ACE_SOCK_ACCEPTOR> TCPService;
>>> typedef ACE_Acceptor<TLSServer, ACE_SSL_SOCK_ACCEPTOR> TLSService;
>>> where TCPServer and TLSServer are derived from ACE_Svc_Handler as below
>>> class TCPServer : public ACE_Svc_Handler<ACE_SOCK_STREAM,
>>> ACE_NULL_SYNCH>
>>> class TLSServer : public ACE_Svc_Handler<ACE_SSL_SOCK_STREAM,
>>> ACE_NULL_SYNCH>
>>> Both these TCPServer and TLSServer have implemented the handle_input
>>> callbacks for handling of incoming events from the socket.
>>> 3) Our observation is that while using the TLSServer, we see that
>>> the handle_input is not called in a few cases as listed below.
>>> a) When the application attempts to do a peer().recv(buff,len) on
>>> receiving a handle_input and the application buffer size remaining
>>> is smaller than the number of bytes available on the socket to be
>>> read, only the number of bytes that can be read into the buffer are
>>> actually read. After this the application expects another
>>> handle_input for the remaining bytes that are still on the socket
>>> that are yet to be read. In this scenario, we never receive the
>>> subsequent handle_input event so that we can do the next
>>> peer().recv(buf,len) call to read the remaining bytes from the
>>> socket. As a workaround for this, we changed the buffer allocation
>>> logic to ensure the application doesn't wait for the next
>>> handle_input if there are more bytes than the remaining buffer size.
>>> We solved this problem with the workaround, but we encountered the
>>> issues described in 3b below.
>>> b) On running load tests with a TLS client which keeps sending https
>>> requests continuously, we found that the server randomly freezes
>>> after anywhere between 1 and 30 minutes and when we check it looks
>>> like we never received a handle_input event at the TLSServer. There
>>> is no logic on the client that causes it to stop sending requests.
>>> So, it has to be something to do with the handle_input events not
>>> being notified to the application.
>>> 4) Please note that the issues 3a and 3b are not observed while
>>> using TCPServer and TCP clients. The logic in the handle_input for
>>> both TCP and TLS is exactly the same.
>>>
>>>
>>> REPEAT BY:
>>> [What you did to get the error; include test program or session
>>> transcript if at all possible. ]
>>>
>>> SAMPLE FIX/WORKAROUND:
>>> No fix available. Looking for a solution or an explanation.
>>>
>>> ------------------------------------------------------------------------
>>>
>>> This email has been scanned for email related threats and delivered
>>> safely by Mimecast.
>>> For more information please visit http://www.mimecast.com
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> ace-bugs mailing list
>>>
>>> ace-bugs at list.isis.vanderbilt.edu
> <mailto:ace-bugs at list.isis.vanderbilt.edu>
> <mailto:ace-bugs at list.isis.vanderbilt.edu>
>>>
>>> http://list.isis.vanderbilt.edu/cgi-bin/mailman/listinfo/ace-bugs
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>> This email has been scanned for email related threats and delivered
>>> safely by Mimecast.
>>> For more information please visit http://www.mimecast.com
>>> ------------------------------------------------------------------------
>> _______________________________________________
>> ace-bugs mailing list
>> ace-bugs at list.isis.vanderbilt.edu
> <mailto:ace-bugs at list.isis.vanderbilt.edu>
>> http://list.isis.vanderbilt.edu/cgi-bin/mailman/listinfo/ace-bugs
> 
> 
> 
> ------------------------------------------------------------------------
> This email has been scanned for email related threats and delivered
> safely by Mimecast.
> For more information please visit http://www.mimecast.com
> ------------------------------------------------------------------------