[tao-bugs] Stale connections with BiDirGIOP

[Milan Cvetkovic]

I have created Bug 4207 in bugzilla, and submitted pull request #157 
with automated test and bug fix.

Hope it makes it,

Milan.

Milan Cvetkovic wrote:
> OK, answering to my own question to some extent...
>
> I narrowed the problem to Transport_Cache_Manager_T, and its use of
> Cache_ExtId::index_.
>
> First, how I see it working:
> ============================
>
> Transport_Cache_Manager_T uses ACE_Hash_Map_Manager to keep a mapping
> between Cache_ExtId and Cache_IntId. In case of IIOP (in my case it
> really was SSLIOP, but I doubt there is a difference there), Cache_ExtId
> represents IP-ADDR:PORT/index triple for a connection. IP-ADDR:PORT is
> the address that Transport connects to, and index is used to allow
> multiple connections to same ip/port address. All three values (address,
> port, index) are used to calculate hash when stored to ACE_Hash_Map_Manager
>
> When a new Transport is created, it is registered with cache manager,
> and it would create an entry using ip:port:index(0). When another
> transport is needed again, Transport_Cache_Manager_T::find_i looks up
> for an existing connection, and uses it if it is found and idle.
>
> The problem:
> ============
> Transport_Cache_Manager_T::find_i assumes that indexes of existing
> connections are all consecutive numbers starting with 0. It will try to
> lookup Transport with index=1 *only* if index=0 entry for the same
> IP:port exists, and if it is busy. If IP:port:index=0 entry is
> previously purged from the cache, Transport_Cache_Manager_T::find_i will
> never try to use index=1 (or any other index in the cache).
>
> This scenario is exactly what happens with BiDirGIOP when client
> disappears from the network, and later reconnects( and re-registers
> callback with same IP:PORT) value:
> - server caches first callback with IP:addr:index=0
> - client reconnects/re-registers
> - server caches the second callback with IP:addr:index=1
> - eventually, server cleans up cache entry with IP:addr:index=0
> - but it is never able to access the entry with IP:addr:index=1
>
> I am not too sure on the impact on regular TAO clients, since I didnt
> try it, but I would assume that:
> - if index=0 entry is busy, second transport is created
> - if index=0 entry's transport is closed, index=1
>    entry is purged from cache, and index=1 entry is no
>    longer reachable, until index=0 entry for the same IP:PORT is created.
>
> Potential solutions:
> ====================
> - I could fix Transport_Cache_Manager_T::unbind_i so it made sure
>    that the assumption made in find_i is true: If cache has M elements,
>    when removing an entry at index=N (where N is in [0,M), all remaining
>    entries for same IP:addr should have consecutive indexes
>    in range [0,M-1).
> - Alternatively, Transport_Cache_Manager_T can be rewritten
>    to actually use multi-hashmap. The existing implementation with
>    hash-map and indexes seems inappropriate and sub-optimal.
>    Or there is a good reason not to use multi-hash-map, that I am not
>    aware of...
>    It seems that this would touch more files in TAO though.
>
> I would like to contribute this patch. I would appreciate if someone
> could advise me, which direction should I take.
>
> Thanks, Milan.
>
> Milan Cvetkovic wrote:
>>      TAO VERSION: 2.2.1
>>      ACE VERSION: 6.2.1
>>
>>      HOST MACHINE and OPERATING SYSTEM: Debian wheezy on x86_64
>>
>>      THE $ACE_ROOT/ace/config.h FILE: config-linux.h
>>
>>      THE $ACE_ROOT/include/makeinclude/platform_macros.GNU FILE:
>> c++11 = 1
>> ssl = 1
>> include ${ACE_ROOT}/include/makeinclude/platform_linux.GNU
>>
>>      AREA/CLASS/EXAMPLE AFFECTED:
>>      BiDirGIOP / Transport_Cache_Manager_T / SSLIOP
>>      DOES THE PROBLEM AFFECT:
>>          EXECUTION: YES
>>
>>     SYNOPSIS: After loss of network connection from a client, server is
>> no longer able to invoke callback RPCs, even after client reconnected,
>> and resubmitted its callback IOR.
>>
>> DESCRIPTION:
>>
>> I have BiDirGIOP setup over SSLIOP. Client is behind firewall router on
>> 192.168.12.x network. Client incarnates callback object, listening on
>> 192.168.12.113:7770 and port 7771 for ss. Client contacts the server
>> over the internet, and it sends the IOR to callback object above. Server
>> later uses callback object to send various notifications. This setup
>> utilizes bidirectional GIOP, over SSLIOP.
>>
>> Everything works as desired, until client loses connectivity to server.
>> When client re-registers, server adds the new Transport to Transport
>> cache manager, however in some scenarios it does not remove the old
>> transport, and keeps using it for callbacks, failing on CORBA::TIMEOUT
>>
>> My understanding is that Transport_Cache_Manager keeps the hash map
>> table of all connections. These connections have the same key, being
>> issued from the same IP:port every time (in the example above,
>> 192.168.12.113:7771). In some cases, the server does not replace the
>> existing transport entry, but adds it with an increased index, and keeps
>> using index:0 for making callbacks.
>>
>> I am attaching the portions of TAO logs. Note that second registration
>> binds with index :1. The stale transport is kept with index :0.
>>
>> How do I control the content of Transport_Cache_Manager_T. I removed the
>> references to callback objects from server, however the transport is
>> still cached.
>>
>> Thanks, Milan.
>