[Ace-users] [tao-users] scalability/thread pool performance (500+ clients)

[Andrew Solodovnikov]

  ACE VERSION: 5.6.3

    HOST MACHINE and OPERATING SYSTEM: Windows XP
        If on Windows based OS's, which version of WINSOCK do you
        use?: 2.x

    TARGET MACHINE and OPERATING SYSTEM, if different from HOST: Windows 2003
    COMPILER NAME AND VERSION (AND PATCHLEVEL): VC 2008.

    THE $ACE_ROOT/ace/config.h FILE [if you use a link to a platform-
    specific file, simply state which one]: 

#define ACE_HAS_STANDARD_CPP_LIBRARY 1
#define ACE_DISABLE_WIN32_ERROR_WINDOWS
#define ACE_DISABLE_WIN32_INCREASE_PRIORITY
#define ACE_HAS_GPERF

#include "ace/config-win32.h"

    THE $ACE_ROOT/include/makeinclude/platform_macros.GNU FILE [if you
    use a link to a platform-specific file, simply state which one
    (unless this isn't used in this case, e.g., with Microsoft Visual
    C++)]:

    CONTENTS OF $ACE_ROOT/bin/MakeProjectCreator/config/default.features
    (used by MPC when you generate your own makefiles):

    AREA/CLASS/EXAMPLE AFFECTED:

scalability/performance

    DOES THE PROBLEM AFFECT:
        COMPILATION?
        LINKING?
            On Unix systems, did you run make realclean first?
        EXECUTION?
        OTHER (please specify)?
[Please indicate whether ACE, your application, or both are affected.]

    SYNOPSIS:
Poor scalability/thread pool performance with 500+ clients/

    DESCRIPTION:

I'm developing a grid-like system. Server (coordinator) should
communicate with ~1000 clients (agents). Scenario: client connects to
the server, and then constantly sends data - 512 bytes sequence, 33
seq per second.

I try to create simple server and client (no any data processing on
server, only form stream on client and receive on server). Results:
server can dispatch only ~200 agents on 100 mbit network with 50% cpu
load (without thread pool there is 0% cpu load with same dispatch freq) and
only 50% of the network bandwidth use.

Exactly same WCF-based (sic!) sample can hold 600 clients without any
problem - cpu load about 20-30% while network use ~100%. It seems,
that TAO can't effectively dispatch more than 200 connections (for
example, 100-150 connections causes < 1% cpu load). I know, that on
windows TAO uses old select-based reactor, and seems this is really
not right case for scalable systems... Maybe, i missed something, and
i just should tune thread pool settings/tune single threaded mode?

Now i have this lines in my svc.conf:

static Advanced_Resource_Factory "-ORBInputCDRAllocator null -ORBReactorMaskSignals 0 -ORBConnectionCacheLock null -ORBFlushingStrategy blocking"
static Client_Strategy_Factory "-ORBProfileLock null -ORBClientConnectionHandler RW"
static Server_Strategy_Factory "-ORBPOALock null -ORBAllowReactivationOfSystemids 0"

This causes 0% server load with a single orb thread, and 50-60% with a
pool, with no perfomance gain at all.

OS - Windows XP/2003, Intel Core 2 duo, 100 mbit network.

Thanks!

--
Andrew Solodovnikov