[Ace-users] [tao-users] scalability/thread pool performance (500+ clients)

[Douglas C. Schmidt]

Hi Andrew,

   Thanks for using the PRF.

>  ACE VERSION: 5.6.3
>
>    HOST MACHINE and OPERATING SYSTEM: Windows XP
>        If on Windows based OS's, which version of WINSOCK do you
>        use?: 2.x
>
>    TARGET MACHINE and OPERATING SYSTEM, if different from HOST: Windows 2003
>    COMPILER NAME AND VERSION (AND PATCHLEVEL): VC 2008.
>
>    THE $ACE_ROOT/ace/config.h FILE [if you use a link to a platform-
>    specific file, simply state which one]: 
>
>#define ACE_HAS_STANDARD_CPP_LIBRARY 1
>#define ACE_DISABLE_WIN32_ERROR_WINDOWS
>#define ACE_DISABLE_WIN32_INCREASE_PRIORITY
>#define ACE_HAS_GPERF
>
>#include "ace/config-win32.h"
>
>    THE $ACE_ROOT/include/makeinclude/platform_macros.GNU FILE [if you
>    use a link to a platform-specific file, simply state which one
>    (unless this isn't used in this case, e.g., with Microsoft Visual
>    C++)]:
>
>    CONTENTS OF $ACE_ROOT/bin/MakeProjectCreator/config/default.features
>    (used by MPC when you generate your own makefiles):
>
>    AREA/CLASS/EXAMPLE AFFECTED:
>
>scalability/performance
>
>    DOES THE PROBLEM AFFECT:
>        COMPILATION?
>        LINKING?
>            On Unix systems, did you run make realclean first?
>        EXECUTION?
>        OTHER (please specify)?
>[Please indicate whether ACE, your application, or both are affected.]
>
>    SYNOPSIS:
>Poor scalability/thread pool performance with 500+ clients/
>
>    DESCRIPTION:
>
>I'm developing a grid-like system. Server (coordinator) should
>communicate with ~1000 clients (agents). Scenario: client connects to
>the server, and then constantly sends data - 512 bytes sequence, 33
>seq per second.
>
>I try to create simple server and client (no any data processing on
>server, only form stream on client and receive on server). Results:
>server can dispatch only ~200 agents on 100 mbit network with 50% cpu
>load (without thread pool there is 0% cpu load with same dispatch freq) and
>only 50% of the network bandwidth use.
>
>Exactly same WCF-based (sic!) sample can hold 600 clients without any
>problem - cpu load about 20-30% while network use ~100%. It seems,
>that TAO can't effectively dispatch more than 200 connections (for
>example, 100-150 connections causes < 1% cpu load). I know, that on
>windows TAO uses old select-based reactor, and seems this is really
>not right case for scalable systems... 

Yes, that's correct.  That's why TAO makes it easy to support other
concurrency models.

>Maybe, i missed something, and
>i just should tune thread pool settings/tune single threaded mode?

Right.

>Now i have this lines in my svc.conf:
>
>static Advanced_Resource_Factory "-ORBInputCDRAllocator null -ORBReactorMaskSignals 0 -ORBConnectionCacheLock null -ORBFlushingStrategy blocking"
>static Client_Strategy_Factory "-ORBProfileLock null -ORBClientConnectionHandler RW"
>static Server_Strategy_Factory "-ORBPOALock null -ORBAllowReactivationOfSystemids 0"
>
>This causes 0% server load with a single orb thread, and 50-60% with a
>pool, with no perfomance gain at all.
>
>OS - Windows XP/2003, Intel Core 2 duo, 100 mbit network.

I recommend you check out

TAO_ROOT/docs/configurations.html

and try using various concurrency models to see which one performs the
best for your usecase.  For example, try using "multiple threads,
single ORB, thread-per-connection" and see how that compares with
"multiple threads, single ORB, TAO thread-pool model".  Both
approaches should work better than the default reactive model.

Take care,

     Doug
-- 
Dr. Douglas C. Schmidt                       Professor and Associate Chair
Electrical Engineering and Computer Science  TEL: (615) 343-8197
Vanderbilt University                        WEB: www.dre.vanderbilt.edu/~schmidt
Nashville, TN 37203                          NET: d.schmidt at vanderbilt.edu