### Configuring TAO's Components

### Overview

As described in the options documentation, various components in TAO can be customized by specifying options for those components. This document illustrates how to combine these options in order to affect the behavior and performance of a TAO ORB, particularly its concurrency model.

TAO configures itself using the ACE Service Configurator framework. Thus, options are specified in the familiar `svc.conf` file (if you want to use a different file name, use the `-ORBSvcConf` option). You can also setup default configurations for your programs. Please see the Programming Considerations for more detailed discussion on this.

### Roadmap

> Details for the following configurations are provided.
> - Configurating key components:
> - Server Concurrency Strategy.
> - Number of ORBs.
> - POA.
> - Collocation Table.
> - Forwarding Profile
> - orbsvcs Library
> - Configuration examples
> - Single-threaded, reactive model.
> - Multiple threads, single ORB, thread-per-connection model.
> - Multiple threads, single ORB, TAO thread-pool model.
> - Multiple threads, single ORB, Real-time CORBA thread-pool model.
> - Multiple threads, multiple ORBs, reactive model.
> - Multiple threads, multiple ORBs, thread-per-connection model.
> - Multiple threads, ORB-per-thread, thread-pool reactive model.
> - Each configuration has the following information:

> Typical Use | A brief description of the scenario and its typical use.
> Number of Threads | The number of threads used by ORB-related activities.
> Thread Creator | Identifies the creator of the threads discussed above.
> Thread task | Describes what task is undertaken for each thread.
> Options | Specifies the options for each service in order to utilize this configuration.

> - Programming Considerations
> - Configuration for Homogenous Systems
> - Compile-time options
> - Runtime-time
> - Configuration Suggestions

### Configuring Key ORB Components

- Number of ORBs --
TAO can assign multiple endpoints to an ORB. Therefore, it is not necessary to create multiple ORBs to accept requests from multiple endpoints. However, multiple ORBs can be used to support different policies within the same process, e.g., handling requests in different thread priorities. Multiple ORBs are most commonly used in the "ORB per-priority" pattern to avoid priority inversion in real-time system.

- Server concurrency strategy -- The default server strategy factory provided by down support two different concurrency strategy. It can be specified by adding the `-ORBConcurrency` flag in the `Server_Strategy_Factory` entry of the `svc.conf` file. This specifies the concurrency strategy an ORB uses. This strategy is orthogonal the the number of ORBs or threads that are configured in a server process.

- `reactive`: The ORB handles requests reactively, i.e., the ORB runs in one thread and service multiple requests/connections simultaneously using the ACE_Reactor, which uses `select` or a similar event demultiplexing mechanism supported by the platform.

- `thread-per-connection`: The ORB handles new connections by spawning a new thread whose job is to service requests coming from the connection.

- Thread Pools
-- TAO supports several types of thread pools.

- `reactive`: In this approach, each thread in the thread pool has an ORB that accepts and processes requests reactively. This is also called as the "ORB-per-thread" model

- `leader/followers`: In this model, the user must create several threads, all of which invoke `ORB::run`, the ORB will select one of the threads to wait for incoming requests. This thread is called the leader thread and will process the first request that arrives to the ORB, but before doing so the ORB will selects another thread in the pool to become the leader. In other words the threads in the pool take turns to process the events.

- Collocation Table -- An ORB can have several listening endpoints. If there are several ORBs in a process and a global collocation table is used, then all objects in the same process are considered collocated. If not, only objects reside in the same ORB are considered collocated. You can control the usage of global collocation table by passing the `-ORBCollocation` flag as an argument of `ORB_init` (most often thru the command line flags.)

- Forwarding Profile -- When multiple threads using the same `CORBA::Object` and using forwarding, it is necessary to protect the forwarding `Profile`, which is part of the CORBA::Object, against multiple access. Therefore a mutex lock is used by default to ensure proper access.

  Using the switch `-ORBProfileLock` this policy can be deactivated specifying `-ORBProfileLock null`. The primary reason for doing this is to improve performance when no forwarding is used or no multithreading with access to shared `CORBA::Object`'s. Using a null mutex reduces the overhead compared with using a regular mutex lock.

- orbsvcs Library -- By default, TAO's Makefiles build all the services that TAO supports. To reduce build time you can exclude unused services, by defining the `TAO_ORBSVCS` makefile variable. We recommend using one of these two approaches:

1. In your `$(ACE_ROOT)/include/makeinclude/platform_macros.GNU` file,
2. On the make command line, e.g., `make TAO_ORBSVCS=RTEvent`, or
3. Set (and export) a `TAO_ORBSVCS` environment variable.

Please see the `ORBSVCS Makefile` for the default setting of `TAO_ORBSVCS`.

Please note that the Naming Service will always be built, even if Naming is not specified in `TAO_ORBSVCS`. That's because many examples, tests, and presumably applications use it.

### Configuration Examples

The following are common ORB configurations used by TAO applications.

- Single-threaded, reactive model.

Typical Use | This is the default configuration of TAO, where one thread handles requests from multiple clients via a one Reactor. It is appropriate when the requests (1) take a fixed, relatively uniform amount of time and (2) are largely compute bound.
Number of Threads | 1
Thread Creator | OS or whoever creates the main ORB thread in a process.
Thread task | The single thread processes all connection requests and CORBA messages.
Synchronization considerations | Application servants needn't be concerned with synchronizing their interactions since there's only one thread active with this model.

Options

The default settings should work just fine (by default, the `-ORBReactorType` is `tp`). However, you can apply the following options to improve performance:
`TAO_Advanced_Resource_Factory`: `-ORBReactorType select_st`, `-ORBInputCDRAllocator null`
`TAO_Server_Strategy_Factory`: `-ORBconcurrency reactive` (default), `-ORBPOALock null`
`TAO_Client_Strategy_Factory`: `-ORBConnectorLock null`

Check out the `Grid` for an example of a default configuration and `this` for an example of settings that would provide better performance.

- Multiple threads, single ORB, thread-per-connection model.

Typical Use | This configuration spawns a new thread to serve requests from a new connection. This approach works well when there are multiple connections active simultaneously and each request-per-connection may take a fair amount of time to execute.
Number of Threads | 1 thread for the ORB, plus 1 thread for each connection.
Thread Creator | Programmer must set up the main thread which the ORB lives. The ORB is responsible to create new threads upon new connections.
Thread task | The main thread handles new connections and spawns new threads for them. Other threads handle requests for established connections.
Synchronization considerations | To avoid race conditions, application servants may need to synchronize their methods if multiple clients can access them concurrently.
Options | `TAO_Server_Strategy_Factory`: `-ORBConcurrency thread-per-connection`

`IDL_Cubit` is a good example on using multiple threads, thread-per-connection configuration.

- Multiple threads, single ORB, TAO thread-pool model.

Typical Use | This model implements a highly optimized, TAO-specific thread pool that minimizes context switching and thread creation costs. In this model, the programmer is responsible of spawning a group of threads, creating an ORB instance, and then instructing all the threads to run the ORB event loop. When a request comes in, one of these waiting threads in the pool will handle the request.
Number of Threads | Thread for the ORB, plus the number of threads used by the thread pool.
Thread Creator | Pre-spawned by the main thread.
Thread task | Blocking on the reactor to wait for its turn to handle a request.
Synchronization considerations | To avoid race conditions, application servants may need to synchronize their methods if multiple clients or multiple thread from the same client can access them concurrently.
Options | The default ORB settings support this concurrency configuration, though you'll need to spawn the extra threads in the pool explicitly.

`MT_Server` is a good example on using multiple threads, single ORB, TAO thread-pool configuration.

- Multiple threads, single ORB, Real-time CORBA thread-pool model.

Typical Use

This model implements a highly optimized Real-time CORBA thread pool that minimizes context switching, and thread creation costs. In this model, the application is responsible for starting up the ORB and then calling the Real-time CORBA thread pool factory methods that internally spawn a pool of threads. When a request comes in, one of these waiting threads in the pool will handle the request in accordance with the various CORBA policies.

Number of Threads | Thread for the ORB, plus the number of threads used by the thread pool.
Thread Creator | Static threads are pre-spawned by the ORB, which also spawns any dynamic threads on-demand.
Thread task | Blocking on the reactor to wait for its turn to handle a request.
Synchronization considerations | To avoid race conditions, application servants may need to synchronize their methods if multiple clients or multiple thread from the same client can access them concurrently.
Options | The ORB must be configured to work with Real-time CORBA, at which point the default settings work fine.

`Thread_Pool` is a good example on using multiple threads, single ORB, Real-time CORBA thread-pool configuration.

- Multiple threads, multiple ORBs, reactive model.

Typical Use | In this configuration, there are multiple ORBs in a process with multiple threads. Each thread handles requests reactively. This model is good for hard real-time applications that require different thread priorities for the various ORBs.
Number of Threads | One thread for each ORB.
Thread Creator | The main process (thread).
Thread task | Service the requests from associating ORB.
Synchronization considerations | Application servants needn't be concerned with synchronizing their interactions if there's no cross ORB/thread access.
Options | The default settings just works fine. However, one could use `TAO_Advanced_Resource_Factory`: `-ORBReactorType "which"`, for a thread-safe platform-specific reactor.

- Multiple threads, multiple ORBs, thread-per-connection model.

Typical Use | This approach provides a range of thread priorities plus connections that don't interfere with each others.
Number of Threads | One thread for each ORB, plus one thread for each connection.
Thread Creator | Main threads creates threads running ORBs. They, in turns, create connection handling threads.
Thread task | There are threads running ORB's event loops which handle connection requests and handler threads which service requests from established connections.
Synchronization considerations | To avoid race conditions, application servants may need to synchronize their methods if multiple clients can access them concurrently.
Options | `TAO_Server_Strategy_Factory`: `-ORBConcurrency thread-per-connection`

`MT_Cubit` is a good example on using multiple threads, multiple ORBs, and thread-per-connection configuration.

- Multiple threads, multiple ORBs, thread-pool model.

Typical Use | This model incorporates the advantage of using thread-pool while allowing hard real-time system to handle requests in different priority.
Number of Threads | One thread for each ORB, plus the total number of threads in all thread pools
Thread Creator | Pre-spawned by the main thread.
Thread task | Handle incoming request for the ORB event loop it is waiting on.
Synchronization considerations | Application servants needn't be concerned with synchronizing their interactions if there's no cross ORB/thread access.
Options | The default settings work well for this

### Programming Considerations

There are several ways to pass option flags into TAO's components.

- The plain vanilla approach is do nothing. All TAO components use their default settings.

- The most common use case is to use a file called `svc.conf`. On most platforms, TAO programs automatically search and read in the file. The disadvantage of this approach is you always need a `svc.conf` file if you want to do use non-default configuration.

- You can use `-ORBSvcConf filename` to use a config file that is not called `svc.conf`. Specifying `-ORBSvcConf` exclude the reading of default `svc.conf` file.

- If you don't want the application users to worry about setting up or knowing about `svc.conf` files, you can call `TAO_Internal::default_svc_conf_entries()` before calling the first `ORB_init()` in your program to set up the default svc.conf entries. In this case, if a TAO application cannot find a svc.conf file, it will configure TAO's components using the default settings. You can still use a `svc.conf` file or use `-ORBSvcConf` option to tune the program.

- TAO programs evaluate the configuration settings in the following order:

  1. File specified in `-ORBSvcConf` command-line option, if one exist. Otherwise, the `svc.conf` in the start-up directory will be evaluated, if one exist.
  2. Default entries set by `TAO_Internal::default_svc_conf_entries()`, if ones exist.
  3. Default configuration as specified in this document.

  Notice that the first encountered component settings are always the ones take effect. For example, if you set the entries for `Resource_Factory` and `Server_Strategy_Factory` using `TAO_Internal::default_svc_conf_entries()` in a program and you also have a file called `svc.conf` which has an entry for `Resource_Factory`. This program will use the entry for `Resource_Factory` in the `svc.conf` file, the entry for `Server_Strategy_Factory` set in the program, and the in-stock `Client_Strategy_Factory` that TAO defines.

- Some platforms do not support reading of `svc.conf` files or perhaps you would rather not to use this feature. In these cases, you can define `TAO_PLATFORM_SVC_CONF_FILE_NOTSUP` in your ACE `config.h` file and recompile the TAO libraries. When this flag is set, TAO programs will not try to search for the default `svc.conf` file. You can still use `-ORBSvcConf` to initialize the components (assuming the platform supports it).

  On these platform, you can alter the default settings for TAO components by defining the following macros in your `config.h` file:

  - `TAO_DEFAULT_RESOURCE_FACTORY_ARGS`
  - `TAO_ADVANCED_RESOURCE_FACTORY_ARGS`
  - `TAO_DEFAULT_SERVER_STRATEGY_FACTORY_ARGS`
  - `TAO_DEFAULT_CLIENT_STRATEGY_FACTORY_ARGS`

  The ACE Makefiles `fakesvcconf` flag can be used to define `TAO_PLATFORM_SVC_CONF_FILE_NOTSUP`. To define that macro, just add `fakesvcconf=1` to your `make` invocation.

  See `orbconf.h` for an example.

### Configuration for Homogenous Systems

- Compile-time options

  Many real-time applications run on homogenous environments, TAO (and ACE) can take advantage of this fact by simplifying the server side demarshaling; to enable this feature you have to edit the `$ACE_ROOT/ace/OS.h` file and enable the macro `ACE_DISABLE_SWAP_ON_READ`.

  In this systems it is also common that server and the client startup and shutdown simultaneously, in those circumstances there is no need to check the timestamps in the POA, another macro (`POA_NO_TIMESTAMP`) can be used for this purpose.

  Users running in embebbed systems may also need to modify the default options for TAO, the macros `TAO_DEFAULT_RESOURCE_FACTORY_ARGS`, `TAO_ADVANCED_RESOURCE_FACTORY_ARGS`, `TAO_DEFAULT_CLIENT_STRATEGY_FACTORY_ARGS` and `TAO_DEFAULT_SERVER_STRATEGY_FACTORY_ARGS` can be used for those purposes. If the footprint size is an issue users may consider writing custom strategy factories that only create the right strategies, this eliminates the parsing code for the different options.

- Run-time options

  If the only ORB running is TAO and there is no need to be IIOP interoperable the option `-ORBGIOPlite` can be used to reduce the message size and the processing time.

  Unix systems that support local IPC (formerly known as Unix domain sockets) can take advantage of TAO's UIOP pluggable transport protocol to improve performance considerably. To use TAO's UIOP pluggable protocol, simply specify a UIOP endpoint on the command line using the `-ORBEndpoint` option described in the options documentation. Further performance improvement can be achieved by using the UIOP protocol in combination with the `-ORBGIOPlite` option. Additional information about TAO's UIOP pluggable protocol can be found in the release notes.

  Some embedded systems run without the benefit of a DNS server, in that case they can use the `-ORBDottedDecimalAddresses` option; the ORB will avoid the use of hostnames in the profiles it generates, thus clients don't need to do any name resolution. Use the compile-time define `TAO_USES_DOTTED_DECIMAL_ADDRESSES` in `$TAO_ROOT/tao/orbconf.h` to make this the default behavior.

### Configuration Suggestions

Choosing the right configuration is hard and, of course, depends on your application. In the following section we will attempt to describe some motivations for features in TAO, hopefully that can guide you through the choice of your configuration options.

- ORB-per-thread -- The main motivation behind this options is to minimize priority invertion, since threads share no ORB resources no locking is required and thus, priority is preserved in most cases (assuming proper support from the OS). If you are not too concerned about priority inversion try to use a single ORB, using ORB-per-thread has some tradeoffs (like calling ORB_init on each thread, activation of a servant is more complicated, etc.) Some of the problems, can be minimized, but they require even more careful analysis.

  As the reader will note this is a delicate configuration option, the rule of thumb should be not to use ORB-per-thread unless it is really required.

- Collocation tables -- Why would an application not want to use the global collocation table? Because a collocated method invocation is run in the client's thread-of-control. If objects are to serve requests only at a well known priority the application can be configured with the ORB-per-thread option, and the object is activated only in the thread (ORB) corresponding to the desired priority. But using a global table would subert the priority assignment (because calls would run at the priority of the client).

- Single-threaded vs. Multi-threaded Connection Handlers -- The `Client_Connection_Handler` is the component in TAO that writes the requests to the underlying transport socket; this is also the component that reads the response back from the server.

While waiting for this response new requests to the local ORB can arrive, this is the so-called nested upcall support. TAO supports two mechanisms for handling nested upcalls, the default uses the leader-follower model to allow multiple threads to wait on a single reactor for several concurrent requests; sometimes this configuration can be an overkill, if only one thread is using a reactor at the same time a lighter weight implementation can be used.

This configuration is controlled by the `-ORBClientConnectionHandler` option, good opportunities to use this option are:

- Single threaded servers
- Servers running in ORB-per-thread mode (pseudo single threaded.)
- Pure clients that will never receive a request

- Allocator for input CDR streams -- Normally the application has no access to this buffer, and it is only used on the demarshaling of arguments (or results). It is almost always better to use the "`-ORBInputCDRAllocator null`" option since it will allocate memory from a thread specific allocator and it will not need locks to manage that memory.

In some cases the user may gain access to the CDR stream buffer: TAO makes no copies when demarshaling octet sequences, instead the octet sequence simply points to the CDR buffer, since the octet sequence does not own this buffer a copy must be made if the user wants to keep the buffer after the upcall.

The user can, however, increase the reference count on the CDR stream buffer, thus allowing her to extend the lifetime of this buffer. Still passing this buffer to another thread and attempting to release it in that thread will result in some memory leak or corruption. Users willing to use this feature of TAO can still do so, if they use a global allocator for their input CDR stream, but that will introduce extra locking on the critical path.

As the reader can see this is an option that has limited applicability and requires careful consideration of the tradeoffs involved.

Back to the TAO component options and tuning documentation.