> ## 2 Applications

> Applications are used for "packaging" system components. An application consists of a set of resources, such as modules, registered names, and processes.

> By structuring the system into a well-defined set of applications, the system designer is forced to think about the system in terms of its sub-components and to decide which functionality each application should have.

> ### 2.1 Programming an Application

> Several operations are possible with an application. In particular, we can load, unload, start and stop an application.

> When an application is loaded, the system checks that all the resources the application will need are present. If loading is successful, the application can be started at a later time.

> When an application is started, an `application_master` process is created. Application master is the group leader of all the processes in the application.

> The application master is aware of every process in the application. Thus, if the application was stopped the application master would terminate all processes, for which this application is responsible, in a controlled fashion.

> When an application is unloaded, all code relating to the application is removed.

> Application Controller and Applications

> All operations on applications are coordinated by a single process with the name `application_controller`. The shaded circles in the above diagram represent applications. Each individual application is controlled by an application master.

> ### 2.2 The Application Resource File

> The resources required by each application are defined in an application resource file. These files have the extension `.app`, which specifies the resources required by the application and how the application should be started.

> A resource file (`ApplName.app`) has the following syntax:

>     {application, ApplName,
>       [{description,  String},
>        {vsn,          String},
>        {id,           String},
>        {modules,      [{Mod1,Vsn1}, Mod2, {Mod3,Vsn3} .., {ModN,VsnN}]},
>        {maxP,         Int | infinity},
>        {maxT,         Seconds | infinity},
>        {registered,   [Name1, Name2, ...]},
>        {applications, [Appl1, Appl2, .., ApplN]},
>        {included_applications, [Appl1, Appl2, .., ApplN]},
>        {env,          [{Par1, Val1}, {Par2, Val2} .., {ParN, ValN}]},
>        {mod,          {Mod, StartArgs}},
>        {start_phases, [{Phase, PhaseArgs}]}]}.

> The keys have the following meanings:

> - `description`
>   Textual description of the application.
> - `vsn`
>   Version of the application. It should not contain a directory separator.
> - `id`
>   Product identification of the application.
> - `modules`
>   List of all the modules and their versions introduced by this application. A module can be listed without a version, with only the module name stated. A module can only be defined in one application.
> - `maxP`
>   Maximum allowed number of simultaneous processes which this application can manage (or the atom `infinity`). The key `maxP` is optional and defaults to `infinity`.
> - `maxT`
>   Maximum time that the application can run (or the atom `infinity`). The key `maxT` is optional and defaults to `infinity`.
> - `registered`
>   Lists all the names of registered processes used in this application.
> - `applications`
>   List of applications which must be started before this application can be started. Most applications have dependencies to the `kernel` and `stdlib` applications.
> - `included_applications`
>   List of applications which are included by this application. An included application is loaded, but not started, by the `application_controller`. Processes implemented in an included application should be placed underneath a supervisor in the including application. This key is optional and defaults to an empty list.
> - `env`
>   List of the environment variables in the application. Each parameter `ParX` is an atom, and the associated `ValX` can be any term. The `env` key is optional and defaults to an empty list.
> - `mod`
>   Application call back module of the application behavior. The application master starts the application by evaluating the function `Mod:start(Type, StartArgs)`, refer to the kernel reference manual. When the application has stopped, by command or because it terminates, the application master calls `Mod:stop(State)` to let the application clean up. If no `State` was returned from `Mod:start/2`, then `Mod:stop([])` is called. The `mod` key should be omitted for applications which are code libraries, such as the `stdlib` application. These applications have no dynamic behavior of their own and should not have a start function.
> - `start_phases`
>   List of start phases and the attached start arguments for the application. The application master starts the application by evaluating the function `Mod:start_phase(Phase, Type,PhaseArgs)`, for each defined start phase. (See also the Kernel reference manual). Each parameter `Phase` is an atom, and the associated `PhaseArgs` is a list of any terms. The key `start_phases` is optional, but the behavior of the system is dependent on the key being defined or not (see kernel reference manual and the `Starting Applications` chapter below).

> ### 2.3 The Application Directory

> Each application should be placed in a separate directory. This directory should be divided into several sub-directories. The following sub-directories should exist in the application directory `.../ApplName`:

> - `src`
> - `ebin`
> - `priv`
> - `include`

> This structure is necessary because tools such as `systools` rely on this structure to be able to generate boot scripts and release packages. The release handling procedure on target machines also depends on this structure to be able to upgrade to new releases.

> In the target environment, the application directory name contains the application version.

> The following description of the sub-directories listed also indicates if the sub-directory is mandatory or optional.
> - `src`.
>   This directory contains the source code. This directory is mandatory in the development environment, but optional in the target environment. If source code is written in several different languages, a sub-directory with the name `e_src` can be created below the `src` directory to store the Erlang source code.
> - `ebin`.
>   This directory contains the Erlang object code, for example `beam` files. The application resource file should also be placed here. This directory is mandatory.
> - `priv`.
>   This directory is used for application specific files. For example, `C` executables should be placed here, or in a `bin` directory below the `priv` sub-directory. The application designer can determine the directory structure below `priv`. The function `code:priv_dir/1` should be used to access the `priv` sub-directory. This directory is optional. It can be omitted if the application does not include any application specific files.
> - `include`.
>   Included files must be placed in this directory. The application exports these files for inclusion in other application modules. The `include_lib("ApplName/include/File.hrl")` module attribute should be used to include a file from another application. This directory is optional.

> ### 2.4 Configuring an Application

> Application specific configuration parameters should be specified in the `.env` key in the resource file. The application can then call `application:get_env(ApplName, Parameter)` to retrieve the values for the configuration parameters. You can also override configuration parameters by using the system configuration file. This file is specified with the command line parameter `-config FileName`. The file `FileName.config` contains a list of configuration parameters for the applications. For example:

>     [{sasl, [{sasl_error_logger, tty},
>              {errlog_type, error},
>              {disk_space_check_interval, 10}]},
>      ...
>      {ApplNameN, [{Par1, Val1},
>                   {Par2, Val2}]}].

> The parameters over ride can also be executed from the command line:

>     erl -ApplName Par1 Val1 Par2 Val2 ...

> | Each term should be an Erlang term. However, in the Unix shell, the term must be enclosed in single quotation marks. For example: `'{file, "a.log"}'`.

> ### 2.5 The Application Master

> Each individual application is controlled by an application master.

> Example of an Application

> The application master is responsible for all processes running in an application. The application master can kill all processes which belong to the application.

> The application master assumes that the process started by the start function is responsible for the internal details of the application. This process is assigned the following special role:

> - If this process terminates, then the application master assumes that the application as a whole has terminated.
> - If the process started first terminates with a normal exit, then the application is assumed to have terminated correctly.
> - If the process started first terminates with an abnormal exit, then the application is assumed to have terminated with an error.

> An application can be started in one of three modes: permanent, transient, or temporary. Default value is temporary. The mode specifies what happens if the application dies.

> - If a permanent application dies, all other applications are also terminated.
> - If a transient application dies normally, this is reported and no other applications are terminated. If a transient application dies abnormally, all other applications are also terminated.
> - If a temporary application dies this is reported and no other applications are terminated. In this way, an application can run in test mode, without disturbing the other applications.

> ### 2.6 Included Applications

> An application can include other applications. Processes implemented in an included application should be placed underneath a supervisor in the including application. This way, you include applications in order to group them together.

> Primary Application and Included Applications

> An application which is not included by any other application is called a primary application. An application can only be included by one other application. If you want several applications to include an application, it has to be designed as a library application without a start function (the `mod` key in the .app file).

> When an application is loaded, the application controller ensures that all included applications are also loaded. When an application is started , the application controller will only start the primary application by default (see chapter, `Starting Applications`).

> | The information in the following paragraph only applies to the Erlang/OTP environment.

> When building a release, the `included_applications` key, which is specified in the application resource file, can be overridden with the `.rel` file. Therefore, all applications have the implicit environment (`env`) variable `included_applications` in order to read the current configuration. This is useful when an application has multiple start parts (included applications) depending on customer requirements, and where there is no need install or load excluded parts in the system. Refer to the section Release Structure in the System Architecture Support Libraries Manual for more details.

> ### 2.7 Distributed Applications

> In a distributed system with several cooperating Erlang nodes, there is a need to control certain applications in a distributed manner. Some applications may be specified to run on one of several nodes. The application controllers on the different nodes can be arranged to monitor each other. If one node goes down, the application controller on another node notices this and restart the application on its node. These applications are called distributed applications, as opposed to local applications, which are always started on the local node. An example of a standard local application is `kernel`; there is always one local instance of `kernel` running on an Erlang node.

> With this definition, a local application may be distributed in the sense that it uses services on other nodes, or cooperates with applications on other nodes. For example, the application controller sees the Mnesia DBMS started as a local application. In this description, only the control of applications is discussed.

> Because a distributed application may move between nodes, some addressing mechanism is required to ensure that it can be addressed by other applications, regardless on which node it currently executes. This issue is not addressed here, but the standard Erlang modules `global` or `pg` can be used for this purpose.

> #### 2.7.1 Specifying Distributed Applications

> The configuration parameter `distributed` for the application `kernel` defines which applications are distributed, and on which nodes they may run. This parameter is of the form `{ApplName,[NodeDesc]}` or `{ApplName, Time, [NodeDesc]}`, where `NodeDesc = Node |{Node, ..., Node}`. This data structure specifies a list of nodes where the application `ApplName` may execute, and the order in which these nodes should be used to start the application. If the nodes are specified in a tuple, the order is undefined. If a node crashes and `Time` has been specified, then the application controller will wait for `Time` milliseconds before attempting to restart the application on another node. If `Time` is not specified, it defaults to 0, and if a node goes down the application is restarted immediately on another node.

> All involved nodes have to have the same distribution specification (nodes which an application may run on and restart time) for all distributed applications on all involved nodes. This is easiest accomplished by using the same configuration parameter `distributed` on all involved nodes. All nodes that are connected by normal connections (as opposed to hidden connections) and run the distributed application controller (`dist_ac`) are involved (see the Kernel reference manual for more information on normal/hidden connections and/or on `dist_ac`).

> Distribution specifications can also be set by `application:load/2`. Observe that the management of a distributed application cannot be expected to function correctly until all involved nodes have the same distribution specification of the application.

> For example, suppose that the application `myapp` is defined to be distributed as `{myapp, 5000, [cp1@cave, {cp2@cave, cp3@cave}]}`. Suppose further that all nodes are up and running when `myapp` is started. It is then started at `cp1`, as shown in the figure below.

> Application myapp - Situation 1

> If `cp1` goes down, the system checks which one of the other nodes, `cp2` or `cp3`, has the least number of running applications, but waits for 5 seconds for `cp1` to restart. If `cp1` does not restart and `cp2` runs fewer applications than `cp3,` then `myapp` is restarted on `cp2`.

> Application myapp - Situation 2

> Suppose now that `cp2` goes down as well and does not restart within 5 seconds. `myapp` is now restarted on `cp3`.

> Application myapp - Situation 3

> If `cp2` now restarts, it will not restart `myapp`, because the order between nodes `cp2` and `cp3` is not defined.

> Application myapp - Situation 4

> However, if `cp1` restarts as well, the function `application:takeover/2` moves `myapp` to `cp1`, because `cp1` has a higher priority than `cp3` for this application. In this case, `Mod:start({takeover, cp3@cave}, StartArgs)` is executed at `cp1`.

> Application myapp - Situation 5

> For a distributed application, the `Nodes` list distribution specification must be the same at each involved node. The application controllers check this when they contact each other at start-up. If a mismatch is found, the application controller, and thus the entire Erlang node, terminates with reason `{distribution_mismatch, ApplName, Node}`. The reason for this behavior is that the application may not start at all, it may start at several nodes, or the application controller may be left hanging if there is a mismatch in the specification.

> The `distributed` parameter can be changed at release upgrade. The applications are however not automatically moved to the nodes which have the (new) highest priority; this has to be made manually by the operator after the release upgrade. New applications in the new `distributed` parameter are ignored by `dist_ac`. Such applications have to be loaded (with distribution information) and started manually after the release upgrade has been performed.

> #### 2.7.2 Starting Distributed Applications

> Each node which is included in the distribution specification for a distributed application must make an `application:start` call for the distributed application. The start-up sequence between local and distributed applications is also synchronized.

> To illustrate this point, suppose that a local application named `localapp` uses the distributed application `myapp`, described in the previous example, with the distribution specification `{myapp, 5000, [cp1@cave, {cp2@cave, cp3@cave}]}`. It is assured that `localapp` will not be started on `cp1`, `cp2` or `cp3` until `cp1` starts `myapp`. In this way, the start of distributed applications is a synchronization point for all nodes that may run the application.

> When a local application is dependent on a distributed application but is configured to run on a different node than the distributed application a synchronization problem occurs, this is solved by also starting the distributed application on the node where the local application will run. This ensures that the applications are started in the correct order on the right nodes.

> Please note when the above configuration applies the local application must be loaded before the distributed application is started. Normally this is not a problem because all applications are loaded before any applications are started.

> When distributed applications are specified, it is required that the `sync_nodes` functionality in the `kernel` application (see `kernel(6)`) is used to synchronize all involved nodes. If not used, the distribution mechanism for distributed applications will not function correctly.

> ### 2.8 Starting Applications

> A primary application can be started in one or two steps. The first step is mandatory and the purpose of it is to start the main supervisor of the application and possible permanent children. The second step is optional and its purpose is to synchronize processes within an application.

> #### 2.8.1 Start Supervisors

> The start parameters for the first step are defined in the `mod` key in the resource file for the primary application. The application master will evaluate the function `Mod:start(Type, StartArgs)`.

> `Mod` and `StartArgs` parameters are defined in the `mod` key. The parameter `Type` states what type of start is running, ie. normal, takeover or fail-over.

> Takeover signifies that the application is distributed and is to be moved from another node to this node, due to operator invention or due to this (newly started) node is defined as superior to the other node in the configuration parameter `distributed`.

> Fail-over signifies that this node should start the application due to a crash of the node where the application was previously running.

> All other starts will result in a normal start type.

> Note: Fail-over is only valid if the `start_phases` key is defined, otherwise this start type is denoted as normal. If older application versions without start phases are being used, it is possible to set the `start_phases` key to an empty list.

> #### 2.8.2 Synchronize Processes

> The second step is executed only if the `start_phases` key is defined in the resource file of the primary application.

> If a primary application has a defined key, all its included applications must also have the key defined. There is, however, a possibility to include applications without start phases by wrapping them in an another application (how to do this is explained later).
> In this step the application master evaluates the function `Mod:start_phase(Phase, Type, PhaseArgs)` for each phase in the specified order. `Mod` is defined in the `mod` key, `Phase` and `PhaseArgs` in the `start_phases` key, and `Type` is the same as in the first step. An example showing (a part of) an application's resource file and the evaluated functions:

>     myApp.app
>     :
>     {mod, {myApp, StartArgs},
>     {included_applications, []},
>     {start_phases, [{init, InitArgs}, {go, GoArgs}]},
>     :

>     evaluated functions
>     myApp:start(Type, StartArgs)                % start the main supervisor
>                                                 % and permanent children
>     myApp:start_phase(init, Type, InitArgs)     % start the application
>                                                 % in the init phase
>     myApp:start_phase(go, Type, GoArgs)         % start the application
>                                                 % in the go phase

> A primary application is responsible for starting included applications. The primary application can read the included applications start phases by calling `application:get_key(Application, start_phases)`. There is, however, a possibility to automate the start of included applications, but first an example where the synchronization is taken care by the primary application:

>     primApp.app
>     :
>     {mod, {primApp, PrimAppStartArgs},
>     {included_applications, [inclOne, inclTwo]},
>     {start_phases, [{init, InitArgs}, {go, GoArgs}]},
>     :

>     inclOne.app
>     :
>     {mod, {inclOne, NotUsedArgs},
>     {start_phases, [{go, GoArgs1}]},
>     :

>     inclTwo.app
>     :
>     {mod, {inclTwo, NotUsedArgs},
>     {start_phases, [{init, InitArgs2}, {go, GoArgs2}]},
>     :

>     evaluated functions
>     primApp:start(Type, PrimAppStartArgs)       % start the main supervisor
>                                                 % and permanent children

>     primApp:start_phase(init, Type, InitArgs)   % start the PrimApp and InclTwo
>                                                 % applications in init phase
>     primApp:start_phase(go, Type, GoArgs)       % start all applications
>                                                 % in go phase

> To automate the synchronization of the included applications a predefined module `application_starter` is to be used as the Module in the `mod` key. `application_starter` takes the `Mod` and `StartArgs` as parameters: `{mod, {application_starter, [{Mod, StartArgs}]}`.

> It is possible to have unique start phases in different applications. The order of the phases is defined in the primary applications `start_phases` key for the whole tree. This aserts that all included applications must have their start phases defined in the including applications' resource files.

> The `application_starter` will execute only one start phases at a time, from left to right, searching and executing (where indicated) start phases in the included applications.

> The above example is continued below using an automatic start for included applications:

>     primApp.app
>     :
>     {mod, {application_starter, [primApp, PrimAppStartArgs]},
>     {included_applications, [inclOne, inclTwo]},
>     {start_phases, [{init, InitArgsPrim}, {go, GoArgsPrim}]},
>     :

>     inclOne.app
>     :
>     {mod, {inclOne, NotUsedArgs},
>     {start_phases, [{go, GoArgs1}]},
>     :

>     inclTwo.app
>     :
>     {mod, {inclTwo, NotUsedArgs},
>     {start_phases, [{init, InitArgs2}, {go, GoArgs2}]},
>     :

>     evaluated functions
>     primApp:start(Type, PrimAppStartArgs)         % start the main supervisor
>                                                   % and permanent children
>     primApp:start_phase(init, Type, InitArgsPrim) % start the primApp application
>                                                   % in the init phase
>     inclTwo:start_phase(init, Type, InitArgs2)    % start the inclTwo application
>                                                   % in the init phase
>     primApp:start_phase(go, Type, GoArgsPrim)     % start the primApp application
>                                                   % in the go phase
>     inclOne:start_phase(go, Type, GoArgs1)        % start the inclOne application
>                                                   % in the go phase
>     inclTwo:start_phase(go, Type, GoArgs2)        % start the inclTwo application
>                                                   % in the go phase

> For the reason mentioned above, the subsequently included applications on the same branch must have their start phases defined in all the including applications resource files (`start_phases` key). The `application_starter` will then run the start phases in a sequentially correct start order.

> Note:When starting the application tree the start phase does not descend level by level but follows a branch of the tree (starting applications as it descends) before moving to the next branch.

> There is an example of the recursive use of the application starter below and a diagram of the go start phase flow.

> Flow of the go start phase.

>     primApp.app
>     :
>     {mod, {application_starter, [primApp, PrimAppStartArgs]},
>     {included_applications, [inclOne, inclTwoPrim]},
>     {start_phases, [{prim, PrimArgs}, {init, InitArgs}, {some, SomeArgs},
>                     {spec, SpecArgs}, {go, GoArgs}]},
>     :

>     inclOne.app
>     :
>     {mod, {inclOne, NotUsedArgs},
>     {included_applications, []},
>     {start_phases, [{spec, SpecArgs}, {go, GoArgsOne}]},
>     :

>     inclTwoPrim.app
>     :
>     {mod, {application_starter, [inclTwoPrim, NotUsedArgs]},
>     {included_applications, [incl2A, incl2B]},
>     {start_phases, [{init, []}, {some, []}, {go, []}]},
>     :

>     incl2A.app
>     :
>     {mod, {incl2A, []},
>     {included_applications, []},
>     {start_phases, [{some, SomeArgs2A}, {go, GoArgs2A}]},
>     :

>     incl2B.app
>     :
>     {mod, {incl2B, []},
>     {included_applications, []},
>     {start_phases, [{init, InitArgs2B}]},
>     :

>     evaluated functions
>     primApp:start(Type, PrimAppStartArgs)        % start the main supervisor
>                                                  % and permanent children
>     primApp:start_phase(prim, Type, PrimArgs)
>     primApp:start_phase(init, Type, InitArgs)
>     inclTwoPrim:start_phase(init, Type, [])
>     incl2B:start_phase(init, Type, InitArgs2B)
>     primApp:start_phase(some, Type, SomeArgs)
>     inclTwoPrim:start_phase(some, Type, [])
>     incl2A:start_phase(some, Type, SomeArgs2A)
>     primApp:start_phase(spec, Type, SpecArgs)
>     inclOne:start_phase(spec, Type, SpecArgs)
>     primApp:start_phase(go, Type, GoArgs)
>     inclOne:start_phase(go, Type, GoArgsOne)
>     inclTwoPrim:start_phase(go, Type, [])
>     incl2A:start_phase(go, Type, GoArgs2A)

> Mixing applications with `start_phases` keys together with applications which have no `start_phases` keys is not allowed. The wrapper application's only function is to start an application which has no `start_phases` key. Without a wrapper the application will not start.

>     primApp.app
>     :
>     {mod, {primApp, PrimAppStartArgs},
>     {included_applications, [with, wrapper]},
>     {start_phases, [{init, InitArgs}, {go, GoArgs}]},
>     :

>     with.app
>     :
>     {mod, {with, NotUsedArgs},
>     {included_applications, []},
>     {start_phases, [{init, InitArgsW}, {go, GoArgsW}]},
>     :

>     wrapper.app
>     :
>     {mod, {wrapper, NotUsedArgs},
>     {included_applications, []},
>     {start_phases, [{init, WrapArgs}]},
>     :

>     evaluated functions
>     primApp:start(Type, PrimAppStartArgs)       % start the main supervisor
>                                                 % and permanent children
>     primApp:start_phase(init, Type, InitArgs)   % start the primary application
>                                                 % in the init phase
>     with:start_phase(init, Type, InitArgsW)     % start the with application
>                                                 % in init phase
>     wrapper:start_phase(init, Type, WrapArgs)   % start the without
>                                                 % application
>     primApp:start_phase(go, Type, GoArgs)       % start the primary application
>                                                 % in the go phase
>     with:start_phase(go, Type, GoArgsW)         % start the with application
>                                                 % in go phase

> ### 2.9 An Example

> The following example assumes some familiarity with the Erlang boot script The example shows a simple boot script and config file for nodes `cp1`, `cp2`, `cp3` and `cp4`, and applications `myapp` and `localappl`. These are the nodes and applications which are described in the previous section, with the exception that node `cp4` has been added. This node only executes the local application `localapp`.

> #### 2.9.1 Boot Script

> The following boot script is used at all four nodes. Each node makes an `application:start` call for `myapp`, although this distributed application is started only at one node.

>     {script,{"Dist Test","1.0"},
>      [{preLoaded, [init,erl_prim_loader]},
>       {progress, preloaded},
>       <...>
>       {progress,init_kernel_started},
>       {apply,{application,load,
>               [{application,
>                 myapp,
>                 [{description,"MYAPP"},
>                  {vsn,"1.0"},
>                  {modules, []},
>                  {applications,[kernel, stdlib]},
>                  {env,[]}]}]}},
>       {apply,{application,load,
>               [{application,
>                 localapp,
>                 [{description,"LOCALAPP"},
>                  {vsn,"1.0"},
>                  {modules, []},
>                  {applications,[kernel, stdlib, myapp]},
>                  {env,[]}]}]}},
>       {progress,applications_loaded},
>       {apply,{application,start,[kernel,permanent]}},
>       {apply,{application,start,[stdlib,permanent]}},
>       {apply,{application,start,[myapp,permanent]}},
>       {apply,{application,start,[localapp,permanent]}},
>       {progress,started}]}.

> #### 2.9.2 Config File

> The following configuration file is used at all four nodes.

>        [{kernel, [{sync_nodes_optional, [cp1@cave, cp2@cave, cp3@cave,
>                                               cp4@cave]},
>              {sync_nodes_timeout, 10000},
>              {distributed, [{myapp, [cp1@cave, {cp2@cave, cp3@cave}]}]}]}].

> The application controller notices that `cp4` cannot start `myapp` and then ensures that `cp4` cannot start `localapp` until another node has started `myapp`.

Copyright © 1991-2003 Ericsson Utvecklings AB