SGE_PE(5)

NAME
sge_pe - Grid Engine Enterprise Edition parallel environment configuration file format

DESCRIPTION
Parallel environments are parallel programming and runtime environments allowing for the execution of shared memory or distributed memory parallelized applications. Parallel environments usually require some kind of setup to be operational before starting parallel applications. Examples for common parallel environments are shared memory parallel operating systems and the distributed memory environments Parallel Virtual Machine (PVM) or Message Passing Interface (MPI).

sge_pe allows for the definition of interfaces to arbitrary parallel environments. Once a parallel environment is defined or modified with the -ap or -mp options to the environment can be requested for a job via the -pe switch to together with a request of a range for the number of parallel process to be allocated by the job. Additional -l options may be used to specify the job requirement to further detail.

FORMAT
The format of a sge_pe file is defined as follows:

  pe_name
The name of the parallel environment. To be used in the -pe switch.

  queue_list
A comma separated list of queues to which parallel jobs belonging to this parallel environment have access to. Alternatively the keyword "all" can be used to give access to all queues across the cluster.

  slots
The number of parallel processes being allowed to run in total under the parallel environment concurrently.

  user_lists
A comma separated list of user access list names (see Each user contained in at least one of the enlisted access lists has access to the parallel environment. If the user_lists parameter is set to NONE (the default) any user has access being not explicitly excluded via the xuser_lists parameter described below. If a user is contained both in an access list enlisted in xuser_lists and user_lists the user is denied access to the parallel environment.

  xuser_lists
The xuser_lists parameter contains a comma separated list of so called user access lists as described in Each user contained in at least one of the enlisted access lists is not allowed to access the parallel environment. If the xuser_lists parameter is set to NONE (the default) any user has access. If a user is contained both in an access list enlisted in xuser_lists and user_lists the user is denied access to the parallel environment.

  start_proc_args
The invocation command line of a start-up procedure for the parallel environment. The start-up procedure is invoked by prior to executing the job script. Its purpose is to setup the parallel environment correspondingly to its needs. An optional prefix "user@" specifies the user under which this procedure is to be started. The standard output of the start-up procedure is redirected to the file REQNAME.poJID in the job's working directory (see with REQNAME being the name of the job as displayed by and JID being the job's identification number. Likewise, the standard error output is redirected to REQNAME.peJID.

If the location of the stdout and stderr output has been redirected with the -o/-e flags the following rules apply for the name of the output files of the -PE procedures:

If the basename of the stdout/stderr file is a directory, the PE output files are created with their default names in that directory.

If the basename of the stdout/stderr file is a plain file, the PE output files are joined with the stdout/stderr files of the job.

The following special variables being expanded at runtime can be used (besides any other strings which have to be interpreted by the start and stop procedures) to constitute a command line:

$pe_hostfile
    The pathname of a file containing a detailed description of the layout of the parallel environment to be setup by the start-up procedure. Each line of the file refers to a host on which parallel processes are to be run. The first entry of each line denotes the hostname, the second entry the number of parallel processes to be run on the host and the third entry a processor range to be used in case of a multiprocessor machines.

$host
    The name of the host on which the start-up or stop procedures are started.

$job_owner
    The user name of the job owner.

$job_id
    Grid Engine Enterprise Edition's unique job identification number.

$job_name
    The name of the job.

$pe The name of the parallel environment in use.

$pe_slots
    Number of slots granted for the job.

$processors
    The processors string as contained in the queue configuration (see of the master queue (the queue in which the start-up and stop procedures are started).

$queue
    The master queue, i.e. the queue in which the start-up and stop procedures are started.

  stop_proc_args
The invocation command line of a shutdown procedure for the parallel environment. The shutdown procedure is invoked by after the job script has finished. Its purpose is to stop the parallel environment and to remove it from all participating systems. An optional prefix "user@" specifies the user under which this procedure is to be started. The standard output of the stop procedure is also redirected to the file REQNAME.poJID in the job's working directory (see with REQNAME being the name of the job as displayed by and JID being the job's identification number. Likewise, the standard error output is redirected to REQNAME.peJID
The same special variables as for start_proc_args can be used to constitute a command line.

  allocation_rule
The allocation rule is interpreted by and helps the scheduler to decide how to distribute parallel processes among the available machines. If, for instance, a parallel environment is built for shared memory applications only, all parallel processes have to be assigned to a single machine, no matter how much suitable machines are available. If, however, the parallel environment follows the distributed memory paradigm, an even distribution of processes among machines may be favorable.
The current version of the scheduler only understands the following allocation rules:

<int>:
    An integer number fixing the number of processes per host. If the number is 1, all processes have to reside on different hosts. If the special denominator $pe_slots is used, the full range of processes as specified with the -pe switch has to be allocated on a single host (no matter which value belonging to the range is finally chosen for the job to be allocated).

$fill_up:
    Starting from the best suitable host/queue, all available slots are allocated. Further hosts and queues are "filled up" as long as a job still requires slots for parallel tasks.

$round_robin:
    From all suitable hosts a single slot is allocated until all tasks requested by the parallel job are dispatched. If more tasks are requested than suitable hosts are found, allocation starts again from the first host. The allocation scheme walks through suitable hosts in a best-suitable-first order.

  control_slaves
This parameter can be set to TRUE or FALSE (the default). It indicates whether Grid Engine Enterprise Edition is the creator of the slave tasks of a parallel application via and and thus has full control over all processes in a parallel application, which enables capabilities such as resource limitation and correct accounting. However, to gain control over the slave tasks of a parallel application, a sophisticated PE interface is required, which works closely together with Grid Engine Enterprise Edition facilities. Such PE interfaces are available through your local Grid Engine Enterprise Edition support office.

Please set the control_slaves parameter to false for all other PE interfaces.

  job_is_first_task
This parameter is only checked if control_slaves (see above) is set to TRUE and thus Grid Engine Enterprise Edition is the creator of the slave tasks of a parallel application via and In this case, a sophisticated PE interface is required closely coupling the parallel environment and Grid Engine Enterprise Edition. The documentation accompanying such PE interfaces will recommend the setting for job_is_first_task.

The job_is_first_task parameter can be set to TRUE or FALSE. A value of TRUE indicates that the Grid Engine Enterprise Edition job script already contains one of the tasks of the parallel application, while a value of FALSE indicates that the job script (and its child processes) is not part of the parallel program.

RESTRICTIONS
Note, that the functionality of the start-up, shutdown and signalling procedures remains the full responsibility of the administrator configuring the parallel environment. Grid Engine Enterprise Edition will just invoke these procedures and evaluate their exit status. If the procedures do not perform their tasks properly or if the parallel environment or the parallel application behave unexpectedly, Grid Engine Enterprise Edition has no means to detect this.

SEE ALSO

COPYRIGHT
See for a full statement of rights and permissions.