SUBMIT(1)

NAME

qsub   -
    submit a batch job to Grid Engine.

qsh    -
    submit an interactive X-windows session to Grid Engine.

qlogin -
    submit an interactive login session to Grid Engine.

qrsh   -
    submit an interactive rsh session to Grid Engine.

qalter -
    modify a pending batch job of Grid Engine.

qresub -
    submit a copy of an existing Grid Engine job.

SYNTAX
qsub [ options ] [ command | -- [ command_args ]]

qsh [ options ] [ -- xterm_args ]

qlogin [ options ]

qrsh [ options ] [ command [ command_args ]]

qalter [ options ] wc_job_range_list [ -- [ command_args ]]

qalter [ options ] -u user_list | -uall [ -- [ command_args ]]

qresub [ options ] job_id_list

DESCRIPTION
Qsub submits batch jobs to the Grid Engine queuing system. Grid Engine supports single- and multiple-node jobs. Command can be a path to a binary or a script (see -b below) which contains the commands to be run by the job using a shell (for example, or Arguments to the command are given as command_args to qsub . If command is handled as a script then it is possible to embed flags in the script. If the first two characters of a script line either match '#$' or are equal to the prefix string defined with the -C option described below, the line is parsed for embedded command flags.

Qsh submits an interactive X-windows session to Grid Engine. An is brought up from the executing machine with the display directed either to the X-server indicated by the DISPLAY environment variable or as specified with the -display qsh option. Interactive jobs are not spooled if no resource is available to execute them. They are either dispatched to a suitable machine for execution immediately or the user submitting the job is notified by qsh that appropriate resources to execute the job are not available. xterm_args are passed to the executable. Note, however, that the -e and -ls xterm options do not work with qsh .

Qlogin is similar to qsh in that it submits an interactive job to the queueing system. It does not open an window on the X display, but uses the current terminal for user I/O. Usually, qlogin establishes a connection with the remote host, using standard client- and server-side commands. These commands can be configured with the qlogin_daemon (server-side, Grid Engine telnetd if not set, otherwise something like /usr/sbin/in.telnetd) and qlogin_command (client-side, Grid Engine telnet if not set, otherwise something like /usr/bin/telnet) parameters in the global and local configuration settings of The client side command is automatically parameterized with the remote host name and port number to which to connect, resulting in an invocation like

    /usr/bin/telnet my_exec_host 2442

for example. Qlogin is invoked exactly like qsh and its jobs can only run on INTERACTIVE queues. Qlogin jobs can only be used if the is running under the root account.

Qrsh is similar to qlogin in that it submits an interactive job to the queuing system. It uses the current terminal for user I/O. Usually, qrsh establishes a connection with the remote host. If no command is given to qrsh, an session is established. The server-side commands used can be configured with the rsh_daemon and rlogin_daemon parameters in the global and local configuration settings of An Grid Engine rshd or rlogind is used if the parameters are not set. If the parameters are set, they should be set to something like /usr/sbin/in.rshd or /usr/sbin/in.rlogind. On the client-side, the rsh_command and rlogin_command parameters can be set in the global and local configuration settings of If they are not set, special Grid Engine and binaries delivered with Grid Engine are used. Use the cluster configuration parameters to integrate mechanisms like ssh or the and facilities supplied with the operating system.

Qrsh jobs can only run in INTERACTIVE queues unless the option -now no is used (see below). They can also only be run, if the is running under the root account.

Qrsh provides an additional useful feature for integrating with interactive tools providing a specific command shell. If the environment variable QRSH_WRAPPER is set when qrsh is invoked, the command interpreter pointed to by QRSH_WRAPPER will be executed to run qrsh commands instead of the users login shell or any shell specified in the qrsh command-line. The options -cwd, -v, -V, and -display only apply to batch jobs.

Qalter can be used to change the attributes of pending jobs. For array jobs with a mix of running and pending tasks (see the -t option below), modification with qalter only affects the pending tasks. Qalter can change most of the characteristics of a job (see the corresponding statements in the OPTIONS section below), including those which were defined as embedded flags in the script file (see above). Some submit options, such as the job script, cannot be changed with I. qalter.

Qresub allows the user to create jobs as copies of existing pending or running jobs. The copied jobs will have exactly the same attributes as the ones from which they were copied, except with a new job ID. The only modification to the copied jobs supported by qresub is assignment of a hold state with the -h option. This option can be used to first copy a job and then change its attributes via qalter.

Only a manager can use qresub on jobs submitted by another user. Regular users can only use qresub on their own jobs.

For qsub, qsh, qrsh, and qlogin the administrator and the user may define default request files (see which can contain any of the options described below. If an option in a default request file is understood by qsub and qlogin but not by qsh the option is silently ignored if qsh is invoked. Thus you can maintain shared default request files for both qsub and qsh.

A cluster wide default request file may be placed under $SGE_ROOT/$SGE_CELL/common/sge_request. User private default request files are processed under the locations $HOME/.sge_request and $cwd/.sge_request. The working directory local default request file has the highest precedence, then the home directory located file and then the cluster global file. The option arguments, the embedded script flags and the options in the default request files are processed in the following order:

    left to right in the script line,
    left to right in the default request files,
    from top to bottom of the script file (qsub only),
    from top to bottom of default request files,
    from left to right of the command line.

In other words, the command line can be used to override the embedded flags and the default request settings. The embedded flags, however, will override the default settings.

Note, that the -clear option can be used to discard any previous settings at any time in a default request file, in the embedded script flags, or in a command-line option. It is, however, not available with qalter.

The options described below can be requested either hard or soft. By default, all requests are considered hard until the -soft option (see below) is encountered. The hard/soft status remains in effect until its counterpart is encountered again. If all the hard requests for a job cannot be met, the job will not be scheduled. Jobs which cannot be run at the present time remain spooled.

OPTIONS

-@ optionfile
    Forces qsub, qrsh, qsh, or qlogin to use the options contained in optionfile. The indicated file may contain all valid options. Comment lines must start with a "#" sign.

-a date_time
    Available for qsub and qalter only.

    Defines or redefines the time and date at which a job is eligible for execution. Date_time conforms to [[CC]]YY]MMDDhhmm.[ss], where:

    CC	denotes the century in 2 digits.
    YY	denotes the year in 2 digits.
    MM	denotes the month in 2 digits.
    DD	denotes the day in 2 digits.
    hh	denotes the hour in 2 digits.
    mm	denotes the minute in 2 digits.
    ss	denotes the seconds in 2 digits (default 00).

    If any of the optional date fields is omitted, the corresponding value of the current date is assumed. If CC is not specified, a YY of < 70 means 20YY.
    Usage of this option may cause unexpected results if the clocks of the hosts in the Grid Engine pool are out of sync. Also, the proper behavior of this option very much depends on the correct setting of the appropriate timezone, e.g. in the TZ environment variable (see for details), when the Grid Engine daemons and are invoked.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-ac variable[=value],...
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Adds the given name/value pair(s) to the job's context. Value may be omitted. Grid Engine appends the given argument to the list of context variables for the job. Multiple -ac, -dc, and -sc options may be given. The order is important here.

    Qalter allows changing this option even while the job executes.

-A account_string
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Identifies the account to which the resource consumption of the job should be charged. The account_string may be any arbitrary ASCII alphanumeric string but may not contain "\n", "\t", "\r", "/", ":", "@", "\", "*", or "?". In the absence of this parameter Grid Engine will place the default account string "sge" in the accounting record of the job.

    Qalter allows changing this option even while the job executes.

-b y[es]|n[o]
    Available for qsub, qrsh only. Qalter does not allow changing this option. This option cannot be embedded in the script file itself.

    Gives the user the possibility to indicate explicitly whether command should be treated as binary or script. If the value of -b is 'y', then command may be a binary or script. The command might not be accessible from the submission host. Nothing except the path of the command will be transferred from the submission host to the execution host. Path aliasing will be applied to the path of command before command will be executed.

    If the value of -b is 'n' then command needs to be a script and it will be handled as script. The script file has to be accessible by the submission host. It will be transferred to the execution host. qsub/qrsh will search directive prefixes within script.

    qsub will implicitly use -b n whereas qrsh will apply the -b y option if nothing else is specified.

-c occasion_specifier
    Available for qsub and qalter only.

    Defines or redefines whether the job should be checkpointed, and if so, under what circumstances. The specification of the checkpointing occasions with this option overwrites the definitions of the when parameter in the checkpointing environment (see referenced by the qsub -ckpt switch. Possible values for occasion_specifier are

    n	no checkpoint is performed.
    s	checkpoint when batch server is shut down.
    m	checkpoint at minimum CPU interval.
    x	checkpoint when job gets suspended.
    <interval>	checkpoint in the specified time interval.

    The minimum CPU interval is defined in the queue configuration (see for details). <interval> has to be specified in the format hh:mm:ss. The maximum of <interval> and the queue's minimum CPU interval is used if <interval> is specified. This is done to ensure that a machine is not overloaded by checkpoints being generated too frequently.

-ckpt ckpt_name
    Available for qsub and qalter only.

    Selects the checkpointing environment (see to be used for checkpointing the job. Also declares the job to be a checkpointing job.

-clear
    Available for qsub, qsh, qrsh, and qlogin only.

    Causes all elements of the job to be reset to the initial default status prior to applying any modifications (if any) appearing in this specific command.

-cwd
    Available for qsub, qsh, qrsh and qalter only.

    Execute the job from the current working directory. This switch will activate Grid Engine's path aliasing facility, if the corresponding configuration files are present (see

    In the case of qalter, the previous definition of the current working directory will be overwritten if qalter is executed from a different directory than the preceding qsub or qalter.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-C prefix_string
    Available for qsub and qrsh with script submission (-b n).

    Prefix_string defines the prefix that declares a directive in the job's command. The prefix is not a job attribute, but affects the behavior of qsub and qrsh. If prefix is a null string, the command will not be scanned for embedded directives.
    The directive prefix consists of two ASCII characters which, when appearing in the first two bytes of a script line, indicate that what follows is an Grid Engine command. The default is "#$".
    The user should be aware that changing the first delimiting character can produce unforeseen side effects. If the script file contains anything other than a "#" character in the first byte position of the line, the shell processor for the job will reject the line and may exit the job prematurely.
    If the -C option is present in the script file, it is ignored.

-dc variable,...
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Removes the given variable(s) from the job's context. Multiple -ac, -dc, and -sc options may be given. The order is important.

    Qalter allows changing this option even while the job executes.

-display display_specifier
    Available for qsh only.

    Directs to use display_specifier in order to contact the X server. The display_specifier has to contain the hostname part of the display name (e.g. myhost:1). Local display names (e.g. :0) cannot be used in grid environments. Values set with the -display option overwrite settings from the submission environment and from -v command line options.

-dl date_time
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Specifies the deadline initiation time in [[CC]YY]MMDDhhmm[.SS] format (see -a option above). The deadline initiation time is the time at which a deadline job has to reach top priority to be able to complete within a given deadline. Before the deadline initiation time the priority of a deadline job will be raised steadily until it reaches the maximum as configured by the Grid Engine administrator.
    This option is applicable only for users allowed to submit deadline jobs.

-e [[hostname]:]path,...
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Defines or redefines the path used for the standard error stream of the job. For qsh, qrsh and qlogin only the standard error stream of prolog and epilog is redirected. If the path constitutes an absolute path name, the error-path attribute of the job is set to path, including the hostname. If the path name is relative, Grid Engine expands path either with the current working directory path (if the -cwd switch (see above) is also specified) or with the home directory path. If hostname is present, the standard error stream will be placed in the corresponding location only if the job runs on the specified host. If the path contains a ":" without a hostname, a leading ":" has to be specified.

    By default the file name for interactive jobs is /dev/null. For batch jobs the default file name has the form job_name.ejob_id and job_name.ejob_id.task_id for array job tasks (see -t option below).

    If path is a directory, the standard error stream of the job will be put in this directory under the default file name. If the pathname contains certain pseudo environment variables, their value will be expanded at runtime of the job and will be used to constitute the standard error stream path name. The following pseudo environment variables are supported currently:

    $HOME	home directory on execution machine
    $USER	user ID of job owner
    $JOB_ID	current job ID
    $JOB_NAME	current job name (see -N option)
    $HOSTNAME	name of the execution host
    $TASK_ID	array job task index number

    Alternatively to $HOME the tilde sign "~" can be used as common in or Note, that the "~" sign also works in combination with user names, so that "~<user>" expands to the home directory of <user>. Using another user ID than that of the job owner requires corresponding permissions, of course.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-hard
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Signifies that all resource requirements following in the command line will be hard requirements and must be satisfied in full before a job can be scheduled.
    As Grid Engine scans the command line and script file for Grid Engine options and parameters it builds a list of resources required by a job. All such resource requests are considered as absolutely essential for the job to commence. If the -soft option (see below) is encountered during the scan then all following resources are designated as "soft requirements" for execution, or "nice-to-have, but not essential". If the -hard flag is encountered at a later stage of the scan, all resource requests following it once again become "essential". The -hard and -soft options in effect act as "toggles" during the scan.

-h | -h {u|s|o|n|U|O|S}...
    Available for qsub, qsh, qrsh, qlogin, qalter and qresub.

    List of holds to place on a job, a task or some tasks of a job.

    `u'	denotes a user hold.
    `s'	denotes a system hold.
    `o'	denotes a operator hold.
    `n'	denotes no hold.

    As long as any hold other than `n' is assigned to the job the job is not eligible for execution. Holds can be released via qalter and In case of qalter this is supported by the following additional option specifiers for the -h switch:

    `U'	removes a user hold.
    `S'	removes a system hold.
    `O'	removes a operator hold.

    Grid Engine managers can assign and remove all hold types, Grid Engine operators can assign and remove user and operator holds, and users can only assign or remove user holds.

    In the case of qsub only user holds can be placed on a job and thus only the first form of the option with the -h switch alone is allowed. As opposed to this, qalter requires the second form described above.

    An alternate means to assign hold is provided by the facility.

    If the job is a array job (see the -t option below), all tasks specified via -t are affected by the -h operation simultaneously.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-help
    Prints a listing of all options.

-hold_jid wc_job_list
    Available for qsub, qrsh, and qalter only. See for

    Defines or redefines the job dependency list of the submitted job. A reference by job name or pattern is only accepted if the referenced job is owned by the same user as the referring job. The submitted job is not eligible for execution unless all jobs referenced in the comma-separated job id and/or job name list have completed. If any of the referenced jobs exits with exit code 100, the submitted job will remain ineligible for execution.

    With the help of job names or regular pattern one can specify a job dependency on multiple jobs satisfying the regular pattern or on all jobs with the requested name. The name dependencies are resolved at submit time and can only be changed via qalter. New jobs or name changes of other jobs will not be taken into account.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-i [[hostname]:]file,...
    Available for qsub, and qalter only.

    Defines or redefines the file used for the standard input stream of the job. If the file constitutes an absolute filename, the input-path attribute of the job is set to path, including the hostname. If the path name is relative, Grid Engine expands path either with the current working directory path (if the -cwd switch (see above) is also specified) or with the home directory path. If hostname is present, the standard input stream will be placed in the corresponding location only if the job runs on the specified host. If the path contains a ":" without a hostname, a leading ":" has to be specified.

    By default /dev/null is the input stream for the job.

    It is possible to use certain pseudo variables, whose values will be expanded at runtime of the job and will be used to express the standard input stream as described in the -e option for the standard error stream.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-inherit
    Available only for qrsh and

    qrsh allows the user to start a task in an already scheduled parallel job. The option -inherit tells qrsh to read a job id from the environment variable JOB_ID and start the specified command as a task in this job. Please note that in this case, the hostname of the host where the command will be executed must precede the command to execute; the syntax changes to

    qrsh -inherit [ other options ] hostname command [ command_args ]

    Note also, that in combination with -inherit, most other command line options will be ignored. Only the options -verbose, -v and -V will be interpreted. As a replacement to option -cwd please use -v PWD.

    Usually a task should have the same environment (including the current working directory) as the corresponding job, so specifying the option -V should be suitable for most applications.

    Note: If in your system the qmaster tcp port is not configured as a service, but rather via the environment variable SGE_QMASTER_PORT, make sure that this variable is set in the environment when calling qrsh or qmake with the -inherit option. If you call qrsh or qmake with the -inherit option from within a job script, export SGE_QMASTER_PORT with the option "-v SGE_QMASTER_PORT" either as a command argument or an embedded directive.

-j y[es]|n[o]
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Specifies whether or not the standard error stream of the job is merged into the standard output stream.
    If both the -j y and the -e options are present, Grid Engine sets but ignores the error-path attribute.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-js job_share
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Defines or redefines the job share of the job relative to other jobs. Job share is an unsigned integer value. The default job share value for jobs is 0.

    The job share influences the Share Tree Policy and the Functional Policy. It has no effect on the Urgency and Override Policies (see and the Grid Engine Installation and Administration Guide for further information on the resource management policies supported by Grid Engine).

    In case of the Share Tree Policy, users can distribute the tickets to which they are currently entitled among their jobs using different shares assigned via -js. If all jobs have the same job share value, the tickets are distributed evenly. Otherwise, jobs receive tickets relative to the different job shares. Job shares are treated like an additional level in the share tree in the latter case.

    In connection with the Functional Policy, the job share can be used to weight jobs within the functional job category. Tickets are distributed relative to any uneven job share distribution treated as a virtual share distribution level underneath the functional job category.

    If both the Share Tree and the Functional Policy are active, the job shares will have an effect in both policies, and the tickets independently derived in each of them are added to the total number of tickets for each job.

-l resource=value,...
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Launch the job in a Grid Engine queue meeting the given resource request list. In case of qalter the previous definition is replaced by the specified one.
    describes how a list of available resources and their associated valid value specifiers can be obtained.
    There may be multiple -l switches in a single command. You may request multiple -l options to be soft or hard both in the same command line. In case of a serial job multiple -l switches refine the definition for the sought queue.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-m b|e|a|s|n,...
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Defines or redefines under which circumstances mail is to be sent to the job owner or to the users defined with the -M option described below. The option arguments have the following meaning:

    `b'     Mail is sent at the beginning of the job.
    `e'     Mail is sent at the end of the job.
    `a'     Mail is sent when the job is aborted or
            rescheduled.
    `s'     Mail is sent when the job is suspended.
    `n'     No mail is sent.

    Currently no mail is sent when a job is suspended.

    Qalter allows changing the b, e, and a option arguments even while the job executes. The modification of the b option argument will only be in effect after a restart or migration of the job, however.

-M user[@host],...
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Defines or redefines the list of users to which the server that executes the job has to send mail, if the server sends mail about the job. Default is the job owner at the originating host.

    Qalter allows changing this option even while the job executes.

-masterq wc_queue_list
    Available for qsub, qrsh, qsh, qlogin and qalter. Only meaningful for parallel jobs, i.e. together with the -pe option.

    Defines or redefines a list of cluster queues, queue domains and queue instances which may be used to become the so called master queue of this parallel job. A more detailed description of wc_queue_list can be found in The master queue is defined as the queue where the parallel job is started. The other queues to which the parallel job spawns tasks are called slave queues. A parallel job only has one master queue.

    This parameter has all the properties of a resource request and will be merged with requirements derived from the -l option described above.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-notify
    Available for qsub, qrsh (with command) and qalter only.

    This flag, when set causes Grid Engine to send "warning" signals to a running job prior to sending the signals themselves. If a SIGSTOP is pending, the job will receive a SIGUSR1 several seconds before the SIGSTOP. If a SIGKILL is pending, the job will receive a SIGUSR2 several seconds before the SIGKILL. The amount of time delay is controlled by the notify parameter in each queue configuration (see

    Note that the Linux operating system "misused" the user signals SIGUSR1 and SIGUSR2 in some early Posix thread implementations. You might not want to use the -notify option if you are running multi-threaded applications in your jobs under Linux, particularly on 2.0 or earlier kernels.

    Qalter allows changing this option even while the job executes.

-now y[es]|n[o]
    Available for qsub, qsh, qlogin and qrsh.

    -now y tries to start the job immediately or not at all. The command returns 0 on success, or 1 on failure (also if the job could not be scheduled immediately). For array jobs submitted with the -now option, if all tasks cannot be immediately scheduled, no tasks are scheduled. -now y is default for qsh, qlogin and qrsh
    With the -now n option, the job will be put into the pending queue if it cannot be executed immediately. -now n is default for qsub.

-N name
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    The name of the job. The name can be any printable set of characters except "\n", "\t", "\r", "/", ":", "@", "\", "*", and "?", and it has to start with an alphabetic character. Invalid job names will be denied at submit time.
    If the -N option is not present, Grid Engine assigns the name of the job script to the job after any directory pathname has been removed from the script-name. If the script is read from standard input, the job name defaults to STDIN.
    In the case of qsh or qlogin with the -N option is absent, the string `INTERACT' is assigned to the job.
    In the case of qrsh if the -N option is absent, the resulting job name is determined from the qrsh command line by using the argument string up to the first occurrence of a semicolon or whitespace and removing the directory pathname.

    Qalter allows changing this option even while the job executes.

-noshell
    Available only for qrsh with a command line.

    Do not start the command line given to qrsh in a user's login shell, i.e. execute it without the wrapping shell.

    This option can be used to speed up execution as some overhead, like the shell startup and sourcing the shell resource files, is avoided.

    This option can only be used if no shell-specific command line parsing is required. If the command line contains shell syntax like environment variable substitution or (back) quoting, a shell must be started. In this case, either do not use the -noshell option or include the shell call in the command line.

    Example:
    qrsh echo '$HOSTNAME'
    Alternative call with the -noshell option
    qrsh -noshell /bin/tcsh -f -c 'echo $HOSTNAME'

-nostdin
    Available only for qrsh.

    Suppress the input stream STDIN - qrsh will pass the option -n to the command. This is especially useful, if multiple tasks are executed in parallel using qrsh, e.g. in a process - it would be undefined, which process would get the input.

-o [[hostname]:]path,...
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    The path used for the standard output stream of the job. The path is handled as described in the -e option for the standard error stream.

    By default the file name for standard output has the form job_name.ojob_id and job_name.ojob_id.task_id for array job tasks (see -t option below).

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-ot override_tickets
    Available for qalter only.

    Changes the number of override tickets for the specified job. Requires manager/operator privileges.

-P project_name
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Specifies the project to which this job is assigned. The administrator needs to give permission to individual users to submit jobs to a specific project. (see -aprj option to

-p priority
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Defines or redefines the priority of the job relative to other jobs. Priority is an integer in the range -1023 to 1024. The default priority value for jobs is 0.

    Users may only decrease the priority of their jobs. Grid Engine managers and administrators may also increase the priority associated with jobs. If a pending job has higher priority, it is earlier eligible for being dispatched by the Grid Engine scheduler.

-pe parallel_environment n[-[m]]|[-]m,...
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Parallel programming environment (PE) to instantiate. The range descriptor behind the PE name specifies the number of parallel processes to be run. Grid Engine will allocate the appropriate resources as available. The manual page contains information about the definition of PEs and about how to obtain a list of currently valid PEs.
    You can specify a PE name which uses the wildcard character, "*". Thus the request "pvm*" will match any parallel environment with a name starting with the string "pvm". In the case of multiple parallel environments whose names match the name string, the parallel environment with the most available slots is chosen.
    The range specification is a list of range expressions of the form "n-m", where n and m are positive, non-zero integers. The form "n" is equivalent to "n-n". The form "-m" is equivalent to "1-m". The form "n-" is equivalent to "n-infinity". The range specification is processed as follows: The largest number of queues requested is checked first. If enough queues meeting the specified attribute list are available, all are allocated. If not, the next smaller number of queues is checked, and so forth.
    If additional -l options are present, they restrict the set of eligible queues for the parallel job.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-q wc_queue_list
    Available for qsub, qrsh, qsh, qlogin and qalter.

    Defines or redefines a list of cluster queues, queue domains or queue instances which may be used to execute this job. Please find a description of wc_queue_list in This parameter has all the properties of a resource request and will be merged with requirements derived from the -l option described above.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-R y[es]|n[o]
    Available for qsub, qrsh, qsh, qlogin and qalter.

    Indicates whether a reservation for this job should be done. Reservation is never done for immediate jobs, i.e. jobs submitted using the -now yes option. Please note that regardless of the reservation request, job reservation might be disabled using max_reservation in and might be limited only to a certain number of high priority jobs.

    By default jobs are submitted with the -R n option.

-r y[es]|n[o]
    Available for qsub and qalter only.

    Identifies the ability of a job to be rerun or not. If the value of -r is 'yes', the job will be rerun if the job was aborted without leaving a consistent exit state. (This is typically the case if the node on which the job is running crashes). If -r is 'no', the job will not be rerun under any circumstances.
    Interactive jobs submitted with qsh, qrsh or qlogin are not rerunnable.

    Qalter allows changing this option even while the job executes.

-sc variable[=value],...
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Sets the given name/value pairs as the job's context. Value may be omitted. Grid Engine replaces the job's previously defined context with the one given as the argument. Multiple -ac, -dc, and -sc options may be given. The order is important.
    Contexts provide a way to dynamically attach and remove meta-information to and from a job. The context variables are not passed to the job's execution context in its environment.

    Qalter allows changing this option even while the job executes.

-shell y[es]|n[o]
    Available only for qsub.

    -shell n causes qsub to execute the command line directly, as if by No command shell will be executed for the job. This option only applies when -b y is also used. Without -b y, -shell n has no effect.

    This option can be used to speed up execution as some overhead, like the shell startup and sourcing the shell resource files is avoided.

    This option can only be used if no shell-specific command line parsing is required. If the command line contains shell syntax, like environment variable substitution or (back) quoting, a shell must be started. In this case either do not use the -shell n option or execute the shell as the command line and pass the path to the executable as a parameter.

    If a job executed with the -shell n option fails due to a user error, such as an invalid path to the executable, the job will enter the error state.

    -shell y cancels the effect of a previous -shell n. Otherwise, it has no effect.

    See -b and -noshell for more information.

-soft
    Available for qsub, qsh, qrsh, qlogin and qalter only.

    Signifies that all resource requirements following in the command line will be soft requirements and are to be filled on an "as available" basis.
    As Grid Engine scans the command line and script file for Grid Engine options and parameters, it builds a list of resources required by the job. All such resource requests are considered as absolutely essential for the job to commence. If the -soft option is encountered during the scan then all following resources are designated as "soft requirements" for execution, or "nice-to-have, but not essential". If the -hard flag (see above) is encountered at a later stage of the scan, all resource requests following it once again become "essential". The -hard and -soft options in effect act as "toggles" during the scan.

-sync y[es]|n[o]
    Available for qsub.

    -sync y causes qsub to wait for the job to complete before exiting. If the job completes successfully, qsub's exit code will be that of the completed job. If the job fails to complete successfully, qsub will print out a error message indicating why the job failed and will have an exit code of 1. If qsub is interrupted, e.g. with CTRL-C, before the job completes, the job will be canceled.
    With the -sync n option, qsub will exit with an exit code of 0 as soon as the job is submitted successfully. -sync n is default for qsub.
    If -sync y is used in conjunction with -now y, qsub will behave as though only -now y were given until the job has been successfully scheduled, after which time qsub will behave as though only -sync y were given.
    If -sync y is used in conjunction with -t n[-m[:i]], qsub will wait for all the job's tasks to complete before exiting. If all the job's tasks complete successfully, qsub's exit code will be that of the first completed job tasks with a non-zero exit code, or 0 if all job tasks exited with an exit code of 0. If any of the job's tasks fail to complete successfully, qsub will print out an error message indicating why the job task(s) failed and will have an exit code of 1. If qsub is interrupted, e.g. with CTRL-C, before the job completes, all of the job's tasks will be canceled.

-S [[hostname]:]pathname,...
    Available for qsub, qsh and qalter.

    Specifies the interpreting shell for the job. Only one pathname component without a host specifier is valid and only one path name for a given host is allowed. Shell paths with host assignments define the interpreting shell for the job if the host is the execution host. The shell path without host specification is used if the execution host matches none of the hosts in the list.

    Furthermore, the pathname can be constructed with pseudo environment variables as described for the -e option above.

    In the case of qsh the specified shell path is used to execute the corresponding command interpreter in the (via its -e option) started on behalf of the interactive job.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-t n[-m[:s]]
    Available for qsub and qalter only.

    Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Grid Engine almost like a series of jobs. The option argument to -t specifies the number of array job tasks and the index number which will be associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SGE_TASK_ID. The option arguments n, m and s will be available through the environment variables SGE_TASK_FIRST, SGE_TASK_LAST and SGE_TASK_STEPSIZE.

    Following restrictions apply to the values n and m:

        1 <= n <= MIN(2^31-1, max_aj_tasks)
        1 <= m <= MIN(2^31-1, max_aj_tasks)
        n <= m

    max_aj_tasks is defined in the cluster configuration (see

    The task id range specified in the option argument may be a single number, a simple range of the form n-m or a range with a step size. Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each with the environment variable SGE_TASK_ID containing one of the 5 index numbers.

    All array job tasks inherit the same resource requests and attribute definitions as specified in the qsub or qalter command line, except for the -t option. The tasks are scheduled independently and, provided enough resources exist, concurrently, very much like separate jobs. However, an array job or a sub-array there of can be accessed as a single unit by commands like or See the corresponding manual pages for further detail.

    Array jobs are commonly used to execute the same type of operation on varying input data sets correlated with the task index number. The number of tasks in a array job is unlimited.

    STDOUT and STDERR of array job tasks will be written into different files with the default location

    <jobname>.['e'|'o']<job_id>'.'<task_id>

    In order to change this default, the -e and -o options (see above) can be used together with the pseudo environment variables $HOME, $USER, $JOB_ID, $JOB_NAME, $HOSTNAME, and $SGE_TASK_ID.

    Note, that you can use the output redirection to divert the output of all tasks into the same file, but the result of this is undefined.

    -u username,...

Available for qalter only. Changes are only made on those
jobs which were submitted by users specified in the list of usernames. For managers it is possible to use the qalter -u '*' command to modify all jobs of all users.

If you use the -u switch it is not permitted to specify an additional wc_job_range_list.

-v variable[=value],...
    Available for qsub, qsh, qrsh and qalter.

    Defines or redefines the environment variables to be exported to the execution context of the job. If the -v option is present Grid Engine will add the environment variables defined as arguments to the switch and, optionally, values of specified variables, to the execution context of the job.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

-verbose
    Available only for qrsh and

    Unlike qsh and qlogin, qrsh does not output any informational messages while establishing the session, compliant with the standard and system calls. If the option -verbose is set, qrsh behaves like the qsh and qlogin commands, printing information about the process of establishing the or session.

-verify
    Available for qsub, qsh, qrsh, qlogin and qalter.

    Instead of submitting a job, prints detailed information about the would-be job as though -j were used, including the effects of command-line parameters and the external environment.

-V  Available for qsub, qsh, qrsh with command, qalter and qresub.

    Specifies that all environment variables active within the qsub utility be exported to the context of the job.

-w e|w|n|v
    Available for qsub, qsh, qrsh, qlogin and qalter.

    Specifies a validation level applied to the job to be submitted (qsub, qlogin, and qsh) or the specified queued job (qalter). The information displayed indicates whether the job can possibly be scheduled assuming an empty system with no other jobs. Resource requests exceeding the configured maximal thresholds or requesting unavailable resource attributes are possible causes for jobs to fail this validation.

    The specifiers e, w, n and v define the following validation modes:

    `e'	error - jobs with invalid requests will be
    	rejected; the default for qrsh, qsh
    	and qlogin.
    `w'	warning - only a warning will be displayed
    	for invalid requests.
    `n'	none - switches off validation; the default for
    	qsub and qalter.
    `v'	verify - does not submit the job but prints an
    	extensive validation report.

    Note, that the necessary checks are performance consuming and hence the checking is switched off by default.

    Note also, that the reasons for job requirements being invalid with respect to resource availability of queues are displayed in the "-w v" case using the format as described for the -F option (see description of Full Format in section OUTPUT FORMATS of the manual page.

-wd working_dir
    Available for qsub, qsh, qrsh and qalter only.

    Execute the job from the directory specified in working_dir. This switch will activate Grid Engine's path aliasing facility, if the corresponding configuration files are present (see

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

command
    Available for qsub and qrsh only.

    The job's scriptfile or binary. If not present or if the operand is the single-character string '-', qsub reads the script from standard input.

command_args
    Available for qsub, qrsh and qalter only.

    Arguments to the job. Not valid if the script is entered from standard input.

    Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however.

xterm_args
    Available for qsh only.

    Arguments to the executable, as defined in the configuration. For details, refer to

ENVIRONMENTAL VARIABLES

SGE_ROOT
    Specifies the location of the Grid Engine standard configuration files.

SGE_CELL
    If set, specifies the default Grid Engine cell. To address a Grid Engine cell qsub, qsh, qlogin or qalter use (in the order of precedence):

        The name of the cell specified in the environment variable SGE_CELL, if it is set.

        The name of the default cell, i.e. default.

SGE_DEBUG_LEVEL
    If set, specifies that debug information should be written to stderr. In addition the level of detail in which debug information is generated is defined.

SGE_QMASTER_PORT
    If set, specifies the tcp port on which is expected to listen for communication requests. Most installations will use a services map entry for the service "sge_qmaster" instead to define that port.

DISPLAY
    For qsh jobs the DISPLAY has to be specified at job submission. If the DISPLAY is not set by using the -display or the -v switch, the contents of the DISPLAY environment variable are used as default.

In addition to those environment variables specified to be exported to the job via the -v or the -V option (see above) qsub, qsh, and qlogin add the following variables with the indicated values to the variable list:

SGE_O_HOME
    the home directory of the submitting client.

SGE_O_HOST
    the name of the host on which the submitting client is running.

SGE_O_LOGNAME
    the LOGNAME of the submitting client.

SGE_O_MAIL
    the MAIL of the submitting client. This is the mail directory of the submitting client.

SGE_O_PATH
    the executable search path of the submitting client.

SGE_O_SHELL
    the SHELL of the submitting client.

SGE_O_TZ
    the time zone of the submitting client.

SGE_O_WORKDIR
    the absolute path of the current working directory of the submitting client.

Furthermore, Grid Engine sets additional variables into the job's environment, as listed below.

ARC

SGE_ARCH
    The Grid Engine architecture name of the node on which the job is running. The name is compiled-in into the binary.

SGE_CKPT_ENV
    Specifies the checkpointing environment (as selected with the -ckpt option) under which a checkpointing job executes. Only set for checkpointing jobs.

SGE_CKPT_DIR
    Only set for checkpointing jobs. Contains path ckpt_dir (see ) of the checkpoint interface.

SGE_STDERR_PATH
    the pathname of the file to which the standard error stream of the job is diverted. Commonly used for enhancing the output with error messages from prolog, epilog, parallel environment start/stop or checkpointing scripts.

SGE_STDOUT_PATH
    the pathname of the file to which the standard output stream of the job is diverted. Commonly used for enhancing the output with messages from prolog, epilog, parallel environment start/stop or checkpointing scripts.

SGE_STDIN_PATH
    the pathname of the file from which the standard input stream of the job is taken. This variable might be used in combination with SGE_O_HOST in prolog/epilog scripts to transfer the input file from the submit to the execution host.

SGE_JOB_SPOOL_DIR
    The directory used by to store job related data during job execution. This directory is owned by root or by a Grid Engine administrative account and commonly is not open for read or write access to regular users.

SGE_TASK_ID
    The index number of the current array job task (see -t option above). This is an unique number in each array job and can be used to reference different input data records, for example. This environment variable is set to "undefined" for non-array jobs. It is possible to change the predefined value of this variable with -v or -V (see options above).

SGE_TASK_FIRST
    The index number of the first array job task (see -t option above). It is possible to change the predefined value of this variable with -v or -V (see options above).

SGE_TASK_LAST
    The index number of the last array job task (see -t option above). It is possible to change the predefined value of this variable with -v or -V (see options above).

SGE_TASK_STEPSIZE
    The stepsize of the array job specification (see -t option above). It is possible to change the predefined value of this variable with -v or -V (see options above).

ENVIRONMENT
    The ENVIRONMENT variable is set to BATCH to identify that the job is being executed under Grid Engine control.

HOME
    The user's home directory path from the file.

HOSTNAME
    The hostname of the node on which the job is running.

JOB_ID
    A unique identifier assigned by the when the job was submitted. The job ID is a decimal integer in the range 1 to 99999.

JOB_NAME
    The job name. For batch jobs or jobs submitted by qrsh with a command, the job name is built as basename of the qsub script filename resp. the qrsh command. For interactive jobs it is set to `INTERACTIVE' for qsh jobs, `QLOGIN' for qlogin jobs and `QRLOGIN' for qrsh jobs without a command. This default may be overwritten by the -N. option.

LOGNAME
    The user's login name from the file.

NHOSTS
    The number of hosts in use by a parallel job.

NQUEUES
    The number of queues allocated for the job (always 1 for serial jobs).

NSLOTS
    The number of queue slots in use by a parallel job.

PATH
    A default shell search path of:
    /usr/local/bin:/usr/ucb:/bin:/usr/bin

SGE_BINARY_PATH
    The path where the Grid Engine binaries are installed. The value is the concatenation of the cluster configuration value binary_path and the architecture name $SGE_ARCH environment variable.

PE  The parallel environment under which the job executes (for parallel jobs only).

PE_HOSTFILE
    The path of a file containing the definition of the virtual parallel machine assigned to a parallel job by Grid Engine. See the description of the $pe_hostfile parameter in for details on the format of this file. The environment variable is only available for parallel jobs.

QUEUE
    The name of the cluster queue in which the job is running.

REQUEST
    Available for batch jobs only.

    The request name of a job as specified with the -N switch (see above) or taken as the name of the job script file.

RESTARTED
    This variable is set to 1 if a job was restarted either after a system crash or after a migration in case of a checkpointing job. The variable has the value 0 otherwise.

SHELL
    The user's login shell from the file. Note: This is not necessarily the shell in use for the job.

TMPDIR
    The absolute path to the job's temporary working directory.

TMP The same as TMPDIR; provided for compatibility with NQS.

TZ  The time zone variable imported from if set.

USER
    The user's login name from the file.

RESTRICTIONS

There is no controlling terminal for batch jobs under Grid Engine, and any tests or actions on a controlling terminal will fail. If these operations are in your .login or .cshrc file, they may cause your job to abort.

Insert the following test before any commands that are not pertinent to batch jobs in your .login:

    if ( $?JOB_NAME) then
        echo "Grid Engine spooled job"
        exit 0
    endif

Don't forget to set your shell's search path in your shell start-up before this code.

EXIT STATUS
The following exit values are returned:

0   Operation was executed successfully.

25  It was not possible to register a new job according to the configured max_u_jobs or max_jobs limit. Additional information may be found in

>0  Error occurred.

EXAMPLES
The following is the simplest form of a Grid Engine script file.
=====================================================
#!/bin/csh
   a.out
=====================================================

The next example is a more complex Grid Engine script.
=====================================================
#!/bin/csh

# Which account to be charged cpu time
#$ -A santa_claus
# date-time to run, format [[CC]yy]MMDDhhmm[.SS]
#$ -a 12241200
# to run I want 6 or more parallel processes
# under the PE pvm. the processes require
# 128M of memory
#$ -pe pvm 6- -l mem=128
# If I run on dec_x put stderr in /tmp/foo, if I
# run on sun_y, put stderr in /usr/me/foo
#$ -e dec_x:/tmp/foo,sun_y:/usr/me/foo
# Send mail to these users
#$ -M santa@nothpole,claus@northpole
# Mail at beginning/end/on suspension
#$ -m bes
# Export these environmental variables
#$ -v PVM_ROOT,FOOBAR=BAR
# The job is located in the current
# working directory.
#$ -cwd
a.out
==========================================================

FILES
$REQUEST.oJID[.TASKID]	STDOUT of job #JID
$REQUEST.eJID[.TASKID]	STDERR of job
$REQUEST.poJID[.TASKID]	STDOUT of par. env. of job
$REQUEST.peJID[.TASKID]	STDERR of par. env. of job
$REQUEST.hostsJID[.TASKID]	hosts file of par. env. of job

$cwd/.sge_aliases	cwd path aliases
$cwd/.sge_request	cwd default request
$HOME/.sge_aliases	user path aliases
$HOME/.sge_request	user default request
<sge_root>/<cell>/common/sge_aliases
	cluster path aliases
<sge_root>/<cell>/common/sge_request
	cluster default request
<sge_root>/<cell>/common/act_qmaster
	Grid Engine master host file

SEE ALSO

COPYRIGHT
If configured correspondingly, qrsh and qlogin contain portions of the rsh, rshd, telnet and telnetd code copyrighted by The Regents of the University of California. Therefore, the following note applies with respect to qrsh and qlogin: This product includes software developed by the University of California, Berkeley and its contributors.

See as well as the information provided in <sge_root>/3rd_party/qrsh and <sge_root>/3rd_party/qlogin for a statement of further rights and permissions.