There are certain scenarios where a node may require maintenance that does not require powering it down. In these cases it would be nice to be able to suspend jobs that are on those nodes so they aren't lost or requeued. This will be accomplished by providing two new pseudo signals to qsig (like suspend or resume). The first will suspend the job and put all of the job's nodes into a new state. The second will resume the job and the nodes will leave the new state.
CHANGE CONTROL: Public/Stable
SYNOPSIS: New node state which will allow the admin to perform maintenance on the node.
DETAILS:
A node enters the "maintenance" state when the first job is suspended with the new suspend signal. If other jobs are running on a node in the 'maintenance' state, they will continue to run until they are subsequently suspended(or they end). The node will leave the 'maintenance' state when the last job is resumed with the new resume pseudo signal. The scheduler will not run new jobs on a node in this state. It will also not resume any jobs in the suspended state.
Change Control: Public/Stable
PERMISSIONS: operator or manager
Synopsis: Suspend a job and put its nodes in the 'maintenance' state
Details:
The pseudo signal is given by the admin when they want to suspend jobs to start maintenance on the job's nodes. It is given via the standard -s option to qsig (qsig -s admin-suspend <job id>). When the admin-suspend signal is given to a job, two things will happen. First, job will be put in the suspended('S') state and the job's processes will be suspended. Second, the job's vnodes will be put into the 'maintenance' state.
CHANGE CONTROL: Public/Stable
PERMISSIONS: operator or manager
SYNOPSIS: Resume a job which was suspended with the admin-suspend pseudo signal
DETAILS:
The admin-resume pseudo signal is different than the resume pseudo signal. When a job receives the resume pseudo signal, it doesn't actually resume the job. The job's substate is changed to let the scheduler know to resume the job. The admin-resume pseudo signal will directly resume the job (no waiting for the scheduler). When the last admin-suspended job is admin-resumed, the job's vnodes will leave the 'maintenance' state.
CHANGE CONTROL: Public/Stable
PERMISSIONS: manager read only
PYTHON TYPE: string
SYNOPSIS: New vnode attribute which contains a list of admin-suspended jobs on the vnode
TYPE: Array of strings
DETAILS:
PBS will keep a list of jobs that are on a vnode that are admin-suspended. This attribute is read only for managers.
## Submit some jobs
[bmann@mars pbspro]$ qsub -l select=1:ncpus=1 -- /bin/sleep 1000
1351.mars
[bmann@mars pbspro]$ qsub -l select=1:ncpus=1 -- /bin/sleep 1000
1352.mars
## See jobs in running state
[bmann@mars pbspro]$ qstat
Job id Name User Time Use S Queue
---------------- ---------------- ---------------- -------- - -----
1351.mars STDIN user 00:00:00 R workq
1352.mars STDIN user 00:00:00 R workq
## Find all jobs running on a vnode
$ pbsnodes -v mars | grep jobs
jobs = 1351.mars/0, 1352.mars/1
## admin-suspend jobs
$ qsig -s admin-suspend 1351
$ qsig -s admin-suspend 1352
[bmann@mars pbspro]$ qstat
Job id Name User Time Use S Queue
---------------- ---------------- ---------------- -------- - -----
1351.mars STDIN user 00:00:00 S workq
1352.mars STDIN user 00:00:00 S workq
# See vnode in new state
$ pbsnodes -v mars | grep state
state = maintenance
<Perform Maintenance>
## See which jobs need to be resumed
$ pbsnodes -v mars | grep maintenance_jobs
maintenance_jobs = 1351.mars, 1352.mars
## Resume the first job (and see the vnode state remain in 'maintenance')
$ qsig -s admin-resume 1351
$ pbsnodes -v mars | grep state
state = maintenance
## Resume the last job (and see the vnode state leave 'maintenance')
$ qsig -s admin-resume 1352
$ pbsnodes -v mars | grep state
state = free