...
Change Control: Public/Stable
PERMISSIONS: operator or manager
Synopsis: Suspend a job and put its nodes in the 'maintenance' state
...
The pseudo signal is given by the admin when they want to suspend jobs to start maintenance on the job's nodes. It is given via the standard -s option to qsig (qsig -s admin-suspend <job id>). When the admin-suspend signal is given to a job, two things will happen. First, job will be put in the suspended('S') state and the job's processes will be suspended. Second, the job's nodes vnodes will be put into the 'maintenance' state.
...
CHANGE CONTROL: Public/Stable
PERMISSIONS: operator or manager
SYNOPSIS: Resume a job which was suspended with the admin-suspend pseudo signal
...
The admin-resume pseudo signal is different than the resume pseudo signal. When a job receives the resume pseudo signal, it doesn't actually resume the job. The job's substate is changed to let the scheduler know to resume the job. The admin-resume pseudo signal will directly resume the job (no waiting for the scheduler). When the last admin-suspended job is admin-resumed, the job's nodes vnodes will leave the 'maintenance' state.
Interface 4: 'maintenance_jobs'
...
vnode attribute
CHANGE CONTROL: Public/Stable
...
PYTHON TYPE: string
SYNOPSIS: New node vnode attribute which contains a list of admin-suspended jobs on the nodevnode
TYPE: Array of strings
DETAILS:
PBS will keep a list of jobs that are on a node vnode that are admin-suspended. This attribute is read only for managers.
...
- If a job is suspended via normal means, it can not be resumed with an admin-resume pseudo signal. The request will be rejected with the following error message: "Job can not be resumed with the requested resume signal"
- If a job is suspended with the admin-suspend pseudo signal, it can not be resumed with the resume pseudo signal. The request will be rejected with the following message: "Job can not be resumed with the requested resume signal"
- If there are multiple jobs on a nodevnode, it is not recommended to mix and match suspend signals. If this happens it is possible for a node vnode to be put back into a schedulable state prior to all of the non admin-suspended jobs being resumed. The scheduler could then run jobs on the resources owned by the non admin-suspended jobs that are still suspended.
...
- Before admin-suspending jobs, it is recommended to disable scheduling and wait for the current scheduling cycle to finish. The scheduler only queries the node vnode state at the start of the cycle. If a node vnode moves into 'maintenance' after the cycle starts, the scheduler may still consider the node vnode as schedulable. It is possible for new jobs to start during the current cycle.
- If an admin wants to perform maintenance on a node vnode that has no jobs running on it, they should put the node vnode in the offline state and perform maintenance.
- Any reservations on nodes vnodes in the maintenance state will be marked degraded. PBS will search for alternate nodes vnodes for the reservations.
- Sub jobs are requeued upon server restart. Any node vnode which only had admin-suspended subjobs will return to the free state after a server restart.
- If a job is running on some but not all of the vnodes of a multi-vnoded host, only the vnodes the job is running on will be put into maintenance.
...
## Find all jobs running on a nodevnode
$ pbsnodes -v mars | grep jobs
...
[bmann@mars pbspro]$ qstat
Job id Name User Time Use S Queue
---------------- ---------------- ---------------- -------- - -----
1351.mars STDIN user 00:00:00 S workq
1352.mars STDIN user 00:00:00 S workq
# See node vnode in new node statenew state
$ pbsnodes -v mars | grep state
state = maintenance
...
## Resume the first job (and see the node vnode state remain in 'maintenance')
...
## Resume the last job (and see the node vnode state leave 'maintenance')
...