This is a design proposal to configure PBS in a way that it releases only limited number of resources (as specified by the admin) when a job is suspended.
PBS in its current form releases all the consumable resources requested by the job when it is suspended. In most cases suspended job holds on to the memory it would have consumed and just releases ncpus, in some cases admin might have configured an alternate suspend signal which would make the job release a few resources (like licenses) upon suspension. Therefore, it would be better if PBS has a way for admins to specify what all resources can be released from a job upon suspension.
Interface 1: New scheduler attribute to specify which resources can be released.
- Visibility: Public
- Change Control: Stable
- Details:
- A new scheduler attribute “res_released_on_susp” can be used to specify a list of comma separated list of resource names that can be release when a job is suspended.
- This scheduler attribute is of type “array_string” and can only be set by a manager.
- If a manager is trying to set the attribute with a resource which is non existent then following error will be thrown on the console by qmgr command -
# qmgr -c "s sched res_released_on_susp = ‘ncpus, abcd'"
qmgr obj=abcd svr=default: Unknown resource
qmgr: Error (15035) returned from server
- By default, this attribute is unset and after suspending the job PBS will release all the consumable resources requested by the job.
- If this attribute is unset PBS will get back to default behavior of releasing all the consumable resources upon job suspension.
- PBS manager can also add/remove resources to/from "res_released_on_susp" attribute by using "+="/"-=" operators.
- The resources specified in this new scheduler attribute will be released (provided job has requested for them) every time a job is suspended (by preemption or qsig).
Interface 2: New Job attribute “resources_released”
- Visibility: Public
- Change Control: Stable
- Details:
• A new job attribute “resources_released” is added.
• This attribute is of type string and can only be read by user/operator/manager. This attribute is internally set by server when a job is suspended.
• It stores a string that depicts the amount of resources that are released on each chunk that the job was running on (provided these resources are also part of “res_released_on_susp” string). The format of the string is similar to that of exec_vnode
example: qstat -f 1 | grep resources_released
resources_released = (host1:ncpus=2)+(host2:ncpus=4:license=2)
• This job attribute is populated only if “res_released_on_susp” scheduler attribute is set and has a list of legitimate resources to be released.
• This attribute is set by scheduler whenever it tries to preempt a job using suspension. Scheduler will populate this job attribute by sending a ModifyJob batch request to server.
⁃ If scheduler is unable to set “resources_released” attribute of the job then it will bail out and not try preemption for the job in hand.
Interface 3: New Job attribute “resource_released_list”
- Visibility: Public
- Change Control: Stable
- Details:
• A new job attribute “resource_released_list” is added.
• This attribute is of type “resource” and can only be read by user/operator/manager. This attribute is internally set by server when a job is suspended.
• It stores the cumulative value of all the consumable resources requested by the job (provided these resources are also part of “res_released_on_susp” string).
using example in interface 2: qstat -f 1 | grep resource_released_list
resources_released_list.license = 2
resources_released_list.ncpus = 6
• This job attribute is populated only if “res_released_on_susp” scheduler attribute is set and has a list of legitimate resources to be released.
• This job attribute is used to release consumable resources on queue/server objects.
Interface 4: New server log message
- Visibility: Public
- Change Control: Stable
- Details:
- If server is unable to populate “resources_released” job attribute while suspending a job then it will log following log message at LOG_INFO log level.
“Unable to create resource released list”
Interface 5: New scheduler log message
- Visibility: Public
- Change Control: Stable
- Details:
- If scheduler is unable to populate “resources_released” job attribute while suspending a job then it will log following log message at LOG_INFO log level.
“Failed to populate resources released”
Interface 6: New error message while deleting a custom resource
- Visibility: Public
- Change Control: Stable
- Details:
- If an admin tries to delete a custom resource that is part of the res_released_on_susp scheduler attribute then qmgr command will fail with “resource busy” error code.
example: # qmgr -c “d r res1”
qmgr obj=res1 svr=default: Resource busy on scheduler
qmgr: Error (15174) returned from server