Forum discussion link :http://community.pbspro.org/t/external-design-document-for-pp-824-cray-ramp-rate-limiting/693
- Interface 1: Server attribute: power_ramprate_enable
- Change Control: Stable
- Synopsis: power_ramprate_enable
- Details: Server attribute. When set to True, PBS can use power ramp rate limiting feature for Cray platform.
- Setting this attribute will also enables attributes node_idle_time and max_ramprate_limit.
- Unset of power_ramprate_enable attribute will not unset node_idle_time and max_ramprate_limit.
- PBS type: Boolean
- Default: unset
- Python type: bool
- Manager has read and write permission and others have read permission.
- A pbshook PBS_Power get enabled as well when this attribute is set.
- Example:
qmgr -c "set server power_ramprate_enable = True"
qmgr -c "set server power_ramprate_enable = 0"At hook: s=pbs.server(); print s.power_ramprate_enable
- Interface 2: New server attribute: node_idle_time
- Change Control: Stable
- Synopsis: node_idle_time
- Details: This new server attribute will define the minimum idle time for nodes to be considered for power ramp down.
- Enabled when server attribute power_ramprate_enable is set.
- The default value is set to 1800 seconds.
- Manager and Operator has set permission. All have read permission.
- To modify the default value use qmgr:
- qmgr -c "set server node_idle_time = <new_value>"
- <new_value> is the time in seconds and should be a non zero positive number.
- PBS type: long
- Python type: int
- Example:
qmgr -c "set server node_idle_time = 2000"
At hook: s=pbs.server(); print s.node_idle_time
- Interface 3: New server attribute: max_ramprate_limit
- Change Control: Stable
- Synopsis: max_ramprate_limit
- Details: This new server attribute will define the set maximum number of nodes that are allowed to drop to C-6 (least possible sleep state).
- Enabled when server attribute power_ramprate_enable is set.
- The default value is set to 5.
- Manager and Operator has set permission. All have read permission..
- To modify the default value use qmgr:
- qmgr -c "set server max_ramprate_limit = <new_value>"
- <new_value> should be a non zero positive number.
- PBS type: long
- Python type: int
- Example:
qmgr -c "set server max_ramprate_limit = 20"
At hook: s=pbs.server(); print s.max_ramprate_limit
- Interface 4: DELETED
- Interface 5: New node attribute: last_used_time
- Change Control: Stable
- Synopsis: last_used_time
- Details: This new node attribute will be updated with time stamp at the end of any job or reservation.
- If node is released early from a running job this timestamp gets updated.
- Node status command pbsnodes will convert internal date format (seconds since epoch) to human readable format and display the value of this attribute in "MON DD YY HH:MM:SS" format.
- Attribute will be reset when node is ramped up.
- Managers and Operators have read permission.
- For vnodes this attribute will be updated for the first time with the current timestamp when they are created or when the nodes are rebooted.
- This attribute can now be used in sched_config as a node_sort_key. This will hep sort the nodes based on their last used time.
- PBS type: long
- Python type: int
- Example:
- node_sort_key: "last_used_time HIGH"
- node_sort_key: "last_used_time LOW"
- Interface 6: New node state: asleep
- Change Control: Stable
- Synopsis: asleep
- Details: This new node state will be set when nodes are ramped down by PBS via power ramp rate limiting. Scheduler will be able to schedule jobs and reservations on these nodes with "asleep" state. If selected by the scheduler, server will ramp these nodes up when required to run jobs or for reservations.
- A server periodic hook (pbs hook PBS_power provided as part of PBS package) runs every $freq seconds and takes list of vnodes to power ramp down the nodes and marks them in new asleep node state.
- At most $max_ramprate_limit nodes will be ramped down every $freq seconds.
- Interface 7: New node state: ramp-up
- Change Control: Stable
- Synopsis: ramp-up
- Details: This new node state will be set when nodes are being ramped up by PBS via power ramp rate limiting.
- New server side hook power_provisioning will take node in asleep state but are assigned to upcoming jobs or reservations and ramps it up.
- While nodes are being ramped up through this this hook, node is marked with this new state "ramp-up".
- This hook interfaces with vendor power api's through generic PMI interface to power ramp up the nodes.
- Interface 8: Change to server hook event: provision
- Change control: Stable
- Synopsis: Server hook event provision
- Details: This server side hook event will now support multiple hooks.
- Hook will have access to below mentioned data:
- pbs.event().vnode or pbs.event().vnode_list: Name/names of vnode to be provisioned.
- pbs.event().prov_type: Type of provision to be performed. Possible types are provision, power and ramplimit.
- pbs.event().aoe: AOE value to be provisioned in case of os or application provision.
- For prov_type = ramplimit:
- This hook takes names of only those nodes in asleep state but are assigned to upcoming jobs or reservations and ramps them up.
- For a job or reservation if there are nodes more than max_ramprate_limit to be ramped up, at a time maximum max_ramprate_limit nodes will be ramped up in anticipation of use. Once the nodes are provisioned next nodes in queue to be provisioned are considered.
- This hook interfaces with vendor power api's through generic PMI interface to power ramp down the nodes.
- For this type pbs.event().aoe and pbs.event().vnode are unset.
- For prov_type == provision:
- This type is used for os and application provisioning.
- Takes value of aoe and vnode name as input.
- pbs.event().vnode_list is unset here.
- If there are any issues during provisioning such nodes are marked offline.
- Example:
if pbs.event(). prov_type == "ramplimit" or pbs.event().prov_type == "power":
print pbs.event().vnode_list
if pbs.event().prov_type == "provision":
print pbs.event().aoe
print pbs.event().vnode
- Interface 9: Log/Error messages.