View Source

Forum discussion link :http://community.pbspro.org/t/external-design-document-for-pp-824-cray-ramp-rate-limiting/693

Interface design:

Interface 1: Changes to Power hook.
- Change control: Stable
- Synopsis: Power hook PBS_power
- Details: Enabling/disabling pbshook PBS_power will enable/disable power management feature.
  - Through qmgr changing hook attribute "enabled" would enable or disable the hook and power management feature.
  - This hook will use server periodic event to ramp up/down and power on/off nodes.
  - Example:
    - qmgr -c "s pbshook PBS_power enabled=true"
    - qmgr -c "s pbshook PBS_power enabled=false"
  - This change overrides existing way to enable power feature i.e. setting server attribute power_provisioning.
    - Server attribute power_provisioning is now read-only and gets enabled or disabled when power hook is enabled or disabled.
Interface 2: Hook configuration file
- Change Control: Stable
- Synopsis: JSON hook config file
- Details: The configuration file allows the administrator to set global parameters, which are used to modify the behavior of the PBS_power hook across all nodes in the PBS Pro complex. The file must conform to JSON syntax. A sample configuration file is displayed and described below:
  {
  "power_ramp_rate_enable": "True",
  "power_on_off_enable": "False",
  "node_idle_limit": "1000",
  "min_node_down_delay": "600",
  "max_jobs_analyze_limit": "80"
  }
- Parameters:

Parameter Name	Default value	Description
power_ramp_rate_enable	False	Enabling would make PBS perform ramp rate limiting across the PBS cluster running on a CRAY CLE 6.0 platform. Nodes will be ramped-up and kept at sleep state C1 and for ramp down nodes will be put to sleep state C6.
power_on_off_enable	False	Enabling would make PBS power on and off nodes on the nodes where node attribute poweroff_eligible is true.
node_idle_limit	1800	How long any node should be left idle before it to be considered for powering down or ramp down.
min_node_down_delay	1800	The time limit before a powered-off node can be considered to be brought up.
max_jobs_analyze_limit	100	The limit indicating maximum number of jobs that are analyzed for power on/ramp-up. The jobs considered here are those which have estimated start_time and exec_vnode updated on them. To have these attributes updated one should have strict_ordering set to true and submit jobs with walltime.
max_concurrent_nodes	5	Defines how many nodes can be power on/off or ramped up/down at a time. For ramp rate, while stepping up or down sleep states, hook will sleep X seconds (where 1<=X<=10) between each level of sleep state. If a node supports 5 levels of sleep states, in worst case scenario hook can wait for 50 seconds for single node. So while increasing the value of this attribute one should also consider increasing the PBS_Power hook frequency and alarm time so that hook instances do not overlap or timeout.

Interface 3: New node attribute: poweroff_eligible
- Change control: Stable
- Synopsis: Node attribute for power control.
- Details: This new node attribute will control if a node can be allowed to power off or not.
  - PBS type: Boolean
  - Python type: Boolean
  - Default value: False
  - Manager has set permission. All have read permission.
  - To modify the default value use qmgr:
    - qmgr -c "set node <node_name> poweroff_eligible=True"
Interface 4: New node attribute: last_state_change_time
- Change control: Stable
- Synopsis: Read only node attribute to capture timestamp.
- Details: This new node attribute will be updated with time stamp when the node changes from its current state to a new state.
  - Managers and Operators have read permission.
  - Node status command pbsnodes will convert internal date format (seconds since epoch) to human readable format and display the value of this attribute in “MON DD YY HH:MM:SS” format.
  - PBS type: long
  - Python type: int
Interface 5: New node attribute: last_used_time
- Change Control: Stable
- Synopsis: Read only node attribute to capture timestamp.
- Details: This new node attribute will be updated with time stamp at the end of any job or reservation.
  - If node is released early from a running job this timestamp gets updated.
  - Node status command pbsnodes will convert internal date format (seconds since epoch) to human readable format and display the value of this attribute in "MON DD YY HH:MM:SS" format.
  - Attribute will be reset when node is ramped up.
  - Managers and Operators have read permission.
  - For vnodes this attribute will be updated for the first time with the current timestamp when they are created or when the nodes are rebooted.
  - This attribute can now be used in sched_config as a node_sort_key. This will help sort the nodes based on their last used time.
  - PBS type: long
  - Python type: int
  - Example:
    - node_sort_key: "last_used_time HIGH"
    - node_sort_key: "last_used_time LOW"
Interface 6: New node state: sleep
- Change Control: Stable
- Synopsis: New node state that shows node is put down by PBS.
- Details: This new node state will be set when nodes are ramped down or powered-off by PBS via power ramp rate limiting or power on/off feature.
  - A server periodic hook (pbs hook PBS_power provided as part of PBS package) runs every $freq seconds and takes list of vnodes to power ramp down/power-off the nodes and marks them in new sleep node state.
  - At most max_concurrent_power_limit nodes will be ramped down/powered-off every freq seconds, freq being the server periodic hook frequency.
  - Scheduler can consider the nodes in sleep state to run jobs now.
  - Server periodic hook can ramp-up/power-on the nodes which are in sleep state based on the requirement. Requirement is calculated based on analyzing the jobs estimated start time and the exec_vnode list.

Interface 7: Log/Error messages.

Change Control: Unstable
Synopsis: New log/error messages.

Details: Below listed are the new log and error messages introduced by power ramp limiting feature.

#	Scenario	Log/error message
1	Nodes are being ramped down	In server logs: Job;power_ramp_down;launch: /opt/cray/capmc/default/bin/capmc set_sleep_state_limit --nids 24-25 --limit 4 Job;power_ramp_down;launch: finished Log level: LOG_INFO
2	Nodes are being ramped up	In server logs: Job;power_ramp_up;launch: /opt/cray/capmc/default/bin/capmc set_sleep_state_limit --nids 24-25 --limit 0 Job;power_ramp_up;launch: finished Log level: LOG_INFO
3	Server periodic hook output	In server logs: power_ramp_limit: nodes to ramp up: <node_list> power_ramp_limit: nodes to ramp down: <node_list> Log level: LOG_INFO
4	Nodes are being powered off	In server logs: 03/29/2016 02:05:59;0008;Server@sdb;Job;node_power_off;launch: /opt/cray/capmc/default/bin/capmc node_off --nids 24-25 03/29/2016 02:06:01;0008;Server@sdb;Job;node_power_off;launch: finished Log level: LOG_INFO
5	Nodes are being powered on	In server logs: 03/29/2016 02:05:59;0008;Server@sdb;Job;node_power_on;launch: /opt/cray/capmc/default/bin/capmc node_on --nids 24-25 03/29/2016 02:06:01;0008;Server@sdb;Job;node_power_on;launch: finished Log level: LOG_INFO