PP-325: Support Cgroups

Overview:

The objective of  PP-325 - Getting issue details... STATUS is to provide a hook that restricts resource availability for job processes by utilizing the Linux cgroup infrastructure. An overview of the Linux kernel cgroup implementation may be found here: https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt It is assumed that the reader is familiar with the various cgroup subsystems and how they may be utilized. When the cgroup hook is enabled, it runs on every node assigned to the job. When a job is started, the hook creates a set of directories for the configured subsystems based on the resource requirements of the job and then places the job process within the cgroup. The kernel then enforces resource restrictions based on the newly created cgroup settings while the job is running. The cgroup hook may be configured to periodically poll the job's cgroup while the job is running and updates resource utilization. When the job completes, the final resource utilization measurement is taken and the hook removes the cgroup directories it created when the job was initialized.

Interface 1: cgroup configuration file

• Visibility: Public
• Change Control: Experimental

• Synopsis: JSON hook config file

• Details: The configuration file allows the administrator to adjust the behavior of the cgroup hook across their cluster. The file must conform to JSON syntax. A sample configuration file is displayed and described below:

{
    "cgroup_prefix"         : "pbspro",
    "exclude_hosts"         : ["node001", "node002"],
    "exclude_vntypes"       : ["disable_cgroups", "login_node"],
    "run_only_on_hosts"     : [],
    "periodic_resc_update"  : true,
    "vnode_per_numa_node"   : false,
    "online_offlined_nodes" : true,
    "use_hyperthreads"      : false,
    "cgroup" : {
        "cpuacct" : {
            "enabled"         : true,
            "exclude_hosts"   : ["node001"],
            "exclude_vntypes" : ["red_node"]
        },
        "cpuset" : {
            "enabled"         : true,
            "exclude_hosts"   : ["node002"],
            "exclude_vntypes" : ["green_node"]
        },
        "devices" : {
            "enabled"         : false,
            "exclude_hosts"   : [],
            "exclude_vntypes" : [],
            "allow"           : [
                "b *:* rwm",
                "c *:* rwm",
                ["mic/scif", "rwm"],
                ["nvidiactl", "rwm", "*"],
                ["nvidia-uvm", "rwm"]
            ]
        },
        "hugetlb" : {
            "enabled"         : false,
            "exclude_hosts"   : [],
            "exclude_vntypes" : [],
            "default"         : "0MB",
            "reserve_percent" : "0",
            "reserve_amount"  : "0MB"
        },
        "memory" : {
            "enabled"         : true,
            "exclude_hosts"   : [],
            "exclude_vntypes" : ["blue_node"],
            "soft_limit"      : false,
            "default"         : "256MB",
            "reserve_percent" : "0",
            "reserve_amount"  : "1GB"
        },
        "memsw" : {
            "enabled"         : true,
            "exclude_hosts"   : [],
            "exclude_vntypes" : ["grey_node"],
            "default"         : "256MB",
            "reserve_percent" : "0",
            "reserve_amount"  : "1GB"
        }
    }
}

Global Parameters:

• Config Option: online_offlined_nodes allows the cgroup hook to online nodes that were offlined by the cgroup hook due to orphan cgroups not cleaning up. Valid values are true or false.

• Config Option: cgroup allows the admin to specify which subsystems PBS will use for the job. Valid subsystem values are cpuacct, cpuset, devices, hugetlb (where supported), memory, and memsw.
  • Valid Config Options for all Subsystems
    • enabled valid options true/false
    • exclude_hosts see above
    • exclude_vntypes see above for valid options

• • Config Option: cpuacct subsystem allows the admin to use the cpuacct subsystem, which tracks the cput of all of the pids assigned to the cgroup. Valid keys in the cpuacct subsystem are enabled (valid options true/false), exclude_hosts (see interface 3), exclude_vntypes (see interface 4).

• • Config Option: cpuset subsystem allows the admin to use the cpuset subsystem, which assigns the cores and memory socket(s) for use by all of the pids assigned to the cgroup. Valid keys in the cpuset subsystem are enabled (valid options true/false), exclude_hosts (see interface 3), exclude_vntypes (see interface 4).

• • Config Option: devices subsystem allows the admin to use the devices subsystem, which assigns the devices for use by all of the pids assigned to the cgroup. Valid keys in the devices subsystem are enabled (valid options true/false), exclude_hosts (see interface 3), exclude_vntypes (see interface 4), allow (list of devices to allow access to).

• • • Config Option: allow allows the admin to use the devices subsystem, which allows access to the listed devices for use by all of the pids assigned to the cgroup. Valid ways to reference allowable devices are as follows
      • "b *:* rwm" (This exact string will be used in the allowed string)
      • ["mic/scif","rwm"] (This will look for the major and minor number of the mic/scif device and set it to rwm (i.e. if /dev/mic reported "crw-rw-rw- 1 root root 244, 1 Mar 30 14:50 scif" then the line added to the allow file would look like "c 244:1 rwm"))
      • ["nvidiactl","rwm", "*"] (This will look for the major number of the nvidiactl device and set it to rwm (i.e. if /dev/nvidiactl reported "crw-rw-rw- 1 root root 284, 1 Mar 30 14:50 nvidiactl" then the line added to the allow file would look like "c 284:* rwm"))
    • Notes: The devices subsystem has not been extensively tested. However manual tests on systems with GPUs and Mics seem to work as expected if the correct devices have been added to the allow list

• • Config Option: hugetlb subsystem allows the admin to use the hugetlb subsystem, which allows access to the hugetlb memory. Valid keys in the devices subsystem are enabled (valid options true/false), exclude_hosts (see interface 3), exclude_vntypes (see interface 4).

• • Config Option: memory subsystem allows the admin to use the memory subsystem, which allows the admin to monitory and limit memory used by all of the pids assigned to the cgroup. Valid keys in the memory subsystem are enabled (valid options true/false), default (memory assigned if the job did not request any), reserve_memory (memory to reserve for processes outside of PBS), exclude_hosts (see interface 3), exclude_vntypes (see interface 4).
    • Config Option: default allows the admin to assign memory to a job if it did not request any
    • Config Option: reserve_memory allows the admin to reserve memory for processes outside of PBS jobs

• • Config Option: memsw subsystem allows the admin to monitory and limit swap used by all of the pids assigned to the cgroup.
    • Config Option: default allows the admin to assign memory to a job if it did not request any
    • Config Option: reserve_memory allows the admin to reserve memory for processes outside of PBS jobs
    • Note: To limit swap you must add vmem to the resources line in the sched_config file

Setup:

• Run the following commands in qmgr from the directory where the cgroups.py and cgroups.json file are located
  
  • create hook cgroups
  • set hook cgroups event = "execjob_begin,execjob_launch,execjob_attach,execjob_epilogue,execjob_end,exechost_startup,exechost_periodic"
  • set hook cgroups freq = 120
  • set hook cgroups fail_action = offline_vnodes
  • import hook cgroups application/x-python default cgroups.py
  • import hook cgroups application/x-config default cgroups.json