PP-325: Support Cgroups

Overview:

The objective of  PP-325 - Getting issue details... STATUS is to provide a hook that restricts resource availability for job processes by utilizing the Linux cgroup infrastructure. An introductory video to cgroups together with links to other resources may be found here: https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups The definitive Linux kernel cgroup documentation may be found here: https://www.kernel.org/doc/Documentation/cgroup-v1/ It is assumed that the reader is familiar with the various cgroup subsystems and how they may be utilized. When the cgroup hook is enabled, it runs on every node assigned to the job. When a job is started, the hook creates a set of directories for the configured subsystems based on the resource requirements of the job and then places the job process within the cgroup. The kernel then enforces resource restrictions based on the newly created cgroup settings while the job is running. The cgroup hook may be configured to periodically poll the job's cgroup while the job is running and updates resource utilization. When the job completes, the final resource utilization measurement is taken and the hook removes the cgroup directories it created when the job was initialized.

Interface 1: cgroup configuration file

• Visibility: Public
• Change Control: Experimental

• Synopsis: JSON hook config file

• Details: The configuration file allows the administrator to adjust the behavior of the cgroup hook across their cluster. The file must conform to JSON syntax. A sample configuration file is displayed and described below:

{
    "cgroup_prefix"         : "pbspro",
    "exclude_hosts"         : ["node001", "node002"],
    "exclude_vntypes"       : ["disable_cgroups", "login_node"],
    "run_only_on_hosts"     : [],
    "periodic_resc_update"  : true,
    "vnode_per_numa_node"   : false,
    "online_offlined_nodes" : true,
    "use_hyperthreads"      : false,
    "cgroup" : {
        "cpuacct" : {
            "enabled"         : true,
            "exclude_hosts"   : ["node001"],
            "exclude_vntypes" : ["red_node"]
        },
        "cpuset" : {
            "enabled"         : true,
            "exclude_hosts"   : ["node002"],
            "exclude_vntypes" : ["green_node"]
        },
        "devices" : {
            "enabled"         : false,
            "exclude_hosts"   : [],
            "exclude_vntypes" : [],
            "allow"           : [
                "b *:* rwm",
                "c *:* rwm",
                ["mic/scif", "rwm"],
                ["nvidiactl", "rwm", "*"],
                ["nvidia-uvm", "rwm"]
            ]
        },
        "hugetlb" : {
            "enabled"         : false,
            "exclude_hosts"   : [],
            "exclude_vntypes" : [],
            "default"         : "0MB",
            "reserve_percent" : "0",
            "reserve_amount"  : "0MB"
        },
        "memory" : {
            "enabled"         : true,
            "exclude_hosts"   : [],
            "exclude_vntypes" : ["blue_node"],
            "soft_limit"      : false,
            "default"         : "256MB",
            "reserve_percent" : "0",
            "reserve_amount"  : "1GB"
        },
        "memsw" : {
            "enabled"         : true,
            "exclude_hosts"   : [],
            "exclude_vntypes" : ["grey_node"],
            "default"         : "256MB",
            "reserve_percent" : "0",
            "reserve_amount"  : "1GB"
        }
    }
}

Global Parameters:

The global parameters are used to modify the behavior of the cgroup hook across all nodes in the PBS Pro complex. They are not specific to any cgroup subsystem.

cpuacct Subsystem:

The cpuacct subsystem provides support for measuring CPU usage within a cgroup.

cpuset Subsystem:

The cpuset subsystem is used by the Linux kernel to restrict access to both CPU and memory based on available sockets and NUMA nodes.

devices Subsystem:

The devices subsystem is used to grant or restrict access to devices on the system. This is most commonly used for accelerator cards such as GPUs, and MICs.

hugetlb Subsystem:

The hugetlb subsystem restricts the amount of huge page memory that may be used in a cgroup.

memory Subsystem:

memsw Subsystem:

Interface 2: nmics

Interface 3: ngpus

Interface 4: hpmem

• • Config Option: hugetlb subsystem allows the admin to use the hugetlb subsystem, which allows access to the hugetlb memory. Valid keys in the devices subsystem are enabled (valid options true/false), exclude_hosts (see interface 3), exclude_vntypes (see interface 4).

• • Config Option: memory subsystem allows the admin to use the memory subsystem, which allows the admin to monitory and limit memory used by all of the pids assigned to the cgroup. Valid keys in the memory subsystem are enabled (valid options true/false), default (memory assigned if the job did not request any), reserve_memory (memory to reserve for processes outside of PBS), exclude_hosts (see interface 3), exclude_vntypes (see interface 4).
    • Config Option: default allows the admin to assign memory to a job if it did not request any
    • Config Option: reserve_memory allows the admin to reserve memory for processes outside of PBS jobs

• • Config Option: memsw subsystem allows the admin to monitory and limit swap used by all of the pids assigned to the cgroup.
    • Config Option: default allows the admin to assign memory to a job if it did not request any
    • Config Option: reserve_memory allows the admin to reserve memory for processes outside of PBS jobs
    • Note: To limit swap you must add vmem to the resources line in the sched_config file

Setup:

• Run the following commands in qmgr from the directory where the cgroups.py and cgroups.json file are located
  
  • create hook cgroups
  • set hook cgroups event = "execjob_begin,execjob_launch,execjob_attach,execjob_epilogue,execjob_end,exechost_startup,exechost_periodic"
  • set hook cgroups freq = 120
  • set hook cgroups fail_action = offline_vnodes
  • import hook cgroups application/x-python default cgroups.py
  • import hook cgroups application/x-config default cgroups.json