PP-586: On a Cray X-series, create a vnode per compute node.

PP-586 - Getting issue details... STATUS

Forum discussion (EDD review).

Overview:

In order to support Cray features like requesting particular NUMA nodes, or some number of NUMA nodes, PBS had created one vnode for each NUMA node.
Sites have requested the ability to change the way PBS reports the Cray nodes, so that they can have fewer vnodes to deal with and not
encounter some of the differences that multiple vnodes per host brings.

The default behavior of PBS will be changed to create a vnode per compute node by default.
 

Interface: vnode_per_numa_node

• Visibility: PBS Private
• Change Control: Experimental
• Details:
  • New interface: new mom_priv/config variable.
  • vnode_per_numa is a Boolean. Setting it to "TRUE" will cause PBS to create one vnode per Cray numa node.
    Setting it to "FALSE" will cause PBS to create one vnode per Cray compute node.  This is the default behavior
    when $vnode_per_numa_node is not set in mom_priv/config.
  • Note: when $vnode_per_numa is set to "FALSE", it will not be possible for PBS to schedule based on segments.

Administrator's instructions.

• When changing from one version of PBS using one type of vnode create versus another, you must first:
  
  • Quiesce the system (ensure there are no running jobs).
  • Remove all existing vnodes of the vntype cray_compute. The easiest way to do this is to delete all vnodes using 
    • qmgr -c "delete node @default"
  • Stop the version of the mom that create vnodes per NUMA node
  • Start the new version of MoM that will create vnodes per compute node
  • All moms on a Cray X* system must be of the same PBS version.
  • On the server add the MoMs back using qmgr: qmgr -c "create node <mom name>"

Definition of technical terms, spelling out acronyms and abbreviations.