PP-587: Have only one mom report the compute node information to the server.

PP-587 - Getting issue details... STATUS


Overview:
This enhancement only applies to Cray systems.
In order to improve the performance of PBS on very large scale systems (20 K+ compute nodes), a number of changes were made to reduce
the time and CPU cycles required for the reporting of the list of compute nodes and resources (inventory) by the collective set of pbs_mom daemons
running on the system. The specific changes were mostly internal to PBS with the exception of the addition of a new node attribute (vnode_pool)
used to associate the inventory reported by a singe pbs_mom as applying to all pbs_moms on the same Cray system.

Interface #1: new node attribute: vnode_pool
  • Visibility: Public
  • Change Control: Experimental
  • Details:
    • Comments on the interface.
      • Interface standing: new interface.
      • Interface type: node attribute.
    • Set by PBS Administrator when defining a new node for a 'login node' on a Cray system on which a PBS Mom is to be run.
      • Qmgr: create node <name> vnode_pool=1
        Results in only one of the Moms which share the same vnode_pool to report the compute node.
    • PBS type: int, Python type: int.
    • Legal value > 0; [default: unset(0)]
    • Can only be set when the node is created (i.e. before server makes first contact with mom). Cannot be altered or unset once node is created. Can 
      only be set by PBS Manager. Can be seen by anyone.
      • Attempt to set without Manager privilege will result in standard "Invalid request" error message returned to client.
    • Should only be used for login nodes running PBS Mom on a single Cray system. The same non-zero vnode_pool value should be set for
      all login nodes on the same Cray system.
      • This means that if there is more than one Cray system being managed by the same version of PBS, the value for vnode_pool should
        be unique to each cluster. This will indicate to PBS which Moms belong to which Cray system.
    • The vnode_pool attribute only applies to 'natural' vnodes that would normally report Cray inventory. Vnodes representing the compute nodes
      will not report vnode_pool.
    • The vnode_pool attribute will only be displayed in pbsnodes output when it has been set.
    • Use of vnode_pool on systems other than Cray is not supported and results are not defined.
    • Behavior: all Moms who have the same vnode_pool value will be associated with all of the vnodes being reported by the Inventory Mom in
      that pool. If the Inventory Mom goes down, then another Mom from the same vnode_pool is chosen to be the Inventory Mom.

Interface #2: new Mom log entry: "Hello (no inventory required) from server at <server address>"

  • Visibility: Public
  • Change Control: Experimental
  • Details:
    • Comments on the Interface.
      • Interface standing: new interface.
      • Interface type: log message - log level: PBSEVENT_SYSTEM class PBS_EVENTCLASS_SERVER
    • Recorded when Mom which is in a vnode_pool and not the 'Inventory' Mom is sent a Hello message by the Server.

Interface #3: new Server log entry: "invalid vnode_pool provided"

  • Visibility: Public
  • Change control: Experimental
  • Details:
    • Comments on the interface.
      • Interface standing: new interface.
      • Interface type: log message - log level: PBSEVENT_ADMIN class PBS_EVENTCLASS_SERVER
    • Recorded when a node is created or being recovered from the database and the specified vnode_pool value is negative or equal to zero.

Interface #4: new Server log entry: "Unsupported actions for vnode_pool"

  • Visibility: Public
  • Change Control: Experimental
  • Details:
    • Comments on the interface.
      • Interface standing: new interface.
      • Interface type: log message - log level: PBSEVENT_ADMIN class PBS_EVENTCLASS_SERVER
    • Recorded when an attempt to set or alter vnode_pool on an existing node.

Interface #5: new Server log entry: "Error <#> setting attribute vnode_pool in update for vnode <name>"

  • Visibility: Public
  • Change Control: Experimental
  • Details:
    • Comments on the interface.
      • Interface standing: new interface.
      • Interface type: log message - log level: PBSEVENT_SYSTEM class PBS_EVENTCLASS_NODE
    • Recorded when a yet undefined vnode is reported by a Mom, and the vnode_pool is improperly declared.
    • Note: this attribute should not be defined in a vnode definition file. To do will result in unspecified results.

Interface #6: new Server log entry: "POOL: IS_UPDATE<x> received"

  • Visibility: PBS Private
  • Change Control: Experimental
  • Details:
    • Comments on the interface.
      • Interface standing: new interface.
      • Interface type: log message - log level: PBSEVENT_DEBUG4 class PBS_EVENTCLASS_NODE
    • Recorded when a Mom sends either an IS_UPDATE or IS_UPDATE2 message to the Server and the sending pbs_mom is in a vnode_pool.

Interface #7: new Server log entry: "write_single_node_mom_attr, Failed to update 'Mom' attribute"

  • Visibility: Public
  • Change Control: Experimental
  • Details:
    • Comments on the interface.
      • Interface standing: new interface.
      • Interface type: log message - log level: PBSEVENT_ERROR class PBS_EVENTCLASS_SERVER
    • Recorded when an update of the database failed when updating the 'Mom' attribute of a vnode.

Interface #8: new Server log entry: "Setting inventory_mom for vnode_pool <x> to <Mom name>"

  • Visibility: Public
  • Change Control: Experimental
  • Details:
    • Comments on the interface:
      • Interface standing: new interface
      • Interface type: log message - log level: PBSEVENT_DEBUG
    • Recorded when the inventory_mom is set (including each time the inventory_mom changes)

Interface #9: new Server log entry: "Mom <Mom name> added to vnode_pool <x>"

  • Visibility: Public
  • Change Control: Experimental
  • Details:
    • Comments on the interface:
      • Interface standing: new interface
      • Interface type: log message - log level: PBSEVENT_DEBUG3
    • Recorded when a mom is added to a vnode_pool

Interface #10: new Server log entry: "vnode_pool value is <x>"

  • Visibility: Public
  • Change Control: Experimental
  • Details:
    • Comments on the interface:
      • Interface standing: new interface
      • Interface type: log message - log level: PBSEVENT_DEBUG3
    • Recorded when a node is created or being recovered from the database

Interface #11: new Server log entry: "POOL: cross linking <x> vnodes from <mom>"

  • Visibility: Private
  • Change Control: Experimental
  • Details:
    • Comments on the interface:
      • interface standing: new interface
      • Interface type: log message - log level: PBSEVENT_DEBUG4
    • Recorded during mom creation when a mom is being added to a vnode_pool

Administrator's instructions.

  • Mixed versions of the PBS daemons cannot be run on the system. The pbs_server and all pbs_mom
    daemons must be upgraded at once. An overlay upgrade is possible. The vnode_pool attribute should not be defined 
    in a vnode definition file. To do so will result in unspecified results. Specifying vnode_pool on a non Cray login node 
    is not recommended and will result in undefined behavior.

User's instructions and usage notes.

  • There is no impact on the user functionality.
  • No difference in usage.


Definition of technical terms, spelling out acronyms and abbreviations.

Technical termDescription or definition
InventoryThe list of compute nodes, including the number of CPUs and amount of memory on each, reported as vnodes by pbs_mom on a Cray system.
Inventory MomThe pbs_mom which is to report the inventory of the compute nodes, which is common to all pbs_moms running on the Cray system.