PP-719: Enhance setUp in PTL specifically for Cray platforms

Community discussion is located here:

Interface: setUp() 

• Visibility: Public
• Change Control: Stable
• Synopsis: PBSTestSuite.setUp() needs to be able to revert to the 'out-of-box' PBS configuration on Cray.
  
• Details: setUp() calls several functions. The changes made in the settings in these functions are often not desirable on Cray because the MoM, scheduler, and server settings get reverted to plain Linux default settings.

Out-of-box configuration of PBS on Cray:

• PBS is installed and configured on Cray which result in specific settings. These initial settings are referred to as the out-of-box settings below.  These settings should also apply to the Cray ALPS simulator when PBS is properly configured.  These are settings for a freshly installed and configured PBS on Cray or Cray ALPS simulator.

• As of current support for Cray CLE 5.2 and 6.0, below are the out-of-box configuration settings per PBS daemon:

1)  MoM

• • These settings appear in PBS_HOME/mom_priv/config:
  •         $vnodedef_additive 0
  •         $alps_client /opt/cray/alps/default/bin/apbasil
  •         $usecp '*:/home /home'

2) Scheduler

• • PBS_EXEC/etc/pbs_sched_config will have 'vntype' in the resources line.

3) Server

• • The hook PBS_HOME/server_priv/hooks/PBS_translate_mpp.HK is enabled.
  • Based on what is returned by the ALPS inventory :
    • vnodes representing the compute nodes will get created
    • the Cray-specific custom resources (e.g. PBScraynid, PBScrayhost, etc) will get created in PBS_HOME/server_priv/resourcedef.
    • the appropriate Cray-specific custom resources will be added to the vnodes.
    • since the mom config $vnode_per_numa_node is unset by default, there will be no PBScrayseg attribute on the vnodes representing the compute nodes.
  • Server settings :
    • flatuid = true
    • scheduling = true
    • the default queue is workq 
      
      

Design of PBSTestSuite.setUp() for Cray

The affected functions are shown below:

1) MoM:

• In pbspro/test/fw/ptl/lib/pbs_testlib.py, class MoM, method __init__()
  • if on real Cray or Cray ALPS simulator, then add these (key, value) pairs to the dictionary self.dflt_config :
    • ('$vnodedef_additive', 0)
    • ('$alps_client', '<path to apbasil>')
    • ('$usecp' '*:/home /home')

2) Scheduler:

• In pbspro/test/fw/ptl/lib/pbs_testlib.py, class Scheduler, method revert_to_defaults()
  • After copying the out-of-box scheduler configuration in PBS_EXEC/etc/pbs_sched_config to PBS_HOME/sched_priv/sched_config :
    • If platform is a real Cray or Cray ALPS simulator, then add "vntype" to resources line of PBS_HOME/sched_priv/sched_config.

3) Server:

• • 3a) resources
    • In pbspro/test/fw/ptl/lib/pbs_testlib.py, class Server, method revert_to_defaults()
      • in the "if revertresources:" clause, after removing resourcedef,
        if on Cray or Cray ALPS simulator, then recreate the out-of-box Cray-specific custom resources and vnodes,
        including the ones PBS creates from reading the ALPS inventory, by:
        • Checking that $alps_client and its value exists in PBS_HOME/mom_priv/config, and if so:
          • delete all the vnodes: qmgr -c "delete node @default"
          • add back the MoMs: qmgr -c "c n <MoM hostname>"
          • if successful, then
            
            • the resourcedef file and the vnodes would have reverted to the out-of-box Cray settings
            • return True for revert_to_defaults()
          • If there is an error during one of the following:
            • checking if $alps_client is properly configured
            • deleting all vnodes
            • adding MoMs
            then raise an exception. This will cause revert_to_defaults() to return False.
            
            
  • 3b) hooks
    • In pbspro/test/fw/ptl/lib/pbs_testlib.py, class Server, method revert_to_defaults()
      • in the "if reverthooks:" clause, after all hooks are disabled,
        if on a Cray ALPS simulator, then enable the PBS_translate_mpp.HK hook.
      • If there is an error during enabling of the PBS_translate_mpp.HK hook, then an exception is raised.  This will cause revert_to_defaults() to return False.