Page Comparison

Community discussion is located here: http://community.pbspro.org/t/pp-719-enhance-setup-in-ptl-specifically-for-cray-platforms/463

Interface: PBSTestSuite.setUp() 

...

Overview:

PBSTestSuite.setUp() calls several functions. The changes made in the settings in these functions are often not desirable on Cray because the MoM, scheduler, and server settings get reverted to plain Linux out-of-box configurations.

Out-of-box configuration of PBS on Cray:

...

• • These lines should appear in PBS_HOME/mom_priv/config:
  •         $vnodedef_additive 0
  •         $alps_client /opt/cray/alps/default/bin/apbasil
  •         $usecp *:/home /home
    
    Notes: $usecp setting above is used to enable local copy of job output of jobs that ran on compute nodes.
    $vnodedef_additive 0 will allow MoM to tell the server that there are vnodes missing, which the server will mark as stale.
    Use '$alps_client /opt/alps/apbasil.sh' if on a Cray ALPS simulator.

2) Scheduler

• • PBS_HOME/sched_priv/sched_config will have 'vntype' in the resources line.

...

• • The hook PBS_HOME/server_priv/hooks/PBS_translate_mpp.HK is enabled.
  • Based on what is returned by the ALPS inventory :
    • vnodes representing the compute nodes will get created
    • the Cray-specific custom resources (e.g. PBScraynid, PBScrayhost, etc) will get created in PBS_HOME/server_priv/resourcedef.
    • the appropriate Cray-specific custom resources will be added to the vnodes.
    • since the mom config $vnode_per_numa_node is unset by default, there will be no PBScrayseg attribute on the vnodes representing the compute nodes.
  • Server settings :
    • flatuid = true

Design of PBSTestSuite.setUp() for Cray

In order to revert to out-of-box configuration of PBS on Cray, per PBS daemon:

...

• • • scheduling = true

    • the default queue is workq 

Interface:  MoM.__init__() 

• Visibility: Public
• Change Control: Stable
• Synopsis: Modifications to include initial MoM config settings for Cray.
  
• Standing of the interface : modified interface
  
• Details
  • if on real Cray or Cray ALPS simulator, then initialize the PBS_HOME/mom_priv/config to have these:

  • $vnodedef_additive 0

  • $alps_client /opt/cray/alps/default/bin/apbasil

$usecp *:/home /home

• • Note:

...

• • If on Cray ALPS simulator use '$alps_client /opt/alps/apbasil.sh'.

...

2) Scheduler:

• • $usecp *:/home /home

Interface:  Scheduler.revert_to_defaults() 

• Visibility: Public
• Change Control: Stable
• Synopsis: Modifications to include 'vntype' in sched_config out-of-box settings for Cray.
  
• Standing of the interface : modified interface
  
• Details
  • After copying the out-of-box scheduler configuration in PBS_EXEC/etc/pbs_sched_config to PBS_HOME/sched_priv/sched_config :
    • If platform is a real Cray or Cray ALPS simulator, then add "vntype" to resources line of PBS_HOME/sched_priv/sched_config.

3) Server:

...

Interface:  Server.revert_to_defaults() 

• Visibility: Public
• Change Control: Stable
• Synopsis: Modifications to revert resources, vnodes, and hooks settings for Cray.
  
• Standing of the interface : modified interface
  
• Details

• resources and vnodes
  
  • As part of reverting resources, the PBS_HOME/server_priv/resourcedef file is removed.
    
if
• • • If on Cray or Cray ALPS simulator, then recreate the out-of-box Cray-specific custom resources and vnodes,
      including the ones PBS creates from reading the ALPS inventory, by:
      • Checking that $alps_client and its value exists in PBS_HOME/mom_priv/config, and if so:
        • delete all the vnodes: qmgr -c "delete node @default"
        • add back the MoMs: qmgr -c "
c
• • • • • create n <MoM hostname>"
        • if successful, then
          
          • the resourcedef file and the vnodes would have reverted to the out-of-box Cray configuration.
        • If there is an error during one of the following:
          • checking if $alps_client is properly configured
          • deleting all vnodes
          • adding MoMs
          then raise an exception.
3b)
• hooks
  • As part of reverting hooks, all hooks, except built-in hooks, are disabled.
    
    • if on a Cray ALPS simulator, then enable the PBS_translate_mpp.HK hook.
    • If there is an error during enabling of the PBS_translate_mpp.HK hook, then raise an exception.