Introduction:

Use case: 

As clusters get larger and workloads vary it is becoming critical that the jobs get evaluated in as short as time possible to ensure that the correct workload is being run. Using multiple schedulers to address this issue can allow for different scheduling policies and quicker turnaround time for large number of jobs or nodes.

Gist of design proposal::

PBS scheduler in it's current form can run easily run in multiple instances on the same machine. There are only two major problems that we have to deal with:

• Managing the scheduler - This includes starting the scheduler, configuring them, making PBS server connect to each one of them and then make them run on specific events.
• Make sure that schedulers do not overrun on each other's territory. Make sure that they run on clearly partitioned complex in terms of jobs and nodes.

Design proposal mentioned below tends to address both these problem.

Forum discussion

• Interface 1: Extend PBS to support a list of scheduler objects

  • Visibility: Public
  • Change Control: Stable
  • Details:
    • PBS supports a list of scheduler objects to be created using qmgr. It is similar to how we create nodes in server.
    • qmgr command can be used to create a scheduler object . It must be invoked by a PBS admin/manager.
    • To create a scheduler object and make it run, the following are the mandatory attributes that needs to be set by the user
      • Name of the scheduler is mandatory to be given while creating a scheduler object. 
        • qmgr -c "c sched multi_sched_1"
          • This will create/set the following attributes for the sched object
            
            • port - If not defined by the user, It will start from 15050 and try to run the scheduler on the next available port number.
            • host (read-only for now, Has the same value as PBS server host)
            • partition = "None" (default)
              
            • sched_priv = $PBS_HOME/multi_sched_1_priv (default)
              
            • sched_log = $PBS_HOME/multi_sched_1_log (default)
              
            • scheduling = False (default)
              
            • scheduler_iteration = 600 (default)
              
            • sched_user = <pbs_server user> (default)

            • comment 

• • • sites can use the comment field to
      • notify them if scheduler undergoes restarting 2-3 times due to potential crashes in an hour for example (i.e. comment => “NEEDS_ATTENTION”)
      • tell when a particular scheduler is ready to function again by setting the comment as follows.
        comment => “READY_TO_USE
• Set the priv directory for the scheduler.
  • The directory must be owned by the sched_user specified while creating scheduler object. It should have permissions set as "750". By default a sched object has 
    it's priv directory set as $PBS_HOME/<sched-name>_priv
  • qmgr -c "s sched multi_sched_1 sched_priv=/var/spool/pbs/sched_priv_1"
  • If the priv directory is not accessible and admin tries to set "scheduling" attribute to true then error code is set to "15211" and following error is thrown to the user
    "scheduler <sched object name> can not access it's log or priv directory"
  Set the log directory for the scheduler. 
  • The directory must be owned by the sched_user specified while creating scheduler object. It should have permissions as "755". By default a sched object has 
    it's logs directory set as $PBS_HOME/<sched_name>_logs
  • qmgr -c "s sched multi_sched_1 sched_log=/var/spool/pbs/sched_logs"
  • If the log directory is not accessible and admin tries to set "scheduling" attribute to true then error code is set to "15211" and following error is thrown to the user
    "scheduler <sched object name> can not access it's log or priv directory"
  • If the log directory is the same as another sched object then error code is set to "?" and the following error is thrown to the user
    "scheduler <sched object name> can not set its log dir to the same dir as <sched object name> or priv directory"
  To set scheduling on one of the newly created scheduler object one must make use of scheduler name. 
  • By default a multi-sched object has scheduling set as False.
    qmgr -c " s sched <scheduler name> scheduling = 1"
• By default PBS server will configure a default scheduler which will run out of the box.
  • The name of this default scheduler will be "sched"
  • The sched_priv directory of this default scheduler will be set to the $PBS_HOME/sched_priv
  • Default scheduler will log in $PBS_HOME/sched_logs directory.
  • Default scheduler will be provided with default set of policies as mentioned in Interface 3.

Interface 2: Changes to PBS scheduler

• Visibility: Public
• Change Control: Stable
• Details:
  • Scheduler now has additional attributes which can be set in order to run it.
    • sched_priv - to point to the directory where scheduler keeps the fairshare usage, resource_group, holidays file and sched_config
    • sched_logs - to point to the directory where scheduler logs.
    • partitions - list of all the partitions for which this scheduler is going to schedule jobs.
    • host - hostname on which scheduler is running. For default scheduler it is set to pbs server hostname.
    • port - port number on which scheduler is listening.
    • state - This attribute shows the status of the scheduler. It is a parameter that is set only by pbs server.
  • One can set a partition or a comma separated list of partitions to scheduler object. Once set, given scheduler object will only schedule jobs from the queues attached to specified partition"
    • qmgr -c "s sched multi_sched_1 partitions='part1,part2'"
  • If no partition are specified for a given scheduler object, other than the default scheduler where no partition value can be set, then that scheduler will not schedule any jobs.
  • By default, All new queues created will be scheduled by the default scheduler, until they have been assigned to a specific partition.
  • A partition once attached to a scheduler can not be attached to a second scheduler without removing it from the first scheduler. If tried, then it will throw following error:
    • qmgr -c "s sched multi_sched_1 partitions+='part2'"
      Partition part2 is already associated with scheduler <scheduler name>.
  • Scheduler object "state" attribute will show one of these 3 values  - DOWN, IDLE, SCHEDULING
    • If a scheduler object is created but scheduler is not running for some reason state will be shown as "DOWN"
    • If a scheduler is up and running but waiting for a cycle to be triggered the state will be shown as "IDLE"
    • If a scheduler is up and running and also running a scheduling cycle then the state will be shown as "SCHEDULING"
  • The default sched object is the only sched object that cannot be deleted.

Interface 3: Removed

Interface 4: Changes to PBS server.

• Visibility: Public
  Change Control: StableDetails:
  • PBS does not allow attributes like scheduling, scheduler_iteration to be set on PBS server object.
  • scheduling and scheduler_iteration now belong to the sched object
    • During failover when secondary server takes control it will try to connect to connect to schedulers by using their host attribute.
      
      • If secondary server is unable to connect to scheduler running on remote host then it will start that scheduler locally and update it's "host" attribute.
      • When Primary pbs server takes control from secondary it will always check if scheduler's host attribute matches it's server name, if it doesn't then it will shutdown the remote scheduler and spawn it locally on primary server.
    • If set at the server level, the changes will be applied to the default sched object

Interface 5: Changes to PBS Nodes objects.

• Visibility: Public
• Change Control: Stable
• Details:
  • Node object in PBS will have an additional attribute called "partition" which can be used to associate a node to a particular partition.
    • This attribute will be of type string and it will be settable only by Manager/operator and viewable by all users.
  • If "partition" attribute is not set, node will not belong to any partition and default scheduler will schedule jobs on this node.
  • PBS admin/manager can set node's partition attribute to an existing partition name and it's corresponding scheduler will be scheduling jobs on this node.
  • If nodes are associated to a partition then they can not be linked to any queue which isn't part of that partition. Trying to set a node to a queue which isn't part of it's partition will result into following error:
    • qmgr -c "s n node1 queue=workq3"
      workq3 is not part of partition <node's partition name>

Interface 6: Changes to Queues.

• Visibility: Public
• Change Control: Stable
• Details:
  
  • Queue will have a new "partition" attribute which can be used to associate a node to a particular partition.
    • This attribute will be of type string and it will be settable only by Manager/operator and viewable by all users.
    • If "partition" attribute is not set to anything, queue will not belong to any partition and default scheduler will schedule jobs from this queue.
    • We recommend this attribute to be set as job-wide default at queue level
    • When an admin changes the partition value at the queue level all jobs (except in a finished state) will have their partition attribute changed to match the new value. All new jobs entering the queue after that time will have their partition attribute to match the partition value set for the queue
      
      

Interface 7: How PBS server runs scheduler.

• Visibility: Public
• Change Control: Stable
• Details:
  • Upon startup PBS server will start all schedulers which have their scheduling attribute set to "True"
    • "PBS_START_SCHED" pbs.conf variable is now deprecated and it's value will get overridden by schedulers "scheduling" attribute.
  • PBS server will connect to these schedulers on their respective hostnames and port number.
  • If server is unable to connect to these schedulers it will check to see if the scheduler is running, try to connect 5 times, and finally restart the scheduler.
  • Scheduling cycles for all configured schedulers are started by PBS server when a job is queued, finished, when scheduling attribute is set to True or when scheduler_iteration is elapsed.
    • When a job gets queued or finished, server will check it's corresponding queue and try to connect to it's corresponding scheduler to run a scheduling cycle.
    • If a scheduler is already running a scheduling cycle while server will just wait for the previous cycle to finish before trying to start another one.
    • If job_accumulation_time is set then server will wait until that time has passed after the submission of a job before starting a new cycle.
  • Each scheduler while querying server specifies it's scheduler name and then gets only a chunk of the universe which is relevant to this scheduler.
    • It gets all the running, queued, exiting jobs from the queues it is associated with one of it's partitions.
    • It gets all the list of nodes which are associated with the partition managed by the scheduler.
    • It gets the list of all the global policies like run soft/hard limits set on the server object.
  • PBS's init script will now be reporting status of pbs server only. Schedulers will be managed by server and their status can be fetched using a qmgr command.
    • When pbs_server daemon is stopped using "qterm -s" then, it will also stop all the running scheduler processes.
    • pbs init script while shutting down pbs_server will use the "-s" option to qterm so that all schedulers also come down along with server.

Interface 8: Changes to pbs_rsub command

• Visibility: Public
• Change Control: Stable
• Details:
  
  • Reservations can now be submitted to a specific partition using a new "-p" option with pbs_rsub command.
  • "-p" option in pbs_rsub command takes partition name as input and makes pbs_server to trigger a scheduling cycle of the scheduler that is servicing the partition.
  • If a scheduler servicing the requested partition isn't up and running then pbs server will store the reservation with itself and mark it as "UNCONFIRMED" until it is able to trigger a scheduling cycle of the said scheduler.
    
    

Interface 9: Job level attribute for partition

• Visibility: Public
• Change Control: Stable
• Details:
  
  • Only PBS Managers can set or alter this attribute. 
  • This job attribute facilitates the moving of a queue from one partition to another partition without draining all running jobs of the queue before move.
    
    

Notes:

• What does not work when multiple scheduler objects are present.
  
  • When there are multiple scheduler objects configures following things might be broken.
    • Run limits set on server may seem to be broken because a scheduler object may not have a view of whole of the universe.
    • Fairshare is now only limited to what a specific scheduler views, it can not be done complex wide with multiple schedulers.