...
Interface 1: Extend PBS to support a list of scheduler objects
- Visibility: Public
- Change Control: Stable
- Details:
- PBS supports a list of scheduler objects to be created using qmgr. It is similar to how we create nodes in server.
- qmgr command can be used to create a scheduler object . It must be invoked by a PBS admin/manager.
- To create a scheduler object and make it run, the following are the mandatory attributes that needs to be set by the user
- Name of the scheduler is mandatory to be given while creating a scheduler object.
- qmgr -c "c sched multi_sched_1"
- This will create/set the following attributes for the sched object
- port - If not defined by the user, It will start from 15050 and try to run the scheduler on the next available port number.
- host (read-only for now, Has the same value as PBS server host)
- partition = "None" (default)
- sched_priv = $PBS_HOME/multi_sched_1_priv (default)
- sched_log = $PBS_HOME/multi_sched_1_log (default)
- scheduling = False (default)
- scheduler_iteration = 600 (default)
- sched_user = <pbs_server user> (default)
comment
- This will create/set the following attributes for the sched object
- qmgr -c "c sched multi_sched_1"
- Name of the scheduler is mandatory to be given while creating a scheduler object.
- comment can be set to the following sites can use the comment field to
- notify them if scheduler undergoes restarting 2-3 times due to potential crashes in an hour for example (i.e.
- comment => “NEEDS_ATTENTION”)
- tell when a particular scheduler is ready to function again by setting the comment as follows.
comment => “READY_TO_USE
- comment can be set to the following sites can use the comment field to
- Set the priv directory for the scheduler.
- The directory must be owned by the sched_user specified while creating scheduler object. It should have permissions set as "750". By default a sched object has
it's priv directory set as $PBS_HOME/<sched-name>_priv - qmgr -c "s sched multi_sched_1 sched_priv=/var/spool/pbs/sched_priv_1"
- If the priv directory is not accessible and admin tries to set "scheduling" attribute to true then error code is set to "15211" and following error if is thrown to the user
"scheduler <scheduler <sched object name> can not access it's log or priv directory"
- The directory must be owned by the sched_user specified while creating scheduler object. It should have permissions as "755". By default a sched object has
it's logs directory set as $PBS_HOME/<sched_name>_logs - qmgr -c "s sched multi_sched_1 sched_log=/var/spool/pbs/sched_logs"
- If the log directory is not accessible and admin tries to set "scheduling" attribute to true then error code is set to "15211" and following error if is thrown to the user
"scheduler <scheduler <sched object name> can not access it's log or priv directory" - If the log directory is the same as another sched object then error code is set to "?" and the following error is thrown to the user
"scheduler <sched object name> can not set its log dir to the same dir as <sched object name> or priv directory"
- By default a multi-sched object has scheduling set as False.
qmgr -c " s sched <scheduler name> scheduling = 1"
- The directory must be owned by the sched_user specified while creating scheduler object. It should have permissions set as "750". By default a sched object has
- By default PBS server will configure a default scheduler which will run out of the box.
- The name of this default scheduler will be "sched"
- The sched_priv directory of this default scheduler will be set to the $PBS_HOME/sched_priv
- Default scheduler will log in $PBS_HOME/sched_logs directory.
- Default scheduler will be provided with default set of policies as mentioned in Interface 3.
...
- Visibility: Public
- Change Control: Stable
- Details:
- Scheduler now has additional attributes which can be set in order to run it.
- sched_priv - to point to the directory where scheduler keeps the fairshare usage, resource_group, holidays file and sched_config
- sched_logs - to point to the directory where scheduler logs.
- partitions - list of all the partitions for which this scheduler is going to schedule jobs.
- host - hostname on which scheduler is running. For default scheduler it is set to pbs server hostname.
- port - port number on which scheduler is listening.
- job_accumulation_time - amount of time server will wait after the submission of a job before starting a new cycle.
- state - This attribute shows the status of the scheduler. It is a parameter that is set only by pbs server.
- One can set a partition or a comma separated list of partitions to scheduler object. Once set, given scheduler object will only schedule jobs from the queues attached to specified partition"
- qmgr -c "s sched multi_sched_1 partitions='part1,part2'"
- If no partition are specified with for a given scheduler object, other than the default scheduler where no partition value can be set, then that scheduler will not schedule any jobs.
- By default, All new queues created will be attached to scheduled by the default scheduler, until they have been assigned to a specific partition.
- A partition once attached to a scheduler can not be attached to another a second scheduler without removing it from the first scheduler. If tried, then it will throw following error:
- qmgr -c "s sched multi_sched_1 partitions+='part2'"
Partition part2 is already associated with scheduler <scheduler name>.
- qmgr -c "s sched multi_sched_1 partitions+='part2'"
- Scheduler object "state" attribute will show one of these 3 values - DOWN, IDLE, SCHEDULING
- If a scheduler object is created but scheduler is not running for some reason state will be shown as "DOWN"
- If a scheduler is up and running but waiting for a cycle to be triggered the state will be shown as "IDLE"
- If a scheduler is up and running and also running a scheduling cycle then the state will be shown as "SCHEDULING"
- The default sched object is the only sched object that cannot be deleted.
- Scheduler now has additional attributes which can be set in order to run it.
Interface 3: Removed
Interface 4: Changes to PBS server.
- Visibility: Public
Change Control: StableDetails:- PBS does not allow attributes like scheduling, scheduler_iteration to be set on PBS server object.
- scheduling and scheduler_iteration now belong to the sched object
- During failover when secondary server takes control it will try to connect to connect to schedulers by using their host attribute.
- If secondary server is unable to connect to scheduler running on remote host then it will start that scheduler locally and update it's "host" attribute.
- When Primary pbs server takes control from secondary it will always check if scheduler's host attribute matches it's server name, if it doesn't then it will shutdown the remote scheduler and spawn it locally on primary server.
- If set at the server level, the changes will be applied to the default sched object
- During failover when secondary server takes control it will try to connect to connect to schedulers by using their host attribute.
Interface 5: Changes to PBS Nodes objects.
...
Interface 6: Changes to Queues. Removed..
- Visibility: Public
- Change Control: Stable
- Details:
- Queue will have a new "partition" attribute which can be used to associate a node to a particular partition.
- This attribute will be of type string and it will be settable only by Manager/operator and viewable by all users.
- If "partition" attribute is not set to anything, queue will not belong to any partition and default scheduler will schedule jobs from this queue.
- We recommend this attribute to be set as job-wide default at queue level
- When an admin changes the partition value at the queue level all jobs (except in a finished state) will have their partition attribute changed to match the new value. All new jobs entering the queue after that time will have their partition attribute to match the partition value set for the queue
- Queue will have a new "partition" attribute which can be used to associate a node to a particular partition.
Interface 7: How PBS server runs scheduler.
- Visibility: Public
- Change Control: Stable
- Details:
- Upon startup PBS server will start all schedulers which have their scheduling attribute set to "True"
- "PBS_START_SCHED" pbs.conf variable is now deprecated and it's value will get overridden by schedulers "scheduling" attribute.
- PBS server will connect to these schedulers on their respective hostnames and port number.
- If server is unable to connect to these schedulers it will check to see if the scheduler is running, try to connect 5 times, and finally restart the scheduler.
- Scheduling cycles for all configured schedulers are started by PBS server when a job is queued, finished, when scheduling attribute is set to True or when scheduler_iteration is elapsed.
- When a job gets queued or finished, server will check it's corresponding queue and try to connect to it's corresponding scheduler to run a scheduling cycle.
- If a scheduler is already running a scheduling cycle while server will just wait for the previous cycle to finish before trying to start another one.
- If job_accumulation_time is set then server will wait until that time has passed after the submission of a job before starting a new cycle.
- Each scheduler while querying server specifies it's scheduler name and then gets only a chunk of the universe which is relevant to this scheduler.
- It gets all the running, queued, exiting jobs from the queues it is associated with one of it's partitions.
- It gets all the list of nodes which are associated with the partition managed by the scheduler.
- It gets the list of all the global policies like run soft/hard limits set on the server object.
- PBS's init script will now be reporting status of pbs server only. Schedulers will be managed by server and their status can be fetched using a qmgr command.
- When pbs_server daemon is stopped using "qterm -s" option then, it will also stop all the running scheduler processes.
- pbs init script while shutting down pbs_server will now use the "-s" option to qterm so that all schedulers also come down along with server.
- Upon startup PBS server will start all schedulers which have their scheduling attribute set to "True"
...
- Visibility: Public
- Change Control: Stable
- Details:
- Only PBS Manager Managers can only set or alter this attribute. We recommend this attribute be set at queue level as job-wide default. Through this all jobs that belong to this queue now contain the value for partition job attribute.
- This job attribute helps in moving the facilitates the moving of a queue from one partition to another partition without draining all running jobs of the queue before move.
...