This is a design proposal for PBS to support job submissions with conditional operators to filter out node where the job can run.

Here is the list of Use cases and motivation behind it.

Motivation:  Resilience – ensure jobs run “correctly” and are unlikely to experience faults due to use of nodes with incompatible properties (with respect to the applications)

Use Cases: 

1.      User requests all allocated nodes will have CPU speed > 2 GHz

2.      User requests none of the allocated nodes will be node X, node Y, node Z, …

3.      User requests none of the allocated nodes will be ARM nor POWER architecture

4.      User requests all of the allocated nodes should be running Linux version 6.5 or higher, but none will be running 6.5.2

Gist of design: We can expose a way for each job to pass a limited python expression that can be used to filter out nodes on the basis of their resources_available present on the nodes. This filter will be used to filter out nodes that will be considered to run a job (if submitted with node_filter).

link to  forum discussion

Extend PBS to allow users to submit jobs with a node-filter (node_filter) expression.

• Details:
  • Example: qsub -lselect=3:ncpus=2:mem=18gb -lnode_filter=“ncpus<=8 and model==Skylake or ncpus>=8 and model==Haswell” job.scr
• A user can specify a node filter with each of their jobs and this filter will help scheduler to filter out nodes that this job is allowed to run on.
• There is a new built-in resource “node_filter”. This resource is of type string. Users/operator/manager has privileges to read/write this resource. It is not a host level resource.
• node_filter is an expression created by using resources_available on the nodes.
• node_filter can be a collection of multiple filters separated by 'or' operator. The scheduler will cull the nodes by applying filters and run the job as soon as it finds the node solution.
• If there are many filters separated by 'or' operator, the scheduler will start applying filter. After applying each filter scheduler will try to find a node solution for the job, if it is able to find one then it will use it otherwise it will discard the result of the filter and applies the next filter on all the nodes again. The scheduler may choose to apply filters in any order.
• Users can specify a node filter with node resources using operators like "==,<, >, <=, >=, !=, and, or" 
• While applying the filter, if scheduler encounters a resource in the filter that is not present in the 'resources' line of sched config then the scheduler will ignore that resource while filtering the nodes.
• Caveats:
  
  • node_filter and -lplace=group cannot be used together in the same job. Such a job submission will fail with error "node filter cannot be used with placement grouping"