Date: Fri, 29 Mar 2024 08:02:11 +0000 (UTC)
Message-ID: <882463775.1.1711699331079@9c244d5e6733>
Subject: Exported From Confluence
MIME-Version: 1.0
Content-Type: multipart/related;
boundary="----=_Part_0_623303654.1711699331065"
------=_Part_0_623303654.1711699331065
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Content-Location: file:///C:/exported.html
This is a design proposal for PBS to support job submissions with c=
onditional operators to filter out node where the job can run.
Here is the list of Use cases and motivation behind it.
M=
otivation: Resilience =E2=80=93 ensure jobs run =E2=80=9Ccor=
rectly=E2=80=9D and are unlikely to experience faults due to use of nodes w=
ith incompatible properties (with respect to the applications)
U=
se Cases:
1.=
User requests all allocated nodes will have CPU speed > 2 GHz
2.=
User requests none of the allocated nodes will be node X, node Y, n=
ode Z, =E2=80=A6
3.=
User requests none of the allocated nodes will be ARM nor POWER arc=
hitecture
4.=
User requests all of the allocated nodes should be running Linux ve=
rsion 6.5 or higher, but none will be running 6.5.2
Gist of design: We can expos=
e a way for each job to pass a limited python expression that can be used t=
o filter out nodes on the basis of their resources_available present on the=
nodes. This filter will be used to filter out nodes that will be considere=
d to run a job (if submitted with node_filter).
link to &=
nbsp;forum discussion
Exte=
nd PBS to allow users to submit jobs with a node-filter (node_filter) expre=
ssion.
- Details:
- Example: qsub -lselect=3D3:ncpus=3D2:m=
em=3D18gb -lnode_filter=3D=E2=80=9Cncpus<=3D8 and model=3D=3DSky=
lake or ncpus>=3D8 and model=3D=3DHaswell=E2=80=9D job.scr=
- A user can specify a node filter with each of their jobs =
and this filter will help scheduler to filter out nodes that this job is al=
lowed to run on.
- There is a new built-in resource =E2=80=9Cnode_filter=E2=
=80=9D. This resource is of type string. Users/operator/manager has privile=
ges to read/write this resource. It is not a host level resource.
- node_filter is an expression created by using resources_=
available on the nodes.
- node_filter can be a collection of multiple filters sepa=
rated by 'or' operator. The scheduler will cull the nodes by applying filte=
rs and run the job as soon as it finds the node solution.
- If there are many filters separated by 'or' operator, th=
e scheduler will start applying filter. After applying each filter schedule=
r will try to find a node solution for the job, if it is able to find one t=
hen it will use it otherwise it will discard the result of the filter and a=
pplies the next filter on all the nodes again. The scheduler may choose to =
apply filters in any order.
- Users can specify a node filter with =
node resources using operators like "=3D=3D,<, >, <=3D, >=
=3D, !=3D, and, or"
- While applying the filter, if schedule=
r encounters a resource in the filter that is not present in the 'resources=
' line of sched config then the scheduler will ignore that resource while f=
iltering the nodes.
- Caveats:
- node_filter and -lplace=3Dgroup cannot be used together i=
n the same job. Such a job submission will fail with error "node filter cannot be used with placement grouping=
"
------=_Part_0_623303654.1711699331065--