Versions Compared

Old Version 3

changes.mady.by.user Shrinivas Harapanahalli

Saved on

compared with

New Version Current

changes.mady.by.user Shrinivas Harapanahalli

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

[[This EDD is

...

stalled]]


Target releaseFuture release 
JIRA link

Jira Legacy
serverPBS Pro JIRA
columnskey,summary,priority,status,resolution
serverIda18d00cd-0706-3856-bdfc-a2e3e6110bd7
keyPP-1018

Document status
DRAFT
Document owner

Shrinivas Harapanahalli

Designer

Shrinivas Harapanahalli

Developers


QA
Forum DiscussionPP-1018: Design document review of Placement set sorting feature



Interface: New sched_config "node_group_sort_key" to enable placement set sorting.

  • Change Control:  stable

  • Standing of the interface:  new interface

  • Interface type:  Configure variable  (in sched_config)

  • Synopsis:  A new sched_config named "node_group_sort_key" option is added which will enable admin to configure the order of placement set used for job placement

  • Details:

...

total           Use the resources_available value
assigned   Use the resources_assigned value
unused      Use the value given by resources_available - resources_assigned

**ignored for non-consumable resources

HIGH    to sort descending order i.e, high first and low last
LOW     to sort ascending order i.e, low first and high last

...

New behavior introduced with this EDD: Placement sets are created and partitioned based on the combination of different string values defined at the vnodes under custom string array resources named in node_group_key list, indexed multi dimensionally with custom string array resources named in the node_group_key as the index appearing in that order. This is in addition to the placement sets created as in current behavior, i.e flatened index.  For example, if the server’s node_group_key attribute contains “router,switch”, and router can take the values “R1” and “R2” and switch can take the values “S1”, “S2”, and “S3”, then there are elleven placement sets "R1-S1, R1-S2, R1-S3, R2-S1, R2-S2, R2-S3, R1, R2, S1, S2, S3", in two placement series, in the server’s placement pool.

**The implicit unset placement set and the placement set containing all nodes is not shown here.

Example Scenario of placement set sorting:
  1. Simple Placement set (single Placement series)

Lets assume we have simple placement set, i.e a single placement series configured as below

           set server node_group_key = switch
           set server node_group_enable = True
           set sched do_not_span_psets = True

Lets assume we have below vnode list as below

Vnode

name
resource_available.
ncpus
mem
switch
vn0

4

8GB

"sw3,sw5"
vn1

2

8GB

"sw2"
vn2

8

8GB

"sw4,sw1"
vn3

2

4GB

"sw1"
vn4

8

16GB

"sw2"
vn5

4

16GB

"sw3,sw1"
vn6

8

8GB

"sw4,sw5,sw6"
vn7

4

4GB

"sw3,sw4"


Since here we have 8 vnodes in a single placement series, the partitioning and identification of psets will be same with current and new behavior. Here there are 6 placement sets created and identified as below.

psetsTotalVnodes in pset
ncpusmem
sw1
1428GB
vn2, vn3, vn5
sw2
1024GB
vn1, vn4
sw3
1228GB
vn0, vn5, vn7
sw4
2020GB
vn2, vn6, vn7
sw5
1216GB
vn1, vn6
sw6
88GB
vn6


Now the order of placement sets considered for a job placement for currernt and new behaviour is as described below

** only static resource is considered here for simplicity
Ordering of Placement sets:

Current Beharviour

New Behaviour


psetsTotalVnodes in pset
ncpusmem
sw6
88GB
vn6
sw2
1024GB
vn1, vn4
sw5
1216GB
vn1, vn6
sw3
1228GB
vn0, vn5, vn7
sw1
1428GB
vn2, vn3, vn5
sw4
2020GB
vn2, vn6, vn7


node_group_sort_key: "switch HIGH" all
psetsTotalVnodes in pset
ncpusmem
sw6
88GB
vn6
sw5
1216GB
vn1, vn6
sw4
2020GB
vn2, vn6, vn7
sw3
1228GB
vn0, vn5, vn7
sw2
1024GB
vn1, vn4
sw1
1428GB
vn2, vn3, vn5


...

2. Simple Placement Pool with two placement series

Lets assume a complex with multiple partitions and each partition consists multiple node groups. This can be defined by simple placement pool, with two placement series configured as below.

Now assume admin has a requirement for jobs to be placed to a partition by considering all the partitions in a particular order, and default order for placement sets within partions.

** here the words "router" and "switch" are considered as label and do not necessarily relate to network Router or network Switch hardwares or describe any network topology
 
           set server node_group_key = router,switch
           set server node_group_enable = True
           set sched do_not_span_psets = True

Here the resource "router" describes a partition of a complex and the resource "switch" describe node grouping within a partion (or across partitions)

Lets assume we have below vnode list as below

Vnode

name
resource_available.
ncpus
mem
router
switch
vn10

4

8GB

"rt1"
"sw3"
vn11

2

8GB

"rt1"
"sw2"
vn12

8

8GB

"rt1"
"sw4,sw1"
vn13

2

4GB

"rt1"
"sw1"
vn14

8

16GB

"rt1"
"sw2"
vn15

4

16GB

"rt1"
"sw3,sw1"
vn16

8

8GB

"rt1"
"sw4,"
vn17

4

4GB

"rt1"
"sw3,sw4"
vn20

2

4GB

"rt2"
"sw4,sw3,sw2"
vn21

2

8GB

"rt2"
"sw3,sw1"
vn22

4

8GB

"rt2"
"sw2,sw4,sw1,sw3"
vn23

4

4GB

"rt2"
"sw4,sw3,sw1"
vn24

8

8GB

"rt2"
"sw4,sw2"
vn25

4

4GB

"rt2"
"sw1,sw2,sw3,sw4"
vn26

2

2GB

"rt2"
"sw4"
vn27

2

4GB

"rt2"
"sw2,sw3"
vn30

2

2GB

"rt3"
"sw1,sw2,sw4"
vn31

8

8GB

"rt3"
"sw3"
vn32

4

4GB

"rt3"
"sw4,sw2,sw3"
vn33

4

4GB

"rt3"
"sw4,sw3"
vn34

8

16GB

"rt3"
"sw4"
vn35

8

8GB

"rt3"
"sw2,sw1,sw4"
vn36

4

8GB

"rt3"
"sw3"
vn37

8

16GB

"rt3"
"sw1"
vn40

4

8GB

"rt4"
"sw2"
vn41

2

2GB

"rt4"
"sw2"
vn42

8

16GB

"rt4"
"sw4,sw3,sw2"
vn43

4

4GB

"rt4"
"sw3,sw4"
vn44

8

8GB

"rt4"
"sw1,sw2,sw3,sw4"
vn45

4

4GB

"rt4"
"sw4,sw2"
vn46

8

8GB

"rt4"
"sw1"
vn47

8

8GB

"rt4"
"sw4"
vn50

4

8GB

"rt3,rt1"
"sw4,sw1"
vn51

2

2GB

"rt3,rt1"
"sw2"
vn52

8

16GB

"rt3,rt1"
"sw2"
vn53

4

4GB

"rt3,rt1"
"sw1,sw4"






Partitioning and indetification of Placement sets:
Here we have 36 vnodes in a two placement series, the partitioning and identification of psets will be different with current and new behavior as shown below.

Current Beharviour

New Behaviour


Here there will be 8 placement sets identified as below

psetsTotalVnodes in pset
ncpusmem
sw1
70106GB
vn12, vn13, vn15, vn21, vn22, vn23, vn25,vn30, vn35, vn37, vn44,

vn46, vn50, vn53
sw2
80122GB
vn11, vn14, vn20, vn22, vn24, vn25, vn27, vn30, vn32, vn35, vn40,
vn41, vn42, vn44, vn45, vn51, vn52
sw3
70112GB
vn10, vn15, vn17, vn20, vn21, vn22, vn23, vn25, vn27, vn31, vn32,
vn33, vn36, vn42, vn43, vn44
sw4
110136GB
vn12, vn16, vn17, vn20, vn22, vn23, vn24, vn25, vn26, vn30, vn32,
vn33, vn34, vn35, vn42, vn43, vn44, vn45, vn47, vn50, vn53
rt1
58102GB
vn10, vn11, vn12, vn13, vn14, vn15, vn16, vn17, vn50, vn51, vn52,
vn53
rt2
2842GB
vn20, vn21, vn22, vn23, vn24, vn25, vn26, vn27
rt3
6496GB
vn30, vn31, vn32, vn33, vn34, vn35, vn36, vn37,
vn50, vn51, vn52, vn53
rt4
4658GB
vn40, vn41, vn42, vn43, vn44, vn45, vn46, vn47


node_group_sort_key: "router HIGH" all


Here there will be 24 placement sets identified as below


psetsTotalVnodes in pset
ncpusmem
sw1
70106GB
vn12, vn13, vn15, vn21, vn22, vn23, vn25, vn30, vn35, vn37, vn44,

vn46, vn50, vn53
sw2
80122GB
vn11, vn14, vn20, vn22, vn24, vn25, vn27, vn30, vn32, vn35, vn40,
vn41, vn42, vn44, vn45, vn51, vn52
sw3
70112GB
vn10, vn15, vn17, vn20, vn21, vn22, vn23, vn25, vn27, vn31, vn32,
vn33, vn36, vn42, vn43, vn44
sw4
110136GB
vn12, vn16, vn17, vn20, vn22, vn23, vn24, vn25, vn26, vn30, vn32,
vn33, vn34, vn35, vn42, vn43, vn44, vn45, vn47, vn50, vn53
rt1
58102GB
vn10, vn11, vn12, vn13, vn14, vn15, vn16, vn17, vn50, vn51, vn52,
vn53
rt2
2842GB
vn20, vn21, vn22, vn23, vn24, vn25, vn26, vn27
rt3
6496GB
vn30, vn31, vn32, vn33, vn34, vn35, vn36, vn37,vn50, vn51, vn52,
vn53
rt4
4658GB
vn40, vn41, vn42, vn43, vn44, vn45, vn46, vn47
rt1-sw1
2240GB
vn12, vn13, vn15, vn50, vn53
rt1-sw2
2042GB
vn11, vn14, vn51, vn52
rt1-sw3
1228GB
vn10, vn15, vn17
rt1-sw4
2832GB
vn12, vn16, vn17, vn50, vn53
rt2-sw1
1424GB
vn21, vn22, vn23, vn25
rt2-sw2
2028GB
vn20, vn22, vn24, vn25, vn27
rt2-sw3
1832GB
vn20, vn21, vn22, vn23, vn25, vn27
rt2-sw4
2430GB
vn20, vn22, vn23, vn24, vn25, vn26
rt3-sw1
2638GB
vn30, vn35, vn37, vn50, vn53
rt3-sw2
2432GB
vn30, vn32, vn35, vn51, vn52
rt3-sw3
2024GB
vn31, vn32, vn33, vn36
rt3-sw4
3446GB
vn30, vn32, vn33, vn34, vn35, vn50, vn53
rt4-sw1
1616GB
vn44, vn46
rt4-sw2
2638GB
vn40, vn41, vn42, vn44, vn45
rt4-sw3
2028GB
vn42, vn43, vn44
rt4-sw4
3240GB
vn42, vn43, vn44, vn45, vn47


Ordering of Placement Sets:
Now observe below how the psets gets ordered.

Current Beharviour

New Behaviour


psetsTotalVnodes in pset
ncpusmem
rt2
2842GB
vn20, vn21, vn22, vn23, vn24, vn25, vn26, vn27
rt4
4658GB
vn40, vn41, vn42, vn43, vn44, vn45, vn46, vn47
rt1
58102GB
vn10, vn11, vn12, vn13, vn14, vn15, vn16, vn17, vn50, vn51, vn52,
vn53
rt3
6496GB
vn30, vn31, vn32, vn33, vn34, vn35, vn36, vn37,
vn50, vn51, vn52, vn53
sw1
70106GB
vn12, vn13, vn15, vn21, vn22, vn23, vn25,vn30, vn35, vn37, vn44,

vn46, vn50, vn53
sw3
70112GB
vn10, vn15, vn17, vn20, vn21, vn22, vn23, vn25, vn27, vn31, vn32,
vn33, vn36, vn42, vn43, vn44
sw2
80122GB
vn11, vn14, vn20, vn22, vn24, vn25, vn27, vn30, vn32, vn35, vn40,
vn41, vn42, vn44, vn45, vn51, vn52
sw4
110136GB
vn12, vn16, vn17, vn20, vn22, vn23, vn24, vn25, vn26, vn30, vn32,
vn33, vn34, vn35, vn42, vn43, vn44, vn45, vn47, vn50, vn53


node_group_sort_key: "router HIGH" all


psetsTotalVnodes in pset
ncpusmem
rt4-sw1
1616GB
vn44, vn46
rt4-sw3
2028GB
vn42, vn43, vn44
rt4-sw2
2638GB
vn40, vn41, vn42, vn44, vn45
rt4-sw4
3240GB
vn42, vn43, vn44, vn45, vn47
rt3-sw3
2024GB
vn31, vn32, vn33, vn36
rt3-sw2
2432GB
vn30, vn32, vn35, vn51, vn52
rt3-sw1
2638GB
vn30, vn35, vn37, vn50, vn53
rt3-sw4
3446GB
vn30, vn32, vn33, vn34, vn35, vn50, vn53
rt2-sw1
1424GB
vn21, vn22, vn23, vn25
rt2-sw3
1832GB
vn20, vn21, vn22, vn23, vn25, vn27
rt2-sw2
2028GB
vn20, vn22, vn24, vn25, vn27
rt2-sw4
2430GB
vn20, vn22, vn23, vn24, vn25, vn26
rt1-sw3
1228GB
vn10, vn15, vn17
rt1-sw2
2042GB
vn11, vn14, vn51, vn52
rt1-sw1
2240GB
vn12, vn13, vn15, vn50, vn53
rt1-sw4
2832GB
vn12, vn16, vn17, vn50, vn53
rt4
4658GB
vn40, vn41, vn42, vn43, vn44, vn45, vn46, vn47
rt3
6496GB
vn30, vn31, vn32, vn33, vn34, vn35, vn36, vn37,vn50, vn51, vn52,
vn53
rt2
2842GB
vn20, vn21, vn22, vn23, vn24, vn25, vn26, vn27
rt1
58102GB
vn10, vn11, vn12, vn13, vn14, vn15, vn16, vn17, vn50, vn51, vn52,
vn53
sw1
70106GB
vn12, vn13, vn15, vn21, vn22, vn23, vn25, vn30, vn35, vn37, vn44,

vn46, vn50, vn53
sw3
70112GB
vn10, vn15, vn17, vn20, vn21, vn22, vn23, vn25, vn27, vn31, vn32,
vn33, vn36, vn42, vn43, vn44
sw2
80122GB
vn11, vn14, vn20, vn22, vn24, vn25, vn27, vn30, vn32, vn35, vn40,
vn41, vn42, vn44, vn45, vn51, vn52
sw4
110136GB
vn12, vn16, vn17, vn20, vn22, vn23, vn24, vn25, vn26, vn30, vn32,
vn33, vn34, vn35, vn42, vn43, vn44, vn45, vn47, vn50, vn53



...