Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Follows the PBS Pro Design Document Guidelines.

[Status: DRAFT WIP ]

Overview

This is to enhance the "node ramp down" feature, by introducing a new option "-k <select>" ("k" for "keep") to the pbs command "pbs_release_nodes". This will allow a users or admins to retain some of the sister nodes/vnodes which satisfy the "select" argument, while performing node ramp down operation.

Technical Details

Interface 1:  -k <select statement>

  • Change Control: Stable
  • Synopsis: This new option to "pbs_release_nodes" specifies a select statement that is a subset of the job submission select statement which describes the the nodes/vnodes which are to be kept assigned with the job, while releasing the remaining sister nodes/vnodes. The nodes/vnodes released will then be made available for scheduling other jobs. 
  • Permission: as described in the Ref 1. above

Details:

  • New Syntax :

pbs_release_nodes [-j <job ID>] <vnode> [<vnode> [<vnode>] ...]
pbs_release_nodes [-j <job ID>] -a
pbs_release_nodes [-j <job ID>]  -k  <select statement>
pbs_release_nodes --version 
 

  • Example of usage :
    Lets submit a job with a select string

$ qsub -l select=3:model=abc:ncpus=5+3:model=abc:bigmem=true:ncpus=1+3:model=def:ncpus=32  job.scr
120.pbssrv

Now grepping for assigned vnodes we may see :

$ qstat -f 120| egrep exec_vnode
exec_vnode = (nd_abc_1:ncpus=5)+(nd_abc_2:ncpus=5)+(nd_abc_3[0]:ncpus=5)+(nd_abc_3[1]:ncpus=1)+(nd_abc_4_bm:ncpus=1)+(nd_abc_5_bm:ncpus=1)+(nd_abc_6_bm:ncpus=1)+(nd_def_1:ncpus=32)+(nd_def_2:ncpus=32)

Here the first chunk "(nd_abc_1:ncpus=5)" represents the mother superior node while each of the remaining others represent a sister node.

and node statuses as :

$ pbsnodes -av
nd_abc_1
    Mom = nd_abc_1.pbspro.com
    state = job-busy
    jobs = 120.pbssrv/0
    resources_available.model = abc
    resources_available.ncpus = 10
    resources_assigned.ncpus = 5

nd_abc_2
    Mom = nd_abc_2.pbspro.com
    state = job-busy
    jobs = 120.pbssrv/0
    resources_available.model = abc
    resources_available.ncpus = 10
    resources_assigned.ncpus = 5

nd_abc_3[0]
    Mom = nd_abc_3.pbspro.com
    state = job-busy
    jobs = 120.pbssrv/0
    resources_available.model = abc
    resources_available.ncpus = 5
    resources_assigned.ncpus = 5

nd_abc_3[1]
    Mom = nd_abc_3.pbspro.com
    state = job-busy
    jobs = 120.pbssrv/0
    resources_available.model = abc
    resources_available.ncpus = 5
    resources_assigned.ncpus = 1

nd_abc_4_bm
    Mom = nd_abc_4_bm.pbspro.com
    state = job-busy
    jobs = 120.pbssrv/0
    resources_available.bigmem = True
    resources_available.model = abc
    resources_available.ncpus = 1
    resources_assigned.ncpus = 1

nd_abc_5_bm
    Mom = nd_abc_5_bm.pbspro.com
    state = job-busy
    jobs = 120.pbssrv/0
    resources_available.bigmem = True
    resources_available.model = abc
    resources_available.ncpus = 1
    resources_assigned.ncpus = 1

nd_abc_6_bm
    Mom = nd_abc_6_bm.pbspro.com
    state = job-busy
    jobs = 120.pbssrv/0
    resources_available.bigmem = True
    resources_available.model = abc
    resources_available.ncpus = 1
    resources_assigned.ncpus = 1

nd_def_1
    Mom = nd_def_1.pbspro.com
    state = job-busy
    jobs = 120.pbssrv/0
    resources_available.model = def
    resources_available.ncpus = 32
    resources_assigned.ncpus = 32

nd_def_2
    Mom = nd_def_2.pbspro.com
    state = job-busy
    jobs = 120.pbssrv/0
    resources_available.model = def
    resources_available.ncpus = 32
    resources_assigned.ncpus = 32

Now if we do a pbs_release_nodes with the new "-k" option having a select argument which is a sub statement of select string used in qsub -l :

$ pbs_release_nodes -j 120 -k select=model=abc:ncpus=5+2:model=abc:bigmem=true:ncpus=1

will release the nodes (nd_abc_3[0]:ncpus=5)+(nd_abc_3[1]:ncpus=1)+(nd_abc_6_bm:ncpus=1)+(nd_def_1:ncpus=32)+(nd_def_2:ncpus=32) from the job while retaining the nodes (nd_abc_1:ncpus=5)+(nd_abc_2:ncpus=5)+(nd_abc_4_bm:ncpus=1)+(nd_abc_5_bm:ncpus=1).

The new phase of the job will have below vnodes associated with it 

$ qstat -f 120| egrep exec_vnode
exec_vnode = (nd_abc_1:ncpus=5)+(nd_abc_2:ncpus=5)+(nd_abc_4_bm:ncpus=1)+(nd_abc_5_bm:ncpus=1)

  • Errors and Return codes :
    • Cannot be used in conjunction "-a" option. If used so, pbs_release_nodes will print below error along with usage strings.

      pbs_release_nodes: -a and -k options cannot be used together

    • Cannot be used in conjunction with supplying host/vnode list arguments (<vnode> [<vnode> [<vnode>] ...]). If used so, pbs_release_nodes will print below error along with usage strings.

      pbs_release_nodes: cannot supply node list with -k option

    • <TBD> When the select statement supplied is not a sub statement of the job's qsub select statement
    • <TBD> List all other Failure scenarios, their return code and error messages
  • Accounting Logs :
    • No new accounting logs introduced. See Ref 2. above.
  • Caveats :
    • The order of selection of nodes/vnodes to be released or kept by the "-k <select>" option is "Undefined". Hence user/admin or his/her scripts/tools should not depend/predict on the order of release/keep operation on the nodes/vnodes.
    • If one or more nodes/vnodes targeted for release have one or more job chunks/processes still running in them, then the release operation will result in their abrupt termination.
    • Clubbing the previous two caveats: user/admin should be aware that by using this new option, the running job may lose some of its running job chunks.
    • <TBD>


API level details:

  • The "select" string parameter will be passed to "pbs_relnodesjob()" using its "extend" argument which is of type "char * "




OSS Site Map

Project Documentation Main Page

Developer Guide Pages


  • No labels