Page Comparison

Objective

...

• Visibility: Public
• Change Control: Stable
• Value:  'true' or 'false' (default is 'false')
• Python type: bool
• Synopsis:  
  • When set to 'true', any failure of nodes are tolerated throughout the life of the job. That is, the job is allowed to run even after mother superior mom has detected bad nodes.
  • It can be set via qsub, qalter, or in a Python hook say queuejob hook. If set via qalter and the job is already running, it will be consulted the next time the job is rerun.
  • This can also be specified in the server attribute 'default_qsub_arguments' to allow all jobs to be submitted with tolerate_node_failures attribute set.
  • This option is best used when job is assigned extra nodes using pbs.event().job.select.increment_chunks() method (interface 7).
• Privilege: user, admin, or operator can set it
• Examples:
  • Via qsub:

...

                            qalter -W tolerate_node_failures=false <job_script><jobid>

• • Via a hook:

                            # cat qjob.py
                            import pbs
                            e=pbs.event()
                            e.job.tolerate_node_failures = True
                            # qmgr -c "create hook qjob event=queuejob"
                            # qmgr -c "import hook application/x-python default qjob.py"
                            % qsub job.scr
                            23.borg
                            % qstat -f 23
                              ...
                              tolerate_node_failures = True

• Log/Error messages:
  • When a job that has tolerant_node_failures attribute set to 'true', the following mom_logs messages will appear in the following conditions: sister moms that failed to join job due to either communication error or execjob_begin hook rejects, when a sister mom fails to setup a job like cpuset creation failure, when a sister mom rejects an execjob_prologue hook, when the primary mom fails to poll a sister mom for status, or for any communication error to the sister mom:
    • DEBUG level: "ignoring error as job is tolerant of node failures"
    • DEBUG3 level: "ignoring POLL error from failed mom <mom_host> as job is tolerant of node failures"
    • DEBUG3 level: "ignoring lost communication with <mom_host> as job is tolerant of node failures"

Interface 2: New server accounting record: 's' for secondary start record when job's assigned resources get pruned during job startup

...

• Visibility: Public
• Change Control: Stable
• Details:
  This is the number of seconds that the primary mom will wait to receive acknowledgement from all the sister moms for the IM_JOIN_JOB requests sent, if job's tolerate_node_failures attribute is set to 'true'. That is, just before the job officially launches its program (script/executable), the primary pbs_mom will ignore any errors from sister moms including failed IM_JOIN_JOB requests. Once all the IM_JOIN_JOB requests have been acknowledged or when the 'sister_join_job_alarm' value wait time has been exceeded, then  pre-starting the job (calling finish_exec()) continues. 
• Default value: set to to the total amount of 'alarm' values associated with enabled execjob_begin hooks. Example, if there are 2 execjob_begin hooks with first hook having alarm=30 and second hook having alarm=20, then the default value of sister_join_job_alarm will be 50 seconds. If there are no execjob_begin hooks, then this is set to 30 seconds.
      To change value, add the following line in mom's config file:
                          $sister_join_job_alarm <# of seconds>
• Log/Error messages:

1. When the $sister_join_job_alarm value is specified, then there'll be PBSEVENT_SYSTEM level message

...

1. that will be shown when mom starts up or kill -HUPed:                                                                      "

...

1. sister_join_job_alarm;<alarm_value>"
2. When

...

1. not all join job request from sister moms have been acknowledged within the $sister_join_job_alarm time limit, then the following

...

1. mom_logs

...

1. message appears at DEBUG2 level:                         

...

1. "sister_join_job_alarm

...

                                    "sister_join_job_alarm wait time <alarm_value> secs exceeded"

Interface 4: job_launch_delay mom config option

...

1. wait time <alarm_value> secs exceeded"

Interface 4: job_launch_delay mom config option

• Visibility: Public
• Change Control: Stable
• Details:
  This is the number of seconds that the primary mom will wait before launching (executing the job script or executable), if the job that has tolerate_node_failures set to "true". This wait time can be used to let execjob_prologue hooks finish execution  to capture or report any node failures, or for mother superior to notice of any communication problems with other nodes. pbs_mom will not necessarily wait fot the entire time but proceed to execute execjob_launch hook (when specified) once all prologue hook acknowledgements have been received from sister moms.
• Default value: set to to the total amount of 'alarm' values associated with enabled execjob_prologue hooks. For example, if there are 2 execjob_prologue hooks, where first hook has alarm=30 and second hook has alarm=60, then the default job_launch_delay value will be 90 seconds. If there are no execjob_prologue hooks, then this is set to 30 seconds.
  To change value, add the following line in mom's config file:
                 $job_launch_delay <number of seconds>
• Log/Error messages:
  

1. When $job_launch_delay value is set, there'll be PBSEVENT_SYSTEM level message

...

1. upon mom startup or when it is kill -HUPed:                                               

...

• 1. Before oficially launching a tolerant job, it would wait up to 'job_launch_delay' time  for any report on failed sister moms, which will later used for determining the entries of vnodelist_fail parameter (see interface 5)  in execjob_launch hook (if any). The following DEBUG2 level log message will be shown:

                 "Job;<job-id>;waiting up to <job_launch_delay_value> secs ($job_launch_delay) for mom hosts status and prologue hooks ack"

  2. When primary mom notices that not all acks were received from the sister moms in regards to execjob_prologue hook execution, then mom_logs would show the DEBUG2 level message:

1.                                                       "job_launch_delay;<delay_value>"
2. When primary mom notices that not all acks were received from the sister moms in regards to execjob_prologue hook execution, then mom_logs would show the DEBUG2 level message:                                                                                           

...

Interface 5: pbs.event().vnode_list_fail[] hook parameter

...

1.                                                                                                                                                                                                              "not all prologue hooks to sister moms completed, but job will proceed to execute"

Interface 5: pbs.event().vnode_list_fail[] hook parameter

• Visibility: Public
• Change Control: Stable
• Python Type: dict (dictionary of pbs.vnode objects keyed by vnode name)
• Details:
  This is a new event parameter for the execjob_prologue and execjob_launch hook. It will contain the list of vnodes and their assigned resources that are managed by unhealthy moms. This can include those vnodes from sister moms that failed to join the job, that rejected an execjob_begin hook or execjob_prologue hook request, encountered communication error while primary mom is polling the sister mom host. This dictionary object is keyed by vnode name. And one can walk through this list and start offlining the vnodes, for example:

  for vn in e.vnode_list_fail:
      v = e.vnode_list_fail[vn]
      pbs.logmsg(pbs.LOG_DEBUG, "offlining %s" % (vn,))
      v.state = pbs.ND_OFFLINE

Interface 6: Allow execjob_launch hooks to modify job and vnode attributes

• Visibility: Public
• Change Control: Stable
• Detail: With this feature, execjob_launch hooks are now allowed to modify job and vnode attributes, in particular, job's Execution_Time, Hold_Types, resources_used, and run_count values. This is the same with vnode object attributes like state and resources_available.
• Examples:

                           Set a job's Hold_Types in case the hook script rejects the execjob_launch event:

                                pbs.event().job.Hold_Types = pbs.hold_types('s')

                           Set a vnode's state to offline:

                               pbs.event().vnode_list[<node_name>].state = pbs.ND_OFFLINE

• Log/Error messages:
1. If an execjob_prologue hook or an execjob_launch hook requested to offline a vnode, server_logs would show the message under PBSEVENT_DEBUG2 level:

   ";Server@borg;Node;<node name>;Updated vnode <node_name>'s attribute state=offline per mom hook request" 

Interface 6: Allow execjob_launch hooks to modify job and vnode attributes

• Visibility: Public
• Change Control: Stable
• Detail: With this feature, execjob_launch hooks are now allowed to modify job and vnode attributes, in particular, job's Execution_Time, Hold_Types, resources_used, and run_count values. This is the same with vnode object attributes like state and resources_available.
• Examples:

                           In previous version of PBS, when a job or vnode attribute/resource is set in execjob_launch, the hook rejects the request and returns the following message:

                                     "Can only set progname, argv, env event parameters under execjob_launch hook"

                           Now, setting vnode and job attributes are allowed and would no longer give the above message. If something else get set in the hook, like a server attribute, then

                           Set a job's Hold_Types in case the hook script rejects the execjob_launch eventthis will now be the new DEBUG2 level mom_logs message:

                                pbs.event().job.Hold_Types = pbs.hold_types('s')                            Set a vnode's state to offline:

                               pbs.event().vnode_list[<node_name>].state = pbs.ND_OFFLINE

• Log/Error messages:

                           In previous version of PBS, when a job or vnode attribute/resource is set in execjob_launch, the hook rejects the request and returns the following message:

                                     "Can only set progname, argv, env event parameters under execjob_launch hook"

                           Now, setting vnode and job attributes are allowed and would no longer give the above message. If something else get set in the hook, like a server attribute, then"Can only set progname, argv, env event parameters as well as job, resource, vnode under execjob_launch hook."

Interface 7: pbs.select.increment_chunks(increment_spec)

• Visibility: Public
• Change Control: Stable
• Return Python Type: pbs.select
• Details:
  This is a new method in the pbs.select type where 'increment_spec' number of chunks are added to each chunk in the chunk specification. So given a select spec of "[N:][chunk specification][+[N:]chunk specification]", this function will return [N+increment:][chunk specification][+[N+increment:]chunk specification]". A missing 'N' values means 1.
• Input:
  • if 'increment_spec' is a number (int or long), then it will be the amount to add to the number of chunks spcified for each chunk in the pbs.select spec.
  • if 'increment_spec' is a numeric string (int or long), then it will also be the amount to add to the number of chunks spcified for each chunk in the pbs.select spec.
  • if 'increment_spec' is a numeric string that ends with a percent sign (%), then this will be the percent amount of chunks to increase each chunk in the pbs.select spec. The resultingamount is rounded up (i.e. ceiling) (e.g. 1.23 rounds up to 2).
  • Finally, if 'increment_spec' is a dictionary with elements of the form:
                     

...

                                     "Can only set progname, argv, env event parameters as well as job, resource, vnode under execjob_launch hook."

Interface 7: pbs.select.increment_chunks(increment, first_chunk=False)

• Visibility: Public
• Change Control: Stable
• Return Python Type: pbs.select
• Details:
  This is a new method in the pbs.select type where 'increment' number of chunks are added to each chunk in the chunk specification. So given a select spec of "[N:][chunk specification][+[N:]chunk specification]", this function will return [N+increment:][chunk specification][+[N+increment:]chunk specification]". A missing 'N' values means 1. By default, first_chunk=False means no increment is added to the first chunk in the specs. Example:

...

• • {<chunk_index_to_select_spec> : <increment>, ...}
    where <chunk_index_to_select_spec> starts at 0 for the first chunk, and <increment> can be numeric, numeric string or a percent increase value. This allows for individually specifying the number of chunks to increase original value.
• Example:

Given:
      sel=pbs.select("ncpus=3:mem=1gb+1:ncpus=2:mem=2gb+2:ncpus=1:mem=3gb")

Calling sel.increment_chunks(2) would return a string:
     "3:ncpus=3:mem=1gb+3:ncpus=2:mem=2gb+4:ncpus=1:mem=3gb"

Calling sel.increment_chunks("3") would return a string:
     "4:ncpus=3:mem=1gb+4:ncpus=2:mem=2gb+5:ncpus=1:mem=3gb"

Calling sel.increment_chunks("23.5%"), would return a pbs.select value mapping to:
      "2:ncpus=3:mem=1gb+2:ncpus=2:mem=2gb+

...

3:ncpus=1:mem=

...

3gb"

for the first and second chunk, 23.5 % increase to the  number of chunks (1) would round up to 2, while for the
third chunk, a 23.5 % increase to the number of chunks (2) would round up to 3.

Calling sel.increment_chunks({0: 0, 1: 4, 2: "50%"}), would return a pbs.select value mapping to:
     "1:ncpus=3:mem=1gb+5:ncpus=2:mem=2gb+

...

3:ncpus=1:mem=

...

3gb"

where no increase (0) for chunk 1, additional 4 chunks for chunk 2, 50% increase for chunk 3 resulting in
3.

Interface 8: pbs.event().job.release_nodes(keep_select) method

...

• Examples:

         Given an execjob_prologue hook, a hook writer can release a set of nodes from a job by doing:

              pj = e.job.release_nodes(keep_select="ncpus=2:mem=2gb+ncpus=2:mem=2gb+ncpus=1:mem=1gb")
              if pj != None:
                  pbs.logmsg(pbs.LOG_DEBUG, "pj.exec_vnode=%s" % (pj.exec_vnode,))
              else:               # returned None job object, so we can put a hold on the job and requeue it, rejecting the hook event
                  e.job.Hold_Types = pbs.hold_types("s")
                  e.job.rerun()
                  e.reject("unsuccessful at LAUNCH")

  
  

• Log/Error messages:
  • When job's assigned nodes get pruned (nodes released to satisfy 'keep_select') , mom_logs will show the following info under PBSEVENT_JOB log level:
    

    ";Job;<jobid>;pruned from exec_vnode=<original value>"
    ";Job;<jobid>;pruned to exec_node=<new value>"
    
    

  • When a multinode job's assigned resources have been modified, primary mom will do a quick 5 seconds wait  for an acknowledgement from the sister moms that they have updated their nodes table.  There's be this DEBUG2 level mom_logs message:

                       "waiting up to 5 secs for job update acks from sister moms"

...

• • from the sister moms that they have updated their nodes table. When not all acknowledgements were received by primary mom during that 5 seconds wait, then there'll be this

...

• • DEBUG2 level mom_logs message:

                       "not all job updates to sister moms completed"

                   Eventually, all node updates will complete in the background.  Seeing this log message means that a job can momentarily receive an error when doing tm_spawn or pbsdsh to a node that did not complete the nodes table update yet.

• • When mother superior fails  to prune currently assigned chunk resource, then the following detailed mom_logs message are shown in DEBUG2 level:
    

    • "could not satisfy select chunk (<resc1>=<val1> <resc2>=<val2> ...<rescN>=valN) 

    • "NEED chunks for keep_select (<resc1>=<val1> <resc2>=<val2> ...<rescN>=valN)
    • "HAVE chunks from job's exec_vnode (<exec_vnode value>

• • When a sister mom updated its internal nodes tableupdated its internal nodes table due to some nodes getting released as  a result of the release_nodes() call, then mom_logs on the sister host would show the message in PBSEVENT_JOB level:                                                                                                                                                                                     ";<jobid>;updated nodes info"
  • Calling release_nodes() from a hook that is not execjob_prologue or execjob_launch hook would return None as this is currently not supported.
  • Upon successful execution of release_nodes() call, it is normal to receive messages in the mom_logs of the form:

...

e.job.Resource_List["site"] = str(e.job.Resource_List["siteselect"])

Next, add extra chunks to current select:

...