Jira Legacy server System JIRA serverId 32008a99-7831-3ff8-9638-3db0cd01164d key PP-479 - Discussion forum: http://community.pbspro.org/t/pp-479-running-subjobs-to-be-able-to-survive-a-pbs-server-restart/835
Overview:
Currently on server restart, job arrays that have running sub jobs are terminated due to them only being stored in memory. With this RFE the behavior is changed so that running subjobs continue to run after a server restart . It also enables storing the information that is unique to each subjob such as run_count, resources_used, comments, and hence qstat of the sub jobs does not return just the parent information once the job is finished.
Interface Design:
- Interface 1:
- Change control: Stable
- Synopsis: Status of running Subjobs of Array Job persistent across pbs_server restarts
- Details:
- Currently when a server restarts (peacefully or abruptly) any running subjobs of Array Job are killed and re-queued and start from beginning as the whole parent Array job is re-queued.
- With this RFE we make subjob and array job status persistent across server restarts. So any running subjobs of Array Job continue to run when server is restarted
- This is achieved by making a running subjob is now made on par with a single job and storing each subjob's job object and its attributes into the pbs database's job table and job_attr table which gets recovered during subsequent server start (pbsd_init())
- "pbs.subjob_track" db Table is removed from pbs db schema
- Change control: Stable
- Synopsis: Non-Rerunnable Array jobs
- Details:
- Array Jobs can now be submitted with "-r n" command line argument to qsub.
- i.e Array Jobs can now be Non-Rerunnable
- i.e Restriction of Array Jobs to be Rerunnable always is removed.
- More accurate job comment field is updated for subjob as listed below
- When Job History is enabled
- For running subjob: "Job run at <date> at <time> on <exec_vnode>"
- For finished subjob: "Job run at <date> at <time> on <exec_vnode> and <finished | failed | terminated>"
- For MoM rejected subjob: "Not Running: PBS Error: Execution server rejected request"
- When Job History is disabled
- For running subjob: "Job run at <date> at <time> on <exec_vnode>"
- When Job History is enabled
- Interface 2: < deleted >
- Interface 3:
- Change control: Stable
- Synopsis: Impact on running sub jobs when terminating the pbs server using "qterm -t quick"
- Details:
- When the PBS server is terminated with "qterm -t quick" (i.e type of shutdown requested is "quick"), any running subjob is not requeued and will be left running after the server shutdown.
...
- Interface 4:
- Change control: Stable
- Synopsis: content of "qstat -xtf" with respect to subjob
- Details:
- Now the contents displayed under command "qstat -xft" for each of the attributes / resources of finished subjobs are obtained from the corresponding subjob job object instead of copying from the parent array job obj
- Hence it displays the information that is unique to each subjob such as run_count, resources_used, comments, and hence qstat of the sub jobs does not return just the parent information once the job is finished.
- Interface 5:Change control < deleted >
- Interface 6: Stable
- Synopsis: Display of state of subjob in a finished array job under "qstat -xt"
- Details:
- The status of a subjob in the a finished array job is shown as "X" instead of "F" (i.e the status of subjob remain as "expired" instead of changing to "finished")
- This change is a restriction discovered with experimental internal change
Interface 6: - Change control: Stable
- Synopsis: "run_count" and "run_version" attributes for subjob is made valid Details:
- Currently "run_count" and "run_version" attributes for subjob was never incremented and remained as "1" even after a rerun. Now with this RFE "run_count" and "run_version" attributes for subjob will be incremented for each rerun of the subjob < deleted >
P.S. : Admin should take note of an inconsistency caused due to a limitation originating from interface 1
As stated in the interface 1, we are progressing towards making subjobs equivalent to regular jobs.
Specifically, this means that like a regular job, when a subjob finishes, the information about the subjob (exit status etc) can be seen only as long as the subjob is in job history.
If a subjob finishes and is no longer in job history (or history is disabled), then information specific to that subjob is no longer available.
In that case, a stat on such subjobs (no longer in history) will wrongly show default values about the subjob (like state = finished, exit_status = 0 etc.)
This leads to the below inconsistencies with respect to running Array job when the server is restarted
- A failed subjob will be shown as finished with Exit_status = 0, and Job comment for that subjob will become "Subjob finished" from "Subjob failed"
- A terminated (deleted using qdel) subjob will be shown as finished with Exit_status = 0, and Job comment for that subjob will become "Subjob finished" from "Subjob terminated"
- For both the above situations the Exit status and the Job comment of the finished Array job can be wrong.
...
...