- PP-479 - Getting issue details... STATUS
- Discussion forum: http://community.pbspro.org/t/pp-479-running-subjobs-to-be-able-to-survive-a-pbs-server-restart/835
Overview:
Currently on server restart, job arrays that have running sub jobs are terminated due to them only being stored in memory. With this RFE the behavior is changed so that running subjobs continue to run after a server restart . It also enables storing the information that is unique to each subjob such as run_count, resources_used, comments, and hence qstat of the sub jobs does not return just the parent information once the job is finished.
Interface Design:
- Interface 1:
- Change control: Stable
- Synopsis: Status of running Subjobs of Array Job persistent across pbs_server restarts
- Details:
- Currently when a server restarts (peacefully or abruptly) any running subjobs of Array Job are killed and re-queued and start from beginning as the whole parent Array job is re-queued.
- With this RFE we make subjob and array job status persistent across server restarts. So any running subjobs of Array Job continue to run when server is restarted
- This is achieved by making a running subjob on par with a single job and storing each subjob's job object and its attributes into the pbs database's job table and job_attr table which gets recovered during subsequent server start (pbsd_init())
- "pbs.subjob_track" db Table is removed from pbs db schema
- The PBS_EVENTCLASS_JOB debug log message msg_nosave_histarrayjob = "History of individual array subjobs not saved" has been removed as it is no more valid.
- More accurate job comment field is updated for subjob as listed below
- When Job History is enabled
- For running subjob: "Job run at <date> at <time> on <exec_vnode>"
- For finished subjob: "Job run at <date> at <time> on <exec_vnode> and <finished | failed | terminated>"
- For MoM rejected subjob: "Not Running: PBS Error: Execution server rejected request"
- When Job History is disabled
- For running subjob: "Job run at <date> at <time> on <exec_vnode>"
- When Job History is enabled
- Interface 2: < deleted >
- Interface 3:
- Change control: Stable
- Synopsis: Impact on running sub jobs when terminating the pbs server using "qterm -t quick"
- Details:
- When the PBS server is terminated with "qterm -t quick" (i.e type of shutdown requested is "quick"), any running subjob is not requeued and will be left running after the server shutdown.
- Interface 4:
- Change control: Stable
- Synopsis: content of "qstat -xtf" with respect to subjob
- Details:
- Now the contents displayed under command "qstat -xft" for each of the attributes / resources of finished subjobs are obtained from the corresponding subjob job object instead of copying from the parent array job obj
- Hence it displays the information that is unique to each subjob such as run_count, resources_used, comments, and hence qstat of the sub jobs does not return just the parent information once the job is finished.
- Interface 5: < deleted >
- Interface 6: < deleted >