Overview
PTL performance tests run large number of jobs. The tearDown() of these tests or setUp() of subsequent tests clean up by deleting these jobs using qdel. This qdel operation takes very long. Consequently these tests timeout. This is because ,the scheduler is constantly cycling and it may tell the server to start the jobs we are trying to delete. This makes server busy and the delete job requests have to wait to get processed until the server is done talking to the scheduler at the start of the sched cycle. Also, a lot of MoM and server<->MoM activity happens when terminating a running job. This adds delay.
...
In fw/ptl/utils/pbs_testlib.py,
- Interface: _cleanup_large_num_jobs(job_ids=None, runas=None)
Visibility: Private
Synopsis: Helper function to delete large number of jobs. Will be called from cleanup_jobs if number of jobs for deletion is greater than 100.
Details: This function will get the process ids of the running jobs and kill them manually. It would then delete all jobs from server using ‘qdel -Wforce’.
Interface updates
Following are the updates (highlighted in green) that will be done to existing interfaces
fw/ptl/utils/pbs_testlib.py,
- Interface: cleanup_jobs(extend=None, runas=None)
Visibility: Public
Synopsis: Helper function to delete all jobs. By default this method will determine whether job_history_enable is on and will cleanup all history jobs. Specifying an extend parameter could override this behavior. Handles deletion of large number of jobs differently.
Details: Deletes jobs by calling low-level function deljob() if number of jobs for deletion is less than 100. Calls internal function _cleanup_large_num_jobs when number of jobs to delete is greater than 100. - Interface: tearDown()
Visibility: Public
Synopsis: Cleans up jobs and reservations at the end of test execution
Details: tearDown() will call cleanup_jobs() to make sure jobs are deleted at the end of every test
...
Project Documentation Main Page
...