PP-302: Implement save of PBS data for post-run analysis

Interface: New option (--post-analysis-data=<dir>) in pbs_benchpress command

• Visibility: Public
• Change Control: Stable
• Synopsis: A new option added to pbs_benchpress command to enable saving post analysis data on test case failure in PTL
  
• Details: To enable saving post analysis data on test case failure in PTL, new command line option is introduced called '--post-analysis-data=<dir>' in pbs_benchpress command. By default PTL will not save any post analysis data in failure case. When enabled, PTL will store post analysis data on failure in given <dir> in below format

  <dir>/<test suite 1 name>/

                                        <test case 1 name>/

                                                                       logfile_<status>

                                                                       PBS_<hostname>.tar.gz

                                        ...

                                        <test case n name>/

                                                                       logfile_<status>

                                                                       PBS_<hostname>.tar.gz

         ...

         <test suite n name>/

                                        <test case 1 name>/

                                                                       logfile_<status>

                                                                       PBS_<hostname>.tar.gz

                                        <test case 2 name>/

                                                                       logfile_<status>

                                                                       PBS_<hostname>.tar.gz

                                        ...

                                        <test case n name>/

                                                                       logfile_<status>

                                                                       PBS_<hostname>.tar.gz

  
  This post analysis data includes following data:
  • logfile_<status of test case>
    • This file contains PTL logs of current test case for which post analysis data is being stored
    • <status of test case> can be one of following:
      • PASS
      • SKIP
      • FAIL
      • ERROR
      • TIMEDOUT
    • Also in last line of this file will have time taken by current test case
  • PBS_<hostname>.tar.gz
    • This file is nothing but PBS diag saved by pbs_diag command
      • PTL will run pbs_diag with following options
        • -f: for running pbs_diag in non-interactive mode
        • -j <jobid1>,<jobid2>,...,<jobidn>: comma separated list of job ids (if any)
        • -d 2: To save PBS logs for last two days
          • 2 days because there might be chances that date can switch (like midnight) while PTL is running test
      • Also PTL will run pbs_diag command with -c <path to core file> if all core files found in PBS_HOME directory
        
        • The output of above command will be store in <core file name>.out
        • After taking pbs_diag output for core file, core file will be deleted from PBS_HOME directory

Interface: New option (--max-post-analysis-data=<count>) in pbs_benchpress command

• Visibility: Public
• Change Control: Stable
• Synopsis: A new option added to pbs_benchpress command to override default value of max count for saving post analysis data per testsuite.
  
• Details: When saving post analysis data enabled, by default PTL will save post data for first 3 failure per testsuite. After saving post data for 3rd failure in testsuite PTL will disable saving post data for that testsuite. But this option provides user to override that default value of max count for saving post data per testsuite. If this option is given then PTL will save post data for first <count> failed cases before it disables saving post data for that testsuite.