Page Comparison

...

Introduction:

This tool is meant to replace the 'pbs_diag' script which is currently the means to capture data from PBS for diagnostics.

"pbs_snapshot" will be written in Python and will make use of PTL libraries to interact with the PBS system that it is capturing. This will mean that any major changes to PBS will need very minor (if any) refactoring of pbs_snapshot as PTL gets updated in tandem with PBS now, so pbs_snapshot will automatically work with the latest version of PBSPro. 

Also, a new set of utilities (PBSSnapUtils) will be added to PTL itself for this tool, which will be directly available for PTL test writers and developers to write PTL tests/debugging tools which may need the ability to take snapshots of PBS.

The first version of the tool will also come with the ability to anonymize/obfuscate PBS data to enable users with sensitive data to obfuscate and share snapshots for bug reporting and debugging.

Shape and Form of a 'snapshot':

A 'snapshot', which will be the output produced by the pbs_snapshot tool, will be a tarball (.tgz file) containing the following directory structure & files:

...

...

...

...

...

...

Introduction:

This tool is meant to replace the 'pbs_diag' script which is currently the means to capture data from PBS for diagnostics.

"pbs_snapshot" will be written in Python and will make use of PTL libraries to interact with the PBS system that it is capturing. This will mean that any major changes to PBS will need very minor (if any) refactoring of pbs_snapshot as PTL gets updated in tandem with PBS now, so pbs_snapshot will automatically work with the latest version of PBSPro. 

Also, a new set of utilities (PBSSnapUtils) will be added to PTL itself for this tool, which will be directly available for PTL test writers and developers to write PTL tests/debugging tools which may need the ability to take snapshots of PBS.

The first version of the tool will also come with the ability to anonymize/obfuscate PBS data to enable users with sensitive data to obfuscate and share snapshots for bug reporting and debugging.

Shape and Form of a 'snapshot':

A 'snapshot', which will be the output produced by the pbs_snapshot tool, will be a tarball (.tgz file) named as "snapshot_<timestamp>.tgz" containing the following directory structure & files:

• server/
  • qstat_B.out: output of "qstat -B"
  • qstat_Bf.out: output of "qstat -Bf"
  • qmgr_ps.out: output of "qmgr print server"
  • qstat_Q.out: output of "qstat -Q"
  • qstat_Qf.out: output of "qstat -BQf"
  • qstatqmgr_Bfpr.out: output of "qstat -Bfqmgr print resource"
  • qmgr_pspq.out:  output of "qmgr print serverqueue @default"
• server_priv/: a copy of the 'server_priv' directory inside PBS_HOME, may or may not include accounting logs (see the -L option under "Interface Documentation"), core files are not captured core files are captured separately (see core_file_bt/).
  server_logs
  • accounting/: contains
  server
  • accounting logs from
  the
  • PBS_HOME/server_
  logs
  • priv/accounting/ directory for the number of days specified by --accounting-
  L
  • logs option
  job/
• qstat.out: output of "qstat"
qstat_f.out:
• server_logs/ : contains server logs from the PBS_HOME/server_logs directory for the number of days specified by --daemon-logs option
• job/
  • qstat.out: output of "qstat"
  • qstat_f.out: output of "qstat -f"
  • qstat_t.out: output of "qstat -t"
  • qstat_tf.out: output of "qstat -tf"
  • qstat_x.out: output of "qstat -x"
  • qstat_xf.out: output of "qstat -xf"
  • qstat_ns.out: output of "qstat -ns"
  • qstat_fx_F_dsv.out: output of "qstat -fx -F dsv"
  • qstat_f_F_dsv.out: output of "qstat -f -F dsv"
  node/
  
  • pbsnodes_vaqstat_f_F_json.out:  output of "pbsnodes -vaqstat -f -F json"
• node/
  
  • pbsnodes_va.out: output of "pbsnodes -va"
  • pbsnodes_a.out: output of "pbsnodes -a"
  • pbsnodes_avSj.out: output of "pbsnodes -avSj"
  • pbsnodes_aSj.out: output of "pbsnodes -aSj"
  • pbsnodes_avS.out: output of "pbsnodes -avS"
  • pbsnodes_aS.out: output of "pbsnodes -aS"
  • pbsnodes_aFdsv.out: output of "pbsnodes -aFdsv"
  • pbsnodes_avFdsv.out: output of "pbsnodes -avFdsv"
    
  • pbsnodes_avFjson.out: output of "pbsnodes -avFjson"
    

  • qmgr_pn_default.out: output of "qmgr print node @default"

• mom_priv/

  Copies

  : a copy of the

  following files: 'config', 'prologue', 'epilogue', 'mom.lock'config.d/: contains copy of all vnode def files from inside PBS_HOME/mom_priv/config.d/

  'mom_priv' directory inside PBS_HOME, core files are captured separately (see core_file_bt/)

• mom_logs/: contains mom logs from the PBS_HOME/mom_logs directory for the number of days specified by --daemon-L logs option
• comm/comm_logs/: contains comm logs from the PBS_HOME/comm_logs directory for the number of days specified by -L by --daemon-logs option
• queue/
  • qstat_Q.out: output of "qstat -Q"
  • qstat_Qf.out: output of "qstat -Qf"
• hook/
  qmgr_ph_defaultsched_priv/: a copy of the 'sched_priv' directory inside PBS_HOME with all the files, core files are not captured (see core_file_bt/).
• sched_logs/: contains scheduler logs from the PBS_HOME/sched_logs directory for the number of days specified by --daemon-logs option
• reservation/
  
  • pbs_rstat_f.out: output of "qmgr print hook @defaultpbs_rstat -f"
  • qmgrpbs_lpbshookrstat.out: output of "qmgr list pbshookpbs_rstat"
• scheduler/
  
  • qmgr_lsched.out: output of "qmgr list sched"
    sched_priv/: 
  • qmgr_psched.out: output of "qmgr print sched"

• hook/

  • qmgr_ph_default.out: output of "qmgr print hook @default"

  • qmgr_lpbshook.out: output of "qmgr list pbshook"

• datastore/

  • pg_log/: a copy of the

    'sched_priv' directory inside PBS_HOME with all the files, core files are not captured (see

    "PBS_HOME/datastore/pg_log" directory for the number of days specified by --daemon-logs option

  
  

• core_file_bt/ (stack backtrace from core files)

  .

• sched_

logs/: contains scheduler logs from the

• priv/: files containing the output of "thread apply all backtrace full" on all core files captured from PBS_HOME/sched_

logs directory for the number of days specified by -L option
• reservation/
  
  • pbs_rstat_f.out: output of "pbs_rstat -f"
  • pbs_rstat.out: output of "pbs_rstat"
resource/
qmgr_pr

• priv

• server_priv/: files containing the output of "thread apply all backtrace full" on all core files captured from PBS_HOME/server_priv
• mom_priv/: files containing the output of "thread apply all backtrace full" on all core files captured from PBS_HOME/mom_priv
• misc/: files containing the output of "thread apply all backtrace full" on any other core files found inside PBS_HOME
• system/
  • pbs_probe_v.out: output of "qmgr print resource"
  • rscs_all (derived from the resourcedef file): Will list out built-in as well as custom resources in the following format:

        Name: <resource name>
             type = <resource type attribute>
             flag = <resource flag attribute>

        Name: <resource name>
             type = <resource type attribute>
             flag = <resource flag attribute>

        ...
        ...

• datastore/

  • pg_log/: a copy of the "PBS_HOME/datastore/pg_log" directory

• pbs/

  • pbs.conf: a copy of the pbs.conf file for the PBS system
    

  • pbs_probe_v.out: output of "pbs_probe -v"

  • pbs_hostn_v.out: output of "pbs_hostn -v $(hostname)"
  • pbs_environment: copy of PBS_HOME/pbs_environment file

• core_file_bt/ (stack backtrace from core files)

• sched_priv/: files containing the output of "thread apply all backtrace full" on all core files captured from PBS_HOME/sched_priv

• server_priv/: files containing the output of "thread apply all backtrace full" on all core files captured from PBS_HOME/server_priv
• mom_priv/: files containing the output of "thread apply all backtrace full" on all core files captured from PBS_HOME/mom_priv
• system/
  • os_info: Information about the OS: version, flavour of linux etc. (output of "uname -a" and "cat /etc/*release*" for linux)
  • process_info: List of processes running on the system when the snapshot was taken (output of "ps -ef | grep pbs | grep -v grep" for linux)
  • lsof_pbs.out: output of "lsof | grep pbs | grep -v grep", only on linux systems
  • ps_aux_pbs.out: output of "ps -aux | grep pbs | grep -v grep", only on linux systems
  • etc_hosts: Copy of "/etc/hosts" file, only on linux systems.
    
  • etc_nsswitch_conf: Copy of "/etc/nsswitch.conf" file, only on linux systems.
  • vmstat.out: Output of the command 'vmstat', only on linux systems.
  • df_h.out: Output of the command 'df -h', only on linux systems.
  • dmesg.out: Output of the 'dmesg' command, only on linux systems.
• ctime: this will log the time (since epoch) when the snapshot was taken.
• pbs_snapshot.log: captures the logs generated by pbs_snapshot if the -l option is provided.

Interface Documentation:

The interface for pbs_snapshot will be as follows:

sudo pbs_snapshot [OPTION]

-d <pbs_diag>: diag directory to use as input

-H <hostname>: hostname to operate on. Defaults to the value of PBS_SERVER

-l <loglevel>: set log level to one of INFO, INFOCLI, INFOCLI2, DEBUG, DEBUG2,

                      WARNING, ERROR, FATAL

-o <dir>: output directory

--service-logs=<num days> number of days of service logs to collect

--accounting-logs=<num days> number of days of accounting logs to collect

--additional_hosts=<hostname>: capture additional logs from the hosts specified

                                                        'hostname' is a comma separated list of hosts to take logs from

--map=<file>: path to filename to store the mapping of obfuscated data

--obfuscate: obfuscates euser, egroup, project, account_name, hostnames,

                     IP Addressses, PBS dataservice username

                     Deletes mail endpoints, owner, managers, operators, variable_list

                     ACLs, group_list, job name, jobdir

--version: print version number and exit

sudo - Currently pbs_snapshot will need to be run as a user with sudo privileges because it needs to access protected PBS information (e.g - information inside the PBS_HOME/ _priv directories)

Interface: Option -d <pbs_diag>

• Synopsis: Option to provide path to a pbs_diag directory to be used to generate the snapshot
• Details:
  • This option is meant to make pbs_snapshot be usable on diags generated by the pbs_diag script.
  • <pbs_diag> should be path to a pbs_diag directory that's generated by unwrapping the tarball that pbs_diag produces.
  • This option will instruct pbs_snapshot to not query a live PBS system and instead use the information captured inside the diag to create the snapshot
  • No sudo privileges are needed when running pbs_snapshot using this optionpbs_probe -v"
  • pbs_hostn_v.out: output of "pbs_hostn -v $(hostname)"
  • pbs_environment: copy of PBS_HOME/pbs_environment file
  • os_info: Information about the OS
  • process_info: List of processes running on the system when the snapshot was taken (output of "ps -aux | grep [p]bs" on linux systems and "tasklist /v" on windows systems)
  • ps_leaf.out: output of ps -leaf, only on linux systems
  • lsof_pbs.out: output of "lsof | grep [p]bs", only on linux systems
  • etc_hosts: Copy of "/etc/hosts" file, only on linux systems.
    
  • etc_nsswitch_conf: Copy of "/etc/nsswitch.conf" file, only on linux systems.
  • vmstat.out: Output of the command 'vmstat', only on linux systems.
  • df_h.out: Output of the command 'df -h', only on linux systems.
  • dmesg.out: Output of the 'dmesg' command, only on linux systems.
• pbs.conf: a copy of the pbs.conf file for the PBS system
• ctime: this will log the time (since epoch) when the snapshot was taken.
• pbs_snapshot.log: captures the logs generated by pbs_snapshot.
  
  

Interface Documentation:

The interface for pbs_snapshot will be as follows (output of pbs_snapshot --help):

Usage: pbs_snapshot -o <path to existing output directory> [OPTION]

    Take snapshot of a PBS system and optionally capture logs for diagnostics

    -H <hostname>                     primary hostname to operate on
                                      Defaults to local host
    -l <loglevel>                     set log level to one of INFO, INFOCLI,
                                      INFOCLI2, DEBUG, DEBUG2, WARNING, ERROR
                                      or FATAL
    -h, --help                        display this usage message
    --daemon-logs=<num days>          number of daemon logs to collect
    --accounting-logs=<num days>      number of accounting logs to collect
    --additional-hosts=<hostname>     collect data from additional hosts
                                      'hostname' is a comma separated list
    --map=<file>                      file to store the map of obfuscated data
    --obfuscate                       obfuscates sensitive data
    --with-sudo                       Uses sudo to capture privileged data
    --version                         print version number and exit

Caveat - Currently pbs_snapshot will need to be run as root because it needs to access protected PBS information (e.g - information inside the PBS_HOME/ _priv directories). So, it could either be run with sudo, or as root user. If it is run with restricted privileges, it won't be able to query all of the data.

Interface: Option -o <path to target directory>

• Synopsis: Path to the directory where the snapshot tarball will be generated
• Details:
  
  • This is a mandatory option to pbs_snapshot.
  • The target directory must exist.
  • As an example, if -o is passed "/temp", then path to the generated snapshot would be "/temp/snapshot_<timestamp>.tgz"

Interface: Option -H <hostname>

...

• Synopsis: Option to set the desired log level for debugging pbs_snapshot
• Details:
  
  • The <loglevel> can be set to INFO, INFOCLI, INFOCLI2, DEBUG, DEBUG2, WARNING, ERROR or FATAL.
  • The logging becomes more comprehensive going from FATAL to INFO.
  • By default, the log level will be set to INFOCLI2.
  • The generated logs will also be written out in the file 'pbs_snapshot.log' inside the snapshot directory.

Interface: Option -o <dir>-daemon-logs=<num days>

• Synopsis: Option to

...

• instruct pbs_snapshot to capture daemon logs for the given number of days going back from the current day
• Details:
  

...

• <timestamp> will be in the format: YYYYMMDD_HH_MM_SS

• • This will capture all the daemons' logs available on the host that's running PBS Server.
  • If this option is not specified, a default of 5 days of logs will be collected.
  • The value of <num days> should be >= 0.
    • If the value is 0, no logs are captured.
    • If the value is 1, only the logs for the current day are captured.

Interface: Option --serviceaccounting-logs=<num days>

• Synopsis: Option to instruct pbs_snapshot to capture service accounting logs for the given number of days going back from the current day
• Details:
  
  • This will capture all the daemons' logs available on the host that's running PBS Server.
  • If this option is not specified, a default of 5 30 days of accounting logs will be collected.
  • The value of <num days> should be >= 0.
    • If the value is 0, only the logs for the current day are no logs are captured.
    • If the value is 1, only the logs for the current day , and the day before will be are captured.

Interface: Option --accounting-logs=<num days>additional_hosts=<hostname>

• Synopsis: Option to instruct pbs_snapshot to capture accounting logs for the given number of days going back from the current daycapture information from hosts other than the one where PBS Server is running
• Details:
  
  • If this option is not specified, a default of 30 days of accounting logs will be collected.
  • The value of <num days> should be >= 0.
    • If the value is 0, only the logs for the current day are captured.
    • If the value is 1, only the logs for the current day, and the day before will be captured.

Interface: Option --additional_hosts=<hostname>

• Synopsis: Option to capture logs from hosts other than the one where PBS Server is running
• Details:
  
  • This option only works with the -L <days> option.
  • This option will cause pbs_snapshot to capture logs of all the daemons running on the hosts specified, for the days specified in the -L optionThis option only works if daemon logs are being captured (i.e:- if --daemon-logs=0, then this option will not cause any effect).
  • This option will cause pbs_snapshot to capture the following information:
    • mom and comm logs from the hosts specified, for the number of days of logs being captured.
    • mom_priv from the hosts specified, if present.
    • system information from the hosts specified.
  • The 'hostname' argument could either be a single hostname value, or a comma separated list of hostnames to capture the logs from, or blank, in which case logs from all the hosts running PBS daemons will be captured.Warning: This .
  • Warning: This can bloat the size of snapshot by a lot and cause pbs_snapshot to take a lot of time copying possibly large amount of data over the network.

Interface: Option --map=<file>

• Synopsis: Option to generate a map file by the name specified for obfuscated data
• Details:
  Synopsis:
  • This option will cause pbs_snapshot to create a map file by the name specified with "key:value" pair mapping of the data that's obfuscated.
  • This option will work only with the --obfuscate option.

Interface: Option --obfuscate

• • If this option is not specified, a file called "obfuscate.map" will be created by default at the location specified by the -o option.

Interface: Option --obfuscate

• Synopsis: Option to obfuscate/anonymize the PBS data captured
• Details:
  
  • This option will instruct pbs_snapshot to obfuscate euser, egroup, project and account_name. If the --map option is provided, it will generate a map file for these attributes.It will also delete mail endpoints, owner, managers, operators, variable_list, ACLs, group_list, job name and jobdirpbs_snapshot to obfuscate euser, egroup, project, Account_Name, operators, managers, group_list, Mail_Users, User_List, server_host, acl_groups, acl_users, acl_resv_groups, acl_resv_users, sched_host, acl_resv_hosts, acl_hosts, Job_Owner, exec_host, Host, Mom, resources_available.host and resources_available.vnode.
  • It will also delete Variable_List, Error_Path, Output_Path, mail_from, Job_Name, jobdir, Submit_arguments,  Shell_Path_List.

Interface: Option --version

• Synopsis: Option to display the version of pbs_snapshot being used
• Details:
  
  • This will just display the version and exit
  • The versions would display the version of PBSPro

Sample Usage:

• sudo pbs_snapshot -o /tmp: Will capture snapshot of the system without any logs, and will store the output a snapshot inside /tmptemp/snapshot_<timestamp>
• sudo pbs_snapshot -L 10 -o mysnapshot: Will capture a snapshot at $PWD/mysnapshot along with 11 days of logs going back from the present day (including the present day's logs)
• sudo pbs_snapshot -L 10 -o mysnapshot .tgz along with 30 days of accounting logs and 5 days of daemon logs from the machine that's running PBS Server
• pbs_snapshot --daemon-logs=1 --accounting-logs=1 -o /tmp --obfuscate --map=mapfile.txt: Will capture a snapshot at $PWD/mysnapshot along with 11 days of inside /temp/snapshot_<timestamp>.tgz along with 1 day of accounting and daemon logs, will obfuscate the data and store data mapping in the map file named 'mapfile.txt'.

...