Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Page Properties


Target release17.1.1
Epic

Jira Legacy
serverJIRA (pbspro.atlassian.net)
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId32008a99-7831-3ff8-9638-3db0cd01164d
keyPP-758

Document status
Status
titleDRAFT
Forum Discussion/Reviewhttp://community.pbspro.org/t/pp-758-add-pbs-snapshot-tool-to-capture-state-logs-from-pbs/520/22
Document owner
Designer
Developers
QA


...

  • server/
    • qstat_B.out: output of "qstat -B"
    • qstat_Bf.out: output of "qstat -Bf"
    • qmgr_ps.out: output of "qmgr print server"
    • server_priv/: a copy of the 'server_priv' directory inside PBS_HOME, core files are captured separately (see core_file_bt/)
      • accounting/: contains accounting logs from PBS_HOME/server_priv/accounting/ directory for the number of days specified by --accounting-logs option
    • server_logs/ : contains server logs from the PBS_HOME/server_logs directory for the number of days specified by --service-logs option
  • job/
    • qstat.out: output of "qstat"
    • qstat_f.out: output of "qstat -f"
    • qstat_t.out: output of "qstat -t"
    • qstat_tf.out: output of "qstat -tf"
    • qstat_x.out: output of "qstat -x"
    • qstat_xf.out: output of "qstat -xf"
    • qstat_ns.out: output of "qstat -ns"
    • qstat_fx_F_dsv.out: output of "qstat -fx -F dsv"
    • qstat_f_F_dsv.out: output of "qstat -f -F dsv"
  • node/
    • pbsnodes_va.out: output of "pbsnodes -va"
    • pbsnodes_a.out: output of "pbsnodes -a"
    • pbsnodes_avSj.out: output of "pbsnodes -avSj"
    • pbsnodes_aSj.out: output of "pbsnodes -aSj"
    • pbsnodes_avS.out: output of "pbsnodes -avS"
    • pbsnodes_aS.out: output of "pbsnodes -aS"
    • pbsnodes_aFdsv.out: output of "pbsnodes -aFdsv"
    • pbsnodes_avFdsv.out: output of "pbsnodes -avFdsv"
    • qmgr_pn_default.out: output of "qmgr print node @default"
    • mom_priv/

      • Copies of the following files: 'config', 'prologue', 'epilogue', 'mom.lock'

      • config.d/: contains copy of all vnode def files from inside PBS_HOME/mom_priv/config.d/

    • mom_logs/: contains mom logs from the PBS_HOME/mom_logs directory for the number of days specified by --service-logs option
  • comm/
    • comm_logs/: contains comm logs from the PBS_HOME/comm_logs directory for the number of days specified by --service-logs option
  • queue/
    • qstat_Q.out: output of "qstat -Q"
    • qstat_Qf.out: output of "qstat -Qf"
  • hook/
    • qmgr_ph_default.out: output of "qmgr print hook @default"
    • qmgr_lpbshook.out: output of "qmgr list pbshook"
  • scheduler/
    • qmgr_lsched.out: output of "qmgr list sched"
    • sched_priv/: a copy of the 'sched_priv' directory inside PBS_HOME with all the files, core files are not captured (see core_file_bt/).
    • sched_logs/: contains scheduler logs from the PBS_HOME/sched_logs directory for the number of days specified by --service-logs option
  • reservation/
    • pbs_rstat_f.out: output of "pbs_rstat -f"
    • pbs_rstat.out: output of "pbs_rstat"
  • resource/
    • qmgr_pr.out: output of "qmgr print resource"
    • rscs_all (derived from the resourcedef file): Will list out built-in as well as custom resources in the following format:

          Name: <resource name>
               type = <resource type attribute>
               flag = <resource flag attribute>

          Name: <resource name>
               type = <resource type attribute>
               flag = <resource flag attribute>

          ...
          ...

  • datastore/

    • pg_log/: a copy of the "PBS_HOME/datastore/pg_log" directory

  • pbs/

    • pbs.conf: a copy of the pbs.conf file for the PBS system

    • pbs_probe_v.out: output of "pbs_probe -v"

    • pbs_hostn_v.out: output of "pbs_hostn -v $(hostname)"
    • pbs_environment: copy of PBS_HOME/pbs_environment file
  • core_file_bt/ (stack backtrace from core files)

    • sched_priv/: files containing the output of "thread apply all backtrace full" on all core files captured from PBS_HOME/sched_priv

    • server_priv/: files containing the output of "thread apply all backtrace full" on all core files captured from PBS_HOME/server_priv
    • mom_priv/: files containing the output of "thread apply all backtrace full" on all core files captured from PBS_HOME/mom_priv
    • misc/: files containing the output of "thread apply all backtrace full" on any other core files found inside PBS_HOME
  • system/
    • os_info: Information about the OS: version, flavour of linux etc. (output of "uname -a" and "cat /etc/*release*" for linux)
    • process_info: List of processes running on the system when the snapshot was taken (output of "ps -ef | grep pbs | grep -v grep" for linux)
    • lsof_pbs.out: output of "lsof | grep pbs | grep -v grep", only on linux systems
    • ps_aux_pbs.out: output of "ps -aux | grep pbs | grep -v grep", only on linux systems
    • etc_hosts: Copy of "/etc/hosts" file, only on linux systems.
    • etc_nsswitch_conf: Copy of "/etc/nsswitch.conf" file, only on linux systems.
    • vmstat.out: Output of the command 'vmstat', only on linux systems.
    • df_h.out: Output of the command 'df -h', only on linux systems.
    • dmesg.out: Output of the 'dmesg' command, only on linux systems.
  • ctime: this will log the time (since epoch) when the snapshot was taken.
  • pbs_snapshot.log: captures the logs generated by pbs_snapshot.

...

The interface for pbs_snapshot will be as follows:

sudo pbs_snapshot -o <path to output file> [OPTION]

...

--version: print version number and exit


sudo Caveat - Currently pbs_snapshot will need to be run as a user with sudo privileges root because it needs to access protected PBS information (e.g - information inside the PBS_HOME/ _priv directories). So, it could either be run with sudo, or as root user. If it is run with restricted privileges, it won't be able to query all of the data.


Interface: Option -o <path to output file>

...

  • Synopsis: Option to display the version of pbs_snapshot being used
  • Details:
    • This will just display the version and exit
    • The versions would display the version of PBSPro

Sample Usage:

  • sudo pbspbs_snapshot -o mysnapshot: Will capture a snapshot inside $PWD/mysnapshot.tgz along with 30 days of accounting logs and 5 days of service logs from the machine that's runningg PBS Server
  • sudo pbs_snapshot --service-logs=1 --accounting-logs=1 -o mysnapshot --obfuscate --map=mapfile.txt: Will capture a snapshot inside $PWD/mysnapshot.tgz along with 1 days of accounting and service logs, will obfuscate the data and store data mapping in the map file named 'mapfile.txt'.

...