Community discussion is present here: http://community.pbspro.org/t/pp-838-support-for-logging-via-syslog-in-pbs/591
Objective
PBS currently has a facility for logging via syslog. We are reverse engineering to the EDD. We need to have the use cases, test scenarios and test cases (both manual and PTL) reviewed by the open source community.
Use Cases
1. As an admin, I should be able to set PBS logging via syslog so that all daemons log into the same file.
2. As an admin, I should be able to control log events level of the PBS logging.
3. As an admin, I should be able to set the logging via syslog for all nodes in the cluster
4. As an admin, I should be able to set the logging via syslog for all servers in a failover environment.
5. As a admin I should be able to have logging into both local daemons and Syslog enabled together
6. As an admin/user, I should be able to run tracejob for a specific job with Syslog.
Note- This functionality is not currently given by PBS. This is added here because it is nice enhancement for future use.
7. As an admin/user, I should be able to differentiate log messages from each daemon.
8. As an admin/user, I should be able to set logging via syslog for only a particular daemon (Server, Mom, Sched or Comm).
Design
For adding syslog support in PTL we will have to make changes in two files- log_utils.py and pbs_testlib.py
Changes in log_utils.py
Changes in this file are related to the format of the recorded datetime. Syslog records the datetime in a different format than local logs. According to the format we will have to do changes to reading the datetime.
Some syslog files are compressed. Even if we can read a compressed file from python, but I would prefer if we decompress it so that rest of the code in PTL for reading files will work without any changes.
1) Check is file is compressed. Decompress it to same location with same name.
if sys_log_file == compressed
decompress_file_to_text()
_sys_log_path_ = decompressed_file_path
2) Date format.
Change regular expression ‘tm_re ’ to match the date format of the syslog file.
if PBS_SYSLOG=1 (in PBS.conf)
tm_re = "(?P<datetime>\w{3}\s\d\d\s\d\d:\d\d:\d\d)"
def convert_date_time()
if PBS_SYSLOG=1 (in PBS.conf)
Fmt of date = "%b %d %H:%M:%S"
t = time.strptime(datetime, fmt)
Changes in pbs_testlib.py
1) Change log_file path from local logs to syslog. The local log file are in 4 locations- server, mom, sched, comm. For syslog since there is only one file, we only have to point it to that file. Path to the log file can be set through a environment variable before starting the PTL tests or through the test script. If the path is not set the default path "/var/log/messages" will be taken
sys_log_file_path= syslog_path_
def init_logfile_path
if PBS_SYSLOG=1
path_log_file = _sys_log_path_
elif locally
path_log_file= __local_path__
Note- Def log_lines returns the last n lines of the log file.
def log_lines
if PBS_SYSLOG=1
path_log_file = _sys_log_path_
elif locally
path_log_file= __local_path__
Test Scenarios
1) Test that when PBS_SYSLOG=1 is enabled in pbs.conf, PBS logs messages via syslog.
2) Test that according to the PBS_SYSLOGSEVR value in pbs.conf the messages of that severity are logged via syslog. This should include tests for each severity level- Emergency, alert , critical, error, warning, notice, informational, debug.
3) Test that in a multi-node cluster setup, logging via syslog can be enabled.
4) Test that in a failover environment logging via syslog works on all the servers.
5) Test that both local logging and logging via syslog can be enabled simultaneously for both, single node and execution nodes in a cluster.
6) Test that tracejob works when logging via syslog is enabled
(Note: trace job currently does not work with Syslog. Since this is a use case and would be a nice enhancement to have)
7) Test that server and sched and mom messages can be differentiated explicitly in the syslog log file.
8) Test that logging via sys logging can be set for only a particular daemon.