Community discussion is present here: http://community.pbspro.org/t/pp-838-support-for-logging-via-syslog-in-pbs/591
...
The design for adding syslog matching is been written so that there minimal changes for existing functionality.
1) The same logsyslog_match () functions for each class (Server, MoM, Sched and Comm) are been used with changes to read into syslog. For example to check server messages that go into syslog we will have the same method server.log_match(). This will also help the tests differentiate the messages from individual daemons.function will be used. It will have an attribute for syslog. If this attribute is not set, PTL will check the pbs.conf file for syslog setting.
2) PTL will match the messages according to the daemon. For eg: server.logmatch(syslog=true) will match only server log messages from syslog.
3) Multihost support is provided. syslog messages from individual hosts will be read.
Things to be taken intoo consideration-
We should make sure that the user running the tests will have permissions to read syslog file and decompress it. For eg - in test bed machine pbsroot shoud be able to read from syslog file.
New flow of log_match() with syslog support-
For adding syslog support in PTL we will have to make changes in two files- pbs_logutils.py and pbs_testlib.py.
A new class syslog_utils is added for reading the date, syslog file path and the logic for which log messages file to check.
Wiki page for syslog. This can be referred to for understanding basics of syslog
———————syslog_utils.py——————
Here is the top level flow of how the log_match() will behave after support for syslog is added
New Class:
class syslog_utils(object):
facility = //PBS_SYSLOG from pbs.conf
serverity = // PBS_SYSLOGSEVR from pbs.conf (by default NONE)
Methods:
def log_config_values(syslog=0false)
Summary: logic for which messages to read (local loglogs / syslog)
Input
// syslog - by default false
Description:
1) read from PBS.conf - PBS_SYSLOG & PBS_LOCALLOG
2)
...
Logic for which log files to check in log_match(), syslog or local logs whether to read local logs/ syslog we will follow this table -
...
self.file_to_check (/ / file_to_check =1 for syslog, file_to_check=2 for local logs and file_to_check=3 for both)
...
# n= number of lines requested by user
# file name: the file to read # logval - sched/server/mom/comm logs
...
- get syslog utility (rsyslog/syslog-ng) running on host and syslog conf file
- According to utility get list of priorites[] (severity + facility) - get_rsyslog_priorites(), def get_syslogng_priorites()
- With the priorities get list_of_syslog_files[] (if there is no list of files returned, throw PTL error) get_rsyslog_files(), get_syslogng_priorites()
- Get lines from each of the files in list_of_syslog_files[] - def get_syslog_lines()
- Combine and Sort lines by datetime
...
return: Sorted + combined_lines (This list of lines will be used by the match_msg() for log_match
...
Input: severity and priority values
return list_of_priorities[]
...
Input: severity and priority values
return list_of_priorities[]
def get_rsyslog_files(self, list_of_priorities[])
Summary: Returns the path of files to be read on basis of the priorities. Since the syslog messages can be stored in multiple files, a list is returned
Input:
list_of_priorities[]
return list_of_files[]
def get_syslog_ng_files(self, list_of_priorities[])
Summary: Returns the path of files to be read on basis of the priorities. Since the syslog messages can be stored in multiple files, a list is returned
Input:
list_of_priorities[]
return list_of_files[]
def get_syslog_lines(self, hostname=None, n=50, logval=None):
Summary: Return the last block of lines from the syslog files for that particular daemon
Input:
#hostname = host from which to read syslog file
...
syslog: If the user wants to read into syslog file? By default false
Execution steps
...
x= syslog_config.log_config_values(syslog=true)
22) If only syslog messages are to be checked - return _log_match(syslog=true)
33) If only local log messages are to be checked - return _log_match(syslog=false)
4) If both syslog and local log messages are to be checked -
...
def _log_match(syslog=False)
// Additions:
syslog syslog: If the user wants to read into syslog file? By default false
Description:
The The syslog value is further passes to log_lines() and match_msh()
lines lines = self.log_lines(syslog=true)
rv rv = self.logutils.match_msg(syslog=true)
...
def convert_date_time(syslog=false) syslog: If the user wants to read into syslog file?
Description: If the syslog files are to be read the date format in the lines changes. Currently the fmt is set to the default syslog fmt
...
fmt=syslog_format
...
t = time.strptime(datetime, fmt)
if syslog:
now = time.strftime("%Y,%m,%d,%H,%M,%S")
split_now = now.split(',')
t_edit = list(t)
t_edit[0]= int(split_now[0])
t = time.struct_time(tuple(t_edit)) fmt is later passed to strptime funtion.
def match_msg(syslog=false) syslog: If the user wants to read into syslog file?
Description: If the syslog message lines are to be read the match_msg() will check the list of syslog lines
if syslog:
date_length = //length according to date format (by default set to 15)
if lines:
for l in lines:
...
tm = self.convert_date_time(l[:date_length], syslog_flag=syslog)
if endtime is not None:
tm = self.convert_date_time(l[:date_length], syslog_flag=syslog)
Test Scenarios
...