Community discussion is present here: http://community.pbspro.org/t/pp-838-support-for-logging-via-syslog-in-pbs/591
Overview
PBS currently has a facility for logging via syslog. We have to add support to PTL to test the syslog functionality of PBS.
Use Cases
1. As an admin, I should be able to set PBS logging via syslog so that all daemons log into the same file.
2. As an admin, I should be able to control log events level of the PBS logging.
3. As an admin, I should be able to set the logging via syslog for all nodes in the cluster
4. As an admin, I should be able to set the logging via syslog for all servers in a failover environment.
5. As a admin I should be able to have logging into both local daemons and Syslog enabled together
6. As an admin/user, I should be able to run tracejob for a specific job with Syslog.
Note- This functionality is not currently given by PBS. This is added here because it is nice enhancement for future use.
7. As an admin/user, I should be able to differentiate log messages from each daemon.
8. As an admin/user, I should be able to set logging via syslog for only a particular daemon (Server, Mom, Sched or Comm).
Design
The design for adding syslog matching is been written so that there minimal changes for existing functionality.
1) The same log_match() function will be used. It will have an attribute for syslog. If this attribute is not set, PTL will check the pbs.conf file for syslog setting.
2) PTL will match the messages according to the daemon. For eg: server.logmatch(syslog=true) will match only server log messages from syslog.
3) Multihost support is provided. syslog messages from individual hosts will be read.
Things to be taken into consideration-
We should make sure that the user running the tests will have permissions to read syslog file and decompress it. For eg - in test bed machine pbsroot shoud be able to read from syslog file.
New flow of log_match() with syslog support-
For adding syslog support in PTL we will have to make changes in two files- pbs_logutils.py and pbs_testlib.py.
A new class pbs_syslog_utils is added for reading the date, syslog file path and the logic for which log messages file to check.
———————pbs_syslog_utils.py——————
New Class:
class pbs_syslog_utils(object):
facility = //PBS_SYSLOG from pbs.conf
serverity = // PBS_SYSLOGSEVR from pbs.conf (by default NONE)
Methods:
def get_log_config_values(syslog=false)
Summary: logic for which messages to read (local logs / syslog)
Input
syslog - by default false
Description:
1) read from PBS.conf - PBS_SYSLOG & PBS_LOCALLOG.
a) First check if the PBS_SYSLOG & PBS_LOCALLOG are set in the pbs.conf file
b) If the values are not set in the pbs.conf file we will set variable PBS_SYSLOG=0 and PBS_LOCALLOG=1 (line 3 in below table), i.e PTL will execute log_match() in local logs.
2) Logic for whether to read local logs/ syslog we will follow this table -
Note: Return value is the value that will be by log_match() from the PBSService. Since _log_match() will only return one value of _log_match() we will use the below table to decide it.
This info has still been added to this table because the rest of the table is necessary to depict it. Consider it as extra information from the below table.
| |||||||||||||||||||||||||||||||||||||||||||||||||
|
Note: In row 7 where we have to match in both syslog and local log, we would have to call _log_match() twice.
Note: Currently we are only supporting for the current syslog file and not for previous days file. That would be covered in PP-969
Note: The error thrown will be PtlLogMatchError
Return:
self.file_to_check (/ / file_to_check =1 for syslog, file_to_check=2 for local logs and file_to_check=3 for both)
def get_log_syslog_lines(hostname, n, logval):
Summary: This function returns the syslog messages for logmatch
Input:
#hostname = host from which to read syslog file
# n= number of lines requested by user
# logval - sched/server/mom/comm logs
Description:
- get type of syslog utility (rsyslog/syslog-ng) running on host and syslog conf file. Currently we are only supporting rsyslog
- According to utility get list_of_priorites[] (severity + facility) - get_rsyslog_priorites(), def get_syslogng_priorites()
- With the priorities get list_of_syslog_files[] (if there is no list of files returned, throw PTL error) get_rsyslog_files(), get_syslogng_files()
- Get lines from each of the files in list_of_syslog_files[] - def get_syslog_lines()
If no lines are returned throw PTLerror - Combine and Sort lines by datetime
return: Sorted + combined_lines (This list of lines will be used by the match_msg() for log_match
def get_rsyslog_priorites(self, severity=None, facility=None)
Summary: Returns the priorities. Since the syslog messages can be stored in multiple files, a list is returned
# If facility and severity mentioned return the file for that combination from rsyslog.conf
Input: severity and facilty values
return list_of_priorities[]
def get_syslogng_priorites(self, severity=None, facility=None)
Summary: Returns the priorities. Since the syslog messages can be stored in multiple files, a list is returned
# If facility and severity mentioned return the file for that combination from rsyslog.conf
Input: severity and facilty values
return list_of_priorities[]
def get_rsyslog_files(self, list_of_priorities[])
Summary: Returns the path of files to be read on basis of the priorities. Since the syslog messages can be stored in multiple files, a list is returned
Input:
list_of_priorities[]
return list_of_files[]
def get_syslogng_files(self, list_of_priorities[])
Summary: Returns the path of files to be read on basis of the priorities. Since the syslog messages can be stored in multiple files, a list is returned
Input:
list_of_priorities[]
return list_of_files[]
def get_syslog_lines(self, hostname=None, n=50, logval=None, filename=None):
Summary: Return the last block of lines from the syslog files for that particular daemon
Input:
#hostname = host from which to read syslog file
# n= number of lines requested by user
# filename: the file to read
# logval - sched/server/mom/comm logs
return lines[]
Changes to existing methods are made in the following files:-
————pbs_testlib.py——————————
class PBSService(PBSObject)
def log_match(syslog=False)
// Additions:
syslog: If the user wants to read into syslog file? By default false
Description:
1) run pbs_syslog_utils.get_log_config_values(syslog=syslog) to check if local logs/ syslog is to be checked
x= syslog_config.log_config_values(syslog=true)
2) If only syslog messages are to be checked - return _log_match(syslog=true)
3) If only local log messages are to be checked - return _log_match(syslog=false)
4) If both syslog and local log messages are to be checked -
syslog_return = _log_match(syslog=true)
// Read in local logs only if syslog log match is returned true. If syslog match fails there is no need to check in local logs since the test will be marked as a failure.
if syslog_return:
return _log_match(syslog=false)
else:
return syslog_return
def _log_match(syslog=False)
// Additions:
syslog: If the user wants to read into syslog file? By default false
Description:
The syslog value is further passes to log_lines() and match_msg()
lines = self.log_lines(syslog=true)
rv = self.logutils.match_msg(syslog=true)
Note- 'def log_lines' returns the last n lines of the log file.
def log_lines(syslog=False)
syslog: If the user wants to read into syslog file?
Description:
if syslog:
logval = self._instance_to_logpath(logtype)
lines = self.pbs_syslog_utils.get_log_syslog_lines(hostname=self.hostname, n=n, logval=logval)
——————————In pbs_logutils.py—————————————
Class PBSLogUtils(object)
def convert_date_time(syslog=false)
Description: If the syslog files are to be read the date format in the lines changes. Currently the fmt is set to the default syslog fmt
if syslog:
fmt=syslog_format
fmt is later passed to strptime funtion.
def match_msg(syslog=false)
Description: If the syslog message lines are to be read the match_msg() will check the list of syslog lines
if syslog:
date_length = //length according to date format (by default set to 15)
if lines:
for l in lines:
if starttime is not None:
tm = self.convert_date_time(l[:date_length], syslog=syslog)
if endtime is not None:
tm = self.convert_date_time(l[:date_length], syslog=syslog)
Test Scenarios
1) Test that when PBS_SYSLOG=1 is enabled in pbs.conf, PBS logs messages via syslog.
2) Test that according to the PBS_SYSLOGSEVR value in pbs.conf the messages of that severity are logged via syslog. This should include tests for each severity level- Emergency, alert , critical, error, warning, notice, informational, debug.
3) Test that in a multi-node cluster setup, logging via syslog can be enabled.
4) Test that in a failover environment logging via syslog works on all the servers.
5) Test that both local logging and logging via syslog can be enabled simultaneously for both, single node and execution nodes in a cluster.
6) Test that tracejob works when logging via syslog is enabled
(Note: trace job currently does not work with Syslog. Since this is a use case and would be a nice enhancement to have)
7) Test that server and sched and mom messages can be differentiated explicitly in the syslog log file.
8) Check that there is no performance issue with large log files
8) Test that logging via sys logging can be set for only a particular daemon.
(This functionality is not present in PBS)