Community discussion is present here: http://community.pbspro.org/t/pp-838-support-for-logging-via-syslog-in-pbs/591
...
For adding syslog support in PTL we will have to make changes in two files- pbs_logutils.py and pbs_testlib.py.
A new class PBSSyslogUtils is added for reading the date, syslog file path and the logic for which log messages file to check.
———————pbs_syslog_utils.py——————
New Class:
class PBSSyslogUtils———————In pbs_logutils.py——————
Class PBSLogUtils(object):
facility = //PBS_SYSLOG from pbs.conf
severity = // PBS_SYSLOGSEVR from pbs.conf (by default NONE)
Methods:
def get_log_type()
Summary: logic for which messages to read (local logs / syslog)
Input
syslog - by default None
Description:
1) read from PBS.conf - PBS_SYSLOG & PBS_LOCALLOG.
a) Check if the PBS_SYSLOG & PBS_LOCALLOG are set in the pbs.conf file
b) If PBS_SYSLOG is not set, we will locally in code set variable PBS_SYSLOG=0
c) If PBS_LOCALLOG is not set, we will locally in code set variable PBS_LOCALLOG=1
2) Logic for whether to read local logs/ syslog we will follow this table -
Note: Return value is the value that will be by log_match() from the PBSService. Since _log_match() will only return one value of _log_match() we will use the below table to decide it.
This info has still been added to this table because the rest of the table is necessary to depict it. Consider it as extra information from the below table.
...
PBS_SYSLOG
...
PBS_LOCALLOG
...
Which log to check in log_match()
...
0
...
0
...
Throw error
...
0
...
1
...
Local_log
...
set
...
0
...
Syslog
...
set
...
1
...
Match in both logs
...
Note: In row 4 where we have to match in both syslog and local log, we would have to call _log_match() twice.
Note: Currently we are only supporting for the current syslog file and not for previous days file. That would be covered in PP-969
Note: The error thrown will be PtlLogMatchError
Return:
self.file_to_check (/ / file_to_check =1 for syslog, file_to_check=2 for local logs and file_to_check=3 for both)
def
New methods:
def _get_syslog_lines(hostname, n, logval):
...
- get type of syslog utility (rsyslog/syslog-ng) running on host and syslog conf file. Currently we are only supporting rsyslog and not syslog-ng
- get PBS_SYSLOGSEVR from pbs.conf. If it is not set throw error PBSConfigError that set PBS_SYSLOGSEVR
- According to utility get list_of_priorites[] (severity + facility) - _get_rsyslog_priorites()
- With the priorities get list_of_syslog_files[] (if there is no list of files returned, throw PTL error) _get_rsyslog_files()
- Get lines from each of the files in list_of_syslog_files[] . This will consider the n and logval arguments. Only the specified logval messages will be considered
If no lines are returned throw PTLerror - Combine and Sort lines by datetime
...
return: Sorted + combined_lines (This list of lines will be used by the match_msg() for log_match
def _get_rsyslog_priorites(self, severity=None, facility=None)
...
return list_of_priorities[]
def _get_rsyslog_files(self, list_of_priorities[])
...
list_of_priorities[]
return list_of_files[]
// Changes to existing methods in PBSLogUtils class
def convert_date_time(syslog=false)
Description: If the syslog files are to be read the date format in the lines changes. Currently the fmt is set to the default syslog fmt
if syslog:
fmt=syslog_format
fmt is later passed to strptime funtion.
def match_msg(syslog=false)
Description: If the syslog message lines are to be read the match_msg() will check the list of syslog lines
if syslog:
date_length = //length according to date format (by default set to 15)
if lines:
for l in lines:
if starttime is not None:
tm = self.convert_date_time(l[:date_length], syslog=syslog)
if endtime is not None:
tm = self.convert_date_time(l[:date_length], syslog=syslog)
Changes to existing methods are made in the following files:-
...
class PBSService(PBSObject)
New method
def _get_log_type()
Summary: logic for which messages to read (local logs / syslog)
Description:
1) read from PBS.conf - PBS_SYSLOG & PBS_LOCALLOG.
2) Logic for whether to read local logs/ syslog we will follow this table -
|
Note: In row 4 where we have to match in both syslog and local log, we would have to call _log_match() twice.
Note: In case of row 4 the return value for log_match() will return lines from local_logs only.
Note: Currently we are only supporting for the current syslog file and not for previous days file. That would be covered in PP-969
Note: The error thrown will be PtlLogMatchError
Return:
self.file_to_check (/ / file_to_check =1 for syslog, file_to_check=2 for local logs and file_to_check=3 for both)
def log_match()
Description:
1) PBSSyslogUtilsself._get_log_type() to check if local logs/ syslog is to be checked
x= PBSSyslogUtilsPBSLogUtils._get_log_type(syslog=syslog)
2) if x =1 or 3 - read _log_match(syslog=true)
...
logval = self._instance_to_logpath(logtype)
lines = self.PBSSyslogUtilsPBSLogUtils._get_syslog_lines(hostname=self.hostname, n=n, logval=logval)
——————————In pbs_logutils.py—————————————
Class PBSLogUtils(object)
def convert_date_time(syslog=false)
Description: If the syslog files are to be read the date format in the lines changes. Currently the fmt is set to the default syslog fmt
if syslog:
fmt=syslog_format
fmt is later passed to strptime funtion.
def match_msg(syslog=false)
Description: If the syslog message lines are to be read the match_msg() will check the list of syslog lines
if syslog:
date_length = //length according to date format (by default set to 15)
if lines:
for l in lines:
if starttime is not None:
tm = self.convert_date_time(l[:date_length], syslog=syslog)
if endtime is not None:
...
)
Test Scenarios
...