Community discussion is present here: http://community.pbspro.org/t/pp-838-support-for-logging-via-syslog-in-pbs/591
...
The design for adding syslog matching is been written so that there minimal changes for existing functionality. The same log_match() functions for each class (Server, MoM, Sched and Comm) are been used with changes to read into syslog. For example to check server messages that go into syslog we will have the same method server.log_match(). This will also help the tests differentiate the messages from individual daemons.
PTL will read into syslog in these cases-
1) syslog parameter is set in log_match() function
2) PBS_SYSLOG is set in pbs.conf. This case is for running the existing PTL tests which do not have syslog parameter in log_match().
We should make sure that the user running the tests will have permissions to read syslog file and decompress it. For eg - in test bed machine pbsroot shoud be able to read from syslog file.
For adding syslog support in PTL we will have to make changes in two files- pbs_logutils.py and pbs_testlib.py.
A new class syslog_utils is added for reading the date, syslog file path and the logic for which log messages file to check.
———————syslog_utils.py——————
To add the functionality on which logs to check in log_match(), syslog or local logs we will follow this table -
PBS_SYSLOG and PBS_LOCALLOG are values from the pbs.conf file. Syslog_attribute is the attribute to be passed to the log_match() function. Return value is the return value of log_match()
...
PBS_SYSLOG
...
PBS_LOCALLOG
...
Syslog_attribute
...
Which log to check in log_match()
...
Not_set
...
0
...
0
...
Throw error
...
Not_set
...
0
...
1
...
Throw error
...
Not_set
...
1
...
0
...
Local_Log
...
Not_set
...
1
...
1
...
Throw error
...
set
...
0
...
0
...
Syslog
...
set
...
0
...
1
...
Syslog
...
set
...
1
...
0
...
Match in both logs
...
set
...
1
...
1
...
Syslog
...
...
In row 7 where we have to match in both syslog and local log, we would have to call _log_match() twice.
Note: Currently we are only supporting for the current syslog file and not for previous days file. That would be covered in PP-969
import platform
class syslog_utils(object):
du = DSHUtils()
# Syslog conf file data path
syslog_conf_path = {'rsyslogd': '/etc/rsyslog.conf', 'syslog-ng': '/etc/syslog-ng/syslog-ng.conf'}
# PBS supported severity and facilities
p_severity = ['info', 'debug']
p_facility = ['*', 'local7', ....]
facility = //PBS_SYSLOG from pbs.conf
serverity = // PBS_SYSLOGSEVR from pbs.conf (by default NONE)
def log_config_values(syslog=0)
// logic for which file to read will come from the above table
// returns a integer value which is interpreted as which file to check
self.file_to_check=0
// read from PBS.conf PBS_SYSLOG & PBS_LOCALLOG
// Logic of which file to read log messages from- syslog or local
// file_to_check =1 for syslog, file_to_check=2 for local logs and file_to_check=3 for both
return self.file_to_check
def log_syslog_lines(self):
//This function returns the syslog messages for logmatch
# get syslog utility running and conf file
self.cmd_sys = "ps -e | grep syslog | awk '{print $NF}'"
self.utility = self.du.run_cmd(hosts=hostname, cmd=self.cmd_sys, as_script=True, sudo=True,
level=logging.DEBUG2)['out']
self.syslog_conf_file = self.syslog_conf_path.get(self.utility[0])
#get syslog log files to be read.
list_of_priority = []
if self.utility[0] == "rsyslogd":
list_of_priorities = self.get_rsyslog_priorites(severity=severity, facility=facility)
self.list_of_syslog_files = self.get_rsyslog_files(list_of_priorities)
elif self.utility[0] == "syslog-ng":
list_of_priorities = self.get_syslogng_priorites(severity=severity, facility=facility)
self.list_of_syslog_files = self.get_syslogng_files(list_of_priorities)
else:
// throw error
// PtlLogMatchError(rc=1, rv=False, msg=_msg)
# Get lines from each of the files from list_of_syslog_files[]
lines = []
combined_lines = []
for x in self.list_of_syslog_files:
l_sys = self.get_syslog_lines(hostname=hostname, n=n, filename=x, logval=logval)
lines.append(l_sys)
# combine lines from each file
combined_lines = [item for sublist in lines for item in sublist]
# Sort lines by datetime
combined_lines.sort(key=lambda x: datetime.datetime.strptime(x[:15], "%b %d %H:%M:%S"))
return combined_lines
def get_rsyslog_priorites(self, severity=None, facility=None)
# Returns the priorities. Since the syslog messages can be stored in multiple files, a list is returned
# If facility and severity mentioned return the file for that combination from rsyslog.conf
# If severity is NONE read into rsyslog.conf file and return all priotities that contain the severity set in rsyslogd.conf
return list_of_priorities[]
def get_syslogng_priorites(self, severity=None, facility=None)
# Returns the priorities. Since the syslog messages can be stored in multiple files, a list is returned
# If facility and severity mentioned return the file for that combination from rsyslog.conf
# If severity is NONE read into rsyslog.conf file and return all priotities that contain the severity set in syslog-ng.conf
return list_of_priorities[]
def get_rsyslog_files(self, list_of_priorities[])
# Returns the path of files to be read on basis of the priorities. Since the syslog messages can be stored in multiple files, a list is returned
return list_of_files[]
def get_syslog_ng_files(self, list_of_priorities[])
# Returns the path of files to be read on basis of the priorities. Since the syslog messages can be stored in multiple files, a list is returned
return list_of_files[]
def get_syslog_lines(self, hostname=None, n=50, filename=None, logval=None):
# Return the last block of lines from the syslog files for that particular daemon
# n= number of lines requested by user
#hostname = host from which to read syslog file
# file name is the file to read
# is user says all lines, default to 100
if n == "ALL":
n=100
# if hostname is None, default to machine on which ptl is running
if hostname is None:
hostname = socket.gethostname()
if logval is 'sched_logs':
daemon_str = 'pbs_sched'
elif logval is 'server_logs':
daemon_str = 'Server@'
elif logval is 'mom_logs':
daemon_str = 'pbs_mom'
elif logval is 'comm_logs':
daemon_str = 'pbs_comm'
else:
// PtlLogMatchError(rc=1, rv=False, msg=_msg)
# use tail command to get last block of lines from file
cmd = ['/usr/bin/tail']
cmd += [str(n), filename]
lines = self.du.run_cmd(hosts=hostname, cmd=cmd, sudo=True,
level=logging.DEBUG2)['out']
lines = [l for l in lines if daemon_str in l]
return lines
————pbs_testlib.py——————————
Following changes are done to the current log_match() function
1) log_match() calls the _log_match() method.
Therefore according to the table above we should have a conditional statements in log_match()
In case both have to be read we should call the _log_match() twice.We should make sure that the user running the tests will have permissions to read syslog file and decompress it. For eg - in test bed machine pbsroot shoud be able to read from syslog file.
For adding syslog support in PTL we will have to make changes in two files- pbs_logutils.py and pbs_testlib.py.
A new class syslog_utils is added for reading the date, syslog file path and the logic for which log messages file to check.
———————syslog_utils.py——————
Here is the sample flow of how the log_match() will behave after support for syslog is added
New Class:
class syslog_utils(object):
facility = //PBS_SYSLOG from pbs.conf
serverity = // PBS_SYSLOGSEVR from pbs.conf (by default NONE)
Interfaces:
def log_config_values(syslog=0)
Summary: logic for which messages to read (local log/ syslog)
//
Description:
1) read from PBS.conf - PBS_SYSLOG & PBS_LOCALLOG
2) Argument syslog
Logic for which log files to check in log_match(), syslog or local logs we will follow this table -
| |||||||||||||||||||||||||||||||||||||||||||||||||
|
Note:
In row 7 where we have to match in both syslog and local log, we would have to call _log_match() twice.
Note: Currently we are only supporting for the current syslog file and not for previous days file. That would be covered in PP-969
Return:
self.file_to_check (/ file_to_check =1 for syslog, file_to_check=2 for local logs and file_to_check=3 for both)
def log_syslog_lines(hostname, n, logval):
Summary: This function returns the syslog messages for logmatch
Input:
#hostname = host from which to read syslog file
# n= number of lines requested by user
# file name: the file to read
# logval - sched/server/mom/comm logs
Description:
- get syslog utility (rsyslog/syslog-ng) running on host and syslog conf file
- According to utility get list of priorites[] (severity + facility) - get_rsyslog_priorites(), def get_syslogng_priorites()
- With the priorities get list_of_syslog_files[] (if there is no list of files returned, throw PTL error) get_rsyslog_files(), get_syslogng_priorites()
- Get lines from each of the files in list_of_syslog_files[] - def get_syslog_lines()
- Combine and Sort lines by datetime
return: Sorted + combined_lines (This list of lines will be used by the match_msg() for log_match
def get_rsyslog_priorites(self, severity=None, facility=None)
Summary: Returns the priorities. Since the syslog messages can be stored in multiple files, a list is returned
# If facility and severity mentioned return the file for that combination from rsyslog.conf
Input: severity and priority values
return list_of_priorities[]
def get_syslogng_priorites(self, severity=None, facility=None)
Summary: Returns the priorities. Since the syslog messages can be stored in multiple files, a list is returned
# If facility and severity mentioned return the file for that combination from rsyslog.conf
Input: severity and priority values
return list_of_priorities[]
def get_rsyslog_files(self, list_of_priorities[])
Summary: Returns the path of files to be read on basis of the priorities. Since the syslog messages can be stored in multiple files, a list is returned
return list_of_files[]
def get_syslog_ng_files(self, list_of_priorities[])
Summary: Returns the path of files to be read on basis of the priorities. Since the syslog messages can be stored in multiple files, a list is returned
return list_of_files[]
def get_syslog_lines(self, hostname=None, n=50, filename=None, logval=None):
Summary: Return the last block of lines from the syslog files for that particular daemon
Input:
#hostname = host from which to read syslog file
# n= number of lines requested by user
# file name: the file to read
# logval - sched/server/mom/comm logs
return lines[]
Changes to existing methods are made in the following files:-
————pbs_testlib.py——————————
class PBSService(PBSObject)
def log_match(syslog=False)
// Additions: Get the value of whether
syslog: If the user wants to read syslog or locallog filesinto syslog file?
Execution steps
1) run syslog_config.log_config_values(syslog=syslog) to check if syslog: local logs/ syslog is to be checked
x= syslog_config.log_config_values(syslog=true)
// Values to pass to _log_match()
if x==1
return 2) If only syslog messages are to be checked - return _log_match(syslog=true) elif x==2
return 3) If only local log messages are to be checked - return _log_match(syslog=false)
elif x==3 4) If both syslog and local log messages are to be checked -
syslog_return = _log_match(syslog=true)
// Read in local logs only if syslog log match is returned true. If syslog fails there is no need to check in local logs
if syslog_return:
return _log_match(syslog=false)
else:
return syslog_return
else:
_msg= "Log file to check not set"
PtlLogMatchError(rc=1, rv=False, msg=_msg)
def _log_match(syslog=False)
lines def _log_match(syslog=False)
// Additions:
syslog: If the user wants to read into syslog file?
Description:
The syslog value is further passes to log_lines() and match_msh()
lines = self.log_lines(syslog=true)
rv rv = self.logutils.match_msg(syslog=true)
Note- 'def log_lines' returns the last n lines of the log file.
def log_lines(syslog=False)
logval = Nonesyslog: If the user wants to read into syslog file? lines = None
sudo = False
self.syslogUtils = syslog_utils()
Description:
if syslog:
logval = self._instance_to_logpath(logtype)
lines = self.syslogUtils.log_syslog_lines(hostname=self.hostname, n=n, logval=logval)
——————————In pbs_logutils.py—————————————
...
Class PBSLogUtils(object)
def convert_date_time(syslog=false)_date_time(syslog=false)
syslog: If the user wants to read into syslog file?
Description: If the syslog files are to be read the date format in the lines changes. Currently the fmt is set to the default syslog fmt
if syslog:
fmt="%b %d %H:%M:%S"syslog_format
try:
t = time.strptime(datetime, fmt)
if syslog:
now = time.strftime("%Y,%m,%d,%H,%M,%S")
split_now = now.split(',')
t_edit = list(t)
t_edit[0]= int(split_now[0])
t = time.struct_time(tuple(t_edit))
def match_msg(syslog=false)
syslog: If the user wants to read into syslog file?
Description: If the syslog message lines are to be read the
if syslog:
...