A Remote Rotating Log for Python that works instantly
Log the stack path along with each message. This is a debugging feature.
Note: That stack path is intended to be human readable, as a debugging information.
How long ?
→ rrlog.Log for these parameters.
Into a database
Unwanted elements of the path can be skipped, for better readability. → Shorten stack paths
To hide unwanted parts of the stack trace, initialize rrlog.Log with one of these parameters:
For the use case of Use Pythons standard logging, both ways work. We prefer the seFilesExclude way. We don’t need to bother the amount of calls that happen inside the logging library, and there is already a predefined rrlog.logging23.seFilesExclude() specially for the “standard logging” case:
from rrlog.server import filewriter
from rrlog.logging23 import seFilesExclude
# Note: the OPTIONAL traceOffset skips the standard-logging-code from logged stacktraces
# Required value may depend on Python version! If given, seFilesExcluded is redundant. See Manual.
log = filewriter.createLocalLog(
filePathPattern="./demo-log-%s.txt", # "pattern" because %s (or %d) is required for the rotate-number
rotateCount=3, # rotate over 3 files
rotateLineMin=10, #at least 10 lines in each file before rotating
seFilesExclude=seFilesExclude, # optional, remove all "logging.__init__" from logged stack path
# traceOffset=5, # optional, value fits with Python2.5
stackMax=20,
)
All the “logging.__init__” stuff is removed now.
Custom time stamp formats are possible. The create... functions take a tsFormat parameter; for a description, see rrlog.server.LogServer When remote logging, this parameter is found in the server side create... function.
from rrlog.server.filewriter import createLocalLog
# This uses a custom line formatting
def format_line(job, lineCount):
# A named argument ("ip") was added ad hoc in the log calls. It's available here the formatting method:
return "Original message = '%s'. And the ip address is:%d\n"%(job.msg, job.special.get("ip",-1)) # default -1 when ip is missing
log = createLocalLog(
format_line=format_line,
filePathPattern="./demo-log-%s.txt", # "pattern" because %s (or %d) is required for the rotate-number
rotateCount=3, # rotate over 3 files
rotateLineMin=10, #at least 10 lines in each file before rotating
)
# now, log some lines:
for i in xrange(25):
log("customized-line %s"%(i), special={"ip":12345} )
log("customized-line without ip")
Another example for database logging:
from rrlog.server import dbwriter_sa
# This is a RFC-1738 style URL, as required by SQLAlchemy.
# It includes in this order:
# the database type (e.g.mysql,postgres,sqlite...)
# the database user (that has write permission)
# the database name (here: "logtest")
# See SQLAlchemy Doc for a more accurate and up-to-date description.
engineStr = "mysql://logtester@localhost/logtest"
log = dbwriter_sa.createLocalLog(
engineStr = engineStr,
tableNamePattern = "logtable_%s", # "pattern" because %s (or %d) is required for the rotate-number
rotateCount = 3, # rotate over 3 tables
rotateLineMin = 10, # at least 10 lines in each table before rotating
# This adds two custom columns, a string and an integer type:
cols = dbwriter_sa.DBConfig.COLS_DEFAULT + (
("mystring", dbwriter_sa.DBConfig.String(20)), # String(20) becomes VARCHAR(20) on MySQL
("myinteger", dbwriter_sa.DBConfig.Integer),
)
)
# Own columns are possible:
# Use COLS_DEFAULT as a base,
# e.g. mycols = list(COLS_DEFAULT) , and then append or re-order columns.
# The predefined columns (like "msg") usually must be kept, since the default writer expects them.
# now, log some lines using our new fields:
for i in xrange(25):
log("customizeDatabase:", special={"mystring":"Yahoo! %d"%(i), "myinteger":i*3} )
Each formatter method can use the attributes of the rrlog.server.MsgJob to build a line. A custom formatter method can optinally use Custom parameters
The log call can take custom named parameters. These appear in the resulting Job. A custom line formatting method can put these parameters in the logged line. → “special” parameter in rrlog.Log.__call__()
The ‘Standard Case’ is defined to be rotation, along with overwriting of old files/tables.
To disable, set rotateLineMin=None in the create* functions. Rotation is switched off, and the file/table is appended forever. In addition, the rotateCount parameter must be set to 1.
to visualize a call hierarchy:
There’s a filter available that indents each line, depending on how deep in the stack we are.
The filter could be applied with Database logging, too. But surely, it looks best with files or standard out.
Example with standard out:
from rrlog.contrib import stackindent
log = stackindent.createPrintIndentLog(token=" ",stackMax=0) # stackMax=0 suppresses filename/line number
# log as usual. The first call will "tara" the indention to zero.
def egg():
log("egg here")
def spam():
log("spam here")
#log("did tara=0",stackindent_tara=0) # tara would adjust the indent to 0
egg()
log("begin")
spam()
Same example with files:
from rrlog.server import filewriter
from rrlog.contrib import stackindent
# add a filter while log creation:
log = filewriter.createLocalLog(
filePathPattern="./demo-log-%s.txt",
tsFormat=None, # no time stamps
rotateCount=1,
rotateLineMin=None, # no rotation
filters=(stackindent.StackIndentFilter(token="--"),), # indent with one "--" per stack level
)
# log as usual. The first call will "tara" the indention to zero.
def egg():
log("egg here")
def spam():
log("spam here")
#log("did tara=0",stackindent_tara=0) # tara would adjust the indent to 0
egg()
log("begin")
spam()
Messages can trigger actions, and can be manipulated before they are written. Our terminology is:
A predefined observer available is Send mails that sends mails for defined message categories.
A predefined filter available is the function that formats each line (see also Custom line format), or the optional line indention (Indent lines).
Filters and observers can be added n the create... functions. With remote logging, they are at server side. → rrlog.LogServer.__init__() for more documentation of these parameters.
There are on / off methods in rrlog.Log to disable / enable that log instance.
The log can send lovely letters when messages of selected categories (like “E” for error) appear.
The messages are collected for a (configurable) “collect time” and sent bundled. We can send important stuff quickly, otherwise collect messages for a while.
Create a mail notifier ...
from rrlog.contrib import mail
# Assume we want mails for these 4 log message categories:
# I(internal-error), S(ecurity-issue) : we mail these nearly immediately
# E(rror) is business as usual and can wait 60 seconds :)
# W(arnings) are collected at maximum 5 minutes.
rules = (
mail.CatRule( ("I","S",), 15),
mail.CatRule( ("E",), 60),
mail.CatRule( ("W",) ,300),
)
notifier = mail.mailnotifier(
mail.SMTPMailer(
server="smtpserver.example.org", # SMTP server
serverpw="nobodyknows",
to_address="receiver@example.org",
from_address="sender@example.org",
subject="Vi@gra cheap",
loginuser="login@example.org", # optional, by default from_address is used
),
rules=rules,
)
... and hook it as “observer” into the log. That notifier sends mails when messages of the given categories are logged.
# demonstrate with the simple print logger
# observers are available with all log configurations. When remote, observers are in the server side create... function
from rrlog.server import printwriter
log = printwriter.createLocalLog(
observers=(notifier,)
)
# that category will trigger a mail now:
log("Hi my uncle from nigeria like to give yu a million", "S")
Independent line formatting in mails The rrlog.contrib.mail.mailnotifier() takes an own format_line parameter, independently from the formatting in the log files.
Custom mail protocols There is a simple SMTP mailer: rrlog.contrib.mail.SMTPMailer already included. For other mail protocols (e.g. SMTP authentication with starttls, or a sendmail server), you need to extend or replace it.
! Missing overflow protection There is no limit to protect us from very long mails yet.
→ module: rrlog.contrib.mail