Table Of Contents

Previous topic

Examples for rrlog

Next topic


This Page

More Features

A Remote Rotating Log for Python that works instantly

Stack Path logging

Log the stack path along with each message. This is a debugging feature.

Note: That stack path is intended to be human readable, as a debugging information.

How long ?

  • The create... functions take a stackMax parameter to limit the logged stack length.
  • When remote logging, these parameters are found in the client side create... function.
  • See also the extractStack parameter to completely switch off stack extraction (for performance reasons).

rrlog.Log for these parameters.

Into a database

  • Column: The callers path is written into the “path” column.
  • The top element of the path is written redundantly in the cfn/cln columns (“Callers File Name” resp. “Callers Line Number”). Thus, cfn and cln can be evaluated immediately (without parsing the path string).

Unwanted elements of the path can be skipped, for better readability. → Shorten stack paths

Shorten stack paths

To hide unwanted parts of the stack trace, initialize rrlog.Log with one of these parameters:

  • seFilesExclude : a callable telling which parts to skip in the stack path.
  • traceOffset : skips the first components of the stacktrace. We need to know (or to try out) how many to skip. traceOffset is preferrable when we have an own wrapper around the log object. Increase it to 1, to hide exactly the first method call, and so on.

For the use case of Use Pythons standard logging, both ways work. We prefer the seFilesExclude way. We don’t need to bother the amount of calls that happen inside the logging library, and there is already a predefined rrlog.logging23.seFilesExclude() specially for the “standard logging” case:

from rrlog.server import filewriter
from rrlog.logging23 import seFilesExclude

# Note: the OPTIONAL traceOffset skips the standard-logging-code from logged stacktraces
# Required value may depend on Python version! If given, seFilesExcluded is redundant. See Manual.
log = filewriter.createLocalLog(
		filePathPattern="./demo-log-%s.txt", # "pattern" because %s (or %d) is required for the rotate-number
		rotateCount=3, # rotate over 3 files
		rotateLineMin=10, #at least 10 lines in each file before rotating
		seFilesExclude=seFilesExclude, # optional, remove all "logging.__init__" from logged stack path
#		traceOffset=5, # optional, value fits with Python2.5

All the “logging.__init__” stuff is removed now.

Time Stamp format

Custom time stamp formats are possible. The create... functions take a tsFormat parameter; for a description, see rrlog.server.LogServer When remote logging, this parameter is found in the server side create... function.

Custom line format

from rrlog.server.filewriter import createLocalLog

# This uses a custom line formatting
def format_line(job, lineCount):
	# A named argument ("ip") was added ad hoc in the log calls. It's available here the formatting method:
	return "Original message = '%s'. And the ip address is:%d\n"%(job.msg, job.special.get("ip",-1)) # default -1 when ip is missing

log = createLocalLog(
	filePathPattern="./demo-log-%s.txt", # "pattern" because %s (or %d) is required for the rotate-number
	rotateCount=3, # rotate over 3 files
	rotateLineMin=10, #at least 10 lines in each file before rotating

# now, log some lines:
for i in xrange(25):
	log("customized-line %s"%(i), special={"ip":12345} )
log("customized-line without ip")

Another example for database logging:

from rrlog.server import dbwriter_sa
# This is a RFC-1738 style URL, as required by SQLAlchemy.
# It includes in this order:
# the database type (e.g.mysql,postgres,sqlite...)
# the database user (that has write permission)
# the database name (here: "logtest")
# See SQLAlchemy Doc for a more accurate and up-to-date description.
engineStr = "mysql://logtester@localhost/logtest"

log = dbwriter_sa.createLocalLog(
	engineStr = engineStr,
	tableNamePattern = "logtable_%s",  # "pattern" because %s (or %d) is required for the rotate-number
	rotateCount = 3, # rotate over 3 tables
	rotateLineMin = 10, # at least 10 lines in each table before rotating
	# This adds two custom columns, a string and an integer type:
	cols = dbwriter_sa.DBConfig.COLS_DEFAULT + (
		("mystring", dbwriter_sa.DBConfig.String(20)), # String(20) becomes VARCHAR(20) on MySQL
		("myinteger", dbwriter_sa.DBConfig.Integer),

# Own columns are possible:
# Use COLS_DEFAULT as a base,
# e.g. mycols = list(COLS_DEFAULT) , and then append or re-order columns.
# The predefined columns (like "msg") usually must be kept, since the default writer expects them.

# now, log some lines using our new fields:
for i in xrange(25):
	log("customizeDatabase:", special={"mystring":"Yahoo! %d"%(i), "myinteger":i*3} )

Each formatter method can use the attributes of the rrlog.server.MsgJob to build a line. A custom formatter method can optinally use Custom parameters

Custom parameters

The log call can take custom named parameters. These appear in the resulting Job. A custom line formatting method can put these parameters in the logged line. → “special” parameter in rrlog.Log.__call__()

Don’t rotate

The ‘Standard Case’ is defined to be rotation, along with overwriting of old files/tables.

To disable, set rotateLineMin=None in the create* functions. Rotation is switched off, and the file/table is appended forever. In addition, the rotateCount parameter must be set to 1.

Indent lines

to visualize a call hierarchy:

There’s a filter available that indents each line, depending on how deep in the stack we are.


The filter could be applied with Database logging, too. But surely, it looks best with files or standard out.

Example with standard out:

from rrlog.contrib import stackindent
log = stackindent.createPrintIndentLog(token="  ",stackMax=0) # stackMax=0 suppresses filename/line number

# log as usual. The first call will "tara" the indention to zero.
def egg():
	log("egg here")

def spam():
	log("spam here")
	#log("did tara=0",stackindent_tara=0) # tara would adjust the indent to 0

Note: The filter is applied inside the convenient create function

Same example with files:

from rrlog.server import filewriter
from rrlog.contrib import stackindent

# add a filter while log creation:
log = filewriter.createLocalLog(
	tsFormat=None, # no time stamps
	rotateLineMin=None, # no rotation
	filters=(stackindent.StackIndentFilter(token="--"),), # indent with one "--" per stack level

# log as usual. The first call will "tara" the indention to zero.
def egg():
	log("egg here")

def spam():
	log("spam here")
	#log("did tara=0",stackindent_tara=0) # tara would adjust the indent to 0


Observe and filter

Messages can trigger actions, and can be manipulated before they are written. Our terminology is:

  • “observers” read messages after they are written, and eventually trigger an action
  • “filters” are similar, except: they are called before the message is written, and they can manipulate the job object - especially the message text.

A predefined observer available is Send mails that sends mails for defined message categories.

A predefined filter available is the function that formats each line (see also Custom line format), or the optional line indention (Indent lines).

Filters and observers can be added n the create... functions. With remote logging, they are at server side. → rrlog.LogServer.__init__() for more documentation of these parameters.

On / Off

There are on / off methods in rrlog.Log to disable / enable that log instance.

Send mails

The log can send lovely letters when messages of selected categories (like “E” for error) appear.

The messages are collected for a (configurable) “collect time” and sent bundled. We can send important stuff quickly, otherwise collect messages for a while.

Create a mail notifier ...

from rrlog.contrib import mail

# Assume we want mails for these 4 log message categories: 
# I(internal-error), S(ecurity-issue) : we mail these nearly immediately
# E(rror) is business as usual and can wait 60 seconds :)
# W(arnings) are collected at maximum 5 minutes.
rules = (
	mail.CatRule( ("I","S",), 15),
	mail.CatRule( ("E",), 60),
	mail.CatRule( ("W",) ,300),

notifier = mail.mailnotifier(
		server="", # SMTP server
		subject="Vi@gra cheap",
		loginuser="", # optional, by default from_address is used

... and hook it as “observer” into the log. That notifier sends mails when messages of the given categories are logged.

# demonstrate with the simple print logger
# observers are available with all log configurations. When remote, observers are in the server side create... function
from rrlog.server import printwriter 
log = printwriter.createLocalLog(

# that category will trigger a mail now:
log("Hi my uncle from nigeria like to give yu a million", "S")

Independent line formatting in mails The rrlog.contrib.mail.mailnotifier() takes an own format_line parameter, independently from the formatting in the log files.

Custom mail protocols There is a simple SMTP mailer: rrlog.contrib.mail.SMTPMailer already included. For other mail protocols (e.g. SMTP authentication with starttls, or a sendmail server), you need to extend or replace it.

! Missing overflow protection There is no limit to protect us from very long mails yet.

→ module: rrlog.contrib.mail