A Remote Rotating Log for Python that works instantly
Log into standard out:
from rrlog.server import printwriter
log = printwriter.createLocalLog()
# now, log some lines:
for i in xrange(25):
log("Dear log reader, this is line #%s"%(i))
Log rotating into 3 files, each with 10 lines max. (existing files in the working directory are overwritten)
from rrlog.server import filewriter
log = filewriter.createLocalLog(
filePathPattern="./demo-log-%s.txt", # "pattern" because %s (or %d) is required for the rotate-number
rotateCount=3, # rotate over 3 files
rotateLineMin=10, #at least 10 lines in each file before rotating
)
# now, log some lines:
for i in xrange(25):
log("Dear log reader, this is file line #%s"%(i))
Log rotating into 3 tables, each with 10 lines max. (existing tables are overwritten)
from rrlog.server import dbwriter_sa
# This is a RFC-1738 style URL, as required by SQLAlchemy.
# It includes in this order:
# the database type (e.g.mysql,postgres,sqlite...)
# the database user (that has write permission)
# the database name (here: "logtest")
# See SQLAlchemy Doc for a more accurate and up-to-date description.
engineStr = "mysql://logtester@localhost/logtest"
log = dbwriter_sa.createLocalLog(
engineStr = engineStr,
tableNamePattern = "logtable_%s", # "pattern" because %s (or %d) is required for the rotate-number
rotateCount = 3, # rotate over 3 tables
rotateLineMin = 10, # at least 10 lines in each table before rotating
)
# log 31 lines rotating -> first table has exactly one message (the last)
for i in xrange(31):
log("usingDatabase: this is line #%s"%(i))
This is preferred over xmlrpc, because faster.
(Re-)Connection behavior
Any log client will use the log server as long as the server is available. When the server is up again after a downtime (or connection failure), all clients will start to use the server again. While a log server is down, messages are lost silently.
Host and Ports
By default, the connection uses “localhost” and a default port → rrlog.globalconst.DEFAULTPORT_SOCKET
You can specify 1..n ports on both sides. The server uses the first free port. The client uses the first port where a server seems available.
# First, create a pure print server:
# (this is independent from the remote connection type)
from rrlog.server import printwriter
srv = printwriter.createServer()
# Start the server as a socket server:
from rrlog.server import socketserver
socketserver.startServer(
srv,
ports=(9801, 9802, 9803), # try in this order, use the first port available
)
# Now, the server should be waiting for requests.
# First, create a pure file-writing server:
# (this is independent from the connection type, like XMLRPC)
from rrlog.server import filewriter
logServer = filewriter.createRotatingServer(
filePathPattern = "./demo-log-%s.txt", # "pattern" because %s (or %d) is required for the rotate-number
rotateCount=3,
rotateLineMin=10,
)
# Start the server as an XMLRPC server:
from rrlog.server import socketserver
# with no ports/host given, default port and "localhost" are used
socketserver.startServer(
logServer,
ports=(9801, 9802, 9803), # try in this order, use the first port available
)
# Now, the server should be waiting for requests.
# First, create a pure database-writing server:
# (this is independent from the remote connection protocol)
from rrlog.server import dbwriter_sa
engineStr = "mysql://logtester@localhost/logtest"
logServer = dbwriter_sa.createRotatingServer(
engineStr = engineStr,
tableNamePattern = "logtable_%s", # "pattern" because %s (or %d) is required for the rotate-number
rotateCount=3,
rotateLineMin=10,
tsFormat="std1", # Timestamp format: std1 is shorthand for the strftime-format "%H:%M.%S;%3N"
)
# Start the server as a pure socket server:
from rrlog.server import socketserver
socketserver.startServer(
logServer,
ports=(9801, 9802, 9803), # try in this order, use the first port available
)
# Now, the server should be waiting for requests.
from rrlog import socketclient
# with no ports/host given, default port and "localhost" are used
log = socketclient.createClientLog(
errorHandler="stderr",
ports = (9801,9802,9803,) # use the first port where a server is available
)
# now, log some lines:
for i in xrange(31):
log("Socket-client line %s"%(i))
Remote logging is intended to log on a remote machine. Or, which might be a very common use, to log from multiple processes on the same machine into one logfile/table.
This requires two create* calls. One makes the log server, one makes the client in your application. Most parameters are now found in the server create* function, e.g. the rotation configuration.
Host and Ports
By default, the connection uses “localhost” and a default port → rrlog.globalconst.DEFAULTPORT_XMLRPC
You can specify 1..n ports on both sides. The server uses the first free port. The client uses the first port where a server seems available.
# First, create a pure print server:
# (this is independent from the connection type, like XMLRPC)
from rrlog.server import printwriter
logServer = printwriter.createServer()
# Start the server as an XMLRPC server:
from rrlog.server import xmlrpc
xmlrpc.startServer(
logServer,
ports=(9804,9805,9806,), # try in this order, use the first port available
)
# Now, the server should be waiting for requests.
from rrlog.server import filewriter
logServer = filewriter.createRotatingServer(
filePathPattern = "./demo-log-%s.txt", # "pattern" because %s (or %d) is required for the rotate-number
rotateCount=3,
rotateLineMin=10,
)
# Start the server as an XMLRPC server:
from rrlog.server import xmlrpc
xmlrpc.startServer(
logServer,
ports=(9804,9805,9806,), # try in this order, use the first port available
)
# Now, the server should be waiting for requests.
# First, create a pure database-writing server:
# (this is independent from the connection type, like XMLRPC)
from rrlog.server import dbwriter_sa
engineStr = "mysql://logtester@localhost/logtest"
logServer = dbwriter_sa.createRotatingServer(
engineStr = engineStr,
tableNamePattern = "logtable_%s", # "pattern" because %s (or %d) is required for the rotate-number
rotateCount=3,
rotateLineMin=10,
tsFormat="std1", # Timestamp format: std1 is shorthand for the strftime-format "%H:%M.%S;%3N"
)
# Start the server as an XMLRPC server:
from rrlog.server import xmlrpc
xmlrpc.startServer(
logServer,
ports=(9804,9805,9806,), # try in this order, use the first port available
)
# Now, the server should be waiting for requests.
from rrlog import xmlrpc
log = xmlrpc.createClientLog(
ports = (9804,9805,9806,) # use the first port where a server is available
)
# now, log some lines:
for i in xrange(33):
log("Hello xmlrpc - line %s"%(i))
Care for the correct server running ! For example, a socket server erroneously waiting on that port can cause the XMLRPC Client to block stupidly and wait without any Error message.
Note: Integration of standard logging is incomplete. In particular, we have to log already formatted strings - no separate arguments for string formatting yet.
Assume you have a log object. You can register it as a handler, and then use the Python logging system as usual.
import logging
logger = logging.getLogger("Demo")
logger.setLevel(logging.WARNING)
logger.addHandler(log.logging23_handler())
# standard logging calls should work now:
for i in xrange(7):
logger.info("That's an info via standard logging #%s"%(i)) # omitted !
logger.warn("That's a warning via standard logging #%s"%(i))
Now it should already work. But when you log stack traces, they look ugly. Each call path is dominated by the Python logging framework, especially its __init__ method.
→ Shorten stack paths to make the stack path look better.
no custom categories anymore
A drawback of the standard logging module is that we loose our custom message categories.
We have some LEVELs instead; these are mapped to our category values in a LEVELMAP in rrlog.logging23. By default, the error level is mapped to “E” and so on. You can replace that map with your own.
While we have no custom categories anymore, the logging package, on the other hand, gives us a nice way to register separate handlers.