server Package

server Package

Log Server, Part of the log which can (optionally) be in a separate process. @author: Ruben Reifenberg

exception rrlog.server.ColumnConfigurationMismatch

Bases: exceptions.Exception

A configured column name is impossible because already reserved.

class rrlog.server.LogServer(writer, filters=None, observers=None, cfnMode=1, tsFormat=None, jobhistSize=100)

Bases: object

This can (but must not be) in the application process. @cvar CAT_INTERNALERROR: Default “I”. This is used as “cat” when I write a log message about a logging failure (if possible)

CAT_INTERNALERROR = 'I'

CFN_FULL = 2

CFN_SHORT = 1

__init__(writer, filters=None, observers=None, cfnMode=1, tsFormat=None, jobhistSize=100)

Parameters:

• cfnMode –

  One of the MODE...constants.

  CFN_SHORT: Caller File Names minimal,i.e.file name without directory path(=Default) CFN_FULL: Caller File Names with full file path) Default is CFN_SHORT

• tsFormat –

  strftime-format string with an extension: use %3N for milliseconds.

  [Year..second are: %Y/%y,%m,%d,%H,%M,%S] None for no timestamps. Standard Values for convenience: The String “std1” is interpreted as “%H:%M.%S;%3N”, e.g. 13:59.59;999 The String “std2” is interpreted as “%m/%d %H:%M.%S”

• observers –

  list of callables, called with args: jobhist, writer.

  Called each time after(!) a message was written. writer is the specific writer (depends on DB/file/Stdout modus.) jobhist are the N last message-jobs, with the latests at [-1]. The size N of the jobhist is limited by jobhistSize. Increase jobhistSize if your observer needs to see a large job history. The observers are processed in the given order. You can modify job content for following observers, without affecting the log (since the observers are called after the line is written.) For compatibility: An observer can also have an observe() method. If available, this is used (instead of __call__)

• filters –

  list of callable objects, analogous to observers.

  But filters are called before logging. When they modify the message, the change gets visible in the log.

• jobhistSize – The count of recent messages that are available as a list (these may be read by observers). Default=100

Raises:

• AssertionError – if an observer is >1 times in the list
• AssertionError – if a filter is >1 times in the list

_timeStr(dt)

Return type:str Returns:readable time info, based on my _tsFormat

addClient()

addObserver(observer)

appends the observer at the end of the observers list

format_fname(name)

file name in the configured format (cfnMode).

Returns:String,men-Readable Callers File Name (None is name was None) Parameters:name – File file name incl. path, or None

log(jobdata)

Parameters:jobdata – Internal format. Do not rely on that structure since it will probably remain subject of refactorings.

logJob(job)

pathAsStr(path, imin=0)

Parameters:path –

iterable

imin: min.index of path to use.

Returns:call path, formatted as str, Empty str if path is empty Return type:str

class rrlog.server.MsgJob(msgid, pid, tid, threadname, ts, msg, cat, path, tblen, cfunc, special, formatter)

Bases: object

@ivar threadname: part of client identification @ivar msgid: int, id of the message. Unique in the client. @ivar msg: str, msg as created by the client. @ivar ts: str, timestamp @ivar special: dict with custom items (see “special” argument of the log method) @ivar tblen: len of the client traceback when the log method was called @ivar path: client traceback path as sequence of (filename, linenumber). [0] is the latest (where the log call happened)

__init__(msgid, pid, tid, threadname, ts, msg, cat, path, tblen, cfunc, special, formatter)

Parameters:

• special – data for custom observers only
• (str) (ts) – timestamp

cfn()

Return type:str Returns:Callers File Name

cln()

Return type:int Returns:Callers Line Number

copy_update(kwargs)

Returns:new instance with my init kwargs but updated with the given kwargs

getFormattedDict()

DEPRECATED: use format_dict of the DBLogWriter :rtype: dict :returns: already formatted data

pathStr(imin=0)

Parameters:imin –

min.index of path to use.

Fo example, imin==1 will skip the first path item (Note: the first item is also available separately as cfn,cln)

Return type:str Returns:path as formatted str

exception rrlog.server.ObserverAlreadyAdded

Bases: exceptions.Exception

Attempt to add an observer twice into the observer list

class rrlog.server.RotateLogWriter(getNextWriter, rotateLineMin)

Bases: object

Assigned to a list of LogWriters, rotates by creating a new one each time when a line count is exceeded. Maintains a history of old writers. @ivar writers: History of writers, current at [-1], oldest at [0]. Only writers with an existing (i.e.not already overwritten) table/file are available. (Migration note: This is analogous to the getWriteHistory() of version 0.1.1 but the order is ascending.)

__init__(getNextWriter, rotateLineMin)

Parameters:

• getNextWriter – Callable that takes a list (history) of previous writers, and returns the next logWriter to use
• rotateLineMin – rotate when ~ lines are written

_rotate()

Not threadsafe. Maintains the history (self.writers). A new writer is appended.. removes the oldest writer [0] if self._writer count is longer than self._configs.

writeNow(logrecord)

Write without buffering, return when written

class rrlog.server.RotateWriterFactory(configs, writerFactory)

Bases: object

__init__(configs, writerFactory)

nextWriter(history)

Was RENAMED from next(), otherwise Python3 conversion failure. :param history: list of old writers to be maintained, oldest at [0]

Invalid (expired) writers are removed!

dbwriter_sa Module

@summary: DB Writing via SQLAlchemy (SA). @author: Ruben Reifenberg

class rrlog.server.dbwriter_sa.DBConfig(engineStr, tablename, drop=True, cols=None)

Bases: rrlog.server.dbwriter_sa._Coltypes

COLS_DEFAULT = (('pid', <class 'sqlalchemy.types.Integer'>), ('threadname', String(length=32)), ('msgid', <class 'sqlalchemy.types.Integer'>), ('ts', String(length=32)), ('cat', String(length=1)), ('msg', String(length=512)), ('cfunc', String(length=32)), ('cfn', String(length=32)), ('cln', <class 'sqlalchemy.types.Integer'>), ('path', String(length=512)))

__init__(engineStr, tablename, drop=True, cols=None)

Parameters:

• engineStr – SQLAlchemy engine str
• tablename – str
• drop – If False, an eventually existing table is not deleted but extended. Only true makes sense with rotation.
• cols –

  All log table column names and types.

  default=None. If None, the DBConfig.COLS_DEFAULT is used.

  Use the COLS_DEFAULT as a base for you own column configuration. This is a 3-tuple of (col-name:str,col-type,kwargs:dict) where: col-name is the desired DB column. This name can be used as kwarg in the log() calls col-type is DBConfig.Integer or DBConfig.String kwargs (optional) is for the sqlalchemy Column.

  Example: To add an own integer column, take the default columns, and add your own pair of (column-name,column-type) like that: cols=DBConfig.COLS_DEFAULT + ((“mycolumn”,DBConfig.Integer)) To define you column as primary key, use cols=DBConfig.COLS_DEFAULT + ((“mycolumn”,DBConfig.Integer,{“primary_key”:True}))

class rrlog.server.dbwriter_sa.DBLogWriter(config, format_dict=None)

Bases: object

USes SQLAlchemy (sa), Assigned to 1 Table

__init__(config, format_dict=None)

Parameters:config – DBConfig

_format_dict(job)

Default formatting method.

Return type:{} Returns:{colname:fieldcontent} for a single row

close()

static createColumns(colsConfig)

There is no primary key column; these are content columns only.

Returns:[] of SQLAlchemy Column that make up my log table Parameters:colsConfig –

(col-name:str,col-type,kwargs:dict) where

col-type is DBConfig.Integer or DBConfig.String kwargs is optional and contains kwargs for Column() of sqlalchemy example: (“ipadress”,DBConfig.String,{“default”:”127.0.0.1”})

estimateLineCount()

For performance reasons, it is allowed to estimate instead count exactly. (Remark: This implementation is working exactly.)

Returns:count of already written lines

getTable()

Return type:SQLAlchemy Table

writeNow(job)

Write without buffering, return when written

rrlog.server.dbwriter_sa.LOGWRITER_CLASS

alias of DBLogWriter

class rrlog.server.dbwriter_sa._Coltypes

Bases: object

Column types

class Integer(*args, **kwargs)

Bases: sqlalchemy.types._DateAffinity, sqlalchemy.types.TypeEngine

A type for int integers.

_compiler_dispatch(visitor, **kw)

Look for an attribute named “visit_” + self.__visit_name__ on the visitor, and call it with the same kw params.

_expression_adaptations

get_dbapi_type(dbapi)

python_type

class _Coltypes.String(length=None, collation=None, convert_unicode=False, unicode_error=None, _warn_on_bytestring=False)

Bases: sqlalchemy.types.Concatenable, sqlalchemy.types.TypeEngine

The base for all string and character types.

In SQL, corresponds to VARCHAR. Can also take Python unicode objects and encode to the database’s encoding in bind params (and the reverse for result sets.)

The length field is usually required when the String type is used within a CREATE TABLE statement, as VARCHAR requires a length on most databases.

__init__(length=None, collation=None, convert_unicode=False, unicode_error=None, _warn_on_bytestring=False)

Create a string-holding type.

Parameters:

• length – optional, a length for the column for use in DDL and CAST expressions. May be safely omitted if no CREATE TABLE will be issued. Certain databases may require a length for use in DDL, and will raise an exception when the CREATE TABLE DDL is issued if a VARCHAR with no length is included. Whether the value is interpreted as bytes or characters is database specific.
• collation –

  Optional, a column-level collation for use in DDL and CAST expressions. Renders using the COLLATE keyword supported by SQLite, MySQL, and Postgresql. E.g.:

  >>> from sqlalchemy import cast, select, String
  >>> print select([cast('some string', String(collation='utf8'))])
  SELECT CAST(:param_1 AS VARCHAR COLLATE utf8) AS anon_1
  

  New in version 0.8: Added support for COLLATE to all string types.

• convert_unicode –

  When set to True, the String type will assume that input is to be passed as Python unicode objects, and results returned as Python unicode objects. If the DBAPI in use does not support Python unicode (which is fewer and fewer these days), SQLAlchemy will encode/decode the value, using the value of the encoding parameter passed to create_engine() as the encoding.

  When using a DBAPI that natively supports Python unicode objects, this flag generally does not need to be set. For columns that are explicitly intended to store non-ASCII data, the Unicode or UnicodeText types should be used regardless, which feature the same behavior of convert_unicode but also indicate an underlying column type that directly supports unicode, such as NVARCHAR.

  For the extremely rare case that Python unicode is to be encoded/decoded by SQLAlchemy on a backend that does natively support Python unicode, the value force can be passed here which will cause SQLAlchemy’s encode/decode services to be used unconditionally.

• unicode_error – Optional, a method to use to handle Unicode conversion errors. Behaves like the errors keyword argument to the standard library’s string.decode() functions. This flag requires that convert_unicode is set to force - otherwise, SQLAlchemy is not guaranteed to handle the task of unicode conversion. Note that this flag adds significant performance overhead to row-fetching operations for backends that already return unicode objects natively (which most DBAPIs do). This flag should only be used as a last resort for reading strings from a column with varied or corrupted encodings.

_compiler_dispatch(visitor, **kw)

Look for an attribute named “visit_” + self.__visit_name__ on the visitor, and call it with the same kw params.

bind_processor(dialect)

get_dbapi_type(dbapi)

python_type

result_processor(dialect, coltype)

rrlog.server.dbwriter_sa.createLocalLog(engineStr, rotateCount, rotateLineMin, traceOffset=0, tableNamePattern='log_%s', filters=None, observers=None, tsFormat='std1', stackMax=5, drop=True, catsEnable=None, catsDisable=None, seFilesExclude=None, name=None, cols=None, extractStack=True)

Parameters:

• catsEnable – see L{rrlog.Log.__init__}
• catsDisable – see L{rrlog.Log.__init__}
• seFilesExclude – see L{rrlog.Log.__init__}
• filters – see L{rrlog.server.LogServer.__init__}
• observers – see L{rrlog.server.LogServer.__init__}
• tsFormat – timestamp format. See L{rrlog.server.LogServer.__init__}
• tableNamePattern – table name incl.placeholder for int (rotate number), e.g. “mylog%d”
• rotateCount – int >>1, how many tables to use for rotation
• rotateLineMin – rotate when ~ lines are written. None to switch off rotation.
• stackMax – see L{rrlog.Log.__init__}, default: 5 (==log 5 stack levels.)
• drop – if True, drop an eventually existing table. When False, append to an existing one (this is possible with rotateCount==1 only).
• extractStack – see L{rrlog.Log.__init__}
• cols – Custom column configuration. See L{rrlog.server.dbwriter_sa.DBConfig.__init__}

Returns:

callable log object, ready to use

rrlog.server.dbwriter_sa.createRotatingServer(engineStr, rotateCount, rotateLineMin, tableNamePattern='log_%s', tsFormat=None, filters=None, observers=None, logwriterFactory=None, drop=True, cols=None, format_dict=None)

Parameters:

• engineStr – SQLAlchemy engine str
• tableNamePattern – table name incl.placeholder for int (rotate number), e.g. “mylog%d”
• filters – default = () See L{rrlog.server.LogServer.__init__}
• observers – default = () See L{rrlog.server.LogServer.__init__}
• tsFormat – timestamp format. See L{rrlog.server.LogServer.__init__}
• rotateCount – int >=1, how many tables to use for rotation
• rotateLineMin – rotate when ~ lines are written. None to switch off rotation.
• logwriterFactory – Creates LogWriter instances (one per db table). If None, the module variable LOGWRITER_CLASS is used.
• drop – if True, drop an eventually existing table. When False, append to an existing one (this is possible with rotateCount==1 only).
• cols – Custom column configuration. See L{rrlog.server.dbwriter_sa.DBConfig.__init__}
• format_dict – callable taking a job and returning {colname:fieldcontent}. Ignored if a logwriterFactory is given.

Raises AssertionError:  

when drop==False and rotateCount > 1

rrlog.server.dbwriter_sa.default_format_dict(job)

Format the row fields for writing. :returns: {colname:fieldcontent} as to be written into the database row.

filewriter Module

@summary: logging into text files. @author: Ruben Reifenberg

class rrlog.server.filewriter.FileClosedEvent(filePath, lineCount)

Bases: object

The specified log file was closed.

__init__(filePath, lineCount)

Parameters:

• filePath – full log filename incl.path
• lineCount – line count of the closed file

class rrlog.server.filewriter.FileConfig(filepath, drop=True, lazy=True)

Bases: object

Describes a single log file. I.e.for log rotation, a sequence of this is required.

__init__(filepath, drop=True, lazy=True)

Parameters:

• drop – False: Append lines to existing log file (This makes no sense when using file rotation.) True: A new Writer clears the existing logfile.
• filepath – full log filename incl.path
• lazy – If True, file is opened with first log entry (default) If False, file is opened immediately. Use False only if you know hat you’re doing. With socket server, it may result in data loss: the first file is opened immediately before the worker thread starts. Then, the worker thread uses the already opened file which fails. (Another solution for that case might be always close/re-open the file fo reach written line.)

class rrlog.server.filewriter.FileLogWriter(config, format_line=None, fileClosed=None)

Bases: rrlog.server.textwriter.TextlineLogWriter

__init__(config, format_line=None, fileClosed=None)

Parameters:config – FileConfig

_createFile()

close()

writeNow(job)

Write without buffering, return when written. Flushes the file after each write.

rrlog.server.filewriter.LOGWRITER_CLASS

alias of FileLogWriter

rrlog.server.filewriter.createLocalLog(filePathPattern, rotateCount, rotateLineMin, filters=None, observers=None, traceOffset=0, tsFormat='std1', stackMax=5, drop=True, catsEnable=None, catsDisable=None, seFilesExclude=None, format_line=None, name=None, extractStack=True, fileClosed=None)

Parameters:

• catsEnable – see L{rrlog.Log.__init__}
• catsDisable – see L{rrlog.Log.__init__}
• seFilesExclude – see L{rrlog.Log.__init__}
• filters – see L{rrlog.server.LogServer.__init__}
• observers – see L{rrlog.server.LogServer.__init__}
• tsFormat – timestamp format. See L{rrlog.server.LogServer.__init__}
• filePathPattern – full log filename incl.path and placeholder for an int (rotate number). E.g.”./mylog%d.txt”
• rotateCount – int >>1, how many files to use for rotation
• rotateLineMin – rotate when ~ lines are written. None to switch off rotation.
• stackMax – see L{rrlog.Log.__init__}, default: 5 (==log 5 stack levels.)
• drop – if True, drop an eventually existing file. When False, append to an existing one (this is possible with rotateCount==1 only).
• format_line – See L{rrlog.server.textwriter.TextlineLogWriter.__init__}
• name – The log can be identified by its optional name attribute (__repr__ method of the log will use it.)
• extractStack – see L{rrlog.Log.__init__}
• fileClosed – Experimental, undocumented. Use case: Zip a log file after rotation. Parameter May change.

Returns:

a Log ready to use

rrlog.server.filewriter.createRotatingServer(filePathPattern, rotateCount, rotateLineMin, tsFormat='std1', filters=None, observers=None, logwriterFactory=None, drop=True, format_line=None, fileClosed=None)

Parameters:

• filePathPattern – full log filename incl.path and placeholder for an int (rotate number). E.g.”./mylog%d.txt”
• tsFormat – timestamp format. See L{rrlog.server.LogServer.__init__}
• rotateCount – int >>1, how many files to use for rotation
• rotateLineMin – rotate when ~ lines are written. None to switch off rotation.
• filters – default = () See L{rrlog.server.LogServer.__init__}
• observers – default = () See L{rrlog.server.LogServer.__init__}
• logwriterFactory – Creates LogWriter instances (one per file). If None, the module variable LOGWRITER_CLASS is used.
• format_line – See L{rrlog.server.textwriter.TextlineLogWriter.__init__}
• fileClosed – Experimental, undocumented. Use case: Zip a log file after rotation. Parameter May change.

printwriter Module

@summary: Log to stdout. @author: Ruben Reifenberg

rrlog.server.printwriter.LOGWRITER_CLASS

alias of PrintLogWriter

class rrlog.server.printwriter.PrintLogWriter(writeMsgid=False, format_line=None)

Bases: rrlog.server.textwriter.TextlineLogWriter

__init__(writeMsgid=False, format_line=None)

_format_line(job)

Default formatting method.

Return type:str Returns:single log line

close()

writeNow(job)

Write without buffering, return when written

rrlog.server.printwriter.createLocalLog(writer=None, filters=None, observers=None, traceOffset=0, tsFormat=None, writeMsgid=False, stackMax=1, catsEnable=None, catsDisable=None, seFilesExclude=None, format_line=None, name=None, extractStack=True)

Parameters:

• catsEnable – see L{rrlog.Log.__init__}
• catsDisable – see L{rrlog.Log.__init__}
• seFilesExclude – see L{rrlog.Log.__init__}
• filters – see L{rrlog.server.LogServer.__init__}
• observers – see L{rrlog.server.LogServer.__init__}
• tsFormat –

  timestamp format. See L{rrlog.server.LogServer.__init__}

  Here, the default is None (shows no time stamps)

• stackMax – see L{rrlog.Log.__init__}, default: 5 (==log 5 stack levels.)
• format_line – See L{rrlog.server.textwriter.TextlineLogWriter.__init__}
• name – The log can be identified by its optional name attribute (__repr__ method of the log will use it.)
• extractStack – see L{rrlog.Log.__init__}

Returns:

a Log ready to use

rrlog.server.printwriter.createServer(writer=None, tsFormat=None, filters=None, observers=None, writeMsgid=False, format_line=None)

Parameters:

• tsFormat – timestamp format. See L{rrlog.server.LogServer.__init__}
• filters – default = () See L{rrlog.server.LogServer.__init__}
• observers – default = () See L{rrlog.server.LogServer.__init__}
• format_line – See L{rrlog.server.textwriter.TextlineLogWriter.__init__}

rrlog.server.printwriter.print_()

write(str) -> None. Write string str to file.

Note that due to buffering, flush() or close() may be needed before the file on disk reflects the data written.

socketserver Module

Besides the thread(s) reading the socket, there is a single worker thread which feeds the log server. To stop the threaded server process (Ctrl-C prbably won’t work) use the specified module variables to regularly end it without log line loss, or kill the process.

class rrlog.server.socketserver.LogRecordSocketReceiver(host, ports, handler=<class rrlog.server.socketserver.LogRecordStreamHandler at 0x42bee88>)

Bases: SocketServer.TCPServer

simple TCP socket-based logging receiver suitable for testing.

__init__(host, ports, handler=<class rrlog.server.socketserver.LogRecordStreamHandler at 0x42bee88>)

allow_reuse_address = 1

serve_until_stopped()

class rrlog.server.socketserver.LogRecordStreamHandler(request, client_address, server)

Bases: SocketServer.StreamRequestHandler

Handler for a streaming logging request.

This basically logs the record using whatever logging policy is configured locally.

handle()

Handle multiple requests - each expected to be a 4-byte length, followed by the LogRecord in pickle format. Logs the record according to whatever policy is configured locally.

rrlog.server.socketserver._i_am_orphan()

rrlog.server.socketserver.processq()

Loop which forever pops the oldest (left) element from the jobdataq and calls the log server. The global variable processq_stop=True ends the loop, but not before the jobdataq is found empty.

rrlog.server.socketserver.startServer(logServer, host='localhost', ports=(9801, ))

Run the given logServer as an xmlrpc server (forever). :param ports: sequence of portnumbers, at least one number. The first port available is used.

Multiple ports is for development, where sometimes ports may remain blocked. In production, better use a single port only, for best control over which server/client pairs are married.

textwriter Module

@summary: Abstract base for text line logging. @author: Ruben Reifenberg

class rrlog.server.textwriter.Formatter(fmt=None, datefmt=None)

Bases: object

This resembles the L{logging.__init__.Formatter} of the Python Logging Package by using the same __init__ arguments. That allows conveniently replacing a formatter object of “one world” (standard logging resp. rrlog) with a formatter of the “other world”.

__call__(job, lineCount)

Returns:formatting single-line string

__init__(fmt=None, datefmt=None)

Initialize the formatter with specified format strings.

Initialize the formatter either with the specified format string, or a default as described above. Allow for specialized date formatting with the optional datefmt argument (if omitted, you get the ISO8601 format).

class rrlog.server.textwriter.TextlineLogWriter(format_line=None)

Bases: object

Abstract base class for text line writers

__init__(format_line=None)

Parameters:format_line – Callable, takes a job and returns a single message line as str.

See L{MsgJob} for job attributes available. See L{textwriter.Formatter.__call__} for an example.

cfn_cln(job)

Convenient string for callers file name / callers line number

Return type:str Returns:“*<cfn>(<cln>)”, “” if these data are not available (i.e. stack logging disabled)

estimateLineCount()

For performance reasons, it is allowed to estimate instead count exactly. (Intended for implementations which need to read access written data to count; but not required here, we are simply counting exactly.)

Returns:count of already written log lines

rrlog.server.textwriter.tm_structconverter(secs, to_structtime=<built-in function localtime>)

Parameters:secs – float, as returned by time.time() Returns:(struct_time,milliseconds)

where struct_time is a tuple as returned by time.localtime, milliseconds is an int >=0 and <1000

xmlrpc Module

@summary: The XML-RPC version of a LogServer connection. This is a slow way to log (mainly because of the unnecessary overhead of xml-rpc blocking calls), but simple and robust. Use it when the performance is good enough for you. @author: Ruben Reifenberg

class rrlog.server.xmlrpc.LogAdapter(logServer)

Bases: object

__init__(logServer)

Returns:“” if all right, error msg if not

addClient()

Return type:int Returns:unique id

log(logdata_ps)

No threading, xmlrpc blocks until return

rrlog.server.xmlrpc.createSimpleXMLRPCServer(host, ports)

Creates server that does no multithreading. Remark: The SimpleXMLRPCServer seems to do no multithreading, except we use MixIn class. Indeed, I found that a second client is blocked as long as the server is processing another request. (Python 2.4) :param ports: list or iterator of port numbers Multiple ports address the problem that a socket is for some time “already in use” when restarting the server. The first free port of the ports is used.

rrlog.server.xmlrpc.startServer(logServer, host='localhost', ports=(9804, ), readyMsg=True)

Run the given logServer as an xmlrpc server (forever). :param ports: see L{createSimpleXMLRPCServer}