RL01 Pattern Log Reader

Outline

With the pattern log reader you can read line based logs that contain multiple line patterns. Using regular expressions, all relevant information can be extracted and written to one or more log signals. The reader is supplied with predefined Log4J and syslog configurations. A powerful UI helps to define and extend reader configurations.

Platforms:

32/64bit

Requirements:

None

Known limitations:

File size limited by virtual memory and temporary storage

Status:

Stable

Operations:

Load
Stream

Parameters:

None

Configuration:

A Pattern Log reader configuration is required. The configuration defines the structure of the Log input (regular expression, data types and names, domain, ...).

Tutorials:

About pattern logs

In most cases, developers have text log outputs. In the good case, all log line share the same format, but often there is a strange mix of multiple line pattern due to historical reasons.

Up to a certain level, impulse can parse these mixed logs and present and analyze them in a uniform way.
To achieve this, the user can define a set of pattern. Each pattern describes the format and content of one line type.

Here a typical log example:

NOTE     [212 745 133.974 ns] in top.mDist_dsp_access.r2_op : getX4LegChangeDone happened
WARNING  [212 745 133.974 ns] in top.mDist_dsp_access.r4_op : getX4LegChangeDone
ISR at 214435176
ERROR    [214 435 176.000 ns] in top.mDist_dsp_wait : Kernel changed state to: OUTPUT
# Leg ok - wait for next
NOTE00000[216 735 876.000 ns] in top.F4.generics : Calculated load 47%

Usage

The reader can be used to open workspace resources and together with ports (direct connection to the target using TCP, Serial, J-Link, ...).

Content type association

To use a reader for a given input resource,

a valid content type association needs to be in place,
and the reader need to identify the file resource content as valid data (Content detection).

The serializer dialog contains the field "Content type association" with which you can manage file associations (e.g. to add "*.log" as a valid file name).

Content type association is only meaningful for eclipse resources, not for port inputs like TCP oder File/Pipe ports.

Content Detection and Configurations

This reader requires a reader configuration to work. Such a configuration defines the form of the input content and how it shall be read.

Typically a reader has several configurations that can be enabled or disabled.

The reader does a content type detection to:

identify if the content is valid and matches a given configuration,
or to select a valid configuration for the given input, if not done explicitly by the user ("Default" selection).

A file resource has valid content data if at least 1 enabled configuration matches.

Configuration

The Pattern Log Reader requires a configuration (Pattern Log Configuration) to work. Each configuration contains a set of log pattern. Each log pattern describes the form and content of a single log line.

Click the "Add" button to add a new "Pattern Log Configuration"" and fill in the following parameters:

Char Set: Select the char set that is used by the log file (or "Default").
Domain Base: Select the domain base. This represents is smallest domain change (e.g. 1 ms).
Use relative domain value: Check to convert absolute domain values of the input into relative one (relative to the first domain position).
Add log file position: if checked, the reader will add the line position to each log samples ('RecPos' member). You can later navigate from the value table to the log file position.
Content detection: This setting defines one of your log patterns as the one used for content detection. If the specified pattern matches one of the first lines of an input, the configuration is considered valid. You can also specify how many bytes (from the beginning of the input) to use for content detection. A log pattern matches if the regular expression matches the input line.
Updates: This reader has some sample or standard configurations after installation. After an update of impulse, these configurations may also have been updated. To exclude a configuration from updating (e.g. because you have adapted the configuration for your application), you can set the flag "Do not update with latest version".

A log pattern describes the form and content of a single log line.

You may define multiple log pattern:

if you may have different log styles from different source or due to historical reasons,
or to combine multiple log lines into one log event (multi-line log pattern),
or to ignore certain lines. It is often useful to add a log pattern with the regular expression ".*" and the action "Ignore" on lowest priority to ignore all lines that do not match a valid log line and thus avoid parse errors.

When reading an input, the reader takes the first log pattern that matches. Use the buttons on the right side to re-order the priority of your log pattern. If no pattern matches a line, a parse error will be thrown.

Log Pattern

Press "Add" and select "Log Pattern" to create a new log pattern.

A log pattern describes the form and content of one single log line. Regular expressions can be used not only to match text, but also to extract information for further processing. This is done by defining groups of characters and capturing them using the special parenthesis of ( and ) meta characters. Here an example:

(WARNING|NOTE|ERROR|DEBUG)[\s0]*\[([ 0-9a-z\.]+)\]\s*in\s*([^:]*):\s*(.*)

(WARNING|NOTE|ERROR|DEBUG): Group 1: log level - one of these options
[\s0]*: optional white-spaces and zeros
\[([ 0-9a-z\.]+)\]: Group 2: time stamp
\s*: optional white-spaces
in\s*([^:]*): Group 3: logger (location) - starts at 'in' and ends at ':'
\s*: ':' and optional white-spaces
(.*): Group 4: text - remaining line

Create a Log4J 1.2 log pattern

To create an initial Log4J 1.2 log pattern, press "Add" and select "Log Pattern for Log4j 1.2".

Unfortunately there is no 100% pattern conversion possible. You probably have to modify or extend the resulting pattern.

Define the regular expression

The dialogue starts with the following fields:

Pattern: Define the regular expression for the lines. Use the context proposals (press control-space). For more information about java regular expressions see http://docs.oracle.com/javase/tutorial/essential/regex/ Each defined regular expression group (1..N) can be used as input for the event sample.
Test Lines: You may add a few lines of your log file. Press "Test" to validate against your expression and print the matching log line contents for each group.

Construct	Matches
Characters
x	The character x
`\\`	The backslash character
`\t`	The tab character (`'\u0009'`)
`\n`	The newline (line feed) character (`'\u000A'`)
`\r`	The carriage-return character (`'\u000D'`)
Character classes
`[abc]`	`a`, `b`, or `c` (simple class)
`[^abc]`	Any character except `a`, `b`, or `c` (negation)
`[a-zA-Z]`	`a` through `z` or `A` through `Z`, inclusive (range)
Predefined character classes
`.`	Any character
`\d`	A digit: `[0-9]`
`\D`	A non-digit: `[^0-9]`
`\s`	A whitespace character: `[ \t\n\x0B\f\r]`
`\S`	A non-whitespace character: `[^\s]`
`\w`	A word character: `[a-zA-Z_0-9]`
`\W`	A non-word character: `[^\w]`
Boundary matchers
`^`	The beginning of a line
`$`	The end of a line
`\b`	A word boundary
`\B`	A non-word boundary
Greedy quantifiers
X `?`	X, once or not at all
X `*`	X, zero or more times
X `+`	X, one or more times
X `{`n `}`	X, exactly n times
X `{`n `,}`	X, at least n times
X `{`n `,`m `}`	X, at least n but not more than m times
Reluctant quantifiers
X `??`	X, once or not at all
X `*?`	X, zero or more times
X `+?`	X, one or more times
X `{`n `}?`	X, exactly n times
X `{`n `,}?`	X, at least n times
X `{`n `,`m `}?`	X, at least n but not more than m times
Possessive quantifiers
X `?+`	X, once or not at all
X `*+`	X, zero or more times
X `++`	X, one or more times
X `{`n `}+`	X, exactly n times
X `{`n `,}+`	X, at least n times
X `{`n `,`m `}+`	X, at least n but not more than m times
Logical operators
XY	X followed by Y
X `\|`Y	Either X or Y
`(`X `)`	X, as a >capturing group

After this first step is done, you need to define how to use the pattern and its extracted groups.

Select the action

Ignore : Just skip the line;
Start new log sample;
Add to previous sample: The information of the groups will be added to the previously started sample (to handle multi-line pattern). A new sample will be started if no previous is available;
Finish sample: The information of the groups will be added to the previously started sample. Afterwards the sample will be finished (Additional lines can not be added).

Name and configure the sample members

Log signals are struct signal with multiple members.

Select member names and types for all groups that shall have a member in you log samples (e.g. the message).

Select the member names. You can select a predefined member names or enter custom ones.
For custom members you can define the member type and a descriptor. If custom members are used over multiple pattern, the type need to be identical.

Valid member types are:

Float,
Integer,
Text ,
or Enumeration.

The descriptor field can optionally contain a content or member descriptor:

a simple content descriptor e.g. "default", "state", event" or "label", ...
a complete member descriptor e.g. "default<df=Hex>" or "state<df=Text>"

Define the signal naming scheme

Each single sample belongs to one signal. You can organize your samples into one or multiple signals (e.g. a signal per log source). The "Signal/Scope name" section allows to select the signal and scope naming scheme for each log pattern.

Undefined : Either all log samples get into one signal with the name "log" -- or this log pattern is part of a multi-line pattern and the name information is not part of this line.
Explicit name : Define a signal name for all samples of this log pattern.
Explicit hierarchy : Define a signal/scope hierarchy for all samples of this log pattern. The signal and scope names are generated by splitting the value into its contents (e.g. the value "de.toem.logger" creates the scope/signal hierarchy de/toem/logger).
Name from source value : The value of the selected group is taken to define the signal name (.e.g. from the logger group).
Hierarchy from source value : The value of the selected group is taken to define the signal and scope names by splitting the value into its contents (e.g. the value "de.toem.logger" creates the scope/signal hierarchy de/toem/logger).

You may extend the naming scheme with another group value. If enabled, the value of the group will be put after the signal name (in parentheses).

Configure the domain value source and format (usually the time-stamp)

Undefined : In case of multi-line pattern, you may have lines without domain information.
Date: You need to define the date format. Use the content proposals and http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html. The parsing of the date value results into milliseconds after January 1, 1970 00:00:00 GMT.
Float domain value: Parse a float value, optionally parse the domain unit from the group value or assign a unit.
Integer domain value: Parse an integer value, optionally parse the domain unit from the group value or assign a unit.

There may be log lines without a domain value (single line logs). In this case you may want to use:

Same as previous: Use the domain value of the previous sample (all signals).
Same as previous (per Signal): Use the domain value of the previous sample of the used signal.
Incrementing: Increment the domain value with every new sample (all signals).
Incrementing (per Signal): Increment the domain value with every new sample per signals.
Reception time: Use the current time in milliseconds after January 1, 1970 00:00:00 GMT as domain value.

You may extend the domain position with another source value (e.g one group contains the seconds, another nanoseconds):

Undefined : Do not extend.
Float domain value: Parse a float value, optionally parse the domain unit from the source value or assign a unit. The domain extension value will be added to the main domain position.
Integer domain value: Parse an integer value, optionally parse the domain unit from the source value or assign a unit. The domain extension value will be added to the main domain position.

Date formats are specified by date and time pattern strings:

Letter Date or Time Component Presentation Examples

`G`	Era designator	Text	`AD`
`y`	Year	Year	`1996`; `96`
`M`	Month in year	Month	`July`; `Jul`; `07`
`w`	Week in year	Number	`27`
`W`	Week in month	Number	`2`
`D`	Day in year	Number	`189`
`d`	Day in month	Number	`10`
`F`	Day of week in month	Number	`2`
`E`	Day in week	Text	`Tuesday`; `Tue`
`a`	Am/pm marker	Text	`PM`
`H`	Hour in day (0-23)	Number	`0`
`k`	Hour in day (1-24)	Number	`24`
`K`	Hour in am/pm (0-11)	Number	`0`
`h`	Hour in am/pm (1-12)	Number	`12`
`m`	Minute in hour	Number	`30`
`s`	Second in minute	Number	`55`
`S`	Millisecond	Number	`978`
`z`	Time zone	General time zone	`Pacific Standard Time`; `PST`; `GMT-08:00`
`Z`	Time zone	RFC 822 time zone	`-0800`

Set the tag group and identification patterns

Add log tags (Fatal, Error, Warning, Success, Info, Debug, Trace) to all samples. The tag modifies the presentation color and can be made visible in the complementary views. The tag patterns are regular expressions. A typical expression for the "Error" tag could be "error|Error|ERR".

Parse errors

The reader will provide a parse error message if the input can not be read. The message usually contains the error position, the reason for the failure and a stack trace.

If you can not resolve the problem (e.g. obvious format error in the input file), please send this message together with the input file to "This email address is being protected from spambots. You need JavaScript enabled to view it.".

	Reader: de.toem.impulse.serializer.log.pattern 
	Error at position: 1
	Text at position: "1.000 ns Warning 3.5 huhu 3,5"
	Message: No match
	Type: class de.toem.impulse.serializer.ParseException
	Stack trace: 
	
	de.toem.impulse.extension.log.pattern.PatternLogReader.parse(PatternLogReader.java:289)
	de.toem.impulse.serializer.AbstractParsingRecordReader.read(AbstractParsingRecordReader.java:117)
	de.toem.impulse.serializer.AbstractRecordReader.read(AbstractRecordReader.java:199)
	de.toem.pattern.element.Element$1.execute(Element.java:1067)
	de.toem.eclipse.hooks.actives.EclipseActives$3.run(EclipseActives.java:73)

Home