impulse contains a rich set of tools and mechanisms to make complex systems visible and transparent.This series of articles shall give hints and ideas how to improve your systems visibility and to dig into.
Text line based log formats are still very popular. Whether Log4j formats, syslog, Apache logs, CSV files or traditional logs, they are easy to read without a tool, at least up to a certain size.
The only requirement is a line ending, typically a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed. The syntactic format of the lines and its content varies from format to format.
The impulse Pattern Log Reader enables the reading of any text lines with the help of regular expressions. The determined line segments can then be converted into textual and numerical data and stored as log samples.
Quite often, developers have text log outputs. In the good case, all logs share the same format, but often there is a strange mix due to historical reasons. Up to a certain level, impulse can parse these mixed logs and display them in a uniform way. But it might be sometimes tricky to configure. Lets assume the following example:
NOTE [212 745 133.974 ns] in top.HHShell.m4_top.mDist_dsp_access.r2_top : getX4LegChangeDone happened WARNING [212 745 133.974 ns] in top.HHShell.m4_top.mDist_dsp_access.r4_top : getX4LegChangeDone = 0x1008000 ISR at 214435176 ERROR [214 435 176.000 ns] in top.HHShell.m4_top.mDist_dsp_wait.entry : Kernel changed state to: OUTPUT # Leg ok - wait for next NOTE00000[216 735 876.000 ns] in top.F4.generics : Calculated load 47%
Here we got
There are multiple ways to view signal data with impulse. You might :
impulse come with a set a example wave files.
After receviing new signal data from a port or adding annotations, you may want to save your signal data changes.
All tutorials come with a preference wallet that contains preferences and resources (e.g. recording files).
The Pattern Log Reader requires a configuration (Pattern Log Configuration) to work. Each configuration contains a set of log pattern. Each log pattern describes the form and content of a single log line.
To configure the pattern log reader, get to "Preferences->Impulse->Serializers" and double-click "Pattern Log Reader" . Click the "Add" button to add a new "Pattern Log Configuration"" and fill in the following parameters:
The table shows all the configured log pattern. A log pattern describes the form and content of a single log line.
You may define multiple log pattern:
When reading an input, the reader takes the first log pattern that matches. Use the buttons on the right side to re-order the priority of your log pattern. If no pattern matches a line, a parse error will be thrown.
To create an initial Log4J 1.2 log pattern, press "Add" and select "Log Pattern for Log4j 1.2".
Unfortunately there is no 100% pattern conversion possible. You probably have to modify or extend the resulting pattern.
Press "Add" and select "Log Pattern" to create a new log pattern.
A log pattern describes the form and content of one single log line. Regular expressions can be used not only to match text, but also to extract information for further processing. This is done by defining groups of characters and capturing them using the special parenthesis of ( and ) meta characters.
To play with regular expression (not always easy), the log pattern dialogue has a field to paste log lines and test them with your expression. The first expression is:
(WARNING|NOTE|ERROR|DEBUG)[\s0]*\[([ 0-9a-z\.]+)\]\s*in\s*([^:]*):\s*(.*)
Log signals are struct signal with multiple members. Select member names and types for all groups that shall have a member in you log samples (e.g. the message).
Valid member types are:
Each single sample belongs to one signal. You can organize your samples into one or multiple signals (e.g. a signal per log source). The "Signal/Scope name" section allows to select the signal and scope naming scheme for each log pattern.
In our case the logger group contains an hierarchical name using '.' as separator. If we choose Hierarchy from source value , the reader will create a hierarchy of log signals.
Select the logger group as source group and enter '\.' (regular expression) in field Separator . To differentiate logs and nodes, type in '#' (or whatever you like) in field Leaf Prefix . This is required if a node (e.g top.HHShell) has logs but also sub nodes with logs:
NOTE [212 745 133.974 ns] in top.HHShell : a log NOTE [212 745 133.974 ns] in top.HHShell.m4_top : a log
The reader will create folders/scope for all hierarchy nodes and signals for all loggers.
For the Timestamp group we need some extra information to make the reader able to parse: Select Float value (in this case 212745133.974 ns equals 212745133974 ps), the source group (here Group2), and Parse unit from value in the field Domain unit .
Add log tags (Fatal, Error, Warning, Success, Info, Debug, Trace) to all samples. The tag modifies the presentation color and can be made visible in the complementary views. The tag patterns are regular expressions. A typical expression for the "Error" tag could be "error|Error|ERR".
This pattern is much more simpler than the previous:
ISR\sat\s(.*)
Create a new log pattern and select "Timestamp" for group 1. Select "Integer value" and 'ns' for the field "Domain unit". Select "Explicit name" and type 'ISR' in field "Name". The pattern will create a signal with name 'ISR' containing samples without any additional information. That's enough for the second pattern.
To not get a parser exception on any other line, add another log pattern. Type '.*' into field "Expression" ans select the Ignore action.
This pattern should be at the end of all patterns. The reader processes line by line and selects the first pattern that matches. If no pattern matches, the reader throws an exception !
To use a reader for a given input resource,
The serializer dialog contains the field "Content type association" with which you can manage file associations (e.g. to add "*.log" as a valid file name).
Content type association is only meaningful for eclipse resources, not for port inputs like TCP oder File/Pipe ports.
In the inital dialogue, we missed one important setting:
Here we choose the "Default" pattern.