impulse 2.1

U006 Logging and Tracing 1 - Basic pattern logs

Attach, View Anaylze and Solve!

impulse contains a rich set of tools and mechanisms to make complex systems visible and transparent.This series of articles shall give hints and ideas how to improve your systems visibility and to dig into.

Text line logs

Text line based log formats are still very popular. Whether Log4j formats, syslog, Apache logs, CSV files or traditional logs, they are easy to read without a tool, at least up to a certain size.

The only requirement is a line ending, typically a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed. The syntactic format of the lines and its content varies from format to format.

The impulse Pattern Log Reader enables the reading of any text lines with the help of regular expressions. The determined line segments can then be converted into textual and numerical data and stored as log samples.

Using the pattern log reader

Quite often, developers have text log outputs. In the good case, all logs share the same format, but often there is a strange mix due to historical reasons. Up to a certain level, impulse can parse these mixed logs and display them in a uniform way. But it might be sometimes tricky to configure. Lets assume the following example:

	NOTE     [212 745 133.974 ns] in top.HHShell.m4_top.mDist_dsp_access.r2_top : getX4LegChangeDone happened
    WARNING  [212 745 133.974 ns] in top.HHShell.m4_top.mDist_dsp_access.r4_top : getX4LegChangeDone = 0x1008000
    ISR at 214435176 
    ERROR    [214 435 176.000 ns] in top.HHShell.m4_top.mDist_dsp_wait.entry : Kernel changed state to: OUTPUT
    # Leg ok - wait for next
    NOTE00000[216 735 876.000 ns] in top.F4.generics : Calculated load 47%

Here we got

  1. a default format with 'WARNING|NOTE|ERROR|DEBUG [time] where : information'
  2. an ISR format with just a timing information
  3. and some temporary log that shall be ignored (# Leg ok...)
     
H001 Beginners - How to open the viewer

There are multiple ways to view signal data with impulse. You might :

  • Use eclipse file resources :
    1. If no project exists existing, create a new project of any type (File->New->Project->??? );
    2. Import your record files (wave files, logs, traces,..) or drag and drop them into the project;
    3. Double-click the record file to open (or use the context menu of the file).
  • Use the File->Open File ... menu:
    1. Open the menu File->Open File ... ;
    2. Select the file to open;
    3. Press OK.
H002 Beginners - How to import example wave files

impulse come with a set a example wave files.

  1. If no project exists existing, create a new project of any type (File->New->Project->??? );
  2. Open the "Import..." menu (from "File"" menu or from the navigator context menu)
  3. Select "impulse";
  4. Select "Import example wave files";
  5. Select a project/folder as target.
H003 Beginners - How to save my changes

After receviing new signal data from a port or adding annotations, you may want to save your signal data changes.

  1. Open the "File" menu  and select "Save As..".
  2. Enter name, target folder and the file format.
  3. Press "OK"
H004 Beginners - How to install tutorial examples

All tutorials come with a preference wallet that contains preferences and resources (e.g. recording files).

  1. Press "Download Tutorial Examples" on the tutorial page; this downloads the  file "Uxxx.walMZ". Copy the file into any eclipse project.
  2. Double-click the file to open the Preference Wallet editor
  3. Press the "Install" button
    • A folder (named "Uxxx Resources") containing the tutorial content (record files, launcher,..) will be created.
    • All preference elements (views, serializer configurations, ports,..)  from the tutorial will be added.
  4. After using the tutorial content, you might press the "Un-install" button to remove content and preferences.

Screen Cast: Logging and Tracing 1 - Basic pattern logs

Configuration

The Pattern Log Reader requires a configuration (Pattern Log Configuration) to work. Each configuration contains a set of log pattern. Each log pattern describes the form and content of a single log line.

To configure the pattern log reader, get to "Preferences->Impulse->Serializers" and double-click  "Pattern Log Reader" . Click the "Add" button to add a new "Pattern Log Configuration"" and fill in the following parameters:

  • Char Set: Select the char set that is used by the log file (or "Default").
  • Domain Base: Select the domain base. This represents is smallest domain change (here set to 1 ps).
  • Use relative domain value: Check to convert absolute domain values of the input into relative one (relative to the first domain position).
  • Add log file position: if checked, the reader will add the line position to each log samples ('RecPos' member). You can later navigate from the value table to the log file position.

The table shows all the configured log pattern. A log pattern describes the form and content of a single log line.

You may define multiple log pattern:

  • if you may have different log styles from different source or due to historical reasons,
  • or to combine multiple log lines into one log event (multi-line log pattern),
  • or to ignore certain lines. It is often useful to add a log pattern with the regular expression ".*" and the action "Ignore" on lowest priority to ignore all lines that do not match a valid log line and thus avoid parse errors.

When reading an input, the reader takes the first log pattern that matches. Use the buttons on the right side to re-order the priority of your log pattern. If no pattern matches a line, a parse error will be thrown.

Is it a Log4J 1.2 pattern ?

To create an initial Log4J 1.2 log pattern, press "Add" and select "Log Pattern for Log4j 1.2".

Unfortunately there is no 100% pattern conversion possible. You probably have to modify or extend the resulting pattern.

The 'Default' pattern

Press "Add" and select "Log Pattern" to create a new log pattern.

A log pattern describes the form and content of one single log line. Regular expressions can be used not only to match text, but also to extract information for further processing. This is done by defining groups of characters and capturing them using the special parenthesis of ( and ) meta characters.

Define the regular expression

To play with regular expression (not always easy), the log pattern dialogue has a field to paste log lines and test them with your expression. The first expression is:

(WARNING|NOTE|ERROR|DEBUG)[\s0]*\[([ 0-9a-z\.]+)\]\s*in\s*([^:]*):\s*(.*)
(WARNING|NOTE|ERROR|DEBUG)
Group 1: log level - one of these options
[\s0]*
optional white-spaces and zeros
\[([ 0-9a-z\.]+)\]
Group 2: time stamp
\s*
optional white-spaces
in\s*([^:]*)
Group 3: logger (location) - starts at 'in' and ends at ':'
\s*
':' and optional white-spaces
(.*)
Group 4: text - remaining line

Name and configure the sample members

Log signals are struct signal with multiple members. Select member names and types for all groups that shall have a member in you log samples (e.g. the message).

Valid member types are:

  • Float,
  • Integer,
  • Text ,
  • or Enumeration.

Define the signal naming scheme

Each single sample belongs to one signal. You can organize your samples into one or multiple signals (e.g. a signal per log source). The "Signal/Scope name" section allows to select the signal and scope naming scheme for each log pattern.

In our case the logger group contains an hierarchical name using '.' as separator. If we choose Hierarchy from source value , the reader will create a hierarchy of log signals.
Select the logger group as source group and enter '\.' (regular expression) in field Separator . To differentiate logs and nodes, type in '#' (or whatever you like) in field Leaf Prefix . This is required if a node (e.g top.HHShell) has logs but also sub nodes with logs:

NOTE [212 745 133.974 ns] in top.HHShell : a log 
NOTE [212 745 133.974 ns] in top.HHShell.m4_top : a log

The reader will create folders/scope for all hierarchy nodes and signals for all loggers.

Configure the time-stamp

For the Timestamp group we need some extra information to make the reader able to parse: Select Float value (in this case 212745133.974 ns equals 212745133974 ps), the source group (here Group2), and Parse unit from value in the field Domain unit .

Set the tag group and identification patterns

Add log tags (Fatal, Error, Warning, Success, Info, Debug, Trace) to all samples. The tag modifies the presentation color and can be made visible in the complementary views. The tag patterns are regular expressions. A typical expression for the "Error" tag could be "error|Error|ERR".

The 'ISR' Pattern

This pattern is much more simpler than the previous:

ISR\sat\s(.*)
ISR\sat\s
fixed text and whitespaces
(.*)
Group 1: time stamp - remaining line

Create a new log pattern and select "Timestamp" for group 1. Select "Integer value" and 'ns' for the field "Domain unit". Select "Explicit name" and type 'ISR' in field "Name". The pattern will create a signal with name 'ISR' containing samples without any additional information. That's enough for the second pattern.

The 'Ignore' Pattern

To not get a parser exception on any other line, add another log pattern. Type  '.*' into field  "Expression" ans select the Ignore action.
This pattern should be at the end of all patterns. The reader processes line by line and selects the first pattern that matches. If no pattern matches, the reader throws an exception !

Content type association

To use a reader for a given input resource,

  • a valid content type association needs to be in place,
  • and the reader need to identify the file resource content as valid data (Content detection).

The serializer dialog contains the field "Content type association" with which you can manage file associations (e.g. to add "*.log" as a valid file name).

Content type association is only meaningful for eclipse resources, not for port inputs like TCP oder File/Pipe ports.

Content detection

In the inital dialogue, we missed one important setting:

  • Content detection: This setting defines one of your log patterns as the one used for content detection. If the specified pattern matches one of the first lines of an input, the configuration is considered valid. You can also specify how many bytes (from the beginning of the input) to use for content detection. A log pattern matches if the regular expression matches the input line.

Here we choose the "Default" pattern.

Next Tutorial in thes series

Logging and Tracing 2 - Extract numerical data from text logs

In the first article we defined a pattern log configuration to read and extract given log data. As a result the reader was able to read the text and create signals with text information inside. Now assume that we have numerical data in this log to be extracted. The numerical shall be displayed as a line diagram.
toem

technical software and tooling

Company

Contact Us

This email address is being protected from spambots. You need JavaScript enabled to view it.