impulse 2.1

U007 Logging and Tracing 2 - Extract numerical data from text logs

Attach, View Anaylze and Solve!

In the first article we defined a pattern log configuration to read and extract given log data. As a result the reader was able to read the text and create signals with text information inside. Now assume that we have numerical data in this log to be extracted. The numerical shall be displayed as a line diagram.

Numerical data

The part of the log data we need to look at is this:

    NOTE00000[216 735 876.000 ns] in top.F4.generics : Calculated load 47%

So far we just extracted the text behind the first ':' character (Calculated load 47%).

There are 3 ways to extract this numerical data with impulse:

  1. Define another specific log pattern for this line.
  2. Use signal scripts to extract relevant parts and do further processing.
  3. Use the "Value Extract" production.

1a Extract numerical data with an additional log pattern

A simple pattern could look like this:

NOTE[\s0]*\[([ 0-9a-z\.]+)\] in top.F4.generics : Calculated load ([0-9]+).*
NOTE[\s0]*\[
fixed text and whitespaces
([ 0-9a-z\.]+)
Group 1: time stamp
in top.F4.generics : Calculated load ([0-9]+).*
Group 2: numerical data with surrounding fixed text

Create a new log pattern and select "Timestamp" for group 1. Select "Float value" and "Parse unit from value" for the field "Domain unit".

For the value group, typing in the label "Value" and select "Integer" as its type. This tells the log reader to use an integer member and to convert the value, instead of raw text.

Select "Explicit name" and type 'Load' in field "Name". The pattern will create a signal with the name 'Load'.

Please be reminded that additional patterns mean additional processing time for reading. Also it is important to put the pattern before the default pattern. Otherwise it would not hit !

RL01 Pattern Log Reader
H200 View - How to create a new view

A view combines a set of plots organized in rows. You may switch between multiple views using the the combo box in the top right corner.

  1. Use the "Add new view" toolbar button.
  2. Select if the new view shall be empty or populated.
  3. Press ok to create the view.
  4. Use the signal area to drag&drop signal into the new view.
H201 View - How to switch between views

You can use an unlimited number of views for a record file. The upper-right button in the toolbar is used to switch between multiple views.

  1. Use the combo box in the right top corner to switch between views.
  2. You may select the "Filter" fields to limit the views in the combo box to those that 'fit' to your signals.
  3. Use the preferences page "impulse/views" to manage your views.
H202 View - How to add signals to a view

The signal area is used to find signals in the record (wave, trace, log,..) file and convert them into a plot that can be shown in the Plot Area.

  1. Use the signal area to drag&drop signals or scopes into the view.
  2. You may use the filter to find your signals.
  3. Or you may select the context menu of the view tree to add a new plots.
H203 View - How to resize a plot

  1. When you select a plot in the view, there will be a small resize grip left-hand of the name.
  2. Select the grip and move the cursor to resize.
  3. Release the grip to keep the size.
H204 View - How to re-organize views

Inside of a view you find Plots to display the signal data and Folders to organize the presentation in a hierarchical structure. This enables the user to hide/show parts of the presentation.

  • You can add single elements (folders, plots of multiple types) to the hierarchical structure by opening the context menu of the configuration with a right mouse click.
  • You can reorder all elements by selecting and moving them with the mouse.
  • To remove elements you open the context menu, click delete or mark the elements that you want to remove and press the delete key.
  • You can edit an existing plot (or multiple plots) by opening the edit dialogue with the context menu or by double-clicking the element whose configuration you want to change.

Screen Cast: Logging and Tracing 2 - Extract numerical data from text logs

Download Tutorial Examples (Preference Wallet Uxxx.walMZ)

1b Extract the member and show as line

The parsed values is now a separate integer member of your log samples (a signal of type "Struct").

To display the data in a line diagram, you need to extract the member "Value" from the sample. This is done by using the "Member Extract" production.

Create a new Plot, select the signal "Load" as source and "Member Extract" as production. Select the member "Value" and switch to Line diagram.

 

R033 Member Extract Production
03 Views and Plots

2 Extract numerical data using Signal Scripts

This approach is more flexible and can be assigned to any kind of data. But it requires some java script code!

Signal scripts allow you to combine signals with mathematical operations, implement protocol parsers, compare signals, extract statistical information or automatically detect problems.

Signal scripts can analyse existing signals and always create one new resulting signals. This is what we do now with signal "#generics".

To create a signal script, get to your viewer and add a new plot. Set production to "Signal Script", signal type to "Float" and enter the script below.

Finally set "#generics" as primary source

The script iterates over all samples. When the sample is not none, it reads the member value of the member "Message".

After checking if the text is correct, it extracts the value and converts to in. Then it writes the float value into the signal.

Signal scripts are evaluated as soon the viewer wants to display a signal. If required you can iterate over multiple signals. "in0" is the primary source, in1..n are additional sources. Both can be configured in the plot dialogue.  For debugging purpose, you might use the console object to output debug text.

As the final step, set type to Line.

R030 Signal Script Production
10 Signal Script 
// in0         : Primary input (de.toem.impulse.samples.ISamplePointer,de.toem.impulse.samples.IReadableSamples)
// in1..       : Additional inputs (de.toem.impulse.samples.ISamplePointer,de.toem.impulse.samples.IReadableSamples)
// input:      : An array of all inputs
// out         : Output signal writer (? extends de.toem.impulse.samples.ISamplesWriter)
// iter        : Sample iterator  (de.toem.impulse.samples.ISamplesIterator)
// progress    : Progress control (de.toem.impulse.samples.producer.IScriptProgress)
// console     : Console output (de.toem.impulse.scripting.IScriptConsole)

var Float = Java.type("java.lang.Float");
while (iter.hasNext()) {
    var current = iter.next(out);
    if (!in0.isNone()) {
        var message /*:java.lang.String:*/ = in0.stringValueOf('Message');
        if (message != null && message.startsWith('Calculated load')) {
            var val = Float.parseFloat(message.substring(15, 18));
            console.log("found", current, val)
            out.write(current, false, val);
        }
    }
}
progress.cont(); // Indicates that the script can be re-called if the input signals grow

3 Extract numerical data using Value Extract production

The "Value Extract" production filters all non-relevant samples and extract textual information from the relevant ones. This information is converted into a given output signal type.

Create a new plot, change the production to "Text Extract", select "Float" signal type and type in the member name ('Message').
Finally define the extract definition. The extract definition may be either a text fragment or a regular expression.

If text fragment is chosen, the production will try to find the text in the value representation and extracts the following text fragment surrounded by white-spaces.

Example: Sample value = "nx=400 mx=500"; if extract is set to "nx=", the value 400 will be extracted.

If a regular expression is given, the production will try to find a match in the value representation and extracts the 1st group value.

Example: Sample value = "nx=400 mx=500"; if extract is set to "mx\=([0-9]+)", the value 500 will be extracted.

In our case, the extract definition is:

Calculated load ([0-9]*)
R036 Value Extract Production
03 Views and Plots
Next Tutorial in thes series

U008 Logging and Tracing 3 - Live data

In the first two articles it was assumed that all log data comes from one file. But the fact is that complex systems with multiple domains/cores usually have more than a single log! - e.g. a hardware log (systemC), firmware traces from multiple cores and DSPs and application logs. Also it would be convenient, if we could see and analyse the log while simulation/debugging is ongoing...
Read more
All tutorials
More Resources
toem

technical software and tooling

Products

Company

Contact Us

This email address is being protected from spambots. You need JavaScript enabled to view it.