Logging and Tracing 2 - Extract numerical data from text logs

In the first article we defined a pattern log configuration to read and extract given log data. As a result the reader was able to read the text and create signals with text information inside. Now assume that we have numerical data in this log to be extracted. The numerical shall be displayed as a line diagram.

This article is based on impulse version 1.3/1.4

The part of the log data we need to look for is like this:
    NOTE00000[216 735 876.000 ns] in top.F4.generics : Calculated load 47%
So far we just have the text behind the first ':' character. There are 3 ways to extract non-text data with impulse:
  1. Define another specific pattern for that log.
  2. Use signal scripts to extract relevant parts and do further processing.
  3. Use the "Text Extractor" production.

1 Extract numerical data with an additional log pattern

A simple pattern could look like this:
NOTE[\s0]*\[([ 0-9a-z\.]+)\] in top.F4.generics : Calculated load ([0-9]+).*

In this pattern i just extract and define groups for the time stamp and the load information.

Beside this, i use the same settings as for the default pattern (part 1), and give it a fixed name.

For the value i use a specific label format Value<Float> or Value<Integer>. This tells the log reader, to use an integer/float field, instead of text for the group.

Please remind that additional patterns mean additional processing time for reading. Also it is important to put the pattern before the default pattern. Otherwise it would not hit !

Also don't forget to select the right identification pattern!

That's all to extract the data. To display the data you need to extract the member Value in your samples configuration and change to Line diagram as shown here.

2 Extract numerical data using Signal Scripts

This approach is more flexible and can be assigned to any kind of data. But it requires some java script code!

Signal scripts can analyse existing signals and create a new resulting signals. This is what we do now with signal top.F4.generics.

To create a signal script, get to your viewer and add a new plot. Set production to Signal Script, signal type to Float and enter the script below.

Finally add #generics to additional signals.

The script iterates over all samples. When the sample is not none, it reads the member value of Message.

After checking if the text is correct, it extracts the value and converts to float. Then it writes the float value into the signal.

Signal scripts are evaluated as soon the viewer wants to display a signal. If required you can iterate over multiple signals. in0 is the primary signal, in1..n are additional signals. Both can be added in the configuration dialogue.  For debugging purpose, you might use the console object to output debug text.

As the final step, set diagram to Line.

// input: an array of all input signals
// in0: primary input of type ISamplePointer,IReadableSamples
// in1..: additional inputs of type ISamplePointer,IReadableSamples
// out: output signal of type ITextSamplesWriter
// console: console output of type MessageConsoleStream

for (var iter<:SamplesIterator:>  =  new SamplesIterator(input);iter.hasNext();){
	var current<:Long:> = iter.next();	
	if (!in1.isNone())  { 
		var message <:String:> = in1.stringValueOf( 'Message');
     	if (message != null && message.startsWith( 'Calculated load')){
			var val = Float.parseFloat(message.substring( 15,18));

3 Extract numerical data using Text Extractor production

impulse 1.8 comes with a set of new productions. One is the Text Extractor.
In the plot dialog, change the production to Text Extractor , select Float signal type and type in the member name ('Message').
Optionally type in a filter (impulse will handle only those samples that contain the entered text - here i used 'Calculated load')
Finally define the position to extract the number:

  • Type-in the text before the number (regular flag not set) e.g.: 'load'
  • or type-in a regular expression and put the expression for the value in braces: '.*load ([0-9]+).*'

Enough for the second part. Next article will be about combining multiple sources and live data