RL04 JSON Log Reader

Outline

The JSON Log Reader is a configurable log reader for all kinds of JSON (JavaScript Object Notation) based log formats and comes with a pre-defined Log4J and GELF configuration. A powerful UI helps to define and extend reader configurations.

Platforms:

32/64bit

Requirements:

None

Known limitations:

File size limited by virtual memory and temporary storage

Status:

Stable

Operations:

Load
Stream

Parameters:

None

Configuration:

A JSON Log reader configuration is required. The configuration defines the structure of the JSON input (object name, JSON values, data types and names, domain, ...).

Tutorials:

About JSON logs

There is no standard JSON log format existing. Various formats pack the log data into different objects and values.

Below you find the output of the "Log4J JSON Layout" format":

[
{
  "thread" : "ThreadA",
  "level" : "TRACE",
  "loggerName" : "de.toem.impulse.test.primary.Log1",
  "message" : " No Local Variables are initialized for Method [_GTF",
  "endOfBatch" : false,
  "loggerFqcn" : "org.apache.logging.log4j.spi.AbstractLogger",
  "instant" : {
    "epochSecond" : 1587458274,
    "nanoOfSecond" : 777000000
  },
  "threadId" : 11,
  "threadPriority" : 5
}
,....]

Usage

The reader can be used to open workspace resources and together with ports (direct connection to the target using TCP, Serial, J-Link, ...).

Content type association

To use a reader for a given input resource,

a valid content type association needs to be in place,
and the reader need to identify the file resource content as valid data (Content detection).

The serializer dialog contains the field "Content type association" with which you can manage file associations (e.g. to add "*.log" as a valid file name).

Content type association is only meaningful for eclipse resources, not for port inputs like TCP oder File/Pipe ports.

Content Detection and Configurations

This reader requires a reader configuration to work. Such a configuration defines the form of the input content and how it shall be read.

Typically a reader has several configurations that can be enabled or disabled.

The reader does a content type detection to:

identify if the content is valid and matches a given configuration,
or to select a valid configuration for the given input, if not done explicitly by the user ("Default" selection).

A file resource has valid content data if at least 1 enabled configuration matches.

Configuration

The JSON Log Reader requires a configuration (JSON Log Configuration) to work. Each configuration contains a set of log elements. Each log element describes the form and content of a single JSON object.

Click the "Add" button to add a new "JSON Log Configuration"" and fill in the following parameters:

Char Set: Select the char set that is used by the log file (or "Default").
Domain Base: Select the domain base. This represents is smallest domain change (e.g. 1 ms).
Use relative domain value: Check to convert absolute domain values of the input into relative one (relative to the first domain position).
Content detection: This setting defines one of your log elements as the one used for content detection. If the specified log element matches with one of the first object of an input, the configuration is considered valid. You can also specify how many bytes (from the beginning of the input) to use for content detection. A log element matches, if the object path/name matches and at least half of the defined values can be found in that object.
Updates: This reader has some sample or standard configurations after installation. After an update of impulse, these configurations may also have been updated. To exclude a configuration from updating (e.g. because you have adapted the configuration for your application), you can set the flag "Do not update with latest version".

A log element describes the form and content of a single JSON object (object path and values).

You may define multiple log elements:

if you encounter multiple hierarchical JSON objects that form a log entry (which is usually the case),
or to ignore certain objects. It is often useful to add a log element with the object name "*" and the action "Ignore" on lowest priority to ignore all objects that do not match a valid log element (e.g. a root element) and thus avoid parse errors.

When reading an input, the reader takes the first log element that matches. Use the buttons on the right side to re-order the priority of your log elements. If no log element matches a found JSON object, a parse error will be thrown.

Log Element

Press "Add" and select "Log Element" to create a new log element.

A log element describes the form and content of one single JSON object (object name/path and values).

Define the JSON object path and values

The dialogue starts with the following fields:

Object: Enter the name or the path of the JSON object (e.g. record/millis or just millies or leave empty if the object has no name). Enter "*" for any object (e.g. "record/*" for any object below 'record').
Values: Add a comma separated list of values of the JSON object.

All values that are not listed are ignored. If listed values do not occur in an JSON object, an error is only thrown if the value is required for domain value calculation.

After this first step is done, you need to define how to use the element and its extracted values.

Select the action

Ignore : Just skip this object;
Start new log sample;
Add to previous sample: The information of the sources (attributes and text) will be added to the previously started sample (to handle hierarchical objects). A new sample will be started if no previous is available;
Finish sample: The information of the sources will be added to the previously started sample. Afterwards the sample will be finished (Additional objects can not be added).

Name and configure the sample members

Log signals are struct signal with multiple members.

Select member names and types for all sources (JSON values) that shall have a member in you log samples (e.g. the message).

Select the member names. You can select a predefined member names or enter custom ones.
For custom members you can define the member type and a descriptor. If custom members are used over multiple log elements, the type need to be identical.

Valid member types are:

Float,
Integer,
Text ,
or Enumeration.

The descriptor field can optionally contain a content or member descriptor:

a simple content descriptor e.g. "default", "state", event" or "label", ...
a complete member descriptor e.g. "default<df=Hex>" or "state<df=Text>"

Define the signal naming scheme

Each single sample belongs to one signal. You can organize your samples into one or multiple signals (e.g. a signal per log source). The "Signal/Scope name" section allows to select the signal and scope naming scheme for each log element.

Undefined : Either all log samples get into one signal with the name "log" -- or this log element is part of a hierarchical multi-object log and the name information is not part of this element.
Explicit name : Define a signal name for all samples of this log element.
Explicit hierarchy : Define a signal/scope hierarchy for all samples of this log element. The signal and scope names are generated by splitting the value into its contents (e.g. the value "de.toem.logger" creates the scope/signal hierarchy de/toem/logger).
Name from source value : The value of the selected source is taken to define the signal name (.e.g. from the logger source).
Hierarchy from source value : The value of the selected source is taken to define the signal and scope names by splitting the value into its contents (e.g. the value "de.toem.logger" creates the scope/signal hierarchy de/toem/logger).

You may extend the naming scheme with another source value. If enabled, the value of the source will be put after the signal name (in parentheses).

Configure the domain value source and format (usually the time-stamp)

Undefined : In case of hierarchical multi-object logs, you will have objects without domain information.
Date: You need to define the date format. Use the content proposals and http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html. The parsing of the date value results into milliseconds after January 1, 1970 00:00:00 GMT.
Float domain value: Parse a float value, optionally parse the domain unit from the source value or assign a unit.
Integer domain value: Parse an integer value, optionally parse the domain unit from the source value or assign a unit.

There may be log elements without a domain value. In this case you may want to use:

Same as previous: Use the domain value of the previous sample (all signals).
Same as previous (per Signal): Use the domain value of the previous sample of the used signal.
Incrementing: Increment the domain value with every new sample (all signals).
Incrementing (per Signal): Increment the domain value with every new sample per signals.
Reception time: Use the current time in milliseconds after January 1, 1970 00:00:00 GMT as domain value.

You may extend the domain position with another source value (e.g one value contains the seconds, another nanoseconds):

Undefined : Do not extend.
Float domain value: Parse a float value, optionally parse the domain unit from the source value or assign a unit. The domain extension value will be added to the main domain position.
Integer domain value: Parse an integer value, optionally parse the domain unit from the source value or assign a unit. The domain extension value will be added to the main domain position.

Date formats are specified by date and time pattern strings:

Letter Date or Time Component Presentation Examples

`G`	Era designator	Text	`AD`
`y`	Year	Year	`1996`; `96`
`M`	Month in year	Month	`July`; `Jul`; `07`
`w`	Week in year	Number	`27`
`W`	Week in month	Number	`2`
`D`	Day in year	Number	`189`
`d`	Day in month	Number	`10`
`F`	Day of week in month	Number	`2`
`E`	Day in week	Text	`Tuesday`; `Tue`
`a`	Am/pm marker	Text	`PM`
`H`	Hour in day (0-23)	Number	`0`
`k`	Hour in day (1-24)	Number	`24`
`K`	Hour in am/pm (0-11)	Number	`0`
`h`	Hour in am/pm (1-12)	Number	`12`
`m`	Minute in hour	Number	`30`
`s`	Second in minute	Number	`55`
`S`	Millisecond	Number	`978`
`z`	Time zone	General time zone	`Pacific Standard Time`; `PST`; `GMT-08:00`
`Z`	Time zone	RFC 822 time zone	`-0800`

Set the tag source and identification patterns

Add log tags (Fatal, Error, Warning, Success, Info, Debug, Trace) to all samples. The tag modifies the presentation color and can be made visible in the complementary views. The tag patterns are regular expressions. A typical expression for the "Error" tag could be "error|Error|ERR".

Parse errors

The reader will provide a parse error message if the input can not be read. The message usually contains the error position, the reason for the failure and a stack trace.

If you can not resolve the problem (e.g. obvious format error in the input file), please send this message together with the input file to "This email address is being protected from spambots. You need JavaScript enabled to view it.".

Reader: de.toem.impulse.serializer.log.json 
Error at position: 0
Text at position: "null"
Message: Empty time format
Type: class org.xml.sax.SAXException
Stack trace: 

de.toem.impulse.extension.log.json.JsonLogReader.parseLogs(JsonLogReader.java:189)
de.toem.impulse.extension.log.abstrac.AbstractLogReader.parse(AbstractLogReader.java:221)
de.toem.impulse.serializer.AbstractParsingRecordReader.read(AbstractParsingRecordReader.java:117)
de.toem.impulse.serializer.AbstractRecordReader.read(AbstractRecordReader.java:204)
de.toem.pattern.element.Element$1.execute(Element.java:1067)
de.toem.eclipse.hooks.actives.EclipseActives$3.run(EclipseActives.java:73)

Home