Home

RL02 CSV Reader


Outline

The CSV Data and Log Reader is a configurable reader for all typical kinds of CSV (Comma-Separated Values) based data and log formats. CSV is a common data exchange format that is widely supported by consumer, business, and scientific applications. With easy to use dialogs you can define configurations (separators, first line, labels, data types,...) for your specific formats. This article will help you configure and use the reader.

Platforms:
32/64bit 32/64bit 32/64bit
Requirements:
  • None
Known limitations:
  • File size limited by virtual memory and temporary storage
Status:
  • Stable
Operations:
  • Load
  • Stream
Default Parameters:
  • None
Configuration:
  • A CSV reader configuration is required. The configuration defines the structure of the CSV file (separator, first line, labels,..).


About Comma-separated values

A comma-separated values (CSV) file stores tabular data (numbers and text) in plain text. Each line of the file is a data set. Each set consists of one or more fields, separated by commas or other delimiters. The top line may contain labels, associated with the fields.

Here a typical example:

time;osc_out;lvdt_outp;lvdt_outn
0;0;0;0
1.19209e-07;0.364121124438;0.361954160418;-0.361960938184
2.38418e-07;0.549114671567;0.544246145289;-0.544266588099
3.57627e-07;0.496353600022;0.488405852023;-0.488433470485
4.76836e-07;0.488886165659;0.478050306956;-0.478086469891

Beside data you also find CSV files with logs:

0,1587458274777,TRACE,11,ThreadA,5," No Local Variables are initialized for Method [_GTF",org.apache.logging.log4j.spi.AbstractLogger,de.toem.impulse.test.primary.Log1,,,de.toem.impulse.test.AbstractLog.log(AbstractLog.java:630),{},[]
0,1587458274887,WARN,11,ThreadA,5,get Sinus Wave -0.004940748208935772,org.apache.logging.log4j.spi.AbstractLogger,de.toem.impulse.test.secondary.Log3,,,de.toem.impulse.test.secondary.Log3.log(Log3.java:18),{},[]
0,1587458274898,DEBUG,12,ThreadB,5,get Sinus Wave 0.04504322352107653,org.apache.logging.log4j.spi.AbstractLogger,de.toem.impulse.test.secondary.Log3,,,de.toem.impulse.test.secondary.Log3.log(Log3.java:18),{},[]
0,1587458274900,INFO,1,main,5," ACPI: Local APIC address 0xfee00000",org.apache.logging.log4j.spi.AbstractLogger,de.toem.impulse.test.primary.Log1,,,de.toem.impulse.test.AbstractLog.log(AbstractLog.java:630),{},[]


Usage

The reader can be used to open workspace resources and together with ports (direct connection to the target using TCP, Serial, J-Link, ...).



Content type association

To use a reader for a given input resource,

The serializer dialog contains the field "Content type association" with which you can manage file associations (e.g. to add "*.log" as a valid file name).

Content type association is only meaningful for eclipse resources, not for port inputs like TCP oder File/Pipe ports.



Content Detection and Configurations

This reader requires a reader configuration to work. Such a configuration defines the form of the input content and how it shall be read.

Typically a reader has several configurations that can be enabled or disabled.

The reader does a content type detection to:

A file resource has valid content data if at least 1 enabled configuration matches.



Configuration

The CSV Reader requires a configuration (CSV Configuration) to work. A configuration contains information about the char set, the delimiter, labels and and the actual data fields. Press "Add" and select "CSV Configuration" to create a new reader configuration:



Mode 0: Create individual signals per column

This mode is a typical data mode. Each configured column will create a signal of a given type. The signal name comes either from a label in the input or the configuration.


Name and configure the signals

Select member names and types for all columns that shall create a signal.

Valid member types are:

The descriptor field can optionally contain a content or signal descriptor:



Mode 2: Create one signal with individual column members

This mode create only 1 struct signal. Each configured column will create a member of a given type. The member name comes either from a label in the input or the configuration.


Name and configure the members

Select member names and types for all columns that shall create a signal.

Valid member types are:

The descriptor field can optionally contain a content or member descriptor:



Mode 3: Create single/multiple log signals

The mode handles the csv input as a log. Depending on the naming scheme, 1 or log signals will be created .


Name and configure the sample members

Log signals are struct signal with multiple members.

Select member names and types for all columns that shall have a member in you log samples (e.g. the message).

Valid member types are:

The descriptor field can optionally contain a content or member descriptor:


Define the signal naming scheme

Each single sample belongs to one signal. You can organize your samples into one or multiple signals (e.g. a signal per log source). The "Signal/Scope name" section allows to select the signal and scope naming scheme for each log element.

You may extend the naming scheme with another column value. If enabled, the value of the source will be put after the signal name (in parentheses).


Set the tag source and identification patterns

Add log tags (Fatal, Error, Warning, Success, Info, Debug, Trace) to all samples. The tag modifies the presentation color and can be made visible in the complementary views. The tag patterns are regular expressions. A typical expression for the "Error" tag could be "error|Error|ERR".



All modes: Configure the domain value source and format (usually the time-stamp)

There may be CSV logs without a domain value. In this case you may want to use:

You may extend the domain position with another column value (e.g one column contains the seconds, another nanoseconds):

  • Use relative domain value: Check to convert absolute domain values of the input into relative one (relative to the first domain position).
  • Date formats are specified by date and time pattern strings:

    Letter Date or Time Component Presentation Examples
    G Era designator Text AD
    y Year Year 1996; 96
    M Month in year Month July; Jul; 07
    w Week in year Number 27
    W Week in month Number 2
    D Day in year Number 189
    d Day in month Number 10
    F Day of week in month Number 2
    E Day in week Text Tuesday; Tue
    a Am/pm marker Text PM
    H Hour in day (0-23) Number 0
    k Hour in day (1-24) Number 24
    K Hour in am/pm (0-11) Number 0
    h Hour in am/pm (1-12) Number 12
    m Minute in hour Number 30
    s Second in minute Number 55
    S Millisecond Number 978
    z Time zone General time zone Pacific Standard Time; PST; GMT-08:00
    Z Time zone RFC 822 time zone -0800


    Parse errors

    The reader will provide a parse error message if the input can not be read. The message usually contains the error position, the reason for the failure and a stack trace.

    If you can not resolve the problem (e.g. obvious format error in the input file), please send this message together with the input file to "This email address is being protected from spambots. You need JavaScript enabled to view it.".

    Reader: de.toem.impulse.serializer.csv 
    Error at position: 0
    Text at position: "0,1587458274777,TRACE,11,ThreadA,5," No Local Variables are initialized for Method [_GTF",org.apache.logging.log4j.spi.AbstractLogger,de.toem.impulse.test.primary.Log1,,,de.toem.impulse.test.AbstractLog.log(AbstractLog.java:630),{},[]"
    Message: Invalid no of values
    Type: class de.toem.impulse.serializer.ParseException
    Stack trace: 
    
    de.toem.impulse.extension.log.csv.CsvReader.handleLine(CsvReader.java:529)
    de.toem.impulse.extension.log.csv.CsvReader.parse(CsvReader.java:495)
    de.toem.impulse.serializer.AbstractParsingRecordReader.read(AbstractParsingRecordReader.java:117)
    de.toem.impulse.serializer.AbstractRecordReader.read(AbstractRecordReader.java:204)
    de.toem.pattern.element.Element$1.execute(Element.java:1067)
    de.toem.eclipse.hooks.actives.EclipseActives$3.run(EclipseActives.java:73)