- 0 minutes to read

Flat File CSV Search Field Expression Plugin

Easily extract unique values from delimited (CSV-style) string-based messages using the Flat File CSV Search Field Expression Plugin. This page guides you through input, configuration, and result patterns with clear, actionable examples.

✅ Quickly extract single or multiple unique values from CSV or delimited messages ✅ Flexible configuration for delimiter, header rows, fields, and more ✅ Works with semicolon, comma, tab, or any custom delimiter ✅ Designed for small payloads and high performance in log event processing

🎯 Design Note: Perfect for legacy systems, EDI files, and batch processing scenarios!

What does the Flat File CSV plugin do?

The Flat File CSV plugin lets you extract one or more unique values from the payload (Body) of Log Events. Use it to process messages where data is separated by a specific character (such as ; or ,).

Key Features:

  • String-based content with one or more lines
  • Data records separated by a delimiter (e.g., ;, ,, tab)
  • Each line is a record, each record has fields separated by the delimiter
  • Optionally skipping commented lines or header rows
  • Support for quoted values and missing fields

Use Case: Extract data from CSV exports, EDI files, legacy flat file formats, or batch processing outputs.

How it works

The Flat File CSV plugin processes messages in three steps:

  1. Input: The delimited message payload (CSV-style format)
  2. Configuration: Delimiter, field indexes, header rows, and options
  3. Result: Unique value(s) extracted from specified fields
graph LR A["Input: Delimited Payload"] --> B["Configuration: Delimiter + Fields"] B --> C["Result: Unique Values"]

The plugin parses delimited text and extracts values from specified field positions.

Example

Input:

ORD;101;ACME;456;Company Name ACME ;Dieselgate valley 1;123 45;Flameburg;
ORD;102;Northwind;789;Company Name Northwind;Radiator Spring 420;543 21;Apocalypsenburg;

Configuration:

Number of header rows = 0
Delimiter = ;
Fields = 1
Ignore missing fields = checked
Quote = "
Comment = #

Result:

101
102

Tip

Field indexes are zero-based, so Field 1 is the second column. Use comma-separated values for multiple fields: 1,3,5.

Flat File Example Example: Extract order IDs from a semicolon-separated flat file.

Configuration options

Option Description Example
Delimiter Character separating fields ;, ,, \t (tab)
Fields Zero-based field indexes 1 or 1,3,5
Header rows Number of rows to skip 0, 1
Quote Character for quoted values "
Comment Character for comment lines #
Ignore missing fields Skip records with missing fields checked/unchecked

How to use the Flat File CSV plugin

  1. Open the Search Field Wizard from a Log View and either create a new Search Field or edit an existing one
  2. Choose the Flat File CSV plugin from the Expression Type dropdown
  3. Configure the plugin settings:
    • Number of header rows to skip
    • Delimiter character (;, ,, tab, etc.)
    • Fields to extract (comma-separated zero-based indexes)
    • Quote and comment characters
    • Ignore missing fields option
  4. Select the Message Type(s) to apply the field
  5. Save the configuration
  6. Re-index existing data (optional)
  7. The extracted data is now available in your Log Views!

Select Flat File CSV plugin Example: Selecting the Flat File CSV plugin in the Search Field configuration.

Tip

Extracted values are stored for as long as the Message Type retention is configured.

Test Expression

You can test your configuration in the 'Test Expression' tab when setting up a Search Field:

  1. Enter a sample payload in the 'Message Body' tab
  2. Select the Flat File CSV plugin
  3. Enter your configuration settings
  4. Review the output and adjust as needed

Successful Match: ✅ Shows matched values and extraction count.

Test Expression Example: Valid expression with state output, unique values, and total count.

No Match: ❌ Occurs when the configuration is invalid or fields don't exist.

No Result Example: Invalid configuration yielding no result.

Important considerations

Performance:

  • This plugin loads the entire message into memory
  • Use only for small to medium payloads to avoid performance issues

Alternatives for other formats:

Next steps