Skip to content

Conversation

@AlexisSouquiere
Copy link

@AlexisSouquiere AlexisSouquiere commented Dec 2, 2025

The GrokFilter is really useful to parse unstructured data. When I tried to use it with a *BytesArrayInputReader I missed 2 things:

  • A way to put the extracted fields in a given field and not always have to put these field at the top level.
    To solve this, I added support of the target field:

From:

{
  "message": "1970-01-01 00:00:00,000 INFO a dummy log message",
  "timestamp": "1970-01-01 00:00:00,000",
  "level": "INFO"
}

To:

"filters.grok.target": "parsed"
{
  "message": "1970-01-01 00:00:00,000 INFO a dummy log message",
  "parsed": {
    "message": "a dummy log message",
    "timestamp": "1970-01-01 00:00:00,000",
    "level": "INFO"
  }
}
  • I need to read a file as a whole to generate 1 single event. When working with *BytesArrayInputRead, SplitFilter to parse the file content (on \n for instance), we can have a string array on which we want to apply the GrokFilter.
    To solve this, I added support of array string input as well as the existing string input.

Note: The following example is not really relevant as we would generally read a log file line by line (it shows the objective)

From:

{
  "message": ["1970-01-01 00:00:00,000 INFO a dummy log message", "1970-01-01 01:00:00,000 INFO a second dummy log message", "1970-01-01 02:00:00,000 INFO a third dummy log message"]
}

To:

{
  "parsed": [{
    "message": "a dummy log message",
    "timestamp": "1970-01-01 00:00:00,000",
    "level": "INFO"
  }, {
    "message": "a second dummy log message",
    "timestamp": "1970-01-01 01:00:00,000",
    "level": "INFO"
  }, {
    "message": "a third dummy log message",
    "timestamp": "1970-01-01 00:00:00,000",
    "level": "INFO"
  }]
}

This PR enhances the GrokFilter while keeping the original behaviour. Look forward to having your feedback !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant