0

Ingesting another sourcetype that provides insane json output. It starts out like: 

Sep  1 15:52:26 | IdentityValidationApi |  |  |  | {"header":{"tenantId":"X03LHWE3","requestType":"  ...

and has a pipe in between the request and the response, but both are on the same line:

..."serverTime":"2017-09-01T19:52:24.641Z"}}} | {"responseHeader":{"tenantID":

and the json output ends with 

...,"fieldValue":"Engineer"}]}}} | D2C CrossCore Request-Response | IdentityValidationApi.corp-dev.com | /api/Inquiry | 172.30.68.88 |  | True

I've tried jq, using jq .header[], but it hates that | in the middle of the event. End goal is to ingest the entire event into Splunk without the beginning or end text outside the json. Can someone suggest any steps here? Thank you.

Edit: I can use sed to pull out the beginning of the line, but am unsure how to combine that with removing the text from the end. Can I do that?

manderson
  • 15
  • 1
  • 6

2 Answers2

1

jq is designed to work with json data. Your input is not pure json. If you can make certain assumptions about your input, then you can probably process the json parts. Any deviation in any of the inputs will break things.

  1. the pipe (|) is only used as a delimiter throughout the file, kind of like a "pipe separated values" file (a la csv but with no escape sequences)
    jq can consume raw files as strings, if pipes are really only used as delimiters, we don't have to worry about parsing it
  2. data in the file does not span multiple rows and only occupies a single row
    without parsing the data or assuming any patterns in the file, it will be impossible to know which lines belong to a single item and when a new one starts
  3. your json data will always be found in a fixed column of the psv row
    again, it will be impossible to know where the request or response parts are in the row if it isn't in fixed places without further processing

If these assumptions hold true, you could probably use something like this:

$ jq -R 'split("|") | {request:.[5]|fromjson,response:.[6]|fromjson}' input.psv

This should give you objects with which you could access the request and response objects. Then you can operate on these.

Jeff Mercado
  • 129,526
  • 32
  • 251
  • 272
  • Jeff, that's awesome. What does the [5] and [6] mean? I may have additional questions about running this in a script, but that's amazing, thank you! – manderson Sep 07 '17 at 17:19
  • We're reading in the file line by line as an entire string (`-R`). For each of those lines, we're splitting by the pipe putting into an array (`split("|")`). From what I can tell in your question, the request and response are in the 6th and 7th columns respectively. – Jeff Mercado Sep 07 '17 at 17:56
  • Argh. I've learned that this data is coming in via syslog to a file on my syslog server. I need to ingest these events as they come in (or asap anyways). I found a [https://stackoverflow.com/a/18341108/6752652](script) that will monitor the syslog file output, and create a new file w/ the cleaned up output. I'm sure there's a better way, but I don't know it. I'm using Red Hat 6.9 w/out inotify-tools installed. – manderson Sep 07 '17 at 18:01
0

While Jeff's answer pretty much sums it up, here's a specific example assembled from the sample data fragments. If the file data contains

Sep  1 15:52:26 | IdentityValidationApi |  |  |  | {"header":{"tenantId":"X03LHWE3"}, "serverTime":"2017-09-01T19:52:24.641Z"} | {"responseHeader":{"tenantID": "...", "fieldValue":"Engineer"}} | D2C CrossCore Request-Response | IdentityValidationApi.corp-dev.com | /api/Inquiry | 172.30.68.88 |  | True

then 

$ jq -M -Rc './"|" | .[5] | fromjson' data

will produce just the json fragment from column 5:

{"header":{"tenantId":"X03LHWE3"},"serverTime":"2017-09-01T19:52:24.641Z"}

This filter

$ jq -M -Rc './"|" | (.[5]|fromjson) + (.[6]|fromjson)' data

will combine the objects in columns 5 and 6 into one object:

{"header":{"tenantId":"X03LHWE3"},"serverTime":"2017-09-01T19:52:24.641Z","responseHeader":{"tenantID":"...","fieldValue":"Engineer"}}
jq170727
  • 13,159
  • 3
  • 46
  • 56
  • Oh interesting, I didn't realize that we can use division to split strings. – Jeff Mercado Sep 07 '17 at 19:04
  • Yes this morning I learned [Dividing a string by another splits the first using the second as separators.](https://stedolan.github.io/jq/manual/#Multiplication,division,modulo:*,/,and%) – jq170727 Sep 07 '17 at 19:13