Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I currently have a situation in which logstash pulls a JSON array from azure event hubs which i need to split into multiple events.

The logs i get from the event hub look like this:

{"records": [{JSON LOG},{JSON LOG},{JSON LOG},...,{JSON LOG}]}
I have tried using split and the json filter but i can't quite seem to get it to work. I basically want to split up the array so that logstash sends to elastic each of the JSON Logs in records as a separate event, parsed as json.
I also need to rename/parse the individual JSON logs into ECS so currently think i need to parse records as json and then parse the output as json before doing some mutate rename filters before sending to elastic, unless it would be easier to just do the parsing as JSON in logstash with an elastic index pipeline for the parsing to ECS.
My current filter section is:
filter {    
    #Split results into individual events
    json {
        source => "message"
    #add a target_index field for the final index to send to
    mutate {
        replace => [ "[@metadata][target_index]", "logs-eventhub" ]
Would anyone be able to provide some insight into how to do multiple json parses in logstash so that i get each record in it's own event parsed as JSON.
                You're missing the split filter which does the job of creating individual events from the record array
– Val
                Jan 11, 2022 at 9:33
                So i would split it with a field of record and then after that in the logstash put the json filter? filter {	 	#Split results into individual events 	split { 		field => "records" 	} 	 	json { 		source => "records" 	} 	 	#add a target_index field for the final index to send to     mutate { 		replace => [ "[@metadata][target_index]", "logs-eventhub" ]     } }
– 40339109
                Jan 11, 2022 at 9:36
If your input parses the data as JSON, you don't need a json filter, you'd simply need to use the split filter like this:
filter {    
    #Split results into individual events
    split {
        source => "records"
    #add a target_index field for the final index to send to
    mutate {
        replace => [ "[@metadata][target_index]", "logs-eventhub" ]
What the split filter will do is to clone your original event, split the records array and place each sub-record into a clone of the original event.
Try it out!!
                So it appears as if my input doesn't parse the data as JSON. When running it as shown above in elastic i get a message containing : {"records": [{JSON LOG},{JSON LOG},{JSON LOG},...,{JSON LOG}]} and a tag of _split_type_failure. I have tried to modify it so that i has a specified terminator of "," but my logstash still gives me the error above.
– 40339109
                Jan 11, 2022 at 14:42
                I have worked out the issue, after adding codec => "json" it now works with each of the records in it's own field. However the way it's been extrapolated is that my fields now end up as records.catagory, records.properties.DeviceName, etc. Is there an easy way to change the records and possibly the records.properties fields to something else, for example so the logs end up as example.catagory, example.DeviceName, etc? if it's not a quick solution i don't mind opening a new question for it.
– 40339109
                Jan 11, 2022 at 15:25
                Cool, glad you figured out. You can use the rename filter to rename the field from records.properties.DeviceName to example.DeviceName for instance
– Val
                Jan 11, 2022 at 15:29
                The only issue i've got with that is there are 11 types of records, each with their own set of sub-fields which i don't know fully yet. Is there a way i can programmatically remove the records part only, that would be a start i guess and allow me to at least start analysing the data in more depth.
– 40339109
                Jan 11, 2022 at 15:37
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.