Dynamic Lookup using Kibana-CodePudding

I'm trying to Visualize some data in kabana.
Let's take an example:
I've few entries for selling some Products with their transaction IDs as one event and its cost in another event.

Line  1     {"event" : {"transId" : "q4h56", "prod" : "notebook"} }  
Line  2     {"event" : {"transId" : "q4h56", "cost" : 100       } }  
Line  3     {"event" : {"transId" : "rt45e", "prod" : "crayons" } }  
Line  4     {"event" : {"transId" : "rt45e", "cost" : 75        } }  
Line  5     {"event" : {"transId" : "gr56d", "prod" : "pen"     } }  
Line  6     {"event" : {"transId" : "gr56d", "cost" : 83        } }  
Line  7     {"event" : {"transId" : "uh65r", "prod" : "paper"   } }  
Line  8     {"event" : {"transId" : "uh65r", "cost" : 49        } }  
Line  9     {"event" : {"transId" : "yh5er", "prod" : "crayons" } }  
Line 10     {"event" : {"transId" : "yh5er", "cost" : 140       } }  
Line 11     {"event" : {"transId" : "23yg5", "prod" : "pen"     } }  
Line 12     {"event" : {"transId" : "23yg5", "cost" : 83        } }

What I want to achieve:

1: Using Log stash:

Is it possible to merge events with similar transId as single event, Like below:

Line  1     {"event" : {"transId" : "q4h56", "prod" : "notebook",   "cost" : 100} }  
Line  2     {"event" : {"transId" : "rt45e", "prod" : "crayons",    "cost" : 75 } }  
Line  3     {"event" : {"transId" : "gr56d", "prod" : "pen",        "cost" : 83 } }  
Line  4     {"event" : {"transId" : "uh65r", "prod" : "paper",      "cost" : 49 } }  
Line  5     {"event" : {"transId" : "yh5er", "prod" : "crayons",    "cost" : 140} }  
Line  6     {"event" : {"transId" : "23yg5", "prod" : "pen",        "cost" : 83 } }

since log stash parses one line at a time, this seems unachievable.
I need to store bunch of global key/value pair to be used later on, such as"%{transId}":"%{prod}"and if same transId appear again I can insert the prod there assuming it will have cost already.

*[@metadata] is unique for every line log stash parses, and will not be available in next parsing cycle by log stash. if I'm not wrong!

2: Using Kibana:

Is it possible to visualize sum of [cost] for each similar product we sell.
since the cost is not mapped with product name.
looking of these prod in visualization, don't provide the cost.

How to map these, some kind of dynamic lookup / join, which I feel not being supported by Kibana, Again, if I'm not wrong!

Any suggestion ?
Again this is just a sample data, Actually working with syslog having events with similar situation.

Thanks

CodePudding user response：

You can do this in logstash using an aggregate filter. Your use case is similar to example 1 in the documentation.

If the order of events is variable then go with example 3. In that case store both the product name and the cost in the map, and event.cancel all the source events.

CodePudding user response：

Maybe the aggregate filter from Logstash can help you achieve that.

Considering the sample data:

{"event" : {"transId" : "q4h56", "prod" : "notebook"} }
{"event" : {"transId" : "q4h56", "cost" : 100 } }
{"event" : {"transId" : "rt45e", "prod" : "crayons" } }
{"event" : {"transId" : "rt45e", "cost" : 75} }
{"event" : {"transId" : "gr56d", "prod" : "pen" } }
{"event" : {"transId" : "gr56d", "cost" : 83} }
{"event" : {"transId" : "uh65r", "prod" : "paper" } }
{"event" : {"transId" : "uh65r", "cost" : 49} }
{"event" : {"transId" : "yh5er", "prod" : "crayons" } }
{"event" : {"transId" : "yh5er", "cost" : 140 } }
{"event" : {"transId" : "23yg5", "prod" : "pen" } }
{"event" : {"transId" : "23yg5", "cost" : 83} }

The following aggregate filter will aggregate the lines based on the transId value.

aggregate {
    task_id => "%{[event][transId]}"
    code => "
        map['event'] ||= {}
        map['event']['transId'] ||= event.get('[event][transId]')
        map['event']['prod'] ||= event.get('[event][prod]')
        map['event']['cost'] ||= event.get('[event][cost]')
        event.cancel()
    "
    push_previous_map_as_event => true
}

This will give you an event with both the prod and cost field.