django-traffic, a middleware for traffic visualizations in Kibana

I was doing an all-around research about Python packages that could be used in a web app and provide traffic/requests metrics on top of those data. The stack I manage at Madinad consists of 3 production app servers and the ELK stack for internal analytics, log aggregation and data metrics. Requests are flooding in mainly from the EU area and as our business keeps growing, I needed something that could help me visualize our traffic and get insights about it, starting from a map visualization.

Our team heavily uses Kibana for data metrics so a map-tile visualization would fit in just great at this point. Our backend is almost entirely written in Python on top of Django, so I though a middleware would be nice to intercept the requests and send traffic information in ElasticSearch. From there it’s just a matter of time to create a visualization for this purpose.

Strangely, I didn’t found something for this, so I created my own middleware for this. You can find it here.

https://github.com/koslibpro/django-traffic/

It’s a pretty basic solution that requires minimal input from the user, just some variables set in the settings.py file of Django and then no effort at all to work. In fact, it finds the device IP of every request – no worries it if comes out of a load balancer, I’ve got this case covered – and finds it’s latitude and longitude. That’s enough data to create a geo_point in ElasticSearch, assuming there’s an index with the correct mapping created first, which is something the middleware takes care of as well.

Finally, on Kibana go to Visualizations>Map Tiles and select the desired index. Automatically it will load the geo_point type field so hit the Run button and it should display a map with your traffic data.

traffic heatmap

 

Setup Instructions

Feel free to contact me if you need any help with this and of course fork the repo to adjust this project to your needs.