Abstract

It is often necessary to build a proof of concept for a bigdata project to show the feasibility to customers. With OSTMap (Open Source Tweet Map) we proved that it is possible to accomplish this with the right choice of technologies in a short time frame. OSTMap enables the user to view live, geotagged tweets of the last hours on a map, search all collected tweets by a term or user name and view the amount of incoming tweets per minute distinguished by language as a graph. With OSTMap we cover several important areas of a typical big data PoC project: scalability, stream processing and ingest of incoming data, batch processing of stored data, performant queries on the data and visualization of the data. To achieve this, we use Apache Flink and Apache Accumulo as backend technologies, AngularJS and Leaflet for the frontend. OSTMap was developed iteratively using the walking skeleton approach which allowed us to increase the feature set constantly even with a strict deadline.

Slides: Matthias Kricke_Martin Grimmer_Michael Schmeißer – Building a real time Tweet map with Flink in six weeks

Video on YouTube

Details