Since June 2016, Flink-as-a-service has been available to researchers and companies in Sweden from the Swedish ICT SICS Data Center at www.hops.site using the HopsWorks platform. Flink applications can be either deployed as jobs (batch or streaming) or written and run directly from Apache Zeppelin on YARN. Flink applications are run within a project on a YARN cluster with the novel property that Flink applications are metered and charged to projects. Projects are also securely isolated from each other and include support for project-specific Kafka topics that are protected from access by users that are not members of the project. Hopsworks is entirely UI-driven, is open-source, and Flink applications that include Kafka topics can be created in a few mouse clicks. In this talk we will discuss the challenges in building a metered version of Flink-as-a-Service for YARN, experiences with Flink-on-YARN, and some of the possibilities that Hopsworks opens up for building secure, multi-ten

Slides: Jim Dowling – Multi-tenant Flink-as-a-Service on YARN

Video on YouTube


Jim Dowling
Senior Researcher, SICS Swedish ICT