Master Project: Distributed Systems

This winter semester, we offer a master project around adaptive resource management for stream processing jobs.

Distributed Stream Processing (DSP) systems are critical to the processing of vast amounts of data in real-time. It is here where events must traverse a graph of streaming operators to allow for the extraction of valuable information. There are many scenarios where this information is at its most valuable at the time of data arrival and therefore systems must deliver a predictable level of performance. Examples of such scenarios include IoT data processing, click stream analytics, network monitoring, financial fraud detection, spam filtering, news processing, and many more. In order to process these large streams of data, DSP systems such as Storm, Spark, and Flink have been introduced which allow for the deployment of analytics pipelines which utilize the processing power of a cluster of commodity nodes. Applications developed within these frameworks are, in principle, required to operate indefinitely on an unbounded stream of continuous data in an environment where partial failures are to be expected as these applications scale. Consequently, DSP systems feature high availability modes, implement fault tolerance mechanisms by default, and expose a rich set of continuously evolving features.

In this master project, you together with your team will be required to design and implement a streaming analytics pipeline taking advantage of the latest OS virtualization and container orchestration technologies. You will be required to examine existing real-world data streams and create a generator to emulate the underlying data patterns. Once the proof of concept has been presented for your analytics pipeline, your group will then be assigned a topic within the area of adaptive resource management and will be expected to develop a scientific approach for achieving this. Topic areas may include: adaptive fault tolerance, runtime optimization, automatic parameter tuning, monitoring, modeling and runtime prediction, profiling, resource allocation, and fault detection/identification.

Master students with solid programming skills, who would like to gain some hands-on experience in a practical project in the area of distributed and operating systems.

• We expect participants to have a basic understanding of distributed stream processing systems, frameworks such as Flink or Spark, and a keen interest in adaptive resource management. 
• We assume solid programming skills in at least one widely used programming language (Java /Scala / Python) and also expect familiarity with virtualization technologies and Linux/Unix systems.