Data Platform Architectures & Technologies

In recent years, enterprises of almost all industrial sectors have become subject to fundamental paradigm shifts: Large-scale projects, such as in the area of Industry 4.0, are driving the digital transformation and pursue to take a holistic view on value chains in order to enable cross-phase optimizations.

In order to keep up with this development and benefit from it, enterprises need to collect huge amounts of heterogenous data across the entire value chain, organize it in a structured and re-usable manner and exploit it by applying data-driven analysis techniques to gain insights and knowledge.

For storing and managing the collected data, as well as to enable data preparation, processing and analytics applications, different types of data platforms have emerged in the past decades. They range from traditional data warehouses and the more recent data lakes to metadata management platforms such as data catalogs and enterprise data marketplaces, each serving different purposes. For many enterprises, this results in a large, diverse landscape of data platforms with complex architectures and further shortcomings, such as redundant storage of data and slow analytical processes. Hence, efforts to simplify architectures and technology stacks have recently become apparent, driven by novel approaches such as the delta architecture and lakehouse frameworks.

As the range of data platforms is rapidly evolving, the goal of this research project is to investigate and prototype upcoming architectures and technologies and to assess their applicability and potentials for industrial enterprises.

Funded by: Robert Bosch GmbH