Concepts towards the Implementation of a Data Mesh in Industry Practice

Data is one of the most popular resources of our time, as it can be used to gain many valuable insights. These can be used profitably, for example to create new products or services or to optimise existing processes. To do this, the data must be analysed.

With the help of centralised data management systems such as Data Lake, domain experts can work with data from multiple source systems in a consolidated form. Processing and maintaining the data is the task of a centralised IT team. The IT team has to process a large number of different and complex use cases. However, domain knowledge is required for appropriate processing. This is therefore a difficult task for a single team and creates a bottleneck.

Data Mesh is a new approach to managing, sharing and accessing data at scale that uses decentralisation to overcome the challenges of centralised approaches mentioned above. Responsibility for the data is shifted to the domains that own the data. There, domain-orientated, cross-functional data teams are supported by a self-service platform in maintaining and processing the data. The data is exchanged as data products between the domains to enable analyses in other domains.

While more and more progress is being made in the realisation of data mesh, it remains unclear how exactly data mesh can be designed and implemented in a concrete case, including in terms of data products, federated management and interoperability. The aim of this research project is therefore to investigate approaches for realising the data mesh concept in an industrial context.

The project is currently in the initial phase. A literature review is currently being carried out to obtain an overview of the state of research and the state of the art in the industry. Based on this, research gaps are being identified which will be addressed as part of the project. Work is also currently underway on the first publication on the data protection compliance of data products.