Data-M²

Enormous amounts of data are generated today within companies through, for instance, the Internet of Things (IoT) and Industry 4.0 initiatives. This data contains a potential value which may lead to new insights, the discovery of new business models or the expansion into new markets. The potential data value can, however, only be exploited if the company’s employees can find, access and use it for their respective use cases. However, it has been reported that up to two thirds of data in the enterprises remains unused. Therefore, data democratization initiatives with the goal of empowering and motivating employees to find, understand, access, use and share data across the company, are gaining importance. To drive democratization aspects such as data sharing across the company, the use of enterprise data marketplaces has been proposed. In general, data marketplaces are metadata-driven self-serve platforms for trading data and data related services. The enterprise data marketplace is specifically designed to facilitate the exchange of data and data related services within a company between company employees. In order for the enterprise data marketplace to offer most of the data stored within the company, it must integrate with the company’s system landscape for both operational as well as analytical data and the data organizational structure. This means it must support the variety of systems such as enterprise resource planning systems, data warehouses or data lakes and be able to reflect idiosyncrasies of these systems such as a data lake’s zone architecture structuring the contained data according to different processing states.
Recently, there has been an increased realization that the data architectures and organizational structures for analytical data, including data warehouse and data lake architectures, have a set of characteristics that inhibit them scaling organizationally and thus prevent the exploitation of the analytical data value as desired. Amongst others these limitations are based on their monolithic character, centralized ownership and technical partitioning. In consequence, a new organizational paradigm for handling analytical data has emerged called the data mesh. It is a decentralized approach for managing, sharing and accessing analytical data at scale inside and beyond enterprises. It is the objective of this micro-project to investigate how an enterprise data marketplace integrates with the emerging data mesh concept, and can support democratizing data within this specific organizational structure.