Open-Source Architecture for the Data Mesh

Ia (iya) Mgvdliashvili

Introduction

The typical setup for business intelligence

  • Single team as a data practice bottleneck
  • Implementation latency due to stakeholder conflicts and lack of ownership
  • You can purchase a database with boundless storage, but can’t throw money at scaling the schemas

We know how to solve these things

  • Domain focus, accountability -> ownership
  • Ownership requires control

What is a Data Mesh

  • Organizational structure
  • Approach to data practice
  • Platform & tooling

Data Mesh is not a replacement for warehouses

It is a Federation

The principles

  • Domain-driven, decentralized, federated
  • Data as a Product
  • Self-serve, federated platform
  • Data Governance

Data as a Product

  • discoverable
  • addressable
  • trustworthy
  • interoperable
  • secure

Implementing a Data Mesh

Components summary

  • Data/metadata catalog
  • Versioned schemas and pipelines
  • Ingestion
  • Global IAM
  • Visualization & Analytics

Trino & Ranger - query federation

  • External IAM
  • Centralized Access Management
  • Service accounts
FROM <catalog>.<database>.<schema>.<table>

DataHub - Data Catalog

  • Data & Ownership discovery
  • Metadata catalog
  • Lineage - impact analysis

Airbyte - Data Ingestion

  • custom connectors for enterprise systems
  • replicate data internally

Temporal - Durable Execution

  • runtime for Airbyte ingestions
  • dbt pipelines
  • custom pipelines

Operating a Data Mesh

Canonical & Digital Sovereignty

Ops Framework & CharmHub

Apply rigorous engineering practices to operators

  • Charms: Describe application dependencies & behavior via Python Library
  • juju: execute operational logic of charms.

juju & charms in practice

# Example: make a governed analytics path available to users

# Ingest data
juju deploy airbyte-k8s airbyte
juju deploy postgresql-k8s metadata-db --channel 14/stable --trust
juju integrate airbyte metadata-db

# Query across governed sources
juju deploy trino-k8s trino
juju deploy hive-metastore-k8s hive-metastore
juju integrate trino hive-metastore

# Enforce access policy
juju deploy ranger-k8s ranger
juju integrate trino ranger

# Publish discovery and lineage
juju deploy datahub-k8s datahub
juju integrate datahub trino

# Enable data visualization
juju deploy superset-k8s superset
juju integrate superset trino


# Check the full relation graph
juju status --relations

Q&A