Data Mesh Architecture Explained: What It Means for Your Data Platform

Understanding Data Mesh Architecture: How It Relates to Your Data Platform

Introduction

Centralized data platforms are facing a common issue.

Centralized data platforms offer data lakes or warehouses initially and work well, but once scaled up, problems arise.

When you add more teams, pipelines and dependencies, your "one version of the truth" becomes a point of failure.

As Staff or Principal Engineer, this is something you have likely experienced yourself. Slow data delivery. Weak pipelines. Confused ownership.

This is where data mesh architecture comes into play. Not simply an added tool but rather a new approach to developing, owning and scaling data within the organization.

Instead of having data centralized, the data mesh approach allows individual domain teams to take responsibility for their own data while still maintaining governance and interoperability.

What is Data Mesh Architecture?

Data mesh architecture is a decentralized approach toward designing a modern data platform. It treats data as a product and as such has domain teams owning and managing it, rather than a centralized data team.The core concept behind data mesh is changing from a single centralised data platform team responsible for managing all the data pipelines to having domain data ownership, a self-service infrastructure and federated governance.

Another way of putting this is you move from:

• One centralised data platform team managing the pipelines
• One large monolithic data lake or warehouse

to;

• Domain data ownership
• Self-service data platform
• Federated governance

The reason it is important to make this shift:

Centralisation has limitations at scale:

• Missing context across domains
• Team bottlenecks
• Unclear ownership of data

Data Mesh solves these issues by orienting data ownership to business domains.

Example

For example, in a fintech company, this is what your data would look like:

• The payments domain owns transaction data
• The risk domain owns fraud signals
• And the customer domain owns user information

So each of the three domains publishes data products that all the other domains can use.

The main point to remember with this type of architecture is that the data ownership shifts from a platform to a domain, while the governance remains intact.

The 4 Principles for Data Mesh Architecture

Data mesh is built on the foundation of these 4 principles that help explain how data mesh works.

1. Domain-Oriented Ownership

The data team in the domain is responsible for all data-related items from end to end, such as;

• Data pipelines
• Data quality
• Documentation
• Access controls

This takes away dependencies between the domain and the central data team.

2. Data as Product

No longer is data a secondary product. It becomes a product with:

• Service Level Agreements
• Schemas
• Versioning
• Discoverability

Think of data as an application where APIs are needed to access them.

3. Self-Serve Data Platform

The centralised platform team still exists and their role is to build;

• Infrastructure abstraction
• Tooling
• Templates
• Guardrails

Examples of this may look like;

• Managed Data pipelines
• Data Catalog
• Observability Tools

4. Federated Computational Governance

Governance is not removed but distributed over the federated network. Compliance policies are enforced via;

• Automation
• Controls within the platform
• Common standards

This allows for compliance butMain Takeaway: The combination of autonomy and standardization of governance leads to the success of Data Mesh.

Why Data Mesh Architecture Is Crucial Today

The Data Mesh is no longer a hypothetical concept, but rather exists as a solution to a real-world problem and is primarily driven by serious scaling challenges.

Rapid, exponential increase of data sources generating data from:

• Microservices
• IoT
• User Events
• AI Systems

The use of centralized pipelines can't keep pace.

Increased demand for real-time data insights

There is great demand among teams for:

• Near-Real-Time Data Analytics
• Self-Serve Data Access
• Fast Iteration
• Scalability within the Organization

As organizations grow, they tend to see:

• Increased role specialization among teams
• Increased complexity of their domains
• Fragmentation of data ownership
• Varying levels of visibility into industry signals

McKinsey found that organizations with decentralized data ownership adopted analytics 4 times faster, had 10 times better quality, and increased employee proficiency.

The Implications of Not Using Data Mesh

• Increase in data silos
• Degradation in the functionality of data pipelines
• Duplicate work among teams
• Ineffective governance

Main Takeaway: Building a Data Mesh Architecture is a response to scaling, not just a trend.

A Comparison of Data Lakes, Data Fabrics, and Data Meshes

This is one of the most frequently searched-for subjects, so here we provide clarity:

Data Lake

• Centralized data storage
• Schema-on-Read
• No formal governance

Data Fabric

• Focus is on data integration
• Emphasis on the use of Metadata
• Centralized Intelligence Layer

Data Mesh

• Organizational and Architectural Shift
• Decentralized Ownership of Data with an emphasis on product thinking

Summary: Data Mesh is not a software solution but an organizational schema for constructing a data platform.

Putting Data Mesh Architecture Into Practice

Let’s move forward from the theoretical to practical implementations.

Step 1 - Domains

Determine which are the business-aligned domains within the organization. Some examples might include:

• Payments
• Logistics
• Customer Data

Each domain will become responsible for and the owner of its own data.

Establish Data Product Development

In this stage, each Data Domain will publish Data Products that will include:

• Clearly defined schema
• Metadata associated to Ownership of the Data
• Rules for access to the data

Enabling a Self Service Infrastructure

Each Platform Team will provide:

• Implemented CI/CD pipelines for Data
• A Data Catalog system
• Observability layers for Data

Establish Governance

To govern the use of data, we use policy-as-code to enforce:

• Security of Data
• Compliance with regulatory requirements
• Standards for Data

Establish Discoverability of Data

To find and trust the data, we must provide:

• Data Catalogs that allow discovery
• Lineage tracking for all Data
• Metrics for the quality of Data

Every Domain will have real-world tools that will allow them to discover and trust the data

Commonly used Technology to support Data Mesh Architecture are:

• AWS DataZone
• Databricks
• Snowflake
• Kafka
• dbt

Key Takeaway: Data Mesh is an iterative implementation, and not a big bang implementation

Common Challenges to Adopt Data Mesh

Even though there are benefits to implementing a Data Mesh, Organizations face many challenges

1. Organizational Resistance

Most Teams are used to employing a centrally owned process.

Solution: Start with a single Domain, and demonstrate the benefit of a Data Mesh implementation.

2. Lack of Maturity of the Platform

If there is limited tooling in your Organization, your attempt to decentralize will be unsuccessful.

Solution: Investment in the development of a mature Platform

3. Governance Complexity

Organizations with limited governance structure cannot easily manage the flow of data and will often expose Business with unnecessary risk.

Solution: Automate the governance process, and utilize standard contracts for data sharing.

4. Skills Shortage

Domain Teams may not have experience with Data Engineering Disciplines.

Solution: Provide Templates for Data Engineering Provide Training and Enablement programs.

5. Over Engineering

Some Groups will attempt to accomplish all at once by over-engineering the implementation of a Data Mesh.

Solution: Start Small, and build out over time.

Key Takeaway: Treating a Data Mesh as an IT Project vs. an Organizational Change will result in failure.

What can I expect from a Data Mesh Architecture Diagram?

There are numerous implementations of data mesh architecture; however, most of them contain similar four layers:

• Domain data pipelines
• A shared infrastructure layer
• Data product interfaces
• A governance layer

Typical data flow:

• Domain ingests raw data
• Processing of the data to create curated datasets
• Publishing of data products out
• Other domains consume with an api's or queries

Core layers of a data architecture:

• Infrastructure Layer: Storage; Compute
• Platform Layer: Tools and orchestration
• Domain Layer: Business owned pipelines
• Governance Layer: Policies and compliance

The architecture is layered and ownership is distributed.

When Should I Implement Data Mesh Architecture?

Not every company will benefit from implementing data mesh architecture; however, some of the reasons why you should implement it include:

• A large organisation with multiple independent domains
• A data team that’s become a bottleneck
• Difficulties in scaling data pipelines
• Teams requiring faster access to their data

Reasons to not implement include:

• A small organisation
• Low data complexity
• An immature organisation’s data platform

Checklist For Maturity

Before implementing data mesh architecture to ensure you move forward in your digital journey is if:

• Your company has a well-defined cloud architecture/environment
• Your company is able to observe using adequate tools/methods
• Your company has clearly defined boundaries between domains
• Your organisation’s engineer culture supports domain ownership / accountability for their products

The implementation of the data mesh is simply a method for scaling up large organisation’s current data platforms.

Case study - the improved delivery velocity by implementing Data Mesh

In one of Logiciel’s initiatives to modernise its internal platform, the company had all of its data pipelines manually run at the central level, resulting in an extended amount of time (typically weeks) to deliver data products from its teams that were relying directly on data engineering resources.

Once they implemented data mesh and allocated full domain ownership to their teams, the following improvements occurred:

• Full domain ownership for data pipeline execution (from data owners to domain product teams)
• Reduced delivery times by 35%
• An increase in data quality through team owned accountability and ownership of their data products.

These findings align with broader industry experiences where owning data decentralised from a single location speeds up the ability to iterate.

Key Insight: Ownership is Essential for Speed. Data Mesh makes Ownership Clear.

Conclusion: Movement away from data platforms and toward Data Products

When transitioning to a data mesh architecture, the goal is not to replace everything you already have in your data stack, but rather to rethink how your data platforms work at the edge.

This means, for Staff and Principal Engineers:

• Building systems that align with domain ownership instead of aligning with data pipelines;
• Creating platforms that help teams rather than dictate to teams how to do their work;
• Valuing data as a product that is equally important to the business as any other product.

Being able to execute on these principles correctly, quickly will provide organizations with greater velocity in moving to make decisions faster than organizations that do not.

Logiciel Solutions helps technology executives to accelerate their AI adoption. Our AI-first engineering teams build systems that are ready for production in a way to increase their ability to deliver on time, reliably, and at a sustainable cost. Learn how to intelligently scale your organization’s data platform.

Extended FAQs