Data Fabric Architecture Deep Dive - Building a Unified Data Layer

The platforms of the vast majority of organizations are not failing because they lack data; they fail because they have too many different types of data and data sources causing fragmentation throughout their environment (teams working with different data sources, pipelines, storage systems, etc.). This results in all of them working independently, which can result in different governance models being used, as well as lots of disconnected pieces of software.

What this means for an organization is that it’s not only inconvenient—for an Engineer at a Principal or Staff level—but it’s also a significant bottleneck preventing them from performing analytics on the company-wide scale, limiting the functionality of machine learning pipelines, and creating operational risks if things go wrong.

This is where data fabric architecture comes into play.

A data fabric architecture is not just another example of an architectural pattern, but it is a new methodology for how companies will access, govern, and deliver data across their entire distributed environment.

This deep-dived research article will cover what actually constitutes data fabric architecture; the primary components that comprise data fabric reference architecture; how it compares to a data mesh architecture; how to design and implement a unified data layer; and common mistakes and best practices for building data fabrics.

Definition of Data Fabric Architecture

At a basic level, data fabric architecture provides an architectural approach for how companies can both access all relevant data in their environment in a unified, consistent manner, and build and deliver it to users across distributed environments.

Data fabrics help companies abstract away the complexity of working with large-scale distributions of various data types across multiple systems by providing a single interface to work from.

A data fabric provides a way for organizations to manage their data across distributed storage systems, without having to move all of their data into one system.

The data fabric connects and orchestrates data from:

• Database
• Data warehouses
• Streaming systems
• On-premise and cloud systems

How Does a Data Fabric Architecture Benefit My Business?

From a business perspective, the data fabric provides:

• Faster access to trusted data
• Improved governance of data
• Reduce redundancy
• Improves decision making

Due to these benefits, a lot of organizations are asking:

• What is a data fabric architecture?
• What is the importance of a data fabric architecture?

This is because the focus of a data fabric architecture is shifted away from storage and to access and intelligence of data.

Why Traditional Architectures Fail at Scale

Before an organization can build a data fabric, it must first understand why their current architecture has failed.

Siloed systems
Different groups within an organization create different data pipelines and different data storage layers.

Result:

• Duplicated data
• Inconsistent metrics

Data Movement Overhead
Traditional architectures rely on using:

• ETL (extract, transform, load) processes
• Batch processing

Due to this reliance on processing data in batches, organizations are experiencing increased latency and complexity.

Governance Issues
Different policies are in place for different systems; thus limiting the ability to govern data effectively.

Real-Time Access Limitations
Batch processing systems cannot support real-time access to data; therefore, businesses are unable to use data for real-time decision making.

Key Insight
Traditionally, architectures optimally focus on controlling data.
The data fabric optimally focuses on accessing and interacting with data.

Core Components of a Data Fabric Architecture

A complete data fabric consists of several important layers of architecture.

1. Data Integration Layer

This layer processes the following types of data:

• Ingest (the ingestion of data from multiple sources)
• Transform (the transformation of data into a consistent format)
• Stream (the streaming of data from an ingestion source to one or more destinations)

This is the layer used to link all source systems of data.

2. Metadata Layer

The most fundamental layer of a data fabric.

The metadata layer provides the following functional capabilities:

• Cataloging of data
• Lineage of data
• Management of data schemas

Metadata allows organizations to find data more intelligently by following data lineage.

3. Data Governance Layer

The governance layer ensures:

• Security policies are in place
• Access controls are in place
• Compliance is maintained

4. Data Access Layer

The data access layer provides unified access to data by using:

• APIs (application programming interfaces) to get access to data
• Query engines to get access to data
• Virtualization to get access to data

5. Orchestration Layer

The orchestration layer coordinates data workflow and data pipelines across multiple data sources.

6. Intelligence Layer

Utilizes Artificial Intelligence to:

• Automatic data discovery
• Streamline query process
• Identify anomalies
• Visualize architecture

Typical data fabric architecture diagrams include:

• Distributed data sources
• Centralized metadata layer
• Aggregate access layer

Key Takeaway: Metadata is the "glue" to data fabric architecture.

Data Fabric vs. Data Mesh – Key Differences

A frequently asked question is:
What is the distinction between a data mesh and a data fabric?

Data Fabric

• Tech-driven
• Integrate and access data
• Centralized data governance

Data Mesh

• Org-based
• Own data by domain
• Decentralized data governance

When To Utilize Each

• Use data fabrics when: You require a unified access point across your systems
• Use data meshes when: You desire domain ownership of your data

Hybrid Approach

Many enterprises utilize both.

Key Takeaway: Data fabrics and data meshes are not in competition; they are in synergy.

How To Create A Unified Data Layer Using Data Fabrics

In order to develop a data fabric architecture in Cloud technology (i.e., Azure, AWS or combination thereof), a systematic approach must be taken.

1. Identify Your Data Landscape

• Data Sources
• Storage Solutions
• Pipelines

2. Put Together A Solid Metadata Foundation

Focus on:

• Cataloging
• Lineage Tracking
• Classifying Your Data

3. Enable Data Virtualization

Prevent moving data unnecessarily.

Utilize query federation as a method for accessing data to standardize governance by establishing policies for accessing data and creating compliance rules.

Create a unified access API using APIs and query layers.

Artificial intelligence can help to optimize queries, recommend data sets, and identify anomalies.

Example

A large enterprise can link together its cloud-based data warehouse, its on-premise database, and its streaming data pipeline through the use of a data fabric to provide a unified way to access their data without having to move systems or data.

Data Fabric abstracts the complexity of managing different systems.

Data Fabric Architecture Examples

• Microsoft Azure Data Fabric Architecture: Azure Synapse, Azure Data Factory, Azure Purview
• Amazon AWS Data Fabric Architecture: Glue, Lake Formation, Redshift
• IBM Data Fabric and HPE Ezmeral

Relevant Information:
All the platforms above utilize the same architecture to offer their users various benefits.

Best Practices to Build a Data Fabric

• Start with metadata
• Do not restructure your architecture
• Focus on interoperability
• Design for scale
• Provide a developer-friendly data fabric
• Design for business needs

Data Fabric Common Issues

• Believing that a Data Fabric is a tool
• Insufficient governance
• Overly complicated design
• Lack of ownership
• Underestimating integration complexity

Future of Data Fabric Architecture

According to Gartner, several areas will influence the development of data fabric architecture:

Emerging Trends in Data Fabric Architecture

• Autonomous data pipelines
• Intelligent data discovery
• Cloud vs. on-premises interoperability

Strategic Considerations for Data Fabric Architecture

Data fabric architecture will serve as the foundation for AI-first organizations, real-time decision systems, and many more.

Frequently Asked Questions

Conclusion: Building the Next Generation of Data Platforms

Data platforms are no longer centralized; they are being designed as distributed, dynamic, and complex systems. Without a central point of commonality (the unifying layer), it is impossible to manage the complexities naturally associated with today's digital environments.

Data fabric architecture provides the unifying layer needed for managing today's complex environment and provides a platform for achieving:

• Unified access to data
• Consistent governance, and
• Scalable data systems

As engineering leaders, you must view your data fabric architecture as foundational to your business.

Logiciel’s Perspective

At Logiciel Solutions, our company designs AI-first data platforms with data fabric architecture enabling seamless integration, governance, and scalability across your distributed systems.

In addition, our team of data architects will assist you in developing a unified data layer composed of architecture and supporting technologies that will create analytic capabilities and provide real-time processing and support for artificial intelligence workloads in your organization without adding unnecessary complexity.

If you are experiencing fragmentation in your data ecosystem, it is time to adopt a data fabric approach that will scale with your business. By adopting this unified data layer, you can develop analytics capabilities that will sustain your future business, not just support your present business.