As a Data Engineering lead you are expected to deliver scalable systems that support analytics and enable artificial intelligence (AI) while minimizing costs and complexity.
But there’s an underlying issue:
Most organizations build their data infrastructures without understanding what their data architecture looks like.
This leads to:
- Disjointed data pipelines;
- Increased cloud spending;
- Inconsistent data quality;
- Systems that fail to scale.
Understanding the differences between data infrastructure versus data architecture are two of the most significant decisions you’ll ever make as a data engineer—this will be one of the most impactful decisions you’ll ever make.
We provide you with an easy-to-understand, engineering-first guide to differentiate between the two concepts and give you guidance on how to build scalable data systems.
What Is Data Architecture?
The data architecture describes how the data is organized, structured, and moves through your data systems.
Essentially, the data architecture is the strategy you create to organize, store and move data throughout your company's data systems — this includes everything from specifying how data is modeled and what types of data are included in that model to how they will be accessed and used by the respective systems.
Explaining data architecture within a real-world system gives the following example of how data architecture connects:
Operational Systems (Apps, APIs) AnalyThe conceptual data architecture diagram often has the following components included within it:
- Sources of Data
- Ingester Pipeline
- Storage Layer (Data Lake Architecture, Data Warehouse Architecture)
- Processing Layer
- Service Layer
Modern data architecture no longer has a monolithic shape, but has. This is split between different modes of architecture; such as:
- Data Lake Architecture for scalable raw storage
- Data Warehouse Architecture for analytics
- Data Lakehouse Architecture that combines both types of architectural storage (data lake and warehouse)
- Data Mesh Architecture that decentralizes ownership of data in your organization
Data Infrastructure
Definition
The data infrastructure stack is the backbone to facilitate all your data systems through:
- Compute Resources (CPU's, GPU's, et al.)
- Storage Systems
- Network
- Orchestration
Examples of Data Infrastructure
- Cloud Platforms (AWS, Azure, GCP)
- Processing engines (Spark, Flink)
- Storage Layers (S3, BigQuery, Snowflake)
- Container Orchestration (Kubernetes)
Traditional data centre architecture focused on building physical systems.
Now, the way infrastructure is architected is:
- Distributed
- Cloud-native
- API-driven
Data Infrastructure vs Data Architecture
The main difference is:
| Category | Data Architecture | Data Infrastructure |
|---|---|---|
| Purpose | Designing & Structuring | Enablement & Execution |
| Nature | Conceptual | Physical/Technical |
| Focus | Flow of Data, Data modelling | Tools, Compute, Storage |
| Ownership/Accountability | Architects | Engineers/DevOps |
To understand the difference more clearly, think of building a city:
- Your data architecture system is like urban planning.
- Your data infrastructure is everything needed to build the urban city (roads, electric, buildings, etc).
- If you don't have a good architectural plan for your city the infrastructure will be unorganized.
Why this difference impacts Data Engineering Leaders
1. Removes tool driven decision making
If you do not use your architecture to choose your tools, you will:
- End up with vendor lock-in,
- Poorly designed/pipeline
- Duplicate systems
2. Ensures systems can be built to scale
With strong data architecture, you can ensure that your infrastructure; can constantly scale when developed without having to redesign the architecture to support new requirements.
3. Improve data quality & governance
A strong data architecture will define:
- Ownership of data,
- Standards,
- Data lineage.
Advanced analytics and AI support
Strong architecture is necessary to ensure reliable data flows into the various types of AI applications.
Components of Data Architecture
1. Ingesting Data
- Batch Processing
- Streaming Ingestions
2. Where to Store Data
- Data Lakes
- Data Warehouses
- Lake House Systems
3. Processing the Data
- ETL/ELT Pipelines
- Real-time Processing
4. Access the Data
- APIs, BI Tools, Query Engines
5. Govern the Data
- Metadata, Data Lineage, Access Control
Foundational Principles of Modern Data Architecture
1. Scalability
Design systems to support increasing volume and complexity of data.
2. Modularity
Decouple Components for Flexible Designs
3. Interoperability
Ensure Systems Integrate Easily
4. Observability
Monitor health, latency and quality of Pipelines’ Data
5. Data as a Product
Provide Dataset Ownership and Accountability
Data Architect vs Data Engineering
Understanding the difference in focus is important in creating a functional team structure.
Role
Focus
Data Architect
Designing System and Standards
Data Engineer
Implementing and Creating Pipelines
Team Coordination
High-performing teams collaborate to define, implement, and optimize their data architecture:
- Data architects provide the direction
- Data engineers implement and optimize
How to Select Infrastructure That Supports Your Data Architecture
Selecting a Cloud Vendor for your Data Architecture
Key Factors
- The type of workload would determine the amount of batch vs streaming
- The amount of data and the speed at which it arrives
- Your costs will vary based on storage vs processing.
- Choosing the right Tools (ETL, ELT, BI, Data Visualization etc) to support your architecture and meet your organization’s needs.
Ecosystem Compatibility
- Tools Already In Use
Platform Choices for Creating A Scalable Data Architecture
- AWS - Flexibility/Ecosystem
- Azure - Support for Enterprise Integration
- Google Cloud - Analytics/AI Capability
Various Methods of Building Data Platforms
Centralized Architecture
The centralized architecture method has the following:
Pros:
- Strong governance
- Standardization
Cons:
- Bottlenecks
Data Mesh Architecture
The data mesh architecture method is characterized as:
Pros:
- Domain ownership
- Scalability
Cons:
- Requires a mature organization
Hybrid Approach
A hybrid approach combines centralized governance with decentralized execution
Common Patterns in Data Architecture
1. Data Lake Architecture
- Raw storage that can be scaled/shaped based on business needs
- Provides the flexability of a schema (schema on read)
2. Data Warehouse Architecture
- Structured analytical storage based on business needs
- Provides high performance
3. Data Lake House Architecture
- Provides a single store for data & analytics
4. Data Mesh Architecture
- Each domain inside an organization owns their data (data ownership by domain)
Data Security & Governance
Best practices regarding securing sensitive data across the entire architectural framework
Security Strategies
- At rest and in motion encryption
- Role-based access control
- Data masking
Governance Strategies
- Data catalog
- Lineage tracking
- Quality monitoring
Tools for Modeling Data Architecture
Best tools for modeling/visualizing data architecture
- Lucidchart
- ER/Studio
- Draw.io
- ArchiMate
Why Modeling is Important
- Easier to read diagrams
- Improve communication between teams
- Reduce errors
- Align stakeholders
Roles on Data Architecture Teams
Some common roles and responsibilities on Data Architecture teams are:
Data Architect:
- Defines the architecture strategy
- Ensures alignment of architecture strategy with the business strategy
Data Engineer:
- Builds the data pipeline
- Maintains the supporting infrastructure
Platform Engineer:
- Manages the cloud infrastructure
Data Governance Lead:
- Ensures the data is complete, consistent, and secure.
Common Mistakes to Avoid
1. Confusing tools and architecture
Just because you have purchased certain tools does not mean that an architecture has been built.
2. Problems due to overengineering
Overengineering leads to less agility in complex systems.
3. Ignoring governance
Ignoring governance creates compliance-related risks.
4. Lack of documentation
Lack of architectural diagrams leads to confusion.
Future trends in data architecture
1. AI-first data architecture
Systems designed around machine learning workloads.
2. Real-time architectures
Designing pipelines that are streaming-first.
3. Systems driven by metadata
Automation driven by metadata.
4. Multi-cloud strategies
Multi-cloud strategies reduce vendor lock-in.
1. What is data architecture?
2. What is data infrastructure?
3. What is the difference between data architecture and data infrastructure?
4. Is ETL included as part of data architecture?
5. What are the components of the data architecture?
6. What is the best way to create a scalable data architecture?
7. What are the most prevalent types of data architecture?
8. Why is data architecture important for artificial intelligence?
Conclusion
For modern data teams, understanding the differences between data architecture and data infrastructure is crucial.
The architecture defines how a system should operate; however, the infrastructure is the execution of that architecture.
To be an effective Data Engineering Lead, you must align these two concepts.
The most successful companies do not simply buy tools to help them with their data architecture; they build purposefully designed, scalable data architectures that allow them to capitalize on their data infrastructure as a competitive advantage.
If your current infrastructures are complex, brittle or expensive, the problem lies not with the infrastructure but with the design of the architecture.
