AWS Glue is a managed serverless ETL service that combines a data catalog (metadata repository), a job execution engine (Apache Spark-based), and various supporting services for data transformation, quality, and lineage. It is AWS's primary data integration platform, designed for both batch ETL and streaming pipelines on AWS-native data architectures. Glue handles the operational concerns of running Spark clusters so customers can focus on writing transformation logic.
The service launched in 2017 and has grown substantially since. The original offering was primarily Spark-based ETL jobs with an integrated data catalog. Subsequent additions include Glue Studio (visual job authoring), Glue DataBrew (visual data preparation for non-engineers), Glue Data Quality (managed data quality checks), Glue Streaming (real-time ETL), and Glue for Ray (alternative compute engine). The breadth makes Glue a comprehensive data integration platform within the AWS ecosystem.
By 2026 Glue is well-established for AWS-native data integration. Most organizations running data workloads on AWS use Glue for at least the data catalog (which serves as the metadata layer for many other AWS services like Athena, EMR, and Redshift). The ETL job side is more selective; some organizations use Glue extensively while others prefer alternatives like dbt for transformation, leaving Glue mostly for ingestion.
The catalog component is particularly important in modern AWS data architectures. The Glue Data Catalog serves as the unified metadata repository for AWS analytics services. Athena queries reference Glue Catalog tables. EMR uses Glue Catalog for Hive Metastore compatibility. Redshift Spectrum uses Glue Catalog to query S3 data. The catalog ties the AWS data ecosystem together; even teams that do not run Glue ETL jobs benefit from the catalog.
What Glue is not: it is not the only ETL option on AWS. Alternatives include EMR (managed Spark with more control), Step Functions (for orchestrating non-Glue services), Kinesis Data Firehose (for simple streaming ingestion), Lambda (for lightweight transformation), and third-party tools (Fivetran, Airbyte, Matillion). The choice between these depends on workload characteristics, team skills, and operational preferences.
Glue Data Catalog. Managed metadata repository compatible with Apache Hive Metastore. Stores table definitions, schemas, partitions, and other metadata for data stored in S3, Redshift, RDS, and other AWS sources. Other AWS services reference the catalog: Athena queries catalog tables, EMR uses it as Hive Metastore, Redshift Spectrum uses it for external tables.
Glue ETL Jobs. Serverless Spark-based jobs for data transformation. Customers write PySpark or Scala code (or use visual authoring); Glue runs the jobs on managed Spark clusters. Pricing is per DPU-hour (Data Processing Units, where one DPU is roughly equivalent to a small Spark executor). Jobs scale up to handle larger workloads automatically.
Glue Crawlers. Automated schema discovery. Crawlers connect to data stores (S3 buckets, databases, etc.), infer schema from sample data, and populate the Glue Catalog with table definitions. Crawlers can run on schedule to detect schema changes. The service reduces manual catalog maintenance for data with predictable structure.
Glue Studio. Visual job authoring. Drag-and-drop interface for building ETL workflows. Generates underlying Spark code that can be version-controlled and modified. Useful for less technical users or for quickly prototyping pipelines.
Glue DataBrew. Visual data preparation aimed at analysts and data scientists. More than 250 pre-built transformations (cleaning, normalization, enrichment) accessible through point-and-click. Generates recipes that can be applied to similar data sets.
Glue Data Quality. Managed data quality checks integrated with Glue catalog. Define rules (uniqueness, completeness, business invariants); Glue runs them on schedule and reports results. Less feature-rich than dedicated data quality tools but convenient when already using Glue.
Glue Streaming. Real-time ETL for streaming sources like Kinesis. Same Spark-based pattern but processing continuous data rather than batch. Useful for low-latency data integration when the AWS-native streaming stack fits.
For AWS-native data integration where managed serverless execution suits the workload. Glue eliminates the operational burden of running Spark clusters yourself. The serverless model works well for variable workloads that do not justify dedicated cluster ownership.
For organizations already on AWS with existing data in S3 and the need to ETL into warehouses or lakes. The integration with S3, Redshift, RDS, and other AWS services is convenient. Glue handles the connections and credentials within AWS.
For standardizing on a single data catalog across services. The Glue Catalog serves as the metadata layer for the AWS analytics ecosystem. Even teams not using Glue ETL benefit from the unified catalog.
For complex orchestration that needs more than Glue jobs alone, Step Functions or Airflow work alongside Glue. Glue handles the data processing; orchestration handles the workflow logic. The combination scales beyond what Glue's built-in scheduling can do.
For organizations heavy on dbt and ELT, Glue may be overkill. dbt handles transformations directly in the warehouse. Ingestion tools like Fivetran handle the EL part. Glue ends up in a smaller role in such architectures, often just the catalog.
For teams that want maximum control over Spark configuration, EMR provides more flexibility than Glue's serverless model. Glue trades some flexibility for operational simplicity; EMR trades operational simplicity for more control.
Glue ETL jobs price by DPU-hour. Default pricing as of 2026 is around $0.44 per DPU-hour, though pricing can change. A typical small job using 10 DPUs for 30 minutes costs $2.20. A larger job using 50 DPUs for 4 hours costs $88. Monthly costs depend on job frequency, size, and duration.
DPU sizing is one of the optimization levers. Right-sizing DPUs to actual workload requirements significantly affects cost. Over-provisioned jobs cost more without proportional benefit. Under-provisioned jobs run slowly or fail. Cost optimization includes monitoring DPU utilization and adjusting allocations.
Glue Catalog is priced separately. Free for the first million objects (databases, tables, partitions); after that, charges apply per object per month. For most organizations the catalog is effectively free; very large data lakes with millions of partitions can incur catalog costs.
Glue Crawlers price by DPU-hour like jobs. Crawler runs are typically short, so individual crawler costs are small. The aggregate cost depends on how often crawlers run; running every few minutes against every dataset is expensive, while running daily for stable schemas is cheap.
Glue DataBrew prices by session-hour and node-hour for jobs. Pricing structure differs from regular Glue ETL.
Cost monitoring through CloudWatch and Cost Explorer surfaces Glue spending. Organizations using Glue extensively typically have specific dashboards for job costs, catalog costs, and crawler costs.
A managed metadata repository for data assets, compatible with Apache Hive Metastore. Stores table definitions, schemas, partitions, and other metadata for data in S3, Redshift, RDS, and other AWS sources. Used by Athena, EMR, Redshift Spectrum, and other AWS analytics services as their unified metadata layer. The catalog is one of the most-used Glue components even by organizations that do not run Glue ETL jobs. It ties the AWS analytics ecosystem together by providing consistent metadata access. Teams using Athena, EMR, or Redshift Spectrum benefit from the catalog whether or not they use Glue for transformation.
DPU-hours for job execution (around $0.44 per DPU-hour as of 2026). Separate pricing for catalog operations (free for the first million objects, then per-object monthly). DataBrew has its own pricing structure. Data quality checks have additional charges if used. Most organizations find Glue costs manageable when they monitor DPU usage and right-size jobs. Cost surprises usually trace to over-provisioned jobs, frequent crawler runs, or catalog explosion in very large data lakes. Glue vs EMR? Glue is serverless and managed. EMR provides more control over Spark configuration, longer-running clusters for sustained workloads, and access to Spark features that Glue may not expose. The choice depends on operational preferences and workload characteristics. For most organizations, Glue's operational simplicity wins. EMR matters when teams need specific Spark features, when workloads are large enough that long-running clusters are economical, or when the team has strong Spark expertise that justifies the additional operational burden.
Data Processing Unit: a unit of compute capacity in Glue jobs. One DPU provides 4 vCPUs and 16 GB of memory. Glue jobs request a number of DPUs (typically 2 to 100 depending on workload size). Pricing is per DPU-hour with sub-second billing increments. DPU sizing is a key optimization lever. Right-sizing requires understanding workload characteristics. Tools like Glue's job execution metrics help identify over-provisioned jobs that can use fewer DPUs without affecting performance.
Lake Formation builds on Glue Catalog with additional governance capabilities. Permissions, audit, access controls. Lake Formation manages who can access what data through the Glue Catalog. The two services work together as a unified data lake management solution. Lake Formation is more recent than Glue and adds governance features that Glue itself does not provide. Organizations using Glue for ETL plus Lake Formation for governance get integrated metadata and access control across the AWS data lake.
Glue supports streaming jobs for real-time data integration with sources like Kinesis. Same Spark-based pattern as batch jobs but processing continuous data. Latency is typically seconds to minutes, suitable for many real-time use cases. For lower-latency requirements (sub-second), specialized streaming systems (Kinesis Data Analytics, Flink on EMR, custom Lambda processors) usually fit better. Glue Streaming handles the middle ground where minute-level latency is acceptable and the Spark programming model fits the workload.
Glue runs in AWS but can read from and write to non-AWS sources through connectors. Glue can connect to on-premise databases via JDBC, to other clouds through appropriate networking and credentials, to SaaS sources through specific connectors. The connectivity is workable for hybrid and multi-cloud scenarios but Glue itself remains an AWS service. Organizations that want a more cloud-neutral ETL solution often choose third-party tools (Fivetran, Airbyte) over Glue.
Through schema registries and explicit handling in ETL jobs. Glue Crawlers can detect schema changes and update the catalog. Glue ETL jobs can be configured to handle schema differences across input batches. Automatic schema evolution has limitations and edge cases. Production deployments should plan schema evolution explicitly rather than relying on Glue's automatic handling. The pattern that works combines schema registries (for streaming data), automated detection (Glue Crawlers, schema validation), and explicit handling logic in transformation code.
Visual job authoring with drag-and-drop transformations. Generates underlying Spark code that can be version-controlled. Useful for prototyping, for less technical users, or for documenting workflow visually alongside the code. The trade-off is that visual editing can produce code that is harder to maintain than code written directly. Most production usage of Glue Studio writes initial pipelines visually, then maintains them as code in version control. The combination uses Studio's strengths while keeping the engineering rigor of code-based workflows.
Tighter integration with other AWS data services. Improved performance and cost efficiency. Expanded ML and AI capabilities (Glue for Ray brings alternative compute engines). Continued investment as AWS's primary data integration platform. The bigger trend is Glue evolving from individual ETL service into a more unified data integration platform. The catalog, ETL, quality, and lineage components increasingly work together. By 2027 or 2028, expect Glue to be the integrated AWS data integration platform rather than a collection of services.
