AWS History and Timeline regarding AWS Glue - Overview, Functions, Features, Summary of Updates, and Introduction

First Published: 2026-08-29
Last Updated: 2026-08-29

This article is part of a series exploring the history and timeline of AWS services, drawing out key features. Previous articles have covered Amazon S3, Amazon RDS, Amazon Aurora, Amazon DynamoDB, AWS Lambda, and Amazon Redshift, among others.

This article covers AWS Glue. It was announced on December 1, 2016, and general availability began on August 14, 2017. It is a serverless data integration service, initially introduced as a managed ETL service.

AWS Glue frequently appears as a component in other articles. It is present in diagrams of data lakehouses, in explanations of zero-ETL approaches, and behind the scenes of services like Amazon Athena and Amazon Redshift. However, there are surprisingly few resources that clearly outline what AWS Glue itself has evolved into.

This article is not intended to be a comprehensive list of features. Rather, it aims to document how three distinct tracks of AWS Glue have developed at different paces under the same service name.

The first track is the evolution of its job execution engine. From version 0.9 to 6.0, the underlying pieces of Apache Spark, Python, Scala, and Java have kept moving. This section is particularly relevant for users concerned about when their existing jobs might become obsolete.

The second track is the AWS Glue Data Catalog. It was created as a place for AWS Glue jobs to store table definitions, and Amazon Athena and Amazon EMR shared it from the very beginning. It has since evolved into a shared metastore accessible by multiple engines, and now serves as an entry point to other catalogs as well.

The third track is which service is the subject of integration. The sources of zero-ETL integration are growing on the AWS Glue side, not on the side of the destination services. Two announcements, one from December 2024 and one from July 2025, show the shift.

These three tracks, while operating under the same service name, have evolved at significantly different speeds. The job execution engine typically receives updates approximately once per year, although one gap ran longer. The AWS Glue Data Catalog has widened its reach almost every year. Integration changed its subject between the end of 2024 and the middle of 2025. This article will examine these three tracks individually.

Three AWS Glue Tracks That Moved at Different Speeds

This article does not discuss pricing. It will not include information on discount rates or cost reduction percentages, nor any figures related to costs. Where the structure matters, it goes no further than the fact that a billing unit changed.

Several articles already exist regarding AWS Glue. The AWS Data Lakehouse Architecture Guide covers lakehouse design, Zero-ETL Integrations on AWS addresses zero-ETL source and destination compatibility, and Fine-Grained Access Control for AI Data with AWS Lake Formation discusses the permission model for AWS Lake Formation. This article will not delve into those design principles; instead, it will simply document when specific features became available.

Background and Method of Creating AWS Glue Historical Timeline

There are three primary reasons for creating a timeline for AWS Glue.

First, AWS Glue has been generally available for nine years, and during that time, the very nature of the service has changed. What began as a managed ETL service now has three faces: a job execution platform, a shared metastore, and the subject of integration.

Second, the versions of the job execution platform have end-of-life dates, and some of those dates have already passed. Versions 0.9, 1.0, and 2.0 reached end of life on April 1, 2026. End of support and end of life have to be kept apart; collapsing them leads to misreadings.

Third, information about the scope of the AWS Glue Data Catalog is currently scattered across various articles and documentation. Details about which engines gained the ability to read the catalog, and when, are scattered through the documentation for Amazon Athena, Amazon EMR, and Amazon Redshift. It is valuable to compile this information in a single location, along with the corresponding dates.

This article references the following primary sources:

AWS What's New
AWS Big Data Blog and AWS News Blog
AWS Glue Developer Guide, Documentation history
AWS Glue Developer Guide, AWS Glue versions and AWS Glue version support policy

Dates reflect the announcement date. In some cases, AWS What's New and the Documentation history of the AWS Glue Developer Guide list different dates for the same feature. In those cases, the timeline entry names the source it follows.

⚠ Some of the oldest AWS What's New pages are no longer served. Where that is the case, the timeline keeps the original URL, and the link resolves to an AWS index page rather than to the announcement.

This timeline does not represent every update to AWS Glue. It focuses on representative features that moved one of the three tracks. The addition of new regions, increases in the number of connectors, and minor console improvements are generally excluded.

Each row carries a Track column. It takes six values. Three of them are the three tracks this article follows, and the other three sit outside those tracks.

Launch refers to the introduction of the service itself.
Runtime represents changes to the job execution platform: versions, worker types, execution modes, and the open table formats built into the runtime.
Catalog refers to changes to the AWS Glue Data Catalog, including the crawlers that populate it.
Integration involves connecting with other services and data sources, including zero-ETL, Schema Registry, and Data Quality features.
Authoring refers to changes in how a job is written and composed: languages, transforms, orchestration, visual editors, notebooks, and AI assistance.
Lifecycle represents changes in availability status: end of support, end of life, the close of new customer sign-ups, the publication of the policy that governs them, and a preview with no general availability announcement in the sources this article checked.

AWS Glue was announced on December 1, 2016, and generally available on August 14, 2017. Eight months separate the two dates.

AWS Glue Historical Timeline (Updates from December 1, 2016)

The following is a timeline of the features of AWS Glue. It is the tenth year since the first announcement.

Use the following index to jump to a year.

2016 - The service is announced
2017 - General availability, and a shared metastore from day one
2018 - Scala joins Python, and the catalog gains resource-based policies
2019 - The Glue version property arrives, and worker types begin
2020 - Version 2.0, AWS Glue Studio, DataBrew, and Schema Registry
2021 - Version 3.0 arrives, and the version support policy is documented
2022 - Version 4.0, open table formats, and AWS Glue for Ray
2023 - Data Quality reaches general availability, and the catalog takes on management work
2024 - Version 5.0, Data Catalog views reach general availability, and migration gains generative AI
2025 - The zero-ETL subject moves to AWS Glue, and version 5.1
2026 - Ray closes to new customers, older versions reach end of life, and version 6.0

The table can be sorted by clicking a column header. Sorting by Track groups the rows so that one track can be read through on its own.

Date	Track	Summary
2016-12-01	`Launch`	AWS Glue was announced. It was introduced as a fully managed ETL service designed to move data between data stores, with the ability to automate data discovery, transformation, mapping, and job scheduling. The announcement said the service would be available soon and gave no launch date. What the announcement offered was a sign-up for availability updates, not entry into a preview. References: Introducing AWS Glue: Prepare and load data into data stores
2017-08-14	`Launch`	AWS Glue became generally available. When you specify a data location, AWS Glue discovers the data and stores metadata, such as table definitions and schemas, in the AWS Glue Data Catalog. Data listed in the catalog can then be easily searched, queried, and used as a target for ETL processes. AWS Glue generates the code required for data transformation and loading. Jobs run within a fully managed Apache Spark environment. Initially, it was available in only one region: US East (N. Virginia). References: Introducing AWS Glue: A Simple, Flexible, and Cost-Effective Extract, Transfer, and Load (ETL) Service
2017-08-14	`Catalog`	The AWS Glue Data Catalog began serving as a metastore for engines beyond just AWS Glue. AWS recommended moving from the internal Athena catalog to the AWS Glue Data Catalog, and Amazon EMR could now hold the metadata of external Apache Hive and Apache Spark tables in the catalog rather than in a Hive Metastore on the cluster. The Amazon EMR side requires release 5.8.0 or later. References: Amazon Athena Now Integrates With AWS Glue / Use Apache Spark and Hive on Amazon EMR with the AWS Glue Data Catalog
2017-08-15	`Catalog`	Amazon Redshift Spectrum was integrated with the AWS Glue Data Catalog. The data warehouse engine now uses the same catalog to access data stored on Amazon S3. References: Amazon Redshift Spectrum Now Integrates with AWS Glue
2018-01-12	`Authoring`	Scala was added as a language for ETL scripts. Until then, AWS Glue generated Python only. Additionally, trigger activation conditions were expanded, allowing job executions that ended in failure or stoppage to be used as subsequent activation criteria. References: AWS Glue Now Supports Scala in Addition to Python
2018-10-15	`Catalog`	AWS Glue Data Catalog added support for resource-based policies and resource-level permissions. Access can now be controlled at the database and table level in the catalog, laying the groundwork for managing shared metastores across accounts. References: AWS Glue Documentation history — Support for resource-level permission and resource-based policies (October 15, 2018)
2019-01-22	`Runtime`	Python shell jobs were added. This allows smaller tasks to be executed without the need to launch a Spark cluster. This marks the first divergence from a system solely reliant on Spark for job execution. References: Introducing Python Shell Jobs in AWS Glue
2019-03-14	`Catalog`	The AWS Glue Data Catalog can now be configured as an external Apache Hive Metastore. You can now directly execute Apache Spark SQL queries against tables in the catalog from AWS Glue jobs and development endpoints. This marks the point at which AWS Glue's own jobs began utilizing the catalog's Hive Metastore compatibility. References: AWS Glue enables running Apache Spark SQL queries
2019-04-05	`Runtime`	The worker types `G.1X` and `G.2X` were added. Previously, Apache Spark jobs could only run with a configuration of 1 DPU per worker and 2 executors per node. Now, you can select configurations with larger amounts of memory per executor. This introduces the concept of worker types as a configuration option. References: AWS Glue Documentation history — Support for additional configuration options for memory-intensive jobs (April 5, 2019)
2019-06-20	`Authoring`	Workflows were added. Triggers, crawlers, and jobs can be assembled into a directed acyclic graph, so an ETL process spanning several dependent jobs runs and is tracked as a single unit. References: AWS Glue now provides workflows to orchestrate your ETL workloads
2019-07-25	`Runtime`	The `Glue version` job property was introduced. Specifying a version determines the associated Apache Spark and Python versions. Version 1.0 runs on Apache Spark 2.4.3 and allows you to choose between Python 2 and Python 3. Existing jobs created without a specified version are treated as version 0.9. This default value would be updated multiple times in the future. References: AWS Glue now supports the ability to run ETL jobs on Apache Spark 2.4.3 (with Python 3)
2019-08-09	`Authoring`	The FindMatches transformation, a machine learning feature, was added. This feature allows you to match duplicate records that do not perfectly align, even without common identifiers, by using machine learning. It was the first feature to incorporate a pre-trained model into an ETL transformation. References: AWS Glue now provides FindMatches ML transform to deduplicate and find matching records in your dataset
2020-04-27	`Runtime`	Streaming ETL was added. Jobs can now ingest continuous data from sources like Apache Kafka and Amazon Kinesis Data Streams, process it, and make it available in seconds. This adds an always-on execution mode to a job execution platform that previously ran batches only. References: AWS Glue now supports serverless streaming ETL
2020-08-10	`Runtime`	AWS Glue version 2.0 became generally available. Spark ETL jobs started faster, and start times varied less. The minimum billing duration has changed from 10 minutes to 1 minute, and billing is now calculated in 1-second increments. This change makes micro-batch processing and interactive data exploration more practical. References: AWS Glue version 2.0 featuring 10x faster job start times and 1-minute minimum billing duration
2020-09-14	`Catalog`	AWS Glue Data Catalog now supports partition indexing. For tables with a growing number of partitions, it is now possible to limit the partitions read during queries. This represents an expansion of the catalog's role, evolving it from a simple repository for table definitions to a crucial component that impacts query performance. References: AWS Glue Data Catalog now supports PartitionIndex, improving query performance on highly partitioned tables
2020-09-23	`Authoring`	AWS Glue Studio was announced. It allows users to build jobs using a graphical interface and monitor their progress through a dashboard. The aim is to enable users, even those unfamiliar with Apache Spark programming, to create ETL processes. References: Announcing AWS Glue Studio: Visual job authoring and advanced monitoring for AWS Glue
2020-11-11	`Authoring`	AWS Glue DataBrew became generally available. This visual tool allows users to perform data preparation without writing code, by assembling pre-built transformations into recipes. It is offered as a separate entry point from AWS Glue Studio. References: Introducing AWS Glue DataBrew: Visual data preparation tool to clean and normalize data up to 80% faster
2020-11-19	`Integration`	The AWS Glue Schema Registry was added. This feature allows you to register schemas for streaming data, validate changes, and control their evolution. It integrates with Java applications running on Apache Kafka, Amazon MSK, Amazon Kinesis Data Streams, Apache Flink, and AWS Lambda. While part of AWS Glue, this functionality can be used without running any AWS Glue jobs. References: Control the evolution of data streams using the AWS Glue Schema Registry
2020-12-01	`Lifecycle`	AWS Glue Elastic Views were announced as a preview. The concept was to create materialized views using SQL that span multiple data stores, monitoring source changes and automatically updating them. The initial targets were Amazon DynamoDB, Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service, with Amazon RDS and Amazon Aurora planned to follow. Based on searches within AWS What's New, no general availability announcement for this feature has been found, and a dedicated product page is currently unavailable. The goal of joining several stores through SQL reappears later, in zero-ETL integration and in materialized views. References: Announcing AWS Glue Elastic Views Preview
2021-08-19	`Runtime`	AWS Glue version 3.0 was announced. This runtime, built on Apache Spark 3.1.1, features vectorized readers and output committers to accelerate reading and writing to Amazon S3. Access to the AWS Glue Data Catalog is also optimized with partition predicates, and partition indexes let the runtime skip partitions it does not need. References: Introducing optimized Spark 3.1 runtime for data integration with AWS Glue 3.0
2021-09-24	`Lifecycle`	The AWS Glue version support policy was documented in the Developer Guide. The framework for when a version reaches end of support and end of life became public at this point. References: Documentation history for AWS Glue / AWS Glue version support policy
2021-10-15	`Catalog`	Crawlers now support Amazon S3 event notifications. Instead of scanning the entire bucket, crawlers now only examine newly added objects to ingest incremental changes. The method for keeping the catalog up-to-date has shifted from periodic full scans to notifications of changes. References: AWS Glue Crawlers support Amazon S3 event notifications
2022-04-21	`Authoring`	AWS Glue interactive sessions became generally available. You can now run Spark workloads interactively from notebooks. This allows you to write job code without launching a development endpoint. References: Announcing general availability of AWS Glue Interactive Sessions
2022-07-18	`Runtime`	The `G.025X` worker was added for streaming jobs. These are smaller workers, consuming only 1/4 DPU, and are designed to process smaller, intermittent streams. The range of worker types now extends to include smaller options, in addition to larger ones. References: AWS Glue now supports smaller instance types for streaming
2022-08-09	`Runtime`	The Flex execution option was added. This allows jobs to run using available AWS capacity, with variable start and run times, and may involve the reclamation of compute resources during execution. This enables you to route less time-critical workloads to a different execution model. References: AWS Glue now supports Flex execution option
2022-11-28	`Runtime`	AWS Glue version 4.0 was announced. It upgrades the engines to Apache Spark 3.3.0 and Python 3.10, and adds the Cloud Shuffle Storage Plugin, which uses Amazon S3 for shuffling, together with Adaptive Query Execution. References: Introducing AWS Glue 4.0
2022-11-28	`Runtime`	Apache Hudi, Apache Iceberg, and Delta Lake are now natively supported in AWS Glue for Apache Spark. You can now read and write data to Amazon S3 using these table formats without needing to install separate connectors. This marks a significant step as open table formats become an integral part of the execution platform. References: AWS Glue for Apache Spark Native support for Data Lake Frameworks (Apache Hudi, Apache Iceberg, Delta Lake)
2022-11-28	`Runtime`	AWS Glue for Ray was announced as a preview. This is a third job execution engine, separate from Apache Spark and the Python shell, and utilizes Ray, a Python distributed processing framework. References: Announcing AWS Glue for Ray (Preview)
2022-11-30	`Integration`	AWS Glue Data Quality was announced as a preview. It analyzes data to collect statistics, recommends data quality rules, and continuously validates against those rules. References: AWS Glue announces AWS Glue Data Quality (Preview)
2023-05-09	`Runtime`	The `G.4X` and `G.8X` worker types became generally available. These offer 4 DPU and 8 DPU respectively, and are designed for transformations that require significant memory and for handling highly skewed aggregations. References: AWS Glue large instance types are now generally available
2023-06-05	`Runtime`	AWS Glue for Ray became generally available. This follows approximately six months of preview availability. References: AWS Glue for Ray is now generally available
2023-06-06	`Integration`	AWS Glue Data Quality became generally available. It can now evaluate data quality against tables in the AWS Glue Data Catalog and within jobs in AWS Glue Studio. References: AWS Glue Data Quality is now generally available
2023-07-07	`Catalog`	The crawler now supports Apache Iceberg tables. It can scan Iceberg tables on Amazon S3 and register them in the AWS Glue Data Catalog. During the same month, support was also added for Apache Hudi tables. References: AWS Glue Crawlers now supports Apache Iceberg Tables / AWS Glue Crawlers now supports Apache Hudi Tables
2023-11-15	`Catalog`	AWS Glue Data Catalog now supports automatic compaction for Apache Iceberg tables. The catalog now handles the work of combining small objects on Amazon S3. In addition to maintaining metadata, the catalog now performs management tasks related to the actual table data. References: AWS Glue Data Catalog supports automatic compaction for Apache Iceberg tables
2023-11-17	`Catalog`	AWS Glue Data Catalog can now generate column-level statistics. It calculates statistics for tables in Parquet, ORC, JSON, ION, CSV, and XML formats without requiring separate pipelines. References: AWS Glue Data Catalog now supports generating column-level statistics
2023-11-27	`Catalog`	AWS Glue Data Catalog now supports SQL views that can be used with multiple engines. Previously, separate views had to be created for each engine, and permissions had to be granted to both the view and the underlying table. Now, a single view can be accessed from Amazon Athena, Amazon Redshift, and Amazon EMR on EC2's Spark, allowing users to read the view even if they do not have direct access to the underlying table. References: AWS Glue Data Catalog supports multi engine views with AWS Analytics Engines
2024-04-30	`Authoring`	Amazon Q data integration in AWS Glue became generally available. You can now use natural language to build data integration jobs and troubleshoot issues. References: Amazon Q data integration in AWS Glue is now generally available
2024-08-08	`Catalog`	AWS Glue Data Catalog views became generally available with Amazon Athena and Amazon Redshift. You can create, share, and query views across regions, accounts, and organizations. Access is managed through AWS Lake Formation permissions, and requests are logged in AWS CloudTrail. References: AWS Glue Data Catalog views are now GA with Amazon Athena and Amazon Redshift
2024-11-22	`Authoring`	Generative AI upgrades for Apache Spark were announced as a preview. Using Amazon Bedrock, the system analyzes existing Spark job scripts and configurations to create migration plans for newer versions. On the same day, generative AI-powered troubleshooting for Spark jobs was also announced. Upgrading a Spark version has itself become a feature of the product. References: Announcing generative AI upgrades for Apache Spark in AWS Glue (preview) / Announcing generative AI troubleshooting for Apache Spark in AWS Glue (Preview)
2024-12-03	`Runtime`	AWS Glue version 5.0 became generally available. At the time of announcement, it supports Apache Spark 3.5.2, Python 3.11, and Java 17. It includes fine-grained access control through Spark native integration with AWS Lake Formation, and provides support for Amazon SageMaker Lakehouse. References: Introducing AWS Glue 5.0
2024-12-03	`Catalog`	An Apache Iceberg REST endpoint was added to the AWS Glue Data Catalog. This supports API operations defined in the Apache Iceberg REST specification, allowing analytical engines with Iceberg clients to connect directly to the catalog, without using AWS Glue jobs. References: Connecting to the Data Catalog using AWS Glue Iceberg REST endpoint / Documentation history for AWS Glue
2024-12-03	`Integration`	zero-ETL integration was documented in the AWS Glue Developer Guide. In a separate announcement on the same day, AWS described the zero-ETL integration from eight applications as a feature of Amazon SageMaker Lakehouse and Amazon Redshift, with the focus on the destination side. References: Documentation history for AWS Glue / Amazon SageMaker Lakehouse and Amazon Redshift support for zero-ETL integrations from eight applications
2025-03-14	`Catalog`	AWS Glue Data Catalog views can now be created within Apache Spark jobs in AWS Glue 5.0. Previously, view creation was limited to Amazon Athena and Amazon Redshift; now, Spark jobs are also able to create views. References: Announcing support of AWS Glue Data Catalog views with AWS Glue 5.0
2025-07-16	`Integration`	AWS Glue now supports zero-ETL integrations from Amazon DynamoDB and eight applications to Amazon S3 Tables. The December 2024 announcement named the destination services as the subject. This one names AWS Glue. Information on which sources are supported is now available in AWS Glue's documentation. References: AWS Glue now supports zero-ETL integrations from Amazon DynamoDB and eight applications to S3 Tables
2025-07-17	`Runtime`	The worker types `G.12X` and `G.16X`, along with the memory-optimized `R.1X`, `R.2X`, `R.4X`, and `R.8X`, were added. The announcement states that the R series workers offer twice the memory of the G series, making them suitable for memory-intensive operations such as caching, shuffling, and aggregation. ⚠ The change history in the AWS Glue Developer Guide records this feature as being introduced on January 30, 2025, which does not align with the announcement date. References: AWS Glue now supports new workers for larger and memory intensive workloads / Documentation history for AWS Glue
2025-08-29	`Catalog`	Amazon EMR on EC2 now supports AWS Glue Data Catalog views. The announcement states that this feature is available in Amazon EMR release 7.10. You can now query views created from Spark jobs using multiple engines. References: Amazon EMR on EC2 Adds Apache Spark native FGAC and AWS Glue Data Catalog Views Support
2025-11-24	`Catalog`	Federation with external Apache Iceberg catalogs became generally available in the AWS Glue Data Catalog. You can query Iceberg tables registered in other catalogs from AWS analytics engines, without needing to move or replicate the tables. This marks a shift in the catalog's role, evolving from a listing of AWS tables to a gateway to other catalogs. References: AWS Glue announces catalog federation for remote Apache Iceberg catalogs
2025-11-25	`Integration`	AWS Glue's zero-ETL now supports ingesting data from self-managed databases. You can now configure integrations to replicate data from Oracle, SQL Server, MySQL, and PostgreSQL (located on-premises or on Amazon EC2) to Amazon Redshift, without writing any code. References: Announcing AWS Glue zero-ETL for self-managed Database Sources
2025-11-26	`Runtime`	AWS Glue version 5.1 became generally available. It features upgraded versions of Apache Spark 3.5.6, Python 3.11, and Scala 2.12.18, and supports Apache Iceberg format version 3. Fine-grained access control in AWS Lake Formation now covers write operations as well as reads. This version also serves as the default version for jobs created without a specified version. References: Introducing AWS Glue 5.1 / AWS Glue versions
2025-11-30	`Catalog`	AWS Glue now supports materialized views for Apache Iceberg. AWS Glue Data Catalog can now store pre-calculated results as tables in Apache Iceberg format, and update them in response to changes in the original tables. The catalog manages scheduling, change detection, incremental updates, and resource allocation for these calculations. Using this feature requires AWS Glue version 5.1 or later. References: AWS Glue now supports Apache Iceberg based materialized views / Using materialized views with AWS Glue
2026-03-31	`Lifecycle`	AWS announced that AWS Glue for Ray will no longer be available to new customers starting April 30, 2026. Existing customers will continue to be able to use the service, and AWS will continue to focus on security and availability improvements. ⚠ This announcement includes several other services and features, such as AWS App Runner, AWS Audit Manager, and AWS CloudTrail Lake, and represents a change related to one feature of AWS Glue. The core AWS Glue service remains a current offering. References: AWS Service Availability Updates / AWS Glue for Ray end of support
2026-03-31	`Lifecycle`	Python shell 3.6 reached end of support. A note on the support policy page adds a restriction that the general definition of end of support does not carry: new Python shell 3.6 jobs cannot be created after this date, while existing jobs can still be updated and run. References: AWS Glue version support policy
2026-04-01	`Lifecycle`	AWS Glue versions 0.9, 1.0, and 2.0 have reached their end-of-life. These versions no longer support the creation of new jobs and interactive sessions, and it is also no longer possible to start job runs. Existing running jobs and sessions will be stopped, and these versions will be removed from the AWS Glue SDK and API. End-of-support for these versions had previously occurred between 2022 and 2024. References: AWS Glue version support policy
2026-06-17	`Catalog`	Business context and semantic search were announced as a preview for the AWS Glue Data Catalog. You can now add glossary terms and custom metadata fields to catalog tables, and use the new Glue Search API to search based on both structure and meaning. The catalog can also register skills that provide agents with additional context. References: AWS Glue Data Catalog now supports business context and semantic search (Preview)
2026-06-17	`Authoring`	AWS Glue interactive sessions now support Spark Connect. You can connect via the Spark Connect protocol from a lightweight client, and this feature is intended for use in local development environments. References: AWS Glue Interactive Sessions now support Spark Connect for interactive workloads
2026-08-05	`Catalog`	A preview has begun for exporting metadata from the AWS Glue Data Catalog to Amazon S3 Tables. Catalog metadata, including glossary terms, attached metadata, and asset descriptions, is written as tables in Apache Iceberg format, allowing it to be queried, audited, and used for time travel. The catalog's own metadata has become a table in the catalog. References: AWS Glue Data Catalog now supports metadata exports to S3 Tables (Preview)
2026-08-21	`Runtime`	AWS Glue version 6.0 became generally available. The runtime now includes Apache Spark 4.1.1, Python 3.13, and Scala 2.13.17, along with Apache Iceberg 1.11.0, Apache Hudi 1.1.1, and Delta Lake 4.2.0. It supports format version 3 of Apache Iceberg and includes Spark Declarative Pipelines (for writing pipelines declaratively), a real-time mode for stateless streaming, and Arrow-native Python UDFs. EMRFS has been removed, and connections to Amazon S3 are now only supported via S3A. Only version 2 of the AWS SDK for Java remains. AWS Glue 6.0 is available in all AWS Commercial, AWS GovCloud (US), and AWS China regions. The general availability date in this row follows the AWS What's New announcement of 2026-08-21. That announcement is not linked here because its headline carries a pricing figure, which this article does not reproduce. References: Introducing AWS Glue 6.0 for faster and more cost-effective data integration / AWS Glue versions / Migrating AWS Glue for Spark jobs to AWS Glue version 6.0

Current Overview, Functions, Features of AWS Glue

AWS Glue Key Functions and Features

AWS Glue is a serverless data integration service that helps you discover, prepare, move, and integrate data from various sources. Its current capabilities fall into the same three tracks:

First, AWS Glue provides a job execution platform that supports three types of jobs: Apache Spark jobs, Python shell jobs, and Ray jobs. Spark jobs support both batch and streaming workloads. The runtime version is selected through the Glue version job property, and that value determines the versions of Apache Spark, Python, Scala, and Java in use.

Second, the AWS Glue Data Catalog acts as a shared metastore, holding table definitions, schemas, partitions, and statistics. Analytical engines such as Amazon Athena, Amazon Redshift, Amazon EMR, AWS Glue jobs, and REST clients for Apache Iceberg can access this information. The catalog also now manages automated compaction of table data and updates to materialized views.

Third, as the subject of integration, AWS Glue carries zero-ETL integration, the AWS Glue Schema Registry, AWS Glue Data Quality, and a range of connectors. AWS Glue therefore carries both the mechanisms that move data and the ones that act on it where it already sits, such as the AWS Glue Schema Registry and AWS Glue Data Quality.

The AWS Glue Version Lineage

The Glue version property was introduced in July 2019. Jobs created without specifying a version were treated as version 0.9, and from that point a job could name the version it runs on.

The AWS Glue versions page lists the following versions and runtime environments.

AWS Glue Version	Apache Spark	Python	Java	Status
6.0	4.1.1	3.13	17	Current
5.1	3.5.6	3.11	17	Current. Default version for jobs that do not specify a version.
5.0	3.5.4	3.11	17	Current
4.0	3.3.0	3.10	8	Current
3.0	3.1.1	3.7	8	Current
2.0	2.4.3	3.7	Not specified	End of life on April 1, 2026
1.0	2.4.3	2.7 and 3.6	Not specified	End of life on April 1, 2026
0.9	2.2.1	2.7	Not specified	End of life on April 1, 2026

The Scala versions are listed on the same page, specifically for versions 5.0 and later. Version 5.0 and 5.1 use 2.12.18, while version 6.0 uses 2.13.17. Version 4.0 uses 2.12, and its dependencies are listed in the table on the Migrating AWS Glue for Spark jobs to AWS Glue version 6.0 page.

There are two important points to note regarding this table.

First, the default version has moved. At the time of the 2019 announcement, jobs created without specifying a version were treated as version 0.9, and the current Developer Guide states that 5.1 is the default. Both statements describe the value applied to jobs created without specifying a version. Jobs that stayed on version 0.9 stopped running at the end of life date of April 1, 2026.

Second, what is inside a version keeps moving after general availability. The initial announcement for version 5.0 listed Apache Spark 3.5.2, Apache Iceberg 1.6.1, and Delta Lake 3.2.0. However, the current Developer Guide, also regarding version 5.0, lists Apache Spark 3.5.4, Apache Iceberg 1.7.1, and Delta Lake 3.3.0. The values listed in the announcement and the current documentation are different. When copying the versions of bundled libraries into articles or design documents, it is crucial to specify the point in time to which those versions correspond, to avoid potential misinterpretations later on.

End of Support and End of Life Are Different Things

An AWS Glue version has two different endings. They mean different things, and collapsing one into the other changes what a reader decides to do.

The AWS Glue version support policy defines them as follows.

End of Support (EOS) - When an AWS Glue version reaches EOS:
* AWS Glue will no longer apply security patches or other updates to EOS versions.
* AWS Glue jobs on EOS versions are not eligible for technical support.
* AWS Glue may not honor SLAs when jobs are run on EOS versions.

End of Life (EOL) - When an AWS Glue version reaches EOL:
* You can no longer create new AWS Glue jobs or interactive sessions on EOL versions.
* You can no longer start job runs on these AWS Glue versions.
* AWS Glue will stop existing job runs and interactive sessions on EOL versions.
* EOL versions will be removed from AWS Glue SDKs and APIs.

End of support means that patches and technical support stop, while jobs keep running. End of life means that jobs stop running. Running jobs and sessions will be stopped, and that version will be removed from the SDK and API.

The same page writes these dates in month/day/year order. This article uses year-month-day.

Type	Version	End of Support	End of Life
Spark	Glue version 0.9 (Spark 2.2, Scala 2, Python 2)	2022-06-01	2026-04-01
Spark	Glue version 1.0 (Spark 2.4, Python 2)	2022-06-01	2026-04-01
Spark	Glue version 1.0 (Spark 2.4, Scala 2, Python 3)	2022-09-30	2026-04-01
Spark	Glue version 2.0 (Spark 2.4, Python 3)	2024-01-31	2026-04-01
Python shell	Python 2 (AWS Glue Version 1.0)	2022-06-01	2026-04-01
Python shell	PythonShell 3.6 (Glue version 1.0)	2026-03-31	Not applicable
Development endpoint	Zeppelin notebook	2022-09-30	Not applicable

This table is divided into sections for Spark jobs, Python shell jobs, and development endpoints. While there are four entries specifically for the Spark section, the table as a whole contains seven entries. Counting one section and stopping there leaves entries out.

The Python shell version 3.6 only has an end-of-support date, with no end-of-life date specified. A note on the same page adds a restriction that the general definition of end of support does not carry: new Python shell 3.6 jobs cannot be created after that date, while existing jobs can still be updated and run. Reading the general definition alone would miss it, so the operational effect of a date has to be read on the page that sets the date.

AWS provides an upgrade feature powered by generative AI, originally previewed in November 2024, that analyzes existing Spark scripts and configurations to create a migration plan. Generative AI upgrades for Apache Spark in AWS Glue states that the job has to be a PySpark job running on AWS Glue 2.0, 3.0, 4.0, 5.0, or 5.1, and that it can be upgraded to 4.0, 5.0, 5.1, or 6.0. The AWS Glue versions page states the same condition as a range, older Glue versions (≥ 2.0). This article follows the page that enumerates them, because 6.0 satisfies the range but is not a version a job can be upgraded from.

How the AWS Glue Data Catalog Became a Shared Metastore

The AWS Glue Data Catalog was built for AWS Glue jobs, and it did not stay limited to that purpose for long.

How Far the AWS Glue Data Catalog Reaches

On the day of general availability, August 14, 2017, AWS recommended moving from the internal Athena catalog to the AWS Glue Data Catalog, and Amazon EMR gained the ability to hold the metadata of its Apache Hive and Apache Spark external tables there. The following day, Amazon Redshift Spectrum joined. This means the catalog never truly existed as a component solely dedicated to AWS Glue.

Instead of simply supporting more engines that could read the catalog, the catalog's responsibilities expanded. The catalog gained partition indexing in 2020, then automatic compaction and column-level statistics for Apache Iceberg tables in 2023. Its role has evolved from a simple repository for table definitions to a component that contributes to query performance.

The next development involved SQL views accessible from multiple engines. Announced in November 2023, they became generally available in August 2024 for Amazon Athena and Amazon Redshift. In March 2025, AWS Glue 5.0 enabled Apache Spark jobs to create views, and in August 2025, Amazon EMR on EC2 joined them. Access to these views is managed through permissions within AWS Lake Formation. The permission model itself is covered by the earlier Fine-Grained Access Control for AI Data with AWS Lake Formation.

From November 2025, federation with external Apache Iceberg catalogs let the AWS Glue Data Catalog offer a way in to tables that are not registered in it. Apache Iceberg materialized views arrived in the same month, and the catalog took over storing and refreshing the precomputed results. According to Using materialized views with AWS Glue, this feature requires AWS Glue version 5.1 or later.

In 2026, the metadata of the catalog itself became a target for search and analysis. Business context and semantic search entered preview in June, and a preview for writing catalog metadata to Amazon S3 Tables followed in August. Both of these are currently in preview status.

Note that the term Apache Iceberg is used here in two different ways. One refers to the table format itself, while the other refers to the concept of an Iceberg catalog. AWS Glue Data Catalog stores the former (the tables) and also functions as the latter. When this article uses the bare word catalog, it refers to the AWS Glue Data Catalog.

Where AWS Glue Sits in Zero-ETL

In a zero-ETL integration, AWS manages the replication between a source and a destination, so that the user does not build an ETL pipeline.

This article looks at one question only: which service is the subject of that system. The mapping between specific data sources and destinations is already documented in the previously published Zero-ETL Integrations on AWS. That article advocates for using official lists rather than creating custom mapping tables, and this article follows the same approach by not creating its own mapping table.

Two announcements show the shift in subject. An announcement from December 3, 2024, described zero-ETL integrations from eight applications as features of Amazon SageMaker Lakehouse and Amazon Redshift. In this case, the destination was the subject. A subsequent announcement from July 16, 2025, described the integration of the same eight applications, along with data from Amazon DynamoDB, to Amazon S3 Tables, as a feature of AWS Glue.

Subsequent announcements consistently position AWS Glue as the subject. Whether it involves data sources that are self-managed, or the definitions using AWS CloudFormation and AWS CDK, or even regional expansions, AWS announces all of them under the name of AWS Glue. When investigating a list of sources, the zero-ETL section of the AWS Glue Developer Guide serves as the starting point. The documentation for each destination service carries its own limitations, however, and the primary sources do not always agree. This inconsistency is addressed in a section of the previously published Zero-ETL Integrations on AWS.

This shift aligns with the goal of AWS Glue Elastic Views, announced as a preview in 2020. Both set out to combine data held in several stores without the user building the pipeline. One reached general availability, and the other, in the sources this article checked, did not.

Job Types and Worker Types

AWS Glue jobs support three types: Spark, Python shell, and Ray. Spark jobs can operate in both batch and streaming modes, and with the Flex execution option they run on spare capacity.

Worker types arrived as a choice in April 2019. Prior to that, only one configuration was available. The addition of G.1X and G.2X allowed users to select the amount of memory per executor. Since then, this selection has expanded.

April 2019: G.1X and G.2X
July 2022: G.025X for streaming workloads
May 2023: G.4X and G.8X
July 2025: G.12X and G.16X, along with memory-optimized options: R.1X, R.2X, R.4X, and R.8X

The announcement states that the R series offers twice the memory of the G series. These options are designed for workloads that utilize significant amounts of memory, such as caching, shuffling, and aggregation.

The Entry Points for Writing a Job

AWS Glue has introduced several entry points for creating jobs. Key additions include AWS Glue Studio (announced September 2020), AWS Glue DataBrew (generally available November 2020), interactive sessions (generally available April 2022), and Amazon Q data integration (generally available April 2024).

A migration process, Migrating from AWS Glue DataBrew to AWS Glue Studio, is available for AWS Glue DataBrew. This process details how to import recipes into AWS Glue Studio, noting that certain recipes, including those utilizing UNION and JOIN operations, are not supported. No official AWS announcement placing AWS Glue DataBrew in maintenance or in end of life was found in the sources this article checked. This article records that the migration process exists, and stops short of stating the lifecycle status of AWS Glue DataBrew.

Lifecycle Notes

AWS Glue itself is a current service, with version 6.0 generally available as of August 2026. The following notes pertain to specific features within AWS Glue.

AWS Glue for Ray is no longer being offered to new customers as of April 30, 2026. Existing customers can continue to use the service, and AWS states that it will continue to provide security and availability improvements. AWS points to Amazon Elastic Kubernetes Service and KubeRay as the migration path. This announcement included several other services and features, such as AWS App Runner, AWS Audit Manager, and AWS CloudTrail Lake. This change applies to one job execution engine within AWS Glue and does not affect the core AWS Glue service itself.

AWS calls this status maintenance. AWS Lifecycle Changes defines it as a state where existing users can keep using the service, no new customers are onboarded, and no functionality is added. It is a stage distinct from sunset and from full shutdown. The Developer Guide page that carries this change, AWS Glue for Ray end of support, uses that phrase to refer to closing the feature to new customers, not to an AWS Glue version reaching the end of support defined above.

AWS Glue Elastic Views were initially announced as a preview in December 2020. The sources this article checked hold no general availability announcement in AWS What's New, and no dedicated product page is published.

AWS Glue Integration with AWS Services

The primary AWS services currently integrated with AWS Glue are as follows:

Storage and Table Formats: Amazon S3, Amazon S3 Tables, Apache Iceberg, Apache Hudi, Delta Lake.
Analytics Engines: Amazon Athena, Amazon Redshift, Amazon EMR, Amazon SageMaker Unified Studio.
Governance: AWS Lake Formation, AWS Identity and Access Management, AWS CloudTrail.
Streaming: Amazon Kinesis Data Streams, Amazon Managed Streaming for Apache Kafka.
Data Sources: Amazon DynamoDB, Amazon RDS, Amazon Aurora, and self-managed databases and applications.
Generative AI: Amazon Bedrock and Amazon Q.

For broader context, see the following earlier articles: AWS Data Lakehouse Architecture Guide, Zero-ETL Integrations on AWS, and AWS History and Timeline regarding Amazon Redshift.

Frequently Asked Questions about AWS Glue History

When was AWS Glue announced, and when did it become generally available?

The first announcement was made on December 1, 2016, and it said only that the service would be available soon. It became generally available approximately eight months later, on August 14, 2017, initially available in only one region: US East (N. Virginia).

What is the difference between end of support and end of life for an AWS Glue version?

End of support is the state where security patches and updates stop, technical support is no longer available, and AWS may not honor SLAs. The jobs themselves keep running. End of life is the state where jobs stop running. You can no longer create new jobs or interactive sessions, you can no longer start job runs, running jobs and sessions are stopped, and the version is removed from the SDK and the API.

Which AWS Glue versions have reached end of life?

Three versions: 0.9, 1.0, and 2.0. All three reached end of life on April 1, 2026. Support for these versions ended earlier; version 0.9 and version 1.0 (Python 2) ended support on June 1, 2022, version 1.0 (Python 3) ended support on September 30, 2022, and version 2.0 ended support on January 31, 2024. Python shell version 3.6 reached its end of support on March 31, 2026, but does not have a specified end-of-life date.

Which version does a new AWS Glue job use if you do not specify one?

Version 5.1, according to the current Developer Guide. The default has moved over time. When the Glue version property was introduced in 2019, jobs without a specified version were treated as version 0.9. The version a job runs on, when it was created without specifying one, therefore depends on when it was created.

When did the AWS Glue Data Catalog become usable from engines other than AWS Glue?

From the first day of general availability. Amazon Athena and Amazon EMR gained support on August 14, 2017, and Amazon Redshift Spectrum followed the next day, on August 15, 2017. SQL views usable from several engines then became generally available for Amazon Athena and Amazon Redshift in August 2024, followed by support for AWS Glue's Apache Spark jobs in March 2025, and expanded to Amazon EMR on EC2 in August 2025.

Is AWS Glue being retired?

No. AWS Glue itself is currently an active service. Version 6.0 became generally available on August 21, 2026. AWS Glue for Ray, one of the job execution engines, has been closed to new customers since April 30, 2026. Existing users can continue to use the service.

What is the current status of AWS Glue DataBrew?

The Developer Guide carries a migration process to AWS Glue Studio. That process details how to import recipes, and notes that recipes involving UNION and JOIN operations are not supported. No official AWS announcement about maintenance or end of life for AWS Glue DataBrew was found in the sources this article checked.

Summary

This article outlines the history of AWS Glue, organizing its current features and characteristics.

AWS Glue was initially announced in December 2016 and began general availability in August 2017 as a managed ETL service. What followed reads better not as a list of features added to one service, but as three separate tracks.

The first track focuses on the job execution platform. In 2019, the Glue version property arrived, and the lineage now runs to eight versions, from 0.9 to 6.0. End-of-life dates have been set for older versions; versions 0.9, 1.0, and 2.0 reached their end-of-life date on April 1, 2026. End of support and end of life are different concepts. Jobs keep running on a version that has reached end of support. Jobs stop on a version that has reached end of life.

The second track focuses on the shared metastore. Amazon Athena and Amazon EMR have shared the AWS Glue Data Catalog since the first day of general availability, and it has widened its reach to include multi-engine views, Apache Iceberg materialized views, and federation with external catalogs. It now even includes a preview feature for writing metadata to Amazon S3 Tables.

The third track focuses on integration. The subject of zero-ETL integration moved from the destination services in December 2024 to AWS Glue in July 2025. The primary resource to consult when investigating the range of supported sources is currently the AWS Glue Developer Guide.

These three tracks have evolved at different paces. Versions have moved in yearly steps with one longer gap, the catalog has widened almost every year, and integration changed its subject over a short span. They are hard to tell apart while you follow a single service name. Set side by side, they read as three separate lineages.

It will be interesting to watch which features AWS Glue offers next. This timeline will be updated to reflect further changes to AWS Glue.

There are other timelines covering the broader set of AWS services and related topics.

References:
Tech Blog with curated related content

Written by Hidekazu Konishi