AWS Glue now supports the ability to test your Glue ETL scripts on development endpoints using Apache Spark 2.4.3 and Python 3
AWS Glue has updated its Apache Spark infrastructure to support Apache Spark 2.4.3 (in addition to Apache Spark 2.2.1) for Glue scripts submitted on development endpoints. This enables you to take advantage of stability fixes and new features available in this version of Apache Spark.
You can pick the Apache Spark infrastructure that you want your AWS Glue scripts to run on by choosing a Glue version. Glue development endpoints that were created without specifying a Glue version are defaulted to Glue version 0.9. Glue development endpoints with Glue version of 1.0 will run on Apache Spark 2.4.3. In addition to supporting the latest version of Spark, you will also have the ability to choose between Python 2 and Python 3. Previously, you were able to choose Glue versions only for Glue ETL jobs.
To learn more about how you can take advantage of this feature, please visit our documentation.
This feature is now available in all the AWS regions where AWS Glue is available except AWS GovCloud (US-East) and AWS GovCloud (US-West).