Tuesday, February 7, 2023
HomeBig DataIntroducing AWS Glue Flex jobs: Price financial savings on ETL workloads

Introducing AWS Glue Flex jobs: Price financial savings on ETL workloads

AWS Glue is a serverless knowledge integration service that makes it easy to find, put together, and mix knowledge for analytics, machine studying (ML), and utility growth. You need to use AWS Glue to create, run, and monitor knowledge integration and ETL (extract, remodel, and cargo) pipelines and catalog your belongings throughout a number of knowledge shops. Sometimes, these knowledge integration jobs can have various levels of precedence and time sensitivity. For instance, non-urgent workloads akin to pre-production, testing, and one-time knowledge masses usually don’t require quick job startup occasions or constant runtimes through devoted assets.

At present, we’re happy to announce the overall availability of a brand new AWS Glue job run class known as Flex. Flex lets you optimize your prices in your non-urgent or non-time delicate knowledge integration workloads akin to pre-production jobs, testing, and one-time knowledge masses. With Flex, AWS Glue jobs run on spare compute capability as a substitute of devoted {hardware}. The beginning and runtimes of jobs utilizing Flex can range as a result of spare compute assets aren’t available and will be reclaimed through the run of a job

Whatever the run choice used, AWS Glue jobs have the identical capabilities, together with entry to customized connectors, visible authoring interface, job scheduling, and Glue Auto Scaling. With the Flex execution choice, prospects can optimize the prices of their knowledge integration workloads by configuring the execution choice based mostly on the workloads’ necessities, utilizing commonplace execution choice for time-sensitive workloads, and Flex for non-urgent workloads. The Flex execution class is offered for AWS Glue 3.0 Spark jobs.

The Flex execution class is offered for AWS Glue 3.0 Spark jobs.

On this put up, we offer extra particulars about AWS Glue Flex jobs and the right way to allow Flex capability.

How do you employ Versatile capability?

The AWS Glue jobs API now helps an extra parameter known as execution-class, which helps you to select STANDARD or FLEX when working the job. To make use of Flex, you merely set the parameter to FLEX.

To allow Flex through the AWS Glue Studio console, full the next steps:

  1. On the AWS Glue Studio console, whereas authoring a job, navigate to the Job particulars tab
  2. Choose Flex Execution.
  3. Set an applicable worth for the Job Timeout parameter (defaults to 120 minutes for Flex jobs).
  4. Save the job.
  5. After finalizing all different particulars, select Run to run the job with Flex capability.

On the Runs tab, it’s best to be capable of see FLEX listed beneath Execution class.

You can too allow Flex through the AWS Command Line Interface (AWS CLI).

You’ll be able to set the --execution-class setting within the start-job-run API, which helps you to run a selected AWS Glue job’s run with Flex capability:

aws glue start-job-run --job-name my-job 
    --execution-class FLEX 
    --timeout 300 

You can too set the --execution-class through the create-job API. This units the default run class of all of the runs of this job to FLEX:

aws glue create-job 
    --name flexCLI 
    --role AWSGlueServiceRoleDefault 
    --command "Title=glueetl,ScriptLocation=s3://mybucket/myfolder/" 
    --region us-east-2 
    --execution-class FLEX 
    --worker-type G.1X 
    --number-of-workers 10 
    --glue-version 3.0

The next are further particulars concerning the related parameters:

  • –execution-class – The enum string that specifies if a job must be run as FLEX or STANDARD capability. The default is STANDARD.
  • –timeout – Specifies the time (in minutes) the job will run earlier than it’s moved right into a TIMEOUT state.

When must you use Versatile capability?

The Flex execution class is right for lowering the prices of time-insensitive workloads. For instance:

  • Nightly ETL jobs, or jobs that run over weekends for processing workloads
  • One-time bulk knowledge ingestion jobs
  • Jobs working in check environments or pre-production workloads
  • Time-insensitive workloads the place it’s acceptable to have variable begin and finish occasions

Compared, the usual execution class is right for time-sensitive workloads that require quick job startup and devoted assets. As well as, jobs which have downstream dependencies are higher served by the usual execution class.

What’s the typical life-cycle of a Versatile capability Job?

When a start-job-run API name is issued, with the execution-class set to FLEX, AWS Glue will start to request compute assets. If no assets can be found instantly upon issuing the API name, the job will transfer right into a WAITING state. No billing happens at this level.

As quickly because the job is ready to purchase compute assets, the job strikes to a RUNNING state. At this level, even when all of the computes requested aren’t out there, the job begins working on no matter {hardware} is current. As extra Flex capability turns into out there, AWS Glue provides it to the job, as much as a most worth specified by Variety of employees.

At this level, billing begins. You’re charged just for the compute assets which are working at any given time, and just for the length that they ran for.

Whereas the job is working, if Flex capability is reclaimed, AWS Glue continues working the job on the prevailing compute assets whereas it tries to satisfy the shortfall by requesting extra assets. If capability is reclaimed, billing for that capability is halted as effectively. Billing for brand spanking new capability will begin when it’s provisioned once more. If the job completes efficiently, the job’s state strikes to SUCCEEDED. If the job fails as a consequence of varied person or system errors, the job’s state transitions to FAILED. If the job is unable to finish earlier than the time specified by the --timeout parameter, whether or not as a consequence of an absence of compute capability or as a consequence of points with the AWS Glue job script, the job goes right into a TIMEOUT state.

Versatile job runs depend on the provision of non-dedicated compute capability in AWS, which in flip will depend on a number of components, such because the Area and Availability Zone, time of day, day of the week, and the variety of DPUs required by a job.

A parameter of specific significance for Flex Jobs is the --timeout worth. It’s attainable for Flex jobs to take longer to run than commonplace jobs, particularly if capability is reclaimed whereas the job is working. Because of this, deciding on the proper timeout worth that’s applicable to your workload is essential. Select a timeout worth such that the overall price of the Flex job run doesn’t exceed a regular job run. If the worth is about too excessive, the job can look ahead to too lengthy, attempting to amass capability that isn’t out there. If the worth is about too low, the job occasions out, even when capability is offered and the job execution is continuing accurately.

How are Flex capability jobs billed?

Flex jobs are billed per employee on the Flex DPU-hour charges. Because of this you’re billed just for the capability that really ran through the execution of the job, for the length that it ran.

For instance, in case you ran an AWS Glue Flex job for 10 employees, and AWS Glue was solely capable of purchase 5 employees, you’re solely billed for 5 employees, and just for the length that these employees ran. If, through the job run, two out of these 5 employees are reclaimed, then billing for these two employees is stopped, whereas billing for the remaining three employees continues. If provisioning for the 2 reclaimed employees is profitable through the job run, billing for these two will begin once more.

For extra data on Flex pricing, consult with AWS Glue pricing.


This put up discusses the brand new AWS Glue Flex job execution class, which lets you optimize prices for non-time-sensitive ETL workloads and check environments.

You can begin utilizing Flex capability to your current and new workloads right this moment. Nevertheless, notice that the Flex class will not be supported for Python Shell jobs, AWS Glue streaming jobs, or AWS Glue ML jobs.

For extra data on AWS Glue Flex jobs, consult with their newest documentation.

Particular because of everybody who contributed to the launch: Parag Shah, Sampath Shreekantha, Yinzhi Xi and Jessica Cheng,

In regards to the authors

Aniket Jiddigoudar is a Massive Information Architect on the AWS Glue group.

Vaibhav Porwal is a Senior Software program Growth Engineer on the AWS Glue group.

Sriram Ramarathnam is a Software program Growth Supervisor on the AWS Glue group.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments