Saturday, November 26, 2022
HomeBig DataSaying GA of DataFlow Capabilities

Saying GA of DataFlow Capabilities


Right this moment, we’re excited to announce that DataFlow Capabilities (DFF), a characteristic inside Cloudera DataFlow for the Public Cloud, is now usually obtainable for AWS, Microsoft Azure, and Google Cloud Platform. DFF offers an environment friendly, price optimized, scalable method to run NiFi flows in a totally serverless vogue. That is the primary full no-code, no-ops improvement expertise for capabilities, permitting customers to avoid wasting time and sources. 

Fig1: First no-code UI within the trade to rapidly develop and deploy capabilities to cloud suppliers’ serverless compute companies.

First no-code UI for serverless capabilities

Beforehand, builders needed to write code and depend on code samples to get began with capabilities. Now, they’ll use DataFlow’s no-code UI to be extra productive – they’ll rapidly design new NiFi flows after which run them as capabilities in AWS Lambda, Azure Capabilities, and Google Cloud Capabilities.

Fig2: DataFlow Capabilities runtime environments can be found in
AWS Lambda, Azure Capabilities, and Google Cloud Capabilities.

Optimize price and get rid of infrastructure administration

Because the knowledge flows are operating in serverless environments within the public clouds, infrastructure administration is a factor of the previous. The circulate is barely executed when an occasion triggers the operate, providing a really environment friendly method of deploying event-driven use instances with out requiring builders to expend precious sources on operational obligations. For example, a file touchdown in an object retailer (S3, ADLS, or GCS) triggers the execution of an information circulate, which then processes the file and sends the consequence someplace else.

Fig3: A pattern use case the place a file that lands in an object retailer triggers a operate that processes that file and sends outcomes to a vacation spot.

DataFlow Capabilities offers an environment friendly, price optimized, scalable method to run NiFi flows in a totally serverless vogue for event-driven use instances.

The correct runtime to your use instances

There at the moment are two methods to run your Apache NiFi knowledge flows within the Cloudera DataFlow service: DataFlow deployments and DataFlow Capabilities:

  • Deployments runtime is optimized for high-throughput, low-latency streaming use instances
  • Capabilities runtime is greatest fitted to event-driven, short-lived use instances 

Fig4: Runtime choices within the public cloud: DataFlow Deployments and DataFlow Capabilities

Under is a extra detailed breakdown of the 2 NiFi runtime choices within the public cloud: 

Runtime choices within the Public Cloud
Characteristic DataFlow Deployments DataFlow Capabilities
Cloud Runtime NiFi clusters utilizing 

Kubernetes/containers

NiFi flows operating on cloud suppliers’ serverless compute companies (AWS Lambda, Azure Capabilities, and Google Cloud Capabilities)
Use Case Use instances that want low latency for prime throughput workloads requiring always-running NiFi flows Occasion pushed, micro-bursty use instances with no sub-second latency requirement the place NiFi flows don’t must run repeatedly
Advantages Auto-scaling Kubernetes clusters for lengthy operating workflows with centralized monitoring Environment friendly, price optimized, scalable method to run NiFi flows serverless, permitting builders to give attention to enterprise logic

Abstract

DataFlow Capabilities offers a brand new, environment friendly method to run your event-driven Apache NiFi knowledge flows.

With DataFlow Capabilities you may deploy your circulate purposes in minutes by leveraging the serverless structure of all main public cloud suppliers (AWS, Azure, and Google Cloud Platform), and also you would not have to fret concerning the operational overhead of managing and sustaining NiFi circulate runtime environments.

To be taught extra on how one can arrange and run DataFlow Capabilities in AWS Lambda, Azure Capabilities, and Google Cloud Capabilities, checkout our technical weblog, or take a product tour for a light-weight step-by-step expertise.

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular

Recent Comments