Right now, we’re excited to announce that DataFlow Features (DFF), a function inside Cloudera DataFlow for the Public Cloud, is now usually accessible for AWS, Microsoft Azure, and Google Cloud Platform. DFF supplies an environment friendly, price optimized, scalable approach to run NiFi flows in a totally serverless trend. That is the primary full no-code, no-ops improvement expertise for features, permitting customers to avoid wasting time and sources.
Fig1: First no-code UI within the trade to shortly develop and deploy features to cloud suppliers’ serverless compute providers.
First no-code UI for serverless features
Beforehand, builders needed to write code and depend on code samples to get began with features. Now, they’ll use DataFlow’s no-code UI to be extra productive – they’ll shortly design new NiFi flows after which run them as features in AWS Lambda, Azure Features, and Google Cloud Features.
Fig2: DataFlow Features runtime environments can be found in
AWS Lambda, Azure Features, and Google Cloud Features.
Optimize price and get rid of infrastructure administration
Because the information flows are working in serverless environments within the public clouds, infrastructure administration is a factor of the previous. The movement is barely executed when an occasion triggers the perform, providing a really environment friendly approach of deploying event-driven use circumstances with out requiring builders to expend worthwhile sources on operational duties. For example, a file touchdown in an object retailer (S3, ADLS, or GCS) triggers the execution of an information movement, which then processes the file and sends the outcome some other place.
Fig3: A pattern use case the place a file that lands in an object retailer triggers a perform that processes that file and sends outcomes to a vacation spot.
DataFlow Features supplies an environment friendly, price optimized, scalable approach to run NiFi flows in a totally serverless trend for event-driven use circumstances.
The fitting runtime to your use circumstances
There are actually two methods to run your Apache NiFi information flows within the Cloudera DataFlow service: DataFlow deployments and DataFlow Features:
- Deployments runtime is optimized for high-throughput, low-latency streaming use circumstances
- Features runtime is finest fitted to event-driven, short-lived use circumstances
Fig4: Runtime choices within the public cloud: DataFlow Deployments and DataFlow Features
Under is a extra detailed breakdown of the 2 NiFi runtime choices within the public cloud:
| Runtime choices within the Public Cloud | ||
| Characteristic | DataFlow Deployments | DataFlow Features |
| Cloud Runtime | NiFi clusters utilizing
Kubernetes/containers |
NiFi flows working on cloud suppliers’ serverless compute providers (AWS Lambda, Azure Features, and Google Cloud Features) |
| Use Case | Use circumstances that want low latency for prime throughput workloads requiring always-running NiFi flows | Occasion pushed, micro-bursty use circumstances with no sub-second latency requirement the place NiFi flows don’t have to run repeatedly |
| Advantages | Auto-scaling Kubernetes clusters for lengthy working workflows with centralized monitoring | Environment friendly, price optimized, scalable approach to run NiFi flows serverless, permitting builders to deal with enterprise logic |
Abstract
DataFlow Features supplies a brand new, environment friendly approach to run your event-driven Apache NiFi information flows.
With DataFlow Features you possibly can deploy your movement functions in minutes by leveraging the serverless structure of all main public cloud suppliers (AWS, Azure, and Google Cloud Platform), and also you don’t have to fret in regards to the operational overhead of managing and sustaining NiFi movement runtime environments.
To be taught extra on the best way to arrange and run DataFlow Features in AWS Lambda, Azure Features, and Google Cloud Features, checkout our technical weblog, or take a product tour for a light-weight step-by-step expertise.
