Google Announces a New, More Services-Based Architecture Called Runner V2 to Dataflow

Luca Mezzalira explains how to implement micro-front-ends, enabling to scale up a project with tens of developers without reducing the throughput.

This style can provide several benefits to our projects, offering a level of decoupling never seen before in single-page applications or universal architectures

Delta-p statistics is an easier means of communicating results to a non-technical audience than the plain coefficients of a logistic regression model.

In these two episodes Shane Hastie, Lead Editor for Culture & Methods, first spoke to Em Campbell-Pretty about cultural change, the Scaled Agile Framework and her role as a SAFe Fellow.

The panelists discuss DevOps buzzwords and when and where they might have value for organizations seeking performance improvements.

Google Cloud Dataflow is a fully-managed service for executing Apache Beam pipelines within the Google Cloud Platform.

Moreover, Google packaged this framework together with Dataflow Shuffle for batch jobs and Streaming Engine for streaming jobs, allowing them to provide a standard feature set from now on across all language-specific SDKs, as well as share bug fixes and performance improvements. The critical component in the architecture is the worker Virtual Machines , which run the entire pipeline and have access to the various SDKs.

If features or transforms are missing for a given language, they must be duplicated across various SDKs to ensure parity; otherwise, there will be gaps in feature coverage and newer SDKs like Apache Beam Go SDK will support fewer features and exhibit inferior performance characteristics for some scenarios.

Currently, Dataflow Runner v2 is available with Python streaming pipelines and Google recommends developers to test the new Runner out with current non-production workloads before enabling it by default on all new pipelines.

Original article
Author: InfoQ

Curated and peer-reviewed content covering innovation in professional software development, read by over 1 million developers worldwide

InfoQ has recently written 1 articles on similar topics including :
  1. "Researchers at Google have developed a new deep-learning model called BigBird that allows Transformer neural networks to process sequences up to 8x longer than previously possible. Networks based on this model achieved new state-of-the-art performance levels on natural-language processing (NLP) and genomics tasks". (September 2, 2020)
Posted on  , , ,