December 2, 2020 -
Andrey Zagrebin
The Flink community has been working for some time on making Flink a truly unified batch and stream processing system. Achieving this involves touching a lot of different components of the Flink stack, from the user-facing APIs all the way to low-level operator processes such as task scheduling. In this blogpost, we’ll take a closer look at how far the community has come in improving scheduling for batch workloads, why this matters and what you can expect in the Flink 1.
...
Continue reading »
November 11, 2020 -
Tzu-Li (Gordon) Tai
(@tzulitai)
The Apache Flink community released the first bugfix release of the Stateful Functions (StateFun) 2.2 series, version 2.2.1.
This release fixes a critical bug that causes restoring the Stateful Functions cluster from snapshots (checkpoints or savepoints) to fail under certain conditions. Starting from this release, StateFun now creates snapshots with a more robust format that allows it to be restored safely going forward.
We strongly recommend all users to upgrade to 2.
...
Continue reading »
October 15, 2020 -
Arvid Heise
Stephan Ewen
Apache Flink’s checkpoint-based fault tolerance mechanism is one of its defining features. Because of that design, Flink unifies batch and stream processing, can easily scale to both very small and extremely large scenarios and provides support for many operational features like stateful upgrades with state evolution or roll-backs and time-travel.
Despite all these great properties, Flink’s checkpointing method has an Achilles Heel: the speed of a completed checkpoint is determined by the speed at which data flows through the application.
...
Continue reading »
October 13, 2020 -
Tzu-Li (Gordon) Tai
(@tzulitai)
Stateful Functions (StateFun) simplifies the building of distributed stateful applications by combining the best of two worlds: the strong messaging and state consistency guarantees of stateful stream processing, and the elasticity and serverless experience of today’s cloud-native architectures and popular event-driven FaaS platforms. Typical StateFun applications consist of functions deployed behind simple services using these modern platforms, with a separate StateFun cluster playing the role of an “event-driven database” that provides consistency and fault-tolerance for the functions’ state and messaging.
...
Continue reading »
September 28, 2020 -
Tzu-Li (Gordon) Tai
(@tzulitai)
Igal Shilman
(@IgalShilman)
The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.2.0! This release introduces major features that extend the SDKs, such as support for asynchronous functions in the Python SDK, new persisted state constructs, and a new SDK that allows embedding StateFun functions within a Flink DataStream job. Moreover, we’ve also included important changes that improve out-of-the-box stability for common workloads, as well as increased observability for operational purposes.
...
Continue reading »
September 17, 2020 -
Zhu Zhu
(@zhuzhv)
The Apache Flink community released the second bugfix version of the Apache Flink 1.11 series.
This release includes 96 fixes and minor improvements for Flink 1.11.1. The list below includes a detailed list of all fixes and improvements.
We highly recommend all users to upgrade to Flink 1.11.2.
Updated Maven dependencies:
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>1.11.2</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>1.11.2</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>1.11.2</version> </dependency> You can find the binaries on the updated Downloads page.
...
Continue reading »
September 4, 2020 -
Marta Paes
(@morsapaes)
Ah, so much for a quiet August month. This time around, we bring you some new Flink Improvement Proposals (FLIPs), a preview of the upcoming Flink Stateful Functions 2.2 release and a look into how far Flink has come in comparison to 2019.
The Past Month in Flink # Flink Releases # Getting Ready for Flink Stateful Functions 2.2 # The details of the next release of Stateful Functions are under discussion in this @dev mailing list thread, and the feature freeze is set for September 10th — so, you can expect Stateful Functions 2.
...
Continue reading »
September 1, 2020 -
Andrey Zagrebin
Apache Flink 1.11 comes with significant changes to the memory model of Flink’s JobManager and configuration options for your Flink clusters. These recently-introduced changes make Flink adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), providing better control over its memory consumption.
The previous blog post focused on the memory model of the TaskManagers and how it was improved in Flink 1.10. This post addresses the same topic but for the JobManager instead.
...
Continue reading »
August 25, 2020 -
Zhu Zhu
(@zhuzhv)
The Apache Flink community released the second bugfix version of the Apache Flink 1.10 series.
This release includes 73 fixes and minor improvements for Flink 1.10.1. The list below includes a detailed list of all fixes and improvements.
We highly recommend all users to upgrade to Flink 1.10.2.
Note After FLINK-18242, the deprecated `OptionsFactory` and `ConfigurableOptionsFactory` classes are removed (not applicable for release-1.10), please use `RocksDBOptionsFactory` and `ConfigurableRocksDBOptionsFactory` instead. Please also recompile your application codes if any class extending `DefaultConfigurableOptionsFactory` Note After FLINK-17800 by default we will set `setTotalOrderSeek` to true for RocksDB's `ReadOptions`, to prevent user from miss using `optimizeForPointLookup`.
...
Continue reading »
August 20, 2020 -
Robert Metzger
(@rmetzger_)
With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
The Flink community recently put some effort into improving the Docker experience for our users with the goal to reduce confusion and improve usability.
Let’s quickly break down the recent improvements:
Reduce confusion: Flink used to have 2 Dockerfiles and a 3rd file maintained outside of the official repository — all with different features and varying stability.
...
Continue reading »