What's New in the Pulsar Flink Connector 2.7.0

07 Jan 2021 Jianyun Zhao (@yihy8023) & Jennifer Huang (@Jennife06125739)

With the unification of batch and streaming regarded as the future in data processing, the Pulsar Flink Connector provides an ideal solution for unified batch and stream processing with Apache Pulsar and Apache Flink. The Pulsar Flink Connector 2.7.0 supports features in Pulsar 2.7 and Flink 1.12 and is fully compatible with Flink's data format. The Pulsar Flink Connector 2.7.0 will be contributed to the Flink repository soon and the contribution process is ongoing.

Continue reading »

Apache Flink 1.12.0 Release Announcement

10 Dec 2020 Marta Paes (@morsapaes) & Aljoscha Krettek (@aljoscha)

The Apache Flink community is excited to announce the release of Flink 1.12.0! Close to 300 contributors worked on over 1k threads to bring significant improvements to usability as well as new features to Flink users across the whole API stack. We're particularly excited about adding efficient batch execution to the DataStream API, Kubernetes HA as an alternative to ZooKeeper, support for upsert mode in the Kafka SQL connector and the new Python DataStream API! Read on for all major new features and improvements, important changes to be aware of and what to expect moving forward!

Continue reading »

From Aligned to Unaligned Checkpoints - Part 1: Checkpoints, Alignment, and Backpressure

15 Oct 2020 Arvid Heise & Stephan Ewen

Apache Flink’s checkpoint-based fault tolerance mechanism is one of its defining features. Because of that design, Flink unifies batch and stream processing, can easily scale to both very small and extremely large scenarios and provides support for many operational features. In this post we recap the original checkpointing process in Flink, its core properties and issues under backpressure.

Continue reading »

Stateful Functions 2.2.0 Release Announcement

28 Sep 2020 Tzu-Li (Gordon) Tai (@tzulitai) & Igal Shilman (@IgalShilman)

The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.2.0! This release introduces major features that extend the SDKs, such as support for asynchronous functions in the Python SDK, new persisted state constructs, and a new SDK that allows embedding StateFun functions within a Flink DataStream job. Moreover, we’ve also included important changes that improve out-of-the-box stability for common workloads, as well as increased observability for operational purposes.

Continue reading »