Stateful Functions 2.1.0 Release Announcement

09 Jun 2020 Marta Paes (@morsapaes)

The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.1.0! This release introduces new features around state expiration and performance improvements for co-located deployments, as well as other important changes that improve the stability and testability of the project. As the community around StateFun grows, the release cycle will follow this pattern of smaller and more frequent releases to incorporate user feedback and allow for faster iteration.

The binary distribution and source artifacts are now available on the updated Downloads page of the Flink website, and the most recent Python SDK distribution is available on PyPI. For more details, check the complete release changelog and the updated documentation. We encourage you to download the release and share your feedback with the community through the Flink mailing lists or JIRA!

New Features and Improvements

Support for State Time-To-Live (TTL)

Being able to define state expiration and a state cleanup strategy is a useful feature for stateful applications — for example, to keep state size from growing indefinitely or to work with sensitive data. In previous StateFun versions, users could implement this behavior manually using delayed messages as state expiration callbacks. For StateFun 2.1, the community has worked on enabling users to configure any persisted state to expire and be purged after a given duration (i.e. the state time-to-live) (FLINK-17644, FLINK-17875).

Persisted state can be configured to expire after the last write operation (AFTER_WRITE) or after the last read or write operation (AFTER_READ_AND_WRITE). For the Java SDK, users can configure State TTL in the definition of their persisted fields:

@Persisted
PersistedValue<Integer> table = PersistedValue.of(
    "my-value",
    Integer.class,
    Expiration.expireAfterWriting(Duration.ofHours(1)));

For remote functions using e.g. the Python SDK, users can configure State TTL in their module.yaml:

functions:
  - function:
     states:
       - name: xxxx
         expireAfter: 5min # optional key
Note: The state expiration mode for remote functions is currently restricted to AFTER_READ_AND_WRITE, and the actual TTL being set is the longest duration across all registered state, not for each individual state entry. This is planned to be improved in upcoming releases (FLINK-17954).

Improved Performance with UNIX Domain Sockets (UDS)

Stateful functions can be deployed in multiple ways, even within the same application. For deployments where functions are co-located with the Flink StateFun workers, it’s common to use Kubernetes to deploy pods consisting of a Flink StateFun container and the function sidecar container, communicating via the pod-local network. To improve the performance of such deployments, StateFun 2.1 allows using Unix Domain Sockets (UDS) to communicate between containers in the same pod (i.e. the same machine) (FLINK-17611), which drastically reduces the overhead of going through the network stack.

Users can enable transport via UDS in a remote module by specifying the following in their module.yaml:

functions:
  - function:
     spec:
       - endpoint: http(s)+unix://<socket-file-path>/<serve-url-path>

Important Changes

  • [FLINK-17712] The Flink version in StateFun 2.1 has been upgraded to 1.10.1, the most recent patch version.

  • [FLINK-17533] StateFun 2.1 now supports concurrent checkpoints, which means applications will no longer fail on savepoints that are triggered concurrently to a checkpoint.

  • [FLINK-16928] StateFun 2.0 was using the Flink legacy scheduler due to a bug in Flink 1.10. In 2.1, this change is reverted to using the new Flink scheduler again.

  • [FLINK-17516] The coverage for end-to-end StateFun tests has been extended to also include exactly-once semantics verification (with failure recovery).

Release Notes

Please review the release notes for a detailed list of changes and new features if you plan to upgrade your setup to Stateful Functions 2.1.

List of Contributors

The Apache Flink community would like to thank all contributors that have made this release possible:

abc863377, Authuir, Chesnay Schepler, Congxian Qiu, David Anderson, Dian Fu, Francesco Guardiani, Igal Shilman, Marta Paes Moreira, Patrick Wiener, Rafi Aroch, Seth Wiesman, Stephan Ewen, Tzu-Li (Gordon) Tai

If you’d like to get involved, we’re always looking for new contributors — especially around SDKs for other languages like Go, Rust or Javascript.