Announcing three new Apache Flink connectors, the new connector versioning strategy and externalizationAugust 4, 2023 - Elphas Toringepi (@elphastori)
New connectors #
- Amazon DynamoDB - This connector includes a sink that provides at-least-once delivery guarantees.
- MongoDB connector - This connector includes a source and sink that provide at-least-once guarantees.
- OpenSearch sink - This connector includes a sink that provides at-least-once guarantees.
|Connector||Date Released||Supported Flink Versions|
|Amazon DynamoDB sink||2022-12-02||1.15+|
List of Contributors #
The Apache Flink community would like to express gratitude to all the new connector contributors:
Andriy Redko, Chesnay Schepler, Danny Cranmer, darenwkt, Hong Liang Teoh, Jiabao Sun, Leonid Ilyevsky, Martijn Visser, nir.tsruya, Sergey Nuyanzin, Weijie Guo, Yuri Gusev, Yuxin Tan
Externalized connectors #
The community has externalized connectors from Flink’s main repository. This was driven to realise the following benefits:
- Faster releases of connectors: New features can be added more quickly, bugs can be fixed immediately, and we can have faster security patches in case of direct or indirect (through dependencies) security flaws.
- Adding newer connector features to older Flink versions: By having stable connector APIs, the same connector artifact may be used with different Flink versions. Thus, new features can also immediately be used with older Flink versions.
- More activity and contributions around connectors: By easing the contribution and development process around connectors, we will see faster development and also more connectors.
- Documentation: Standardized documentation and user experience for the connectors, regardless of where they are maintained.
- A faster Flink CI: By not needing to build and test connectors, the Flink CI pipeline will be faster and Flink developers will experience fewer build instabilities (which mostly come from connectors). That should speed up Flink development.
The following connectors have been moved to individual repositories:
- Kafka / Upsert-Kafka
- Google Cloud PubSub
- AWS connectors:
Connectors continue to use the same Maven dependency
artifactId. However, the JAR artifact
version has changed and now uses the format,
<major>.<minor>.<patch>-<flink-major>.<flink-minor>. For example, to use the DynamoDB connector for Flink 1.17, add the following dependency to your project:
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-dynamodb</artifactId> <version>4.1.0-1.17</version> </dependency>
You can find the maven dependency for a connector in the Flink connectors documentation for a specific Flink version. Use the Flink Downloads page to verify which version your connector is compatible with.
Similarly, when creating JIRAs to report issues or to contribute to externalized connectors, the
Affects Version/s and
Fix Version/s fields should now use the connector version instead of a Flink version. The format should be
<connector-name>-<major>.<minor>.<patch>. For example, use
opensearch-1.1.0 for the OpenSearch connector. All other fields in the JIRA like
Component/s remain the same.
For more information on how to contribute to externalized connectors, see the Externalized Connector development wiki.