Announcing three new Apache Flink connectors, the new connector versioning strategy and externalization

New connectors #

We’re excited to announce that Apache Flink now supports three new connectors: Amazon DynamoDB, MongoDB and OpenSearch! The connectors are available for both the DataStream and Table/SQL APIs.

• Amazon DynamoDB - This connector includes a sink that provides at-least-once delivery guarantees.
• MongoDB connector - This connector includes a source and sink that provide at-least-once guarantees.
• OpenSearch sink - This connector includes a sink that provides at-least-once guarantees.

List of Contributors #

The Apache Flink community would like to express gratitude to all the new connector contributors:

Andriy Redko, Chesnay Schepler, Danny Cranmer, darenwkt, Hong Liang Teoh, Jiabao Sun, Leonid Ilyevsky, Martijn Visser, nir.tsruya, Sergey Nuyanzin, Weijie Guo, Yuri Gusev, Yuxin Tan

Externalized connectors #

The community has externalized connectors from Flink’s main repository. This was driven to realise the following benefits:

• Faster releases of connectors: New features can be added more quickly, bugs can be fixed immediately, and we can have faster security patches in case of direct or indirect (through dependencies) security flaws.
• Adding newer connector features to older Flink versions: By having stable connector APIs, the same connector artifact may be used with different Flink versions. Thus, new features can also immediately be used with older Flink versions.
• More activity and contributions around connectors: By easing the contribution and development process around connectors, we will see faster development and also more connectors.
• Documentation: Standardized documentation and user experience for the connectors, regardless of where they are maintained.
• A faster Flink CI: By not needing to build and test connectors, the Flink CI pipeline will be faster and Flink developers will experience fewer build instabilities (which mostly come from connectors). That should speed up Flink development.

The following connectors have been moved to individual repositories:

• Kafka / Upsert-Kafka
• Cassandra
• Elasticsearch
• MongoDB
• OpenSearch
• RabbitMQ
• Google Cloud PubSub
• Pulsar
• JDBC
• HBase
• Hive
• AWS connectors:
  • Firehose
  • Kinesis
  • DynamoDB

Versioning #

Connectors continue to use the same Maven dependency groupId and artifactId. However, the JAR artifact version has changed and now uses the format, <major>.<minor>.<patch>-<flink-major>.<flink-minor>. For example, to use the DynamoDB connector for Flink 1.17, add the following dependency to your project:

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-dynamodb</artifactId>
    <version>4.1.0-1.17</version>
</dependency>

You can find the maven dependency for a connector in the Flink connectors documentation for a specific Flink version. Use the Flink Downloads page to verify which version your connector is compatible with.

Contributing #

Similarly, when creating JIRAs to report issues or to contribute to externalized connectors, the Affects Version/s and Fix Version/s fields should now use the connector version instead of a Flink version. The format should be <connector-name>-<major>.<minor>.<patch>. For example, use opensearch-1.1.0 for the OpenSearch connector. All other fields in the JIRA like Component/s remain the same.

For more information on how to contribute to externalized connectors, see the Externalized Connector development wiki.