19 Jan 2021 Xintong Song
The Apache Flink community released the first bugfix version of the Apache Flink 1.12 series.
This release includes 79 fixes and minor improvements for Flink 1.12.0. The list below includes a detailed list of all fixes and improvements.
We highly recommend all users to upgrade to Flink 1.12.1.
Attention: Using unaligned checkpoints in Flink 1.12.0 combined with two/multiple inputs tasks or with union inputs for single input tasks can result in corrupted state.
This can happen if a new checkpoint is triggered before recovery is fully completed. For state to be corrupted a task with two or more input gates must receive a checkpoint barrier exactly at the same time this tasks finishes recovering spilled in-flight data. In such case this new checkpoint can succeed, with corrupted/missing in-flight data, which will result in various deserialisation/corrupted data stream errors when someone attempts to recover from such corrupted checkpoint.
Using unaligned checkpoints in Flink 1.12.1, a corruption may occur in the checkpoint following a declined checkpoint.
A late barrier of a canceled checkpoint may lead to buffers being not written into the successive checkpoint, such that recovery is not possible. This happens, when the next checkpoint barrier arrives at a given operator before all previous barriers arrived, which can only happen after cancellation in unaligned checkpoints.
Updated Maven dependencies:
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>1.12.1</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>1.12.1</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>1.12.1</version> </dependency>
You can find the binaries on the updated Downloads page.
List of resolved issues:
- [FLINK-18897] - Add documentation for the maxwell-json format
- [FLINK-20352] - Rework command line interface documentation page
- [FLINK-20353] - Rework logging documentation page
- [FLINK-20354] - Rework standalone deployment documentation page
- [FLINK-20355] - Rework K8s deployment documentation page
- [FLINK-20356] - Rework Mesos deployment documentation page
- [FLINK-20422] - Remove from .html files in flink documentation
- [FLINK-20485] - Map views are deserialized multiple times
- [FLINK-20601] - Rework PyFlink CLI documentation
- [FLINK-19369] - BlobClientTest.testGetFailsDuringStreamingForJobPermanentBlob hangs
- [FLINK-19435] - Deadlock when loading different driver classes concurrently using Class.forName
- [FLINK-19725] - Logger cannot be initialized due to timeout: LoggerInitializationException is thrown
- [FLINK-19880] - Fix ignore-parse-errors not work for the legacy JSON format
- [FLINK-20213] - Partition commit is delayed when records keep coming
- [FLINK-20221] - DelimitedInputFormat does not restore compressed filesplits correctly leading to dataloss
- [FLINK-20273] - Fix Table api Kafka connector Sink Partitioner Document Error
- [FLINK-20372] - Update Kafka SQL connector page to mention properties.* options
- [FLINK-20389] - UnalignedCheckpointITCase failure caused by NullPointerException
- [FLINK-20404] - ZooKeeper quorum fails to start due to missing log4j library
- [FLINK-20419] - Insert fails due to failure to generate execution plan
- [FLINK-20428] - ZooKeeperLeaderElectionConnectionHandlingTest.testConnectionSuspendedHandlingDuringInitialization failed with "No result is expected since there was no leader elected before stopping the server, yet"
- [FLINK-20429] - KafkaTableITCase.testKafkaTemporalJoinChangelog failed with unexpected results
- [FLINK-20433] - UnalignedCheckpointTestBase.execute failed with "TestTimedOutException: test timed out after 300 seconds"
- [FLINK-20464] - Some Table examples are not built correctly
- [FLINK-20467] - Fix the Example in Python DataStream Doc
- [FLINK-20470] - MissingNode can't be casted to ObjectNode when deserializing JSON
- [FLINK-20476] - New File Sink end-to-end test Failed
- [FLINK-20486] - Hive temporal join should allow monitor interval smaller than 1 hour
- [FLINK-20492] - The SourceOperatorStreamTask should implement cancelTask() and finishTask()
- [FLINK-20493] - SQLClientSchemaRegistryITCase failed with "Could not build the flink-dist image"
- [FLINK-20521] - Null result values are being swallowed by RPC system
- [FLINK-20525] - StreamArrowPythonGroupWindowAggregateFunctionOperator doesn't handle rowtime and proctime properly
- [FLINK-20543] - Fix typo in upsert kafka docs
- [FLINK-20554] - The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI
- [FLINK-20582] - Fix typos in `CREATE Statements` docs
- [FLINK-20607] - a wrong example in udfs page.
- [FLINK-20615] - Local recovery and sticky scheduling end-to-end test timeout with "IOException: Stream Closed"
- [FLINK-20626] - Canceling a job when it is failing will result in job hanging in CANCELING state
- [FLINK-20630] - [Kinesis][DynamoDB] DynamoDB Streams Consumer fails to consume from Latest
- [FLINK-20646] - ReduceTransformation does not work with RocksDBStateBackend
- [FLINK-20648] - Unable to restore job from savepoint when using Kubernetes based HA services
- [FLINK-20664] - Support setting service account for TaskManager pod
- [FLINK-20665] - FileNotFoundException when restore from latest Checkpoint
- [FLINK-20666] - Fix the deserialized Row losing the field_name information in PyFlink
- [FLINK-20669] - Add the jzlib LICENSE file in flink-python module
- [FLINK-20703] - HiveSinkCompactionITCase test timeout
- [FLINK-20704] - Some rel data type does not implement the digest correctly
- [FLINK-20756] - PythonCalcSplitConditionRule is not working as expected
- [FLINK-20764] - BatchGroupedReduceOperator does not emit results for singleton inputs
- [FLINK-20781] - UnalignedCheckpointITCase failure caused by NullPointerException
- [FLINK-20784] - .staging_xxx does not exist, when insert into hive
- [FLINK-20793] - Fix NamesTest due to code style refactor
- [FLINK-20803] - Version mismatch between spotless-maven-plugin and google-java-format plugin
- [FLINK-20841] - Fix compile error due to duplicated generated files
- [FLINK-19013] - Log start/end of state restoration
- [FLINK-19259] - Use classloader release hooks with Kinesis producer to avoid metaspace leak
- [FLINK-19832] - Improve handling of immediately failed physical slot in SlotSharingExecutionSlotAllocator
- [FLINK-20055] - Datadog API Key exposed in Flink JobManager logs
- [FLINK-20168] - Translate page 'Flink Architecture' into Chinese
- [FLINK-20209] - Add missing checkpoint configuration to Flink UI
- [FLINK-20298] - Replace usage of in flink documentation
- [FLINK-20468] - Enable leadership control in MiniCluster to test JM failover
- [FLINK-20510] - Enable log4j2 monitor interval by default
- [FLINK-20519] - Extend HBase notice with transitively bundled dependencies
- [FLINK-20570] - The `NOTE` tip style is different from the others in process_function page.
- [FLINK-20588] - Add docker-compose as appendix to Mesos documentation
- [FLINK-20629] - [Kinesis][EFO] Migrate from DescribeStream to DescribeStreamSummary
- [FLINK-20647] - Use yield to generate output datas in ProcessFunction for Python DataStream
- [FLINK-20650] - Mark "native-k8s" as deprecated in docker-entrypoint.sh
- [FLINK-20651] - Use Spotless/google-java-format for code formatting/enforcement
- [FLINK-20682] - Add configuration options related to hadoop
- [FLINK-20697] - Correct the Type of "lookup.cache.ttl" in jdbc.md/jdbc.zh.md
- [FLINK-20790] - Generated classes should not be put under src/ directory
- [FLINK-20792] - Allow shorthand invocation of spotless
- [FLINK-20805] - Blink runtime classes partially ignored by spotless
- [FLINK-20822] - Don't check whether a function is generic in hive catalog
- [FLINK-20866] - Add how to list jobs in Yarn deployment documentation when HA enabled