Support exactly-once semantics when connecting Kafka to ClickHouse
As a developer using Kafka to stream messages to ClickHouse,
I want to be sure that messages as delivered and ingested exactly once, as opposed to at-least-once, to ClickHouse to not have to deal with duplicated data that I have to deduplicate down the line. This removes both a cognitive and operational load from my data pipeline
c.f. https://github.com/ClickHouse/clickhouse-kafka-connect
The exactly-once semantics has been implemented in the newer version of the ClickHouse Kafka Connect Sink (https://github.com/ClickHouse/clickhouse-kafka-connect).
5
votes
AdminArthur Ansquer
(Senior Product Manager, Aiven.io)
shared this idea