🔍Pinterest Introduces Event Time Finalization for CDC Ingestion
Pinterest's new CDC finalization mechanism
TL;DR
Pinterest has introduced an event time-based partition finalization mechanism for its CDC ingestion framework, addressing challenges in continuous data capture and ensuring data completeness without batch-based approaches. This impacts petabyte-scale data and thousands of pipelines.
Pinterest has introduced an event time-based partition finalization mechanism for its next-gen CDC ingestion framework, addressing challenges in continuous data capture. This mechanism uses Apache Flink and Iceberg metadata to ensure data completeness, unlike previous batch-based approaches. It's crucial for teams managing petabyte-scale data and thousands of pipelines, ensuring online changes are typically available within 15 minutes to an hour. The new mechanism runs inside the Flink-to-Iceberg sink, collecting event time statistics and calculating a partition finalization watermark from recent checkpoints.
Key Points
Pinterest's new mechanism uses Apache Flink to measure event time progress and Apache Iceberg metadata to publish a non-regressing finalization watermark.
The approach addresses challenges in continuous CDC ingestion where data can arrive after a partition's wall clock interval has ended.
The platform supports petabyte-scale data and thousands of pipelines, with online changes typically available within 15 minutes to an hour.
The watermark is stored as an Iceberg table property, allowing downstream sensors to read it without a separate coordination service.
Pinterest uses Flink watermarks and Iceberg snapshots to derive a table-specific completeness signal and store it alongside the data.
Why It Matters
If you're managing petabyte-scale data and thousands of pipelines, Pinterest's new event time-based partition finalization mechanism ensures data completeness without relying on batch-based approaches. This impacts teams using CDC, Kafka, Flink, Spark, and Iceberg to move database changes from MySQL, TiDB, and KVStore into analytical tables.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,567 builders reading daily.