🔍Apache Hudi Adds Time-In-Queue Metric for Data Freshness
Measuring Data Freshness Without Pipeline Changes
TL;DR
Apache Hudi now calculates the time-in-queue metric to measure data freshness in data lakes, without altering existing pipelines. This metric helps define and enforce custom freshness SLAs, ensuring data remains up-to-date.
Apache Hudi now calculates a time-in-queue metric to measure data freshness in data lakes, without altering existing pipelines. This metric helps teams define and enforce custom freshness SLAs, ensuring data remains up-to-date. The metric is computed by finding the latest Hudi commit that contains a checkpoint and measuring the timestamp delta between that message and the current time. This approach ensures data freshness without requiring changes to producers, consumers, or existing pipeline infrastructure. The metrics reporter runs every 15 minutes in production, fetching the latest Hudi commit and calculating the lag.
Key Points
Time-in-queue metric measures data freshness in data lakes, computed every 15 minutes.
Metric is calculated by finding the latest Hudi commit with a checkpoint and measuring the timestamp delta.
No changes required to producers, consumers, or existing pipeline infrastructure.
Custom freshness SLAs can be defined and enforced based on the time-in-queue metric.
Metrics reporter suppresses the metric entirely if no valid checkpoint is found within the search depth.
Why It Matters
If you're managing data pipelines in a data lake, Apache Hudi's new time-in-queue metric can help you define and enforce custom freshness SLAs. For example, if you're running Hudi pipelines with Kafka, the time-in-queue metric provides a complete picture of pipeline health, ensuring data freshness without altering existing infrastructure. This is crucial for teams that need to maintain real-time data freshness.
Frequently Asked Questions
Why does this matter?
If you're managing data pipelines in a data lake, Apache Hudi's new time-in-queue metric can help you define and enforce custom freshness SLAs. For example, if you're running Hudi pipelines with Kafka, the time-in-queue metric provides a complete picture of pipeline health, ensuring data freshness without altering existing infrastructure. This is crucial for teams that need to maintain real-time data freshness.
What happened?
Apache Hudi now calculates the time-in-queue metric to measure data freshness in data lakes, without altering existing pipelines. This metric helps define and enforce custom freshness SLAs, ensuring data remains up-to-date.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,316 builders reading daily.