📊Polars 2.0 RC1 Released With 5x Faster Streaming Engine
Polars 2.0 RC1: 5x Faster Streaming Engine, Stricter Defaults
TL;DR
Polars 2.0 RC1 is out, featuring a 5x faster streaming engine and stricter defaults. This release aims to improve performance and catch inconsistencies early, impacting workflows relying on lazy queries and data integrity.
Polars 2.0 RC1 is out, with a 5x faster streaming engine as the main highlight. This release aims to improve performance and catch inconsistencies early, impacting workflows relying on lazy queries and data integrity. The streaming engine will be the default for all LazyFrame queries, leading to massive memory and performance improvements on most queries for users. However, users can opt to maintain the 'in-memory' engine by setting the engine affinity. The release also introduces stricter behavior, such as raising errors for data mismatches and requiring explicit type casting, ensuring inconsistencies are caught instead of silently produced. This is a big deal for anyone using Polars in production, as it promises better defaults and a more robust API.
Key Points
Polars 2.0 RC1 is now available via pip install polars==2.0rc1.
All LazyFrame queries will run on the streaming engine by default, expected to be 5x faster.
Users can opt to maintain the 'in-memory' engine by setting the engine affinity.
The release introduces stricter behavior, such as raising errors for data mismatches.
New typed exceptions, AttributeRemovedError and ArgumentRemovedError, have been added to handle removed attributes and methods.
Why It Matters
If you're using Polars for data processing in production, the 5x faster streaming engine and stricter defaults will significantly improve performance and catch inconsistencies early. However, the removal of implicit behavior and the need for explicit type casting might require adjustments to existing workflows. Anyone relying on lazy queries and data integrity should pay attention to these changes.
Frequently Asked Questions
Why does this matter?
If you're using Polars for data processing in production, the 5x faster streaming engine and stricter defaults will significantly improve performance and catch inconsistencies early. However, the removal of implicit behavior and the need for explicit type casting might require adjustments to existing workflows. Anyone relying on lazy queries and data integrity should pay attention to these changes.
What happened?
Polars 2.0 RC1 is out, featuring a 5x faster streaming engine and stricter defaults. This release aims to improve performance and catch inconsistencies early, impacting workflows relying on lazy queries and data integrity.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,460 builders reading daily.