Analyzing Data 180,000x Faster with Rust
Analyzing Data 180,000x Faster with Rust
How to hash, index, profile, multi-thread, and SIMD your way to incredible speeds.
Their actual speedup for switching languages was 8x. The rest was all about using better data structures and parallelism.
Would be interesting to see how fast polars (a dataframe library written in rust) would be as it can be used in python.
Referring to lazy Python as "baseline" is a joke.
But isn't it kind of obvious that if you are able to do 180k times improvement, then the baseline is probably not very impressive to begin with. Still, that doesn't take away that the optimizations were impressive, and that it was interesting to read about it.
Yeah, this one really had me scratching my head:
✓Note: there are lots of ways we could make the Python code faster, but the point of this post isn’t to compare highly-optimized Python to highly-optimized Rust. The point is to compare “standard-Jupyter-notebook” Python to highly-optimized Rust.
With rust is the joke as if you couldn't do it otherwise. Maybe c would be only 179,999x faster, or FORTRAN 180,001x, (numbers made up). Python could probably be made 60,000x faster as well.
Yet it's a fine baseline. The actual speedup for switching to rust was 8x, the rest was all about changing data structures, using SIMD, parallelism and batching.
I think it's a great baseline. Within academic context, Python (and perhaps Matlab) are extremely common for data analysis. I doubt many would transition code to other languages unless strictly needed such as the case in the article. Showing how to "simply" speed up code like the article does is a great way to snag speed even if you don't analyze timing, and just replicate steps from this article.
Having done stuff myself as part of research, and having people I know go from developer jobs to research jobs, I can safely say scientists generally do not make good code. Regardless of language. An article like this gives good steps to take from start to end, and would be a valuable tool in a possible transition to better code.
Move infinitly faster with teleportation!!!
Walking baseline