Switch from pandas to Polars to Speed Up ETL

Most ETL processes still rely on pandas, which works well for small to medium-sized datasets. But once your data reaches hundreds of megabytes or more, pandas can start slowing things down. Polars is a faster alternative that uses multiple CPU threads to process data more efficiently.

Pandas is single-threaded and optimized for flexibility, not speed. It doesn't take full advantage of modern multi-core CPUs, so even simple operations like groupby or joins can become expensive at scale.

Polars is designed for speed. Built in Rust, it uses parallelism and a columnar memory model. It's great for filtering, grouping, and transforming large datasets without needing to rewrite all your code.

By default, Polars uses all available CPU cores. If you're running in a shared environment or want to limit resource usage, you can set the number of threads:

import os
os.environ["POLARS_MAX_THREADS"] = "4"

import polars as pl
print(pl.thread_pool_size())

Using pandas:

import pandas as pd

df = pd.read_csv("large_dataset.csv")
df["amount"] = df["amount"].fillna(0)
grouped = df.groupby("category")["amount"].sum()
print(grouped)

Using Polars:

import polars as pl

df = pl.read_csv("large_dataset.csv")
df = df.with_columns([
    pl.col("amount").fill_null(0)
])
grouped = df.groupby("category").agg([
    pl.col("amount").sum()
])
print(grouped)

If your ETL jobs are handling large files and you're starting to hit performance limits with pandas, try Polars. It works well out of the box, and you can often reuse your logic with minimal changes.

You don't always need a GPU to speed things up. Simply switching from pandas to Polars can help you handle larger data, faster. It's an easy win for teams dealing with increasingly heavy ETL workloads.

Polars Team. (2024). Performance Tips. Polars User Guide. Retrieved from https://pola-rs.github.io/polars-book/user-guide/performance/