(opens in a new tab)

DuckDB vs Polars vs Pandas: Which to Use in 2026

August 31, 2026 by Chat2DB Team

Three tools now compete for the same slot in a Python data workflow. pandas has fifteen years of ecosystem gravity. Polars offers a fast, expressive DataFrame API in Rust. DuckDB offers SQL over an embedded columnar engine. They overlap enough that people argue about them, and differ enough that the argument has a real answer depending on what you are doing.

This is a practical comparison: how each one executes, where each breaks down, and — the part that resolves most of the debate — how cheaply you can move between them.

The short version

pandas is an eager, single-threaded, NumPy/Arrow-backed DataFrame library. Every operation materialises immediately. It has the largest ecosystem by far and the most StackOverflow answers, and it falls over on datasets approaching your RAM.

Polars is a multi-threaded, Arrow-native DataFrame library in Rust with a lazy API and a query optimiser. It is typically several times faster than pandas on the same machine and handles larger-than-memory data through streaming.

DuckDB is an embedded columnar SQL database. You write SQL instead of method chains, it is multi-threaded and vectorised, it spills to disk transparently, and it reads Parquet, CSV and JSON — local or on S3 — without loading them first.

Execution models, and why they matter

Eager vs lazy

pandas evaluates line by line. This filter-then-select does the full filter first, materialising every column of the matching rows, then throws most of them away:

import pandas as pd
 
df = pd.read_csv("events.csv")
result = df[df["amount"] > 100][["user_id", "amount"]]

Polars in lazy mode sees the whole plan before running anything, so it pushes the projection into the scan and never reads the unused columns off disk:

import polars as pl
 
result = (
    pl.scan_csv("events.csv")
      .filter(pl.col("amount") > 100)
      .select(["user_id", "amount"])
      .collect()
)

DuckDB does the same optimisation, because it is a database and that is what query planners do:

import duckdb
 
result = duckdb.sql("""
    SELECT user_id, amount
    FROM   'events.csv'
    WHERE  amount > 100
""").df()

Note what DuckDB did not require: no read step. It queried the file directly, inferring the schema, reading only the two columns it needed.

Parallelism

pandas uses one core for almost everything. Polars and DuckDB both parallelise scans, filters, joins and aggregations across all available cores by default. On an 8-core laptop that difference alone accounts for much of the gap people observe.

Memory

pandas needs the whole frame in RAM, plus headroom for intermediates — a groupby on a 6 GB frame can peak well above 12 GB. Polars streams in lazy mode. DuckDB spills operators to disk when they exceed the memory limit, so a join across data several times the size of RAM completes rather than raising MemoryError.

You can cap DuckDB's memory explicitly:

duckdb.sql("SET memory_limit = '4GB'")
duckdb.sql("SET threads = 4")

The same task, three ways

Load events, filter to a date range, aggregate by two keys, sort.

pandas:

import pandas as pd
 
df = pd.read_parquet("events.parquet")
df["created_at"] = pd.to_datetime(df["created_at"])
 
out = (
    df[df["created_at"] >= "2026-01-01"]
      .groupby(["event_type", df["created_at"].dt.to_period("M")])
      .agg(n=("id", "count"), revenue=("amount", "sum"))
      .reset_index()
      .sort_values("revenue", ascending=False)
)

Polars:

import polars as pl
 
out = (
    pl.scan_parquet("events.parquet")
      .filter(pl.col("created_at") >= pl.date(2026, 1, 1))
      .group_by(["event_type", pl.col("created_at").dt.truncate("1mo").alias("month")])
      .agg(
          pl.len().alias("n"),
          pl.col("amount").sum().alias("revenue"),
      )
      .sort("revenue", descending=True)
      .collect()
)

DuckDB:

import duckdb
 
out = duckdb.sql("""
    SELECT event_type,
           date_trunc('month', created_at) AS month,
           count(*)    AS n,
           sum(amount) AS revenue
    FROM   'events.parquet'
    WHERE  created_at >= DATE '2026-01-01'
    GROUP  BY 1, 2
    ORDER  BY revenue DESC
""").df()

The DuckDB version is the shortest and the one a SQL-fluent colleague can review without learning an API. The Polars version is the one that composes best inside a larger Python program. The pandas version is the one that runs everywhere without a new dependency.

Where each one clearly wins

pandas wins on ecosystem

If your next step is scikit-learn, statsmodels, matplotlib, seaborn, or any of a thousand libraries that accept a DataFrame, pandas is the lingua franca. For data that fits comfortably in memory — say under a couple of gigabytes — the performance difference rarely justifies rewriting working code.

Polars wins on expressive transformation

Complex row-wise logic, window functions over groups, conditional expressions and struct/list columns are more natural in Polars' expression API than in SQL:

out = df.with_columns(
    pl.when(pl.col("amount") > 500).then(pl.lit("high"))
      .when(pl.col("amount") > 100).then(pl.lit("mid"))
      .otherwise(pl.lit("low"))
      .alias("tier"),
    pl.col("amount").rank("dense").over("user_id").alias("rank_in_user"),
)

It is also the better fit when the transformation is one step inside a Python pipeline with types and tests around it.

DuckDB wins on data that lives in files or databases

This is the decisive case. DuckDB queries Parquet, CSV, JSON, S3 objects, and live PostgreSQL, MySQL or SQLite databases from the same SQL:

duckdb.sql("INSTALL httpfs; LOAD httpfs;")
duckdb.sql("""
    CREATE SECRET (TYPE s3, KEY_ID 'AKIA...', SECRET 'xxx', REGION 'us-east-1')
""")
 
duckdb.sql("""
    SELECT c.segment, count(*) AS sessions
    FROM   read_parquet('s3://logs/2026/*.parquet') s
    JOIN   'customers.csv' c ON c.id = s.customer_id
    GROUP  BY 1
""").df()

Joining an S3 Parquet glob to a local CSV in one statement is not something the DataFrame libraries do without a load step per source. And because DuckDB reads Parquet row-group statistics, a filtered query over a partitioned dataset skips most files entirely.

For querying operational databases interactively rather than from a script, a dedicated client is usually a better fit than any of these three. Chat2DB (opens in a new tab) connects to DuckDB, PostgreSQL, MySQL and more with AI-assisted SQL generation, which pairs well with a notebook workflow — explore and shape the query in the client, then paste the finished SQL into your Python.

Moving between them is nearly free

All three speak Apache Arrow, so conversions are typically zero-copy. This is why "which one" is rarely an either/or:

import duckdb, polars as pl, pandas as pd
 
# DuckDB -> pandas / Polars / Arrow
pdf = duckdb.sql("SELECT * FROM 'events.parquet'").df()
pldf = duckdb.sql("SELECT * FROM 'events.parquet'").pl()
arrow = duckdb.sql("SELECT * FROM 'events.parquet'").arrow()
 
# Query an existing DataFrame with SQL — DuckDB finds it by variable name
result = duckdb.sql("SELECT event_type, count(*) FROM pldf GROUP BY 1").pl()
 
# Polars <-> pandas
pldf2 = pl.from_pandas(pdf)
pdf2 = pldf.to_pandas()

That third block is worth pausing on: DuckDB can run SQL directly against a pandas or Polars DataFrame sitting in your session, with no registration step and no copy. A perfectly reasonable workflow is Polars for transformation, DuckDB when a query is easier to express as a join and a GROUP BY, pandas at the end for the plotting library.

Comparison table

How to choose

Ask three questions.

Does the data fit comfortably in memory, and is this a small script? Use pandas. The ecosystem advantage is real and the performance gap will not matter.

Is this a transformation step inside a larger Python program, with complex per-row or per-group logic? Use Polars. The expression API is more maintainable than string SQL when it lives inside typed, tested code.

Does the data live in files, object storage, or a database — or is it bigger than RAM? Use DuckDB. Querying in place with a planner that pushes predicates down beats loading everything and filtering after, and SQL is the more reviewable artifact for analytical logic.

And when the answer is mixed, remember the Arrow bridge. Choosing one does not lock out the others.

Wrapping up

pandas is the default that everything integrates with. Polars is the fast, modern DataFrame library for transformation-heavy Python. DuckDB is the analytical engine that meets your data where it already lives and speaks the language most analytical logic is already written in. In 2026 the productive answer for most teams is not to pick a winner but to use DuckDB as the ingestion and aggregation layer, hand results to Polars or pandas for the last mile, and let Arrow make the handoff cost nothing.