DuckDB Is Fast Because It Skips the Part Everyone Forgets

The first time DuckDB embarrassed me, I had a 6 GB Parquet file and a deadline. The “real” pipeline involved uploading it somewhere, waiting for a warehouse to wake up, and writing a CREATE TABLE I’d delete an hour later. Instead I typed SELECT * FROM 'orders.parquet' on my laptop. The answer came back before I finished reaching for coffee.

I assumed it was a trick. It wasn’t. It was the absence of a trick.

Greybeam’s Kyle Cheung wrote a three-part deep dive into DuckDB internals, and Part 1 is the one that hit the HN front page. It follows a single query from the moment it enters the engine to the moment it’s ready to run. What makes the post good is that it keeps answering the same blunt question at every stage: why is this fast?

The story in one sentence

DuckDB is an in-process analytical SQL database - a single binary under 20 MB, no server, no port, no daemon - that you load like a library and point at any pile of Parquet, CSV, or JSON files as if they were already a database.

That’s the whole shape of it. pip install duckdb. Then SQL over files. No connection string.

The part everyone forgets

Latency comparison: In-process memory execution vs. network protocol serialization:

Here’s the design choice the post builds everything around, and it’s not in the query engine at all. It’s the network.

Most analytical databases are servers. Snowflake, BigQuery, Redshift, Postgres. You open a connection, send SQL over a socket, and the results come back across the wire. Along the way every value in the result gets serialized into a byte format, pushed through TCP, and parsed back on the other side.

In 2017 Mark Raasveldt and Hannes Mühleisen - the people who’d go on to build DuckDB - published a paper with the best title in database research: Don’t Hold My Data Hostage. They measured what actually happens when you pull a result set out of a warehouse. The client protocol itself, ODBC and JDBC, was often the slowest single step in the whole query. Sometimes it took longer than computing the answer did.

Two reasons. A gigabit link caps out around 125 MB/s, so a big result can take longer to transmit than to compute. And ODBC hands back data one value at a time - on a 100-million-row result that’s hundreds of millions of function calls, each doing its own little memory copy and type check.

DuckDB sidesteps both by living in the same process as your code. There is nothing to serialize. When you query a pandas dataframe, a “replacement scan” lets DuckDB read the buffers your Python process already owns. If NumPy says “here are a million int64 values,” DuckDB often reads that exact buffer. Zero copy.

From SQL to a plan, in about a millisecond

Once your SQL is inside the engine, it goes through the usual march: parse, bind, plan, optimize. DuckDB forks the Postgres parser, which is why the dialect feels so familiar.

The optimizer is where the post got me to actually open a terminal. DuckDB exposes its optimizer as a list of small, named passes you can inspect and turn off one by one:

SELECT * FROM duckdb_optimizers();
-- 33 rows: filter_pushdown, join_order, row_group_pruner,
-- late_materialization, common_subexpressions, ...

You can run SET disabled_optimizers = 'filter_pullup, join_order' and watch what a pass was doing by its absence. The whole optimization phase usually finishes in about a millisecond.

The heaviest one is join order. A query joining six tables has 30,240 possible tree shapes, and the gap between the best and worst can be orders of magnitude. DuckDB models the query as a graph and uses dynamic programming (DPhyp, DPccp) to avoid re-exploring orderings it already solved. The same kind of trick that makes Fibonacci fast, applied to deciding which tables meet first.

The other half: never read what you don’t need

DuckDB’s native file is a single .duckdb, inspired by SQLite. Inside, data is columnar, broken into 256 KB blocks, each carrying a checksum so a flipped bit on a laptop SSD becomes an error instead of a wrong answer.

The detail that matters for speed is the zone map. Each row group (up to 122,880 rows) stores the min and max of every column plus a null count. When you run WHERE event_date > '2026-01-01', DuckDB checks each row group’s max first and skips the whole group if it can’t possibly match. It never reads the data.

This isn’t exotic. It’s the same idea the expensive warehouses use under prettier names:

And when you query Parquet directly, DuckDB doesn’t even need its own format. Parquet already stores per-row-group min/max stats, so DuckDB reads the footer, decides which row groups can satisfy the predicate, and fetches only those column chunks. For a remote file it issues an HTTP request for just the footer, then range-requests the bytes it needs. A good WHERE clause becomes a way to download less of the file.

What HN is actually arguing about

The thread is mostly love, which is its own data point - DuckDB rarely draws the usual HN skepticism. steve_adams_86 summed up the consensus: he adopted it because it was easy, stayed because it turned out to be “absurdly capable,” and it “still impresses me regularly.” anitil pointed at the thing that makes it special: most projects are good at small problems or large ones. DuckDB is good at both.

0xferruccio had the most 2026 use case in the thread: piping every engineer’s Claude Code session logs from S3 into DuckDB to find gaps in developer experience. The unglamorous superglue work.

But the sharpest comment was the dissent. willtemperley pointed out that “just a library” has a hard edge: if you can’t use dynamic linking - say, in an App Store context - DuckDB is a rough fit, because statically linking its extensions is genuinely hard. “DuckDB is excellent, but it’s more a black box than a library.” For that world he reaches for Arrow C++, which builds portably. The same in-process design that makes DuckDB fast also makes it something you embed on its terms, not yours.

Should you read the original?

This is Part 1 of three, and it stops right before execution: vectorized processing and morsel-driven parallelism are being held for later. Which is a slightly evil cliffhanger for a blog post about a database.

But the lesson of Part 1 lands on its own. We spend enormous effort making query engines clever - better join orders, tighter optimizers, smarter pruning. DuckDB does all of that too. Its biggest speedup, though, came from noticing that the fastest network request is the one you never send.

Related Database & System Architectures

If you are exploring database performance, durable workflows, and minimal network overhead, check out these companion architectures:

• PG Durable: In-Database Execution: Turn PostgreSQL into a durable workflow engine without running a separate Temporal or Redis daemon.
• ZeroStack: 8MB Native Rust Coding Agent: Cutting runtime bloat at the process level — single binary execution with zero VM baggage.
• Context Hub: Curated Docs for Coding Agents: High-density context ingestion without bloating model tokens or running slow network scrapes.
• Pi Mono Explained: Autonomous Coding Agent Architecture: The architectural blueprint of composable agent runtimes and extensible tooling.

---

Discussion on Hacker News · Source: greybeam.ai · Submitted by marklit