Linux, macOS, FreeBSD

Columnar storage is an on-disk data layout where each column of a table is stored contiguously, instead of each row. The layout reduces I/O for queries that read a subset of columns from a wide table, and enables compression ratios of 5–10× by grouping values of the same type adjacent on disk. ClickHouse, Snowflake, BigQuery, and DuckDB all use it as their default storage model.

## How columnar storage works

A row-oriented database writes each tuple as a contiguous record on a page — every column of row 1, then every column of row 2. A columnar layout inverts this: every value of column A is stored together, every value of column B is stored together. A query that touches three columns of a 50-column table reads ~6% of the bytes a row store would read for the same query.

Adjacent values in a column share a type and often share patterns — sorted timestamps, repeated categoricals, monotonic IDs — so encodings like dictionary, run-length, delta, and Gorilla compress them far better than the mixed-type tuples in a row store. A general-purpose codec (LZ4, ZSTD) sits on top of the encoded column. See [how columnar compression works](how-columnar-compression-works.md) for the encoding stack in detail.

Most engines store per-block metadata — min, max, count, and sometimes Bloom filters — so the planner can skip whole blocks during predicate evaluation without decompressing them. This is "predicate pushdown into the storage layer" and is central to how columnar engines stay fast on selective scans.

| Layout | Example engines |
|---|---|
| Row | Postgres heap, MySQL InnoDB |
| Column | Snowflake, Parquet, ORC, ClickHouse, DuckDB, IBM DB2 BLU |

## Inside columnar storage formats

PAX (Partition Attributes Across) was introduced by Ailamaki et al. in the 2001 VLDB paper *[Weaving Relations for Cache Performance](https://www.vldb.org/conf/2001/P169.pdf)*. It groups rows into pages like a row store, then arranges columns within each page contiguously, preserving cache locality for full-row reads while keeping the I/O wins for column-subset scans. This inspired many of today's columnar formats.

Most modern columnar formats — Snowflake micro-partitions, Parquet, ORC, DB2 BLU — are PAX-inspired and use **row groups**. A file or partition is split into bounded chunks of rows (anywhere from a few thousand rows to hundreds of MB), and within each chunk, columns are stored contiguously. The row group is a first-class boundary: it has its own metadata, its own min/max stats, and compression and encoding reset at every row group boundary. Critically, these boundaries are fixed at write time — once a Parquet file is on disk, the row group sizing is baked in. Your only recourse if it turns out wrong is to rewrite the file.

ClickHouse `MergeTree` doesn't do this within a part. Inside a Wide-format part, each column is a single continuous file — `colA.bin` contains every value of colA for the entire part, end to end. Compression blocks span the whole column. Granules (8192 rows by default) exist as index marks, not as container boundaries.

That said, if you zoom out to the table level, parts themselves look a bit like row groups. Each `INSERT` creates a new part, every part holds a bounded set of rows, and a column across the whole table lives in many separate files (`part1/colA.bin`, `part2/colA.bin`, …). But unlike Parquet row groups, parts aren't a fixed write-time decision — they're a continuous operational concern. Background merges combine small parts into larger ones over time, so the on-disk layout you have today isn't the layout you'll have next week. As parts grow, column streams get longer and compression improves. Sizing thresholds (`min_bytes_for_wide_part`, `max_bytes_to_merge_at_max_space_in_pool`) are tunable, and partitioning bounds part size implicitly above that.

## When to use columnar storage

Columnar storage wins when:

- Queries scan many rows but read few columns (the typical analytical pattern).
- Aggregations, filters, and group-bys dominate the workload.
- Data is append-mostly — bulk inserts followed by reads, with few per-row updates.
- Storage cost matters: 5–10× compression typical, up to 30× on low-cardinality columns.

Columnar storage loses when:

- Queries are point lookups by primary key returning whole rows.
- The dominant write pattern is single-row updates.
- Concurrency is in the thousands of small transactional queries per second per node.

For the head-to-head against row stores, see [row-oriented vs column-oriented databases](row-vs-column-database.md).

## How columnar storage differs from a columnar database

Columnar storage is a file layout. A columnar database is a complete system that uses the layout. The database adds:

- A query planner aware of the column-major layout.
- A vectorised execution engine that processes batches of column values per call.
- An indexing strategy (sparse primary indexes, skip indexes, Bloom filters).
- A write path that handles the merge and compaction overhead columnar layouts introduce.

Parquet is a columnar storage format. ClickHouse and DuckDB are columnar databases. A columnar database like ClickHouse reads a columnar storage format like MergeTree and Parquet. See [columnar storage formats](columnar-storage-formats.md) for the format-vs-engine distinction in detail, and the [columnar database overview](what-is-columnar-database.md) for the wider primer.

Frequently asked questions

What is columnar storage?

Columnar databases beat row stores by 10× to 1000× on analytical queries because two principles compound. The first is **efficient execution**: storage layout and vectorised CPU loops do each unit of work as fast as possible. The second is **smart pruning**: data skipping and late materialisation cut the amount of work the query has to do at all. ClickHouse, DuckDB, and Snowflake all stack both.

## TL;DR

- Columnar storage reads only the columns a query references, often cutting raw I/O by 90%+ on wide tables.
- [Vectorised query execution](/resources/engineering/vectorised-query-execution) runs operators on batches of values, amortising interpreter overhead and unlocking SIMD.
- Data skipping (zone maps, Bloom filters, sparse indexes) prunes whole blocks before the engine ever decompresses them.
- Late materialisation defers row assembly until after filters run, cutting downstream column work by 10-1000× on selective queries.
- Each mechanism multiplies the others.

## Why are columnar databases faster than row stores for analytics?

Columnar databases are faster because they read fewer bytes per query, process more values per cycle, and skip data before touching it. A row store walks every byte of every scanned row. A column store reads only referenced columns, processes them in dense batches, and uses precomputed metadata to prune blocks before opening them.

Each mechanism contributes significantly to the efficieny of columnar databases - when combined, they multiply. A query that reads 3 of 50 columns saves ~94% of I/O, then processes the remaining bytes 10× faster per cycle, then prunes 90% of those blocks via skipping indexes. Stack the multipliers and modest queries land 100×-1000x ahead of a row store on the same hardware.

The mechanisms split into two categories. **Efficiency** covers how each unit of work runs faster: columnar storage and vectorised execution. **Smart pruning** covers how the engine avoids work entirely: data skipping indexes and late materialisation.

## How does columnar storage make queries faster?

Columnar storage packs same-type values from one column into contiguous blocks on disk. A query that touches 3 of 50 columns reads ~6% of the bytes a row store reads for the same rows. The savings extend through the page cache and CPU cache: a sequential read fills cache lines with bytes the query actually uses, instead of mixed columns the query will throw away.

The layout also enables much higher compression ratios. Adjacent same-type values dictionary-encode, run-length-encode, or delta-encode 5-10× tighter than mixed rows. A low-cardinality column can hit 30×. Read 8% of the columns at 8× compression and you have cut raw bytes by ~99%.

| Layer | Row store | Column store |
|---|---|---|
| Disk read | Every byte of every scanned row | Only referenced columns |
| Cache locality | Tuples scatter across cache lines | One column fills cache lines tightly |
| Compression | 1.5-3× typical | 5-10×, up to 30× on low cardinality |

The full encoding stack (dictionary, RLE, delta, Gorilla, plus a general codec like LZ4 or ZSTD) lives in [how columnar compression works](/resources/engineering/how-columnar-compression-works). The on-disk shape itself is covered in [what is columnar storage](/resources/engineering/what-is-columnar-storage).

## How does vectorised execution make queries faster?

Vectorised execution processes batches of 1024-4096 values per operator call instead of one row at a time. This amortises interpreter dispatch, keeps tight working sets in CPU cache, and lets the compiler emit SIMD instructions that handle 8-16 values per CPU cycle. The MonetDB/X100 paper (Boncz et al., CIDR 2005) showed that swapping single-row iteration for batched columnar loops eliminated most of the dispatch cost that dominates row-at-a-time engines.

The columnar layout is what makes this practical. Values of one column are already contiguous and identically typed, so a tight loop loads 256-bit or 512-bit chunks straight into a register without unpacking. ClickHouse, DuckDB, Snowflake, Databricks Photon, DataFusion, and Velox all ship variants of this model. Implementation details, vector sizes, and SIMD dispatch strategies are covered in [vectorised query execution](/resources/engineering/vectorised-query-execution).

## How do columnar engines avoid unnecessary work?

Two techniques let the engine skip work entirely. **Data skipping indexes** store precomputed metadata per block (min/max, Bloom filters, sparse marks) so the planner can prune blocks before reading them. **Late materialisation** defers row assembly until after filters run, so the engine only fetches non-predicate columns for surviving rows.

| Pruning mechanism | What it stores | Best for |
|---|---|---|
| Min/max (zone map) | Min and max per block | Range filters on sorted or correlated columns |
| Sparse primary index | One mark every N rows | Range filters on the primary key |
| Bloom filter | Probabilistic set membership | Equality on high-cardinality columns |
| Set index | Distinct values per block | Low-cardinality equality filters |

A time-series table sorted by `(tenant_id, timestamp)` will skip 99%+ of blocks for a typical recent-activity query. Late materialisation then takes the surviving row positions and only fetches the SELECT columns for those. On a 1-in-1000 filter, that cuts downstream column work by ~1000×. The two mechanisms are independent: pruning reduces the bytes the engine reads, materialisation reduces the bytes it processes after reading.

## How does ClickHouse implement these techniques?

ClickHouse's [official "why is ClickHouse so fast?" guide](https://clickhouse.com/docs/concepts/why-clickhouse-is-so-fast) and [academic overview](https://clickhouse.com/docs/academic_overview) give the long-form answer. The short version: ClickHouse stacks every mechanism above and tunes each one aggressively.

- **MergeTree storage.** Data is written as immutable parts, each split into granules of ~8192 rows. Each granule has a sparse primary-key mark, so range scans on a sorted column can skip almost every block.
- **PREWHERE.** An explicit late-materialisation hint. ClickHouse reads the predicate column, evaluates the filter, then fetches the SELECT columns only for surviving rows. Modern versions move predicates into PREWHERE automatically when selectivity warrants.
- **Vectorised execution with CPU dispatch.** Hot loops compile multiple times (SSE4.2, AVX2, AVX-512) and the binary picks the variant at startup based on `cpuid`. See the [CPU dispatch blog](https://clickhouse.com/blog/cpu-dispatch-in-clickhouse).
- **Per-column codecs.** Columns can specify their own codec stack (Delta + LZ4 for timestamps, Gorilla for floats), trading CPU for size on the data shape that suits each column.
- **Skip indexes.** Beyond the sparse primary index, secondary skip indexes (min/max, set, Bloom filter, n-gram) prune blocks for non-primary columns.

<iframe width="560" height="315" src="https://www.youtube.com/embed/7QXKBKDOkJE" title="ClickHouse internals" frameborder="0" allowfullscreen></iframe>

## When are columnar databases slower than row stores?

The same layout that wins on wide aggregations is what makes single-row work expensive: every column file must be opened to reconstruct one row. Columnar engines lose to row stores on point lookups, per-row mutations, and high-concurrency transactional writes.

For mixed workloads, a hybrid pattern (Postgres for OLTP plus ClickHouse for analytics) is usually the right shape rather than forcing one layout to do both jobs. Cross-engine numbers, including the cases where row stores outperform column stores, live in the [row vs column benchmark](/resources/engineering/row-vs-column-benchmark).

Why columnar databases are fast

Columnar storage formats define how column-oriented data is laid out, separately from the engines that read it. Apache Parquet (released July 2013) is the cloud-lake default. Apache ORC (released with Hive 0.12 in October 2013) is Hive-native with deeper per-stripe statistics. Apache Arrow (released February 2016) is an in-memory layout for zero-copy interchange. They are file or memory layouts, not databases.

## TL;DR

- Apache Parquet (July 2013) and Apache ORC (October 2013) are on-disk columnar file formats. Apache Arrow (February 2016) is an in-memory layout for cross-process exchange.
- Parquet groups data into row groups of 64–512 MB, each holding one column chunk per column, with per-row-group statistics in the file footer for predicate pushdown.
- ORC stores data in stripes typically around 200 MB, with statistics at file, stripe, and row-group-within-stripe levels (three layers of skip metadata).
- Arrow defines RecordBatches with columnar buffers laid out for SIMD. Spark, DuckDB, Polars, and pandas use Arrow at the wire level, but rarely as long-term storage.
- Native engine formats (ClickHouse MergeTree parts, Snowflake micro-partitions, DuckDB's native database file) typically read fewer bytes than Parquet on the same data because the engine controls sort order, sparse indexes, and skip indexes.
- Iceberg, Delta Lake, and Apache Hudi are *table* formats on top of Parquet (or ORC/Avro). They add transactions and schema evolution. The data is still Parquet on disk.

## What are columnar storage formats?

Columnar storage formats specify how column-oriented data is organised in files or memory buffers. Each column's values are stored contiguously, with metadata describing types, compression, and statistics. The format is read by a query engine. Parquet, ORC, and Arrow are formats; ClickHouse, DuckDB, and Snowflake are engines that consume them.

The distinction matters because formats and engines evolve on separate cadences. A Parquet file can be read by Spark, Trino, Snowflake, DuckDB, Athena, BigQuery, and ClickHouse. The engine adds query planning, indexes beyond per-block statistics, and a query runtime. The format defines only the bytes on disk and how to decode them. The same separation is described in the [Apache Arrow columnar format spec](https://arrow.apache.org/docs/format/Columnar.html), which positions Arrow as the in-memory counterpart to on-disk formats like Parquet.

Two layout decisions define every columnar format: how to chunk a table horizontally so a reader can parallelise and skip work, and how to encode each column's values for size and decoding speed. Parquet, ORC, and Arrow make different choices on both axes, which is why they coexist instead of converging. For a primer on the underlying layout, see [what is columnar storage](/resources/engineering/what-is-columnar-storage). For the broader category, see the [columnar database overview](/resources/engineering/what-is-columnar-database).

## What is Apache Parquet?

Apache Parquet is an on-disk columnar file format released in July 2013 by Twitter and Cloudera, inspired by Google's Dremel paper. A Parquet file is split into row groups (typically 64–512 MB), each row group holds one column chunk per column, and each column chunk is split into pages. Per-row-group statistics (min, max, null counts) sit in the file footer and drive predicate pushdown.

Parquet 2.x added page indexes, Bloom filters, and column indexes that store per-page statistics, narrowing what was previously ORC's clearest advantage. The default compression codec is Snappy, with ZSTD as the common alternative for archival workloads, and LZ4, GZIP, and Brotli also supported. Encodings include dictionary, RLE/bit-packing, and delta. The format is encoding-rich enough that [compression ratios commonly land in the 5–10× band, with column-specific cases beyond 30×](/resources/engineering/database-compression).

Parquet's reach is the reason it became the cloud-lake default. Spark, Trino, Presto, Hive, ClickHouse, DuckDB, Polars, BigQuery, Snowflake, Athena, and Redshift all read Parquet directly. ClickHouse's [Parquet integration](https://clickhouse.com/blog/clickhouse-and-parquet-a-foundation-for-fast-lakehouse-analytics) reads and writes partitioned Parquet datasets directly from S3-compatible object storage.

## What is Apache ORC?

Apache ORC (Optimized Row Columnar) is an on-disk format released with Apache Hive 0.12 in October 2013, designed to replace Hive's earlier RCFile. An ORC file is split into stripes, each stripe contains the row data plus a stripe footer with column-level statistics, and the file footer holds aggregated statistics across stripes. The [ORC v1 specification](https://orc.apache.org/specification/ORCv1/) notes that stripes are "typically ~200MB". The Hive default `orc.stripe.size` is 64 MB, but production configurations commonly tune this higher.

ORC stores statistics at three levels: file, stripe, and row group within a stripe (default 10,000 rows per row-group index entry per the spec). That depth is the basis of its predicate-pushdown advantage. Default compression is Zlib, with Snappy and ZSTD as alternatives. The format supports ACID transactions via delta files: updates land in delta files alongside the base files, and a compactor merges them periodically. This is what makes ORC the natural format for Hive ACID tables.

ORC's ecosystem is narrower than Parquet's. Inside the Hive/HDFS world it is the dominant format. Outside, support is thinner. Spark, Trino, and Presto read ORC well, while ClickHouse, DuckDB, BigQuery, and Snowflake have varying levels of support. The common pattern is Hive-on-ORC for legacy batch workloads, Parquet everywhere else.

## What is Apache Arrow?

Apache Arrow is an in-memory columnar format released in February 2016, designed for zero-copy data exchange between processes. The Arrow columnar format spec defines RecordBatches as ordered collections of arrays. Each field is a contiguous values buffer plus a validity bitmap, with no per-row overhead and a layout aligned for SIMD instructions.

The critical distinction from Parquet and ORC is that Arrow is in-memory, not on-disk. The Arrow IPC format and Feather V2 do exist as on-disk wrappers, but those are uncommon for production storage. Arrow's primary job is to live in RAM and move between processes. When DuckDB hands a result back to pandas, when Spark passes data to a Python UDF, when Polars exchanges with a JVM client, the layout is Arrow. Wes McKinney's [response to Daniel Abadi's columnar comparison](https://wesmckinney.com/blog/arrow-columnar-abadi/) frames Arrow as complementary to Parquet rather than competing. The on-disk-vs-in-memory split is the design.

Arrow's reach is at the wire level: Spark, DuckDB, Polars, pandas (via PyArrow), Velox, and Apache Datafusion all use Arrow buffers for inter-process or client-server transfer. Arrow is not a competing alternative to Parquet.

## How do row groups, stripes, and record batches compare?

The three formats partition a table horizontally before splitting it into columns, but they name and size those partitions differently. The shape of the partition controls parallelism, predicate-pushdown granularity, and metadata overhead.

| Format | Unit | Typical size | Storage | Default codec | Statistics depth |
|---|---|---|---|---|---|
| Apache Parquet | Row group → column chunk → page | 64–512 MB row group | On-disk | Snappy | File + row group + page (v2.x) |
| Apache ORC | Stripe → column → row group within stripe | ~200 MB stripe | On-disk | Zlib | File + stripe + row group |
| Apache Arrow | RecordBatch → array | KB–MB per batch | In-memory (IPC on-disk) | LZ4 (IPC) | None at format level |

Row group and stripe size is a tuning knob. Too large hurts predicate pushdown and parallelism. Too small hurts compression and explodes metadata overhead. The 64–512 MB band is the production sweet spot for Parquet, with smaller values for latency-sensitive lakes and larger values for archival workloads. Arrow batches are smaller, typically 1024–65,536 rows, because the goal is fast in-process iteration, not skip-on-disk. 

## When do native engine formats beat open formats?

Native engine formats (ClickHouse MergeTree parts, Snowflake micro-partitions, DuckDB's native database file) typically outperform Parquet on the same data when the engine owns the storage. They use the same columnar layout, but the engine controls sort order, sparse indexes, skip indexes, and per-column codecs that an interchange format either does not expose or stores less efficiently.

ClickHouse, for example, layers compression codecs on top of dictionary or delta encodings per column, maintains a sparse primary index keyed on the table's sort key, and stores data skipping indexes (min-max, set, Bloom) that Parquet's footer cannot replicate. On a selective filter against an identical dataset, MergeTree often scans substantially fewer bytes than Parquet because the sort order and skip indexes prune granules before the codec runs.

The native-vs-open choice depends on portability. Open formats win when the same data must be read by multiple engines, or stored in a vendor-neutral lake. Native formats win when one engine owns the workload. The engine can be honest about sort order, indexes, and merge policy in ways a portable spec cannot.

Resource hub

Engineering

What is columnar storage?

Why columnar databases are fast

Columnar storage formats: Parquet, ORC, and Arrow explained