OLAPDBGet new results
Methodology

Where the numbers come from

OLAPDB runs no private benchmarks. Every figure is taken from public ClickBench results that anyone can reproduce, and every calculation on top of them is described here.

Our position

OLAPDB is opinionated. We think ClickHouse is the best default for real-time analytics, and the site says so. We also think a recommendation is only worth as much as the evidence behind it, so we use public data, show our arithmetic and list the results where ClickHouse does not come first.

ClickBench itself is maintained by ClickHouse, Inc. Its scripts, data and results are open, and many vendors submit and tune their own entries. Read it with that in mind, as we do.

The benchmark

ClickBench loads a single table of 99,997,497 rows taken from real web analytics traffic, then runs 43 SQL queries against it: counts, distinct counts, group-bys on high-cardinality columns, string searches, regular expressions, sorting and time-series filters. Each query runs three times. The first run is the cold run; the faster of the second and third is the hot run.

Scripts and raw results are in the ClickBench repository, one folder per system, one JSON file per machine and date.

Groups

Systems are only scored against others in the same group:

  • Self-hosted: ClickHouse, DuckDB and StarRocks on the same AWS c6a.4xlarge (16 vCPU, 32 GiB), latest result for each.
  • Managed: ClickHouse Cloud (2 replicas × 64 GiB on AWS), Snowflake (Medium, Gen2), Databricks SQL (Medium), BigQuery (on-demand) and Redshift Serverless. The latest published result for each; Redshift’s is from 2022.
  • Concurrency: ClickHouse and StarRocks on three bare-metal AWS hosts. Only runs with zero errors are used, as on ClickBench.

Managed services don’t map neatly onto each other’s sizes. We chose mid-sized configurations; on ClickBench, Snowflake’s total hot time is close to 60 seconds from Medium through X-Large, so a larger warehouse would not change its ranking.

Scores

We use ClickBench’s own formula, so our scores match what its site shows for the same selection:

ratio(q)  = (time(q) + 0.01 s) / (fastest_time_in_group(q) + 0.01 s)
score     = exp( mean( ln(ratio(q)) ) )   over all 43 queries

A score of 1.0 means fastest on every query. The 10 ms constant stops sub-millisecond queries from dominating the result. A failed query counts as twice the system’s slowest time, with a 300-second floor.

Ingestion and compression

Ingestion throughput is the row count divided by ClickBench’s reported load_time. That is a one-time bulk load from files, not continuous streaming. Compression is the reported data_size after loading, which each system measures in its own way.

Cost

Cost per 1,000 queries is the average hot-run time over the 43 queries, times 1,000, times the hourly list price, billed per second:

  • ClickHouse: c6a.4xlarge on-demand, AWS us-east-1: $0.612/h
  • DuckDB: c6a.4xlarge on-demand, AWS us-east-1: $0.612/h
  • StarRocks: c6a.4xlarge on-demand, AWS us-east-1: $0.612/h
  • Databricks SQL: Medium serverless SQL warehouse, 24 DBU/h at $0.70/DBU
  • Snowflake: Medium warehouse, 4 credits/h at $2.00/credit (Standard)

It is a compute-only estimate. It leaves out storage, data transfer, discounts, minimum billing periods and idle time. BigQuery, Redshift Serverless and ClickHouse Cloud are left out because their billing depends on usage that ClickBench doesn’t record.

Features

The comparison table reflects each product’s public documentation for the capabilities listed. Marks are our judgement of how well each capability serves real-time analytics.

Corrections

We use the latest published ClickBench result for each system. If a vendor has a newer or better-tuned result, the place to submit it is the ClickBench repository; it will appear here once we refresh the data. To hear when that happens, sign up for updates.