When the Optimizer Guesses Wrong: A Reproducibility Lens on Cardinality Estimation
Presented by:
Alena Rybakina
I am a PostgreSQL database developer in the field of optimization, namely, I write code for the optimizer and statistics. I'm active contributor in PostgreSQL Community: several patches were committed (OR-to-ANY transformation and Values-to-ANY transformation). I have participated and gave talks in conferences: PGConf Canada 2024, 2025, PG Conf Ivory SQL 2025.
No video of the event yet, sorry!
For decades, the query optimizer's hardest problem has not been searching for a plan - it has been predicting how many rows each operator will produce. Cardinality estimation errors, and underestimation especially, compound multiplicatively across joins: estimate one row against a reality of a million, and the optimizer commits to a nested-loop plan that runs orders of magnitude too slow. This single failure mode sits behind most "the optimizer picked a terrible plan" stories. This talk is a critical review of cardinality estimation methods - not join-order search, not end-to-end optimizers, but the prediction problem itself. I start from where traditional statistics break down: histograms, most-common-value lists, and sampling all lean on independence and uniformity assumptions that real correlated data violates. I then work through the learned wave: data-driven models that learn the joint data distribution, query-driven models trained on past executions, pessimistic estimators that bound the error instead of predicting a point value, and plan-aware objectives that optimize for an estimate's effect on the plan rather than its raw accuracy. I line up reported results against standard benchmarks where the comparison is fair, and stay honest about the recurring failure modes - out-of-distribution collapse, training and maintenance cost, poor behavior under data updates, and estimates that improve Q-error on paper without ever changing the chosen plan. The question I want to leave the room with: which of these estimators are reproducible and deployable inside a production optimizer, and which remain research prototypes.
- Date:
- Duration:
- 40 min
- Room:
- Conference:
- PG Down Under 2026
- Language:
- Track:
- Development
- Difficulty:
- Medium