This document discusses how Apache Calcite makes it easier to write database management systems (DBMS) by decomposing them into modular components like a query parser, catalog, algorithms, and storage engines. It presents Calcite as a framework that allows these components to be mixed and matched, with a core relational algebra and rule-based optimization. Calcite powers systems like Apache Hive, Drill, Phoenix, and Kylin by translating SQL and other queries to relational algebra and optimizing queries using over 100 rules before executing them using configurable engines and data sources.

1. One planner fits all
(How Apache Calcite makes it easier to
write a DBMS)
Lightning talk at XLDB 2015
Stanford, California
Julian Hyde (Hortonworks)

2. –Mike Stonebraker (2005)
“One size fits all” is an idea whose time has
come and gone

3. • Hadoop and other open source technologies
have deconstructed the DBMS
• Query parser/API + catalog + authorization +
algorithms + scheduler + engine + data format
+ storage

4. image credit: http://oliviaobryon.com

5. image credit: http://oliviaobryon.com
Interesting
Boring

6. Conventional
DB
Parser
Algebra
Catalog
Data
Algorithms

7. Apache Calcite
DB framework
Parser
Algebra
Engine
Data
Engine
Data
Engine
Data
Schema SPI
Operators,
Rules,
Statistics,
Cost model

8. SELECT products.name, COUNT(*)
FROM sales
JOIN products USING (productId)
WHERE sales.discount IS NOT NULL
GROUP BY products.name
ORDER BY COUNT(*) DESC
scan
[products]
scan
[sales]
join
filter
aggregate
sort
scan
[products]
scan
[sales]
filter’
join’
aggregate
sort
FilterIntoJoinRule
translate SQL
to relational
algebra

9. Relational algebra
• Robust
• Allows re-use
• Complex cost-based
optimization
• Multiple front-ends & back-
ends
• Not just for “flat” relations
SQL
Other
QL
API
Algebra
Rules
Engine
A
Engine
B

10. Thank you!
Download: http://calcite.incubator.apache.org
Use Calcite to build your next database!
Calcite powers Apache Hive, Drill, Phoenix,
Kylin
An Apache Incubator project
since May 2014
@julianhyde
What’s in the box?
• SQL parser & AST
• JDBC/ODBC framework
• Built-in operators (project,
filter, …)
• In-memory engine
• 100+ rules
• Planning engines
• Adapters (CSV, JDBC,
Mongo, …)
• Streaming SQL
• Materialized views
Apache
Calcite
Apache
Calcite