Moving to Riak involves a number of changes from the status quo of RDBMS systems, one of which is taking greater control over your schema design. * How do you structure data when you don't have tables and foreign keys? * When should you denormalize, add links, or create map-reduce queries? * Where will Riak be a natural fit and where will it be challenging?

1. Schema Design
for Riak
Sean Cribbs
Developer Advocate
basho

2. Thinking
Non-Relationally
basho

3. “There is no spoon
schema.”
basho

4. There is an
application.
basho

5. What you lose
basho

6. What you lose
•Tables
basho

7. What you lose
•Tables
•Foreign keys and constraints
basho

8. What you lose
•Tables
•Foreign keys and constraints
•ACID
basho

9. What you lose
•Tables
•Foreign keys and constraints
•ACID
•Sophisticated query planners
basho

10. What you lose
•Tables
•Foreign keys and constraints
•ACID
•Sophisticated query planners
•Declarative query language (SQL)
basho

11. What you gain
basho

12. What you gain
•More flexible, fluid designs
basho

13. What you gain
•More flexible, fluid designs
•More natural data representations
basho

14. What you gain
•More flexible, fluid designs
•More natural data representations
•Scaling without pain
basho

15. What you gain
•More flexible, fluid designs
•More natural data representations
•Scaling without pain
•Reduced operational complexity
basho

16. Lose the crutches
basho

17. Lose the crutches
•Sparse data (optional/multi-value fields)
basho

18. Lose the crutches
•Sparse data (optional/multi-value fields)
•Richer data structures
basho

19. Lose the crutches
•Sparse data (optional/multi-value fields)
•Richer data structures
•Meaningful identifiers
basho

20. Lose the crutches
•Sparse data (optional/multi-value fields)
•Richer data structures
•Meaningful identifiers
•Innovative access patterns
basho

21. Eventual Consistency
basho

22. Eventual Consistency
•You are already dealing with
inconsistency (e.g. caching, read slave)
basho

23. Eventual Consistency
•You are already dealing with
inconsistency (e.g. caching, read slave)
•Plan for EC instead of triaging
basho

24. Eventual Consistency
•You are already dealing with
inconsistency (e.g. caching, read slave)
•Plan for EC instead of triaging
•Determine resolution processes
basho

25. Eventual Consistency
•You are already dealing with
inconsistency (e.g. caching, read slave)
•Plan for EC instead of triaging
•Determine resolution processes
•Weigh risk of conflicts and emergent
issues
basho

26. Know your data
basho

27. Make your Top 10 list
basho

28. Make your Top 10 list
•Frequently requested pages/screens
basho

29. Make your Top 10 list
•Frequently requested pages/screens
•Slow queries
basho

30. Make your Top 10 list
•Frequently requested pages/screens
•Slow queries
•Secondary indexes
basho

31. Make your Top 10 list
•Frequently requested pages/screens
•Slow queries
•Secondary indexes
•Complicated joins or aggregations
basho

32. Analyze
basho

33. Analyze
•Interdependencies, coupling
basho

34. Analyze
•Interdependencies, coupling
•Cardinalities of relationships
basho

35. Analyze
•Interdependencies, coupling
•Cardinalities of relationships
•Access pattern
basho

36. Analyze
•Interdependencies, coupling
•Cardinalities of relationships
•Access pattern
•Shoehorned structures
basho

37. Example
basho

38. basho

39. basho

40. Top queries
basho

41. List Features
basho

42. Reorder Features
basho

43. Regroup Features
basho

44. Edit Feature
basho

45. List Folders
basho

46. Project Statistics
basho

47. projects users
folders
features milestones
project_items
Current Schema
basho

48. projects users
folders
artificial
ordering features milestones
project_items
Current Schema
basho

49. projects users
additional
key for stats
folders
artificial
ordering features milestones
project_items
Current Schema
basho

50. projects users
additional
key for stats
BLOB
folders columns
artificial serialized
ordering features milestones parse tree
formatted
project_items plain text
Current Schema
basho

51. collaborator
projects users
additional
key for stats
BLOB
folders columns
artificial serialized
ordering features milestones parse tree
formatted
project_items plain text
Current Schema
basho

52. collaborator
projects users
additional
key for stats owner
BLOB
folders columns
artificial serialized
ordering features milestones parse tree
formatted
project_items plain text
Current Schema
basho

53. Observations
basho

54. Observations
•Features and milestones are disjunct,
except in list participation
basho

55. Observations
•Features and milestones are disjunct,
except in list participation
•Opacity of serialized BLOB data
basho

56. Observations
•Features and milestones are disjunct,
except in list participation
•Opacity of serialized BLOB data
•Duplication of relationships -
collaborator-owner, project-folder-item
basho

57. List/Reorder Features
basho

58. List/Reorder Features
• AJAX postback was
already “total” ordering
basho

59. List/Reorder Features
• AJAX postback was
already “total” ordering
• List is really a property
of the parent “folder”
basho

60. List/Reorder Features
• AJAX postback was
already “total” ordering
• List is really a property
of the parent “folder”
• Membership, order are
coupled concepts
basho

61. Links in Riak
basho

62. Links in Riak
•Lightweight pointers, like HTML <link>
tags.
basho

63. Links in Riak
•Lightweight pointers, like HTML <link>
tags.
•Composed of bucket, key, tag
basho

64. Links in Riak
•Lightweight pointers, like HTML <link>
tags.
•Composed of bucket, key, tag
•Traversable through “link-walking”
basho

65. Riak: List/Reorder
Features
basho

66. Riak: List/Reorder
Features
•Store list of features and milestones as
links on folder, with index as tag
basho

67. Riak: List/Reorder
Features
•Store list of features and milestones as
links on folder, with index as tag
•Query: Follow links to items, map values
(with index), reduce sort
basho

68. Riak: List/Reorder
Features
•Store list of features and milestones as
links on folder, with index as tag
•Query: Follow links to items, map values
(with index), reduce sort
•One write to order list, easy to link-walk
basho

69. Riak: List/Reorder
Features
•Store list of features and milestones as
links on folder, with index as tag
•Query: Follow links to items, map values
(with index), reduce sort
•One write to order list, easy to link-walk
• Concurrent writes may conflict
basho

70. Regroup Features
basho

71. Regroup Features
• Changing foreign key
to folder
basho

72. Regroup Features
• Changing foreign key
to folder
• Recalculate order for
both lists (multiple
UPDATE statements)
basho

73. Riak: Regroup
Features
basho

74. Riak: Regroup
Features
•Already solved for us by folder owning
list as links
basho

75. Riak: Regroup
Features
•Already solved for us by folder owning
list as links
•List order still recalculated
basho

76. Riak: Regroup
Features
•Already solved for us by folder owning
list as links
•List order still recalculated
•Two writes (vs. N for relational)
basho

77. Riak: Regroup
Features
•Already solved for us by folder owning
list as links
•List order still recalculated
•Two writes (vs. N for relational)
• No transactions, need extra info for
conflict resolution
basho

78. Edit Feature
basho

79. Edit Feature
• Required two
representations (text,
tree), both stored as
BLOBs
basho

80. Edit Feature
• Required two
representations (text,
tree), both stored as
BLOBs
• Application
deserializes tree to
present web form,
recreates text on save
basho

81. Composition
basho

82. Composition
•Sometimes called “denormalization”
basho

83. Composition
•Sometimes called “denormalization”
•Store dependent data in the same object
as the parent
basho

84. Composition
•Sometimes called “denormalization”
•Store dependent data in the same object
as the parent
•JSON, XML, Erlang terms
basho

85. Riak: Edit Feature
basho

86. Riak: Edit Feature
•Already denormalized!
basho

87. Riak: Edit Feature
•Already denormalized!
•Now parse tree can have the same
format (JSON, XML), exposing it to other
types of processing
basho

88. Riak: Edit Feature
•Already denormalized!
•Now parse tree can have the same
format (JSON, XML), exposing it to other
types of processing
•Individual objects larger
basho

89. List Folders
basho

90. List Folders
• Number of folders is
small per project
basho

91. List Folders
• Number of folders is
small per project
• Essentially unordered
within project
basho

92. Riak: List Folders
basho

93. Riak: List Folders
• Store links to folder
objects (or entire list) in
project
basho

94. Riak: List Folders
• Store links to folder
objects (or entire list) in
project
• Simple link-walking or
map-reduce to get
folders
basho

95. Riak: List Folders
• Store links to folder
objects (or entire list) in
project
• Simple link-walking or
map-reduce to get
folders
• Two writes required, or
move feature links/
order to project
basho

96. Riak: List Folders
• Store links to folder • Parameterized bucket
objects (or entire list) in based on project key
project
• Simple link-walking or
map-reduce to get
folders
• Two writes required, or
move feature links/
order to project
basho

97. Riak: List Folders
• Store links to folder • Parameterized bucket
objects (or entire list) in based on project key
project
• One write to add folder
• Simple link-walking or
map-reduce to get
folders
• Two writes required, or
move feature links/
order to project
basho

98. Riak: List Folders
• Store links to folder • Parameterized bucket
objects (or entire list) in based on project key
project
• One write to add folder
• Simple link-walking or
map-reduce to get • List keys is expensive
folders
• Two writes required, or
move feature links/
order to project
basho

99. Project Statistics
basho

100. Project Statistics
• Relational design had
per-feature counters
(maintained by app)
basho

101. Project Statistics
• Relational design had
per-feature counters
(maintained by app)
• Extra foreign key sped
up aggregation by
reducing joins
basho

102. Riak: Project Statistics
basho

103. Riak: Project Statistics
•Use scatter-gather: Map-reduce was
made for this type of problem!
basho

104. Riak: Project Statistics
•Use scatter-gather: Map-reduce was
made for this type of problem!
•Take advantage of now-transparent
internal format of feature objects
basho

105. Riak: Project Statistics
•Use scatter-gather: Map-reduce was
made for this type of problem!
•Take advantage of now-transparent
internal format of feature objects
•Query: Follow links to features, map per-
feature counters, reduce summarize the
counters
basho

106. Awkward Fits
basho

107. Awkward Fits
•User uniqueness constraint
basho

108. Awkward Fits
•User uniqueness constraint
•“double-click” problem
basho

109. Awkward Fits
•User uniqueness constraint
•“double-click” problem
•Existing authentication solutions
basho

110. Awkward Fits
•User uniqueness constraint
•“double-click” problem
•Existing authentication solutions
•Project membership
basho

111. Awkward Fits
•User uniqueness constraint
•“double-click” problem
•Existing authentication solutions
•Project membership
•M-to-N is hard
basho

112. Review
basho

113. Review
•Analyze your relational model
basho

114. Review
•Analyze your relational model
•Identify pain points, take stats
basho

115. Review
•Analyze your relational model
•Identify pain points, take stats
•Design some alternatives
basho

116. Review
•Analyze your relational model
•Identify pain points, take stats
•Design some alternatives
•Test, Measure, Repeat!
basho

117. Plug
Interested in learning about support,
consulting, or Enterprise features?
Email info@basho.com or go to
http://www.basho.com/contact.html to talk
with us.
www.basho.com
basho

118. Plug
Interested in learning about support,
consulting, or Enterprise features?
Email info@basho.com or go to
http://www.basho.com/contact.html to talk
with us.
www.basho.com
sean@basho.com
@seancribbs
basho