Schema management with Scalameta

www.scling.com
Schema management
with Scalameta
Func prog Sweden Meetup, 2023-10-10
Lars Albertsson
Scling
1

www.scling.com
Who’s talking?
...
Google - video conference, productivity
...
Spotify - data engineering
...
Independent data engineering consultant
Banks, media, startups, heavy industry, telco
Founder @ Scling - data-factory-as-a-service
2

www.scling.com
100x 100x
Data artifacts produced
Manual, mechanised, industrialised
3
Spotify's pipelines ~2013

www.scling.com
Craft vs industry
4
● Each step steered by human
○ Or primitive automation
● Improving artifacts
● Craft is primary competence
● Components made for humans
○ Look nice, "easy to use"
○ Most popular
● Autonomous processes
● Improving process that creates artifacts
● Multitude of competences
● Some components unusable by humans
○ Hard, greasy
○ Made for integration
○ Less popular
● Processes often include craft steps

www.scling.com
Road towards industrialisation
5
Data warehouse age -
mechanised analytics
DW
LAMP stack age -
manual analytics
Hadoop age -
industrialised analytics,
data-fed features,
machine learning
Significant change in workflows
Early Hadoop:
● Weak indexing
● No transactions
● Weak security
● Batch transformations

www.scling.com
Simplifying use of new technology
6
DW
Enterprise big data failures
"Modern data stack" -
traditional workflows, new technology
Low-code, no-code

www.scling.com
We have seen this before
7
Difficult adoption
4GL, UML, low-code, no-code
Software engineering education

www.scling.com
Data engineering in the future
8
DW
~10 year capability gap
"data factory engineering"
Enterprise big data failures
"Modern data stack" -
traditional workflows, new technology
4GL / UML phase of data engineering
Data engineering education

www.scling.com
Value of data factories
● Factory value ~ robot movements.
● Data factory value ~ number of datasets.
● Differences are orders of magnitude
○ Enterprises: 100s / day
○ Spotify: 100Ks / day
○ Google: Bs / day
● Cost of DataOps - value of data
○ Low cost - further out the long tail
9
Disruptive value of data, ML
Traditional data warehouse reporting

www.scling.com
Data-factory-as-a-service
10
Data lake
● Data factory
○ Collected, raw data →
processed, valuable data
● Data pipelines customised for client
○ Analytics (BI, reports, A/B testing)
○ Data-fed features (autocomplete, search)
○ Learning systems (recommendations, fraud)
● Compete with data leaders:
○ Quick idea-to-production
○ Operational efficiency
{....}
{....}
{....}

www.scling.com
Success in data factories
11
● Work driven by use cases
○ Teams aligned along the value chain
● Minimal data innovation friction
○ Data democratised - accessible and usable
○ Quick code / test / debug / deploy feedback cycle
● High value / non-value work ratio
○ Guard rails to maintain speed without risk
■ Dev tooling, tests, quality metrics
○ Minimal operational toil
● Build on software engineering processes
○ Composability
○ DevOps, everything as code
○ Strong CI/CD process

www.scling.com
Data lake
Transformation
Cold
pond
Data factories
12
Mutation
Immutable,
shareable

www.scling.com
Data lake
Transformation
Cold
pond
13
Mutation
Immutable,
shareable
Early Hadoop:
● Weak indexing
● No transactions
● Weak security
● Batch transformations
DataOps, functional workflows:
● Immutable, shared data
● Resilient to failure
● Quick error recovery
● Low-risk experiments
Data factories

www.scling.com
● Lowest common denominator = name, type, required
○ Types: string, long, double, binary, array, map, union, record
● Schema specification may support additional constraints, e.g. integer range, other collections
What is a schema?
14
Id Name Age Phone
1 "Anna" 34 null
2 "Bob" 42 "08-123456"
Fields
Name Type Required?
In RDBMS, relations are explicit
In lake/stream datasets, relations are implicit

www.scling.com
Schema definitions
15
{
"type" : "record",
"namespace" : "com.mapflat.example",
"name" : "User",
"fields" : [
{ "name" : "id" , "type" : "int" },
{ "name" : "name" , "type" : "string" },
{ "name" : "age" , "type" : "int" },
{ "name" : "phone" , "type" : ["null", "string"],
"default": null }
]
}
● RDBMS: Table metadata
● Avro: JSON/DSL definition
○ Definition is bundled with avro data files
● Parquet
● pyschema / dataclass
● Scala case classes
● JSON-schema
● JSON: Each record
○ One record insufficient to deduce schema
{ "id": 1, "name": "Alice", "age": "34" }
{ "id": 1, "name": "Bob", "age": "42", "phone": "08-123456" }
case class User(id: String, name: String, age: Int,
phone: Option[String] = None)
val users = Seq( User("1", "Alice", 32),
User("2", "Bob", 43, Some("08-123456")))

www.scling.com
Avro serialisation
16
{
"type" : "record",
"name" : "User",
"fields" : [
{ "name" : "Id" , "type" : "int" },
{ "name" : "Name" , "type" : "string" },
{ "name" : "Age" , "type" : "int" },
{ "name" : "Phone" , "type" : ["null", "string"],
"default": null }
]
}
{ "Id": 1, "Name": "Alice", "Age": "34" }
{ "Id": 1, "Name": "Bob", "Age": "42", "Phone": "08-123456" }
User("1", "Alice", 32)
User("2", "Bob", 43, Some("08-123456"))
Write JSON
Write binary
1001001110101010010101010010111010101010100100111101010
1010101001010101011010101010110111010101010100110010101
Read JSON
Read binary
requires schema
User("2", "Bob", 43, Some("08-123456"))

www.scling.com
Avro in files
17
{
"type" : "record",
"name" : "User",
"fields" : [
{ "name" : "Id" , "type" : "int" },
{ "name" : "Name" , "type" : "string" },
{ "name" : "Age" , "type" : "int" },
{ "name" : "Phone" , "type" : ["null", "string"],
"default": null }
]
}
{ "Id": 1, "Name": "Alice", "Age": "34" }
{ "Id": 1, "Name": "Bob", "Age": "42", "Phone": "08-123456" }
(JSON, line
separated) Schema header +
binary records
1001001110101010010101010010111010101010100100111101010
1010101001010101011010101010110111010101010100110010101
User("2", "Bob", 43, Some("08-123456"))

www.scling.com
Schema on write
18
● Schema defined by writer
● Destination (table / dataset / stream topic) has defined schema
○ Technical definition with metadata (e.g. RDMBS, Kafka + registry)
○ By convention
● Writes not in compliance are not accepted
○ Technically aborted (e.g. RDBMS)
○ In violation of intent (e.g. HDFS datasets)
● Can be technically enforced by producer driver
○ Through ORM / code generation
○ Schema registry lookup
Strict checking philosophy

www.scling.com
Schema on read
19
● Anything (technically) accepted when writing
● Schema defined by reader, at consumption
○ When joining or filtering, unknown fields go through
● Reader may impose requirements on type & value
● Violations of constraints are detected at read
○ Perhaps long after production?
○ By team not owning producer?
Loose checking philosophy
hdfs dfs -cat part-00000.json | jq -c '. | select(.country == "SE")'

www.scling.com
Dynamic vs static typing
20
Schema on write Schema on read
Static typing Dynamic typing
Strict Loose
Possible
Java:
user.setName("Alice");
user2.getName();
Scala:
user = User(name = "Alice", ...)
user2.name
Java:
user.set("name", "Alice");
user2.get("name");
Python:
user.name = "Alice"
user2.name

www.scling.com
Schema on read or write?
21
DB
DB
DB
Service
Service
Export
Business
intelligence
Change agility important here
Production stability important here

www.scling.com
Backward:
E.g. add optional field
Compatibility
22
New consumer
Forward:
E.g. remove optional field
Old producer
= old datasets
Old consumer
New producer

www.scling.com
Incompatible changes
23
● Jobs can accept old input formats
● Translate to new format internally
● Or rerun computations from raw data
● Dedicated job to translate old to new
format - upcasting.

www.scling.com
Language choice, people & preferences
● Java most popular in engineering
+ Good engineering ecosystem
- Boilerplate bad for prototyping & enthusiasm
● Python popular among data scientists
+ Great AI libraries, quick prototypes
- Dynamic typing
● SQL popular among analysts
○ Error handling & data quality difficult
○ Does not compose and scale
● Scala connects both worlds
○ Home of data engineering innovation during the big data hype
24
● Static typing
● Error handling
● Monitor, debug, profile
ecosystem
● Rapid prototyping
● No rituals & boilerplate
● Science innovation

www.scling.com
Scalameta
● Parsing and analysis of scala
source code
25
val a = b() + 3
["val", " ", "a", " ", "=", " ", "b",
"(", ")", " ", "+", " ", "3"]
[val, "a", =, Call("b"), +, Int(3)]
[val, Int(a), =,
Call(com.scling.func.b), +, Int(3)]
lex
parse
semantic
analysis

www.scling.com
Scalameta use cases
● Scalafmt
● Scalafix
○ Static analysis
○ Code transformation
● Online code generation - macros
● Offline code generation
26

www.scling.com
Scalameta use cases
● Scalafmt
● Scalafix
○ Static analysis
○ Code transformation
● Online code generation - macros
● Offline code generation
27
// Example from scio 0.7 -> 0.8 upgrade rules
final class FixTensorflow extends SemanticRule("FixTensorflow") {
override def fix(implicit doc: SemanticDocument): Patch =
doc.tree.collect {
case t @ Term.Select(s, Term.Name(
"saveAsTfExampleFile")) =>
Patch.replaceTree(t, q"$s.saveAsTfRecordFile".syntax)
}.asPatch
}

www.scling.com
Schema & syntax tree
28
Defn.Class(
List(Mod.Annot(Init(Type.Name("PrivacyShielded"), , List())), case),
Type.Name("SaleTransaction"),
List(),
Ctor.Primary(
List(),
,
List(
List(
Term.Param(
List(Mod.Annot(Init(Type.Name("PersonalId"), , List()))),
Term.Name("customerClubId"),
Some(Type.Apply(Type.Name("Option"), List(Type.Name("String")))),
None
),
Term.Param(
List(Mod.Annot(Init(Type.Name("PersonalData"), , List()))),
Term.Name("storeId"),
None
),
Term.Param(
List(),
Term.Name("item"),
None
),
Term.Param(List(), Term.Name("timestamp"), Some(Type.Name("String")), None)
)
)
),
Template(List(), List(), Self(, None), List()))
@PrivacyShielded
case class SaleTransaction(
@PersonalId customerClubId: Option[String],
@PersonalData storeId: Option[String],
item: Option[String],
timestamp: String
)

www.scling.com
Quasiquotes
29
val stat: Stat = "val a = b() + 3".parse[Stat].get
val stat: Stat = q"val a = b() + 3"

www.scling.com
Quasiquotes matching
30
val stat: Stat = "val a = b() + 3".parse[Stat].get
val stat: Stat = q"val a = b() + 3"
stat match {
case q"val $name: $typeOpt = $expr" =>
println(s"Got val declaration $name of type" +
s"${typeOpt.structure} : ${expr.structure} ")
}

www.scling.com
● Expressive
● Custom types
● IDE support
● Avro for data lake storage
Schema definition choice
31
● RDBMS: Table metadata
● Avro: JSON/DSL definition
○ Definition is bundled with avro data files
● Parquet
● pyschema / dataclass
● Scala case classes
● JSON-schema
● JSON: Each record
○ One record insufficient to deduce schema
case class User(id: String, name: String, age: Int,
val users = Seq( User("1", "Alice", 32),
User("2", "Bob", 43, Some("08-123456")))

www.scling.com
Schema offspring Test record
difference render
type classes
32
case classes
test equality
type classes
Avro
definitions
Java Avro
codec classes
Java <-> Scala
converters
Avro type
annotations
MySQL
schemas
CSV codecs
Privacy by
design
machinery
Python
Logical types

www.scling.com
Avro codecs
33
case classes
Avro
definitions
Java Avro
codec classes
Java <-> Scala
converters
{
"name": "JavaUser",
{ "name": "age", "type": "int" }
{ "name": "phone", "type": [ "null", "string" ] }
}
public class JavaUser implements SpecificRecord {
public Integer getAge() { ... }
public String getPhone() { ... }
}
object UserConverter extends AvroConverter[User] {
def fromSpecific(u: JavaUser): User
def toSpecific(u: User): JavaUser
}
case class User(age: Int,

www.scling.com
Quasiquotes in practice
34
q"""
object $converterName extends AvroConverter[${srcClass.clazz.name}
] {
import RecordFieldConverters._
type S = $jClassName
def schema: Schema = $javaClassTerm.getClassSchema()
def tag: ClassTag[S] = implicitly[ClassTag[S]]
def datumReader: SpecificDatumReader[S] = new SpecificDatumReader[$jClassName](classOf[$jClassName])
def datumWriter: SpecificDatumWriter[S] = new SpecificDatumWriter[$jClassName](classOf[$jClassName])
def fromSpecific(record: $jClassName): ${srcClass.clazz.name} =
${Term.Name(srcClass.clazz.name.value)}
(..$fromInits )
def toSpecific(record: ${srcClass.clazz.name}
): $jClassName =
new $jClassName(..$specificArgs)
}
"""

www.scling.com
CSV codecs
35
case classes
CSV codecs
import kantan.csv._
object CsvCodecs {
implicit val userDecoder: HeaderDecoder[User] = ...
}
case class User(age: Int,

www.scling.com
Test equality Test record
difference render
type classes
36
case classes
test equality
type classes
trait REquality[ T] { def equal(value: T, right: T): Boolean }
object REquality {
implicit val double: REquality[Double] = new REquality[Double] {
def equal(left: Double, right: Double): Boolean = {
// Use a combination of absolute and relative tolerance
left === right +- 1e-5.max(left.abs * 1e-5).max(right.abs * 1e-5)
}
}
/** binds the Magnolia macro to the `gen` method */
implicit def gen[T]: REquality[ T] = macro Magnolia. gen[T]
}
object Equalities {
implicit val equalityUser: REquality[User] =
REquality. gen[User]
}

www.scling.com
case class User(
age: Int,
@AvroProp ("sqlType", "varchar(1012)")
Python + RDBMS
37
case classes
Avro
definitions
Avro type
annotations
MySQL
schemas
Python
{
"name": "User",
{ "name": "age", "type": "int" }
{ "name": "phone",
"type": [ "null", "string" ],
"sqlType": "varchar(1012)",
}
}
class UserEgressJob(CopyToTable):
columns = [
( "age", "int"),
( "name", "varchar(1012)"),
]
...

www.scling.com
Logical types
38
case classes
Logical types
case t"""Instant""" =>
JObject(List(JField("type", JString("long")), JField("logicalType",
JString("timestamp-micros"))))
case t"""LocalDate""" => JObject(List(JField("type", JString("int")),
JField("logicalType", JString("date"))))
case t"""YearMonth""" => JObject(List(JField("type", JString("int"))))
case t"""JObject""" => JString("string")
● Avro logical types
○ E.g. date → int, timestamp → long
○ Default is timestamp-millis
■ Great for year > 294441 (!)
● Custom logical types
○ Time
○ Collections
○ Physical

www.scling.com
Stretching the type system
39
● Fail: mixup kW and kWh
● Could be a compile-time error. Should be.
● Physical dimension libraries
○ Boost.Units - C++
○ Coloumb - Scala

www.scling.com
Data
lake
Private
pond
Cold
store
Ingest prepared for deletion
40
Mutation
Landing
pond
Append +
delete
Immutable,
limited
retention

www.scling.com
● PII fields encrypted
● Per-user decryption key table
● Clear single user key => oblivion
- Extra join + decrypt
- Decryption (user) id needed
+ Multi-field oblivion
+ Single dataset leak → no PII leak
+ Handles transformed PII fields
Lost key pattern
41

www.scling.com
Shieldformation
42
@PrivacyShielded
case class Sale(
@PersonalId customerClubId: Option[String],
@PersonalData storeId: Option[String],
timestamp: String
)
case class SaleShielded(
shieldId: Option[String],
customerClubIdEncrypted: Option[String],
storeIdEncrypted: Option[String],
timestamp: String
)
case class SaleAnonymous(
timestamp: String
)
object SaleAnonymize extends SparkJob {
...
}
ShieldForm
object SaleExpose extends SparkJob {
...
}
object SaleShield extends SparkJob {
...
}
case class Shield(
shieldId: String,
personId: Option[String],
keyStr: Option[String],
encounterDate: String
)

www.scling.com
Shield
Shieldformation & lost key
43
SaleShield
Sale
Sale
Shielded
Shield
Deletion
requests
Customer
History
Exposed egress
SaleExpose
Limited retention
SaleAnonymize
Sale
Anonymous
Sale
Stats

www.scling.com
Access limitation models
44
Scramble at ingress (naive)
Validate & filter at access (enterprise)
Locked lake (engineered)
Federated locking (scalable)

www.scling.com
Shield
Shieldformation & locked lake
45
SaleShield
Sale
Sale
Shielded
Shield
Deletion
requests
Customer
History
Exposed egress
SaleExpose
Limited retention
SaleAnonymize
Sale
Anonymous
Sale
Stats
Owned by
target system

www.scling.com
Schema on write!
46
DB
DB
DB
Service
Service
Export
Business
intelligence
Change agility important here
Production stability important here

www.scling.com
Success in data factories
47
○ Composability

www.scling.com
○ Composability
Success in data factories vs data trends
48
Data
mesh
Data
contracts
No code /
low code
SQL / Data
warehouses

www.scling.com
Data factory track record
49
Time to
first flow
Staff size 1st flow
effort, weeks
1st flow cost
(w * 50K ?)
Time to
innovation
Flows 1y
after first
Media 1+ years 10-30 1500? 100M (0.5-1B) 1+ year ?
Finance 2 years 10-50 2000? 100M? Years 10?
Media 3 weeks 4.5 - 8 15 750K 3 months 30
Retail 7 weeks 1-3 7 500K * 6 months 70
Telecom 12 weeks 2-5 30 1500K 6 months 50
Consumer
products
20+ weeks 1.5 30+ 1200+K 6+ months 20
Construction 8 weeks 0.5 4 150K * 7 months 10
Manufacturing 8 weeks 0.5 4 200K * 6 months ?

www.scling.com
● Is Shieldformation open source?
○ No. It might be when it is older and less volatile, and we have grown enough to maintain it properly.
● Is cryptoshredding really acceptable as deletion?
○ Yes.
● Is lost key pattern legally sufficient for all use cases?
○ No.
○ It does not provide complete anonymisation, but pseudonymisation with a limited time span (1 month).
○ Unless data is very sensitive, it has been deemed legally sufficient.
○ Be careful with health & geo data.
Q & A?
50

Schema management with Scalameta

Recommended

Recommended

More Related Content

Similar to Schema management with Scalameta

Similar to Schema management with Scalameta (20)

More from Lars Albertsson

More from Lars Albertsson (20)

Recently uploaded

Recently uploaded (20)

Schema management with Scalameta