While our predecessor product(s) were a COBOL application and later a Windows client/server application distributed on floppies, since 2005 we rock a Java web application that has gone through a lot of changes to keep up with the times. No longer duplicating 5 1/4” 1.2M floppies over the weekend and mailing them to our customers, but a single `git push` will deliver our latest version to our customers.
In this session you will learn how we made our application survive for 18 years and still be relevant. Learn how we rearchitected our application from 2005 to 2022, and more importantly, what we did wrong so you can avoid our mistakes. We migrated from Java 1.4 to Java 17, we moved from tomcat+hibernate+velocity via tomcat+hibernate+wicket+cxf+axis+spring+jgroups+ehcache to wildfly+wicket, from deploying on physical hardware to virtual kubernetes nodes. From manually scp-ing releases to gitops, and from Oracle to Postgresql, causing a 5 week outage.
If you learn only one thing from Martijn Dashorst’s session: always have a plan B.
4. IF YOU LEARN JUST ONE THING
:
ALWAYS HAVE A
PLAN B
5. /WHY FLOPPIES?
/01
A brief history of
our products
/KEEP IT FRESH
/02
An upgrade where we
learned a lesson.
/ARCHITECTURE UPGRADES
/03
Three upgrades,
Three lessons learned
/TABLE OF CONTENTS
/DEPLOYMENTS
/04
Deploying software in
a more civilised age
7. /A BRIEF HISTORY OF TIME
1985-2002
SCHOOLFACT
COBOL·DOS·5¼"FLOPPIES
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
8. /A BRIEF HISTORY OF TIME
1985-2002
2002-2009
SCHOOLFACT
COBOL·DOS·5¼"FLOPPIES
@VO
VB·WINDOWS·3½”FLOPPIES
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
9. /A BRIEF HISTORY OF TIME
1985-2002
2002-
SCHOOLFACT
COBOL·DOS·5¼"FLOPPIES
@VO
VB·WINDOWS·3½”FLOPPIES
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
10. /A BRIEF HISTORY OF TIME
2009-
1985-2002
2002-2009
SCHOOLFACT
COBOL·DOS·5¼"FLOPPIES
@VO
VB·WINDOWS·3½”FLOPPIES
SOMTODAY
WEB·SAAS
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
11. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
PARNASSYS
PARRO
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
12. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
SECONDARY EDUCATION
PARNASSYS
VOCUS
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
13. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
SECONDARY EDUCATION
PARNASSYS
VOCUS
@VO
@VO
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
14. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
SECONDARY EDUCATION
PARNASSYS
@VO
VOCUS @VO SOMTODAY
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
15. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
SECONDARY EDUCATION
VOCATIONAL EDUCATION
PARNASSYS
EDUARTE
2017
@VO
VOCUS @VO SOMTODAY
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
16. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
SECONDARY EDUCATION
VOCATIONAL EDUCATION
PARNASSYS
EDUARTE
2017
@VO
SOMTODAY
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
17. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
SECONDARY EDUCATION
VOCATIONAL EDUCATION
PARNASSYS
EDUARTE
DAYCARE
DIGDAG (PARTOU)
2017
@VO
SOMTODAY
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
18. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
SECONDARY EDUCATION
VOCATIONAL EDUCATION
PARNASSYS
EDUARTE
DAYCARE
DIGDAG (PARTOU)
2017
@VO
SOMTODAY
KEYHUB
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
19. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
SECONDARY EDUCATION
VOCATIONAL EDUCATION
PARNASSYS
EDUARTE
SECURITY
KEYHUB
2017
SOMTODAY
DAYCARE
DIGDAG
01 WHY FLOPPIES? · A BRIEF HISTORY OF TIME
25. “it is completely unworkable for
over a week already. […] it has
the speed of hitting a tennis
ball through a gardenhose.”
02 KEEP IT FRESH · JAVA 6 → JAVA 7 UPGRADE
26. [...] Even so slow that searching is
faster using the old archive (open
archive,
fi
nd folder, browse folder,
transferring the data, storing the
folder, closing the archive). All the
while Parnassys tried to log in. It
didn’t by. the way. And I
fi
nd that
irritating.
02 KEEP IT FRESH · JAVA 6 → JAVA 7 UPGRADE
27. 02 KEEP IT FRESH · JAVA 6 → JAVA 7 UPGRADE
/JAVA 6 → JAVA 7 UPGRADE
/INSTABILITY FOR 2 WEEKS
Code would stall on
synchronization
blocks
/JAVA 7 PART OF RELEASE
NO WAY BACK
:
database and
essential features
and integrations
were released to
customers
/SUSPEND VS SPINNING
Waiting for lock:
suspend or spin. <6
suspend
>
=
6 spin.
28. —WE
“First deploy the new Java
runtime to Test, Acceptance and
Production before writing source
code using the new Java version”
02 KEEP IT FRESH · JAVA 6 → JAVA 7 UPGRADE
32. /ARCHITECTURE UPGRADES
/TOMCAT → WILDFLY
Reevaluate your past
decisions to see if
they still hold true
/SHARED BASE PLATFORM
Code can’t be
preserved, it goes
stale and then bad.
Share the upgrade load.
/ORACLE → POSTGRESQL + K8S
Don’t execute plan
A and B together at
the same time
03 ARCHITECTURE UPGRADES
38. /HOW IT STARTED
TOMCAT
HIBERNATE EHCACHE
WICKET
BROWSERS
BONECP QUARTZ
WEB
SERVICES
AXIS
03 ARCHITECTURE UPGRADES · 01 FROM TOMCAT TO WILDFLY
39. /HOW IT STARTED
TOMCAT
HIBERNATE EHCACHE
WICKET
BROWSERS
BONECP QUARTZ
WEB
SERVICES
AXIS + AXIS 2
03 ARCHITECTURE UPGRADES · 01 FROM TOMCAT TO WILDFLY
40. /HOW IT STARTED
TOMCAT
HIBERNATE EHCACHE
WICKET
BROWSERS
BONECP QUARTZ
WEB
SERVICES
AXIS + AXIS 2
+ CXF
03 ARCHITECTURE UPGRADES · 01 FROM TOMCAT TO WILDFLY
41. /HOW IT STARTED
TOMCAT
BONECP
HIBERNATE
SPRING BEANS
SPRING CONTEXT
EHCACHE
QUARTZ
SPRING
TRANSACTION
AXIS + AXIS 2
+ CXF
WICKET
WEB
SERVICES
BROWSERS
03 ARCHITECTURE UPGRADES · 01 FROM TOMCAT TO WILDFLY
43. /JAVA EE?
SERVLET 3
JSF 2
EJB 3.1
JPA 2
JSP
CDI
JTA
BEAN VALIDATION
JAX-WS
JAX-RPC
JAXR
SAAJ
JAX-RS
JAXB
JMS
JAAS
JASPIC
JACC
JCA
JAVA MAIL
JSR 88
JSR 77
RMI
JNDI
SPRING
BEANS
HIBERNATE
SPRING
CORE
SPRING
CONTEXT
SPRING
TRANSACTION
BATCH
CONCURRENC
QUARTZ QUARTZ
03 ARCHITECTURE UPGRADES · 01 FROM TOMCAT TO WILDFLY
HIBERNATE JAVA MAIL
WICKET
WICKET
CXF
CXF CXF
44. IF IT QUACKS LIKE
JAVA EE, IF IT LOOKS
LIKE JAVA EE, IT
PROBABLY IS JAVA EE
45. /HOW IT STARTED
TOMCAT
BONECP
HIBERNATE
SPRING BEANS
SPRING CONTEXT
EHCACHE
QUARTZ
SPRING
TRANSACTION
CXF
WICKET
WEB
SERVICES
BROWSERS
03 ARCHITECTURE UPGRADES · 01 FROM TOMCAT TO WILDFLY
49. REEVALUATE YOUR OLD DECISIONS
What was once a good decision may no longer be
the best.
/RECOMMENDATION
03 ARCHITECTURE UPGRADES · 01 FROM TOMCAT TO WILDFLY
50. /ARCHITECTURE UPGRADES
/TOMCAT → WILDFLY
Reevaluate your past
decisions to see if
they still hold true
/SHARED BASE PLATFORM
Code can’t be
preserved, it goes
stale and then bad.
Share the upgrade load.
/ORACLE → POSTGRESQL + K8S
Don’t execute plan
A and B together at
the same time
03 ARCHITECTURE UPGRADES
52. /ONE SHARED BASE PLATFORM
ONDERWIJS PARENT
PARNASSYS
SOMTODAY
EDUARTE
DIGDAG
KEYHUB
JAVA EE WILDFLY
JAVA
EVERY 14 DAYS
Upgrade all
dependencies to latest
releases
03 ARCHITECTURE UPGRADES · 01 ONE SHARED BASE PLATFORM
53. STAY UP TO DATE
Code can’t be preserved, it goes stale and then
bad. Share the upgrade load.
/RECOMMENDATION
03 ARCHITECTURE UPGRADES · 01 ONE SHARED BASE PLATFORM
54. /ARCHITECTURE UPGRADES
/TOMCAT → WILDFLY
Reevaluate your past
decisions to see if
they still hold true
/SHARED BASE PLATFORM
Code can’t be
preserved, it goes
stale and then bad.
Share the upgrade load.
/ORACLE → POSTGRESQL + K8S
Don’t execute plan
A and B together at
the same time
03 ARCHITECTURE UPGRADES
55. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
SECONDARY EDUCATION
VOCATIONAL EDUCATION
PARNASSYS
EDUARTE
SECURITY
KEYHUB
2017
SOMTODAY
DAYCARE
DIGDAG
ORA → PG
K8S
SUMMER 2020
03 ARCHITECTURE UPGRADES · 03 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
ORA → PG
56. /WHY MIGRATE FROM ORACLE
/MIGRATION ASSISTENT
Licensing audit
took 3 years,
extortion racket
/PRICE
Our DB server core
count costs €€€
/TOOLING
Requires RAC /
Enterprise license,
€€€
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
57. /SOMTODAY
- ~220 schools
- 8 streets
- 1 schema / street
- 1 database server: Oracle
- column tenant identif
i
er
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
58. 03 ARCHITECTURE UPGRADES · 03 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
/BEFORE
ORACLE
PROD
ADMIN
STUDENT
REST
STREET 1
SCHOOLS 1—25
59. 03 ARCHITECTURE UPGRADES · 03 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
/BEFORE
ORACLE
PROD
ADMIN
STUDENT
REST
STREET 1
ORACLE
PROD
ADMIN
STUDENT
REST
STREET 2
ORACLE
PROD
ADMIN
STUDENT
REST
STREET 3
ORACLE
PROD
ADMIN
STUDENT
REST
STREET 4
ORACLE
PROD
ADMIN
STUDENT
REST
STREET 5
ORACLE
PROD
ADMIN
STUDENT
REST
STREET 6
ORACLE
PROD
ADMIN
STUDENT
REST
STREET 7
ORACLE
PROD
ADMIN
STUDENT
REST
STREET 8
SCHOOLS 1—25 SCHOOLS 26—50 SCHOOLS 51—75 SCHOOLS 76—100 SCHOOLS 101—125 SCHOOLS 126—150 SCHOOLS 151—175 SCHOOLS 176—200
61. 03 ARCHITECTURE UPGRADES · 03 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
/AFTER?
PROD
ADMIN
STUDENT REST
ADMIN ADMIN ADMIN STUDENT STUDENT STUDENT STUDENT
REST REST
KUBERNETES
62. /SCHEMA PER TENANT
Each tenant gets its own schema
for tenant data, one schema for
nation wide data
03 ARCHITECTURE UPGRADES · 03 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
NATIONWIDE
DATA
leerling
vak
resultaat
toets
rooster
absentie
ouder
rapport
examen
kluisje
huiswerk
pakket
opleiding
docent
rol
adres
63. /ORA2PG
/FREE
Dutchies most favorite
criterium!
/CUSTOM MIGRATION
Renumber IDs, move schemas
/BIG BANG MIGRATION
There’s no PG2ORA, so one
-
way only
03 ARCHITECTURE UPGRADES · 03 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
65. /FAILURES
Database processes kept restarting
due to memory errors
/MORE FAILURES
Amount of connections overwhelmed
database, pgbouncer to the rescue
pgbouncer does not support
server
-
side prepared statements
Amount of queries overwhelmed database
due to Java EE caching issues
Amount of long running idle
transactions overwhelmed database
/FAILURE MODE ENGAGED
03 ARCHITECTURE UPGRADES · 03 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
66. 5
WEEKS OUTAGE
START OF SCHOOL YEAR
03 ARCHITECTURE UPGRADES · 03 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
67. 03 ARCHITECTURE UPGRADES · 03 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
/FINAL SOLUTION
PROD
ADMIN
STUDENT REST
ADMIN ADMIN ADMIN STUDENT STUDENT STUDENT STUDENT
REST REST
KUBERNETES
PROD
PGBOUNCER
PGBOUNCER
68. ONLY DO ONE MIGRATION AT A TIME
And…
Always have a plan B
03 ARCHITECTURE UPGRADES · 03 FROM ORACLE TO POSTGRESQL FOR THE 2ND TIME
/RECOMMENDATION
70. /FLOPPY SOFTWARE RELEASES
INDEX.HTML
How do we ship software
currently
/CLOUDY DEPLOYMENT
How did “we” ship software
in them ole days?
/WHAT IS THIS TOPIC ABOUT?
YOUR LOGO HERE
71. /SOFTWARE RELEASE PROCESS
/MODIFY HELM CHART
Apply a sprinkle of
YAML
INDEX.HTML
04 TO CLOUD DEPLOYMENTS · CURRENT SOFTWARE RELEASE PROCESS
72. /SOFTWARE RELEASE PROCESS
/PUSH TO GITHUB
Perform change request
review
/MODIFY HELM CHART
Apply a sprinkle of
YAML
INDEX.HTML
04 TO CLOUD DEPLOYMENTS · CURRENT SOFTWARE RELEASE PROCESS
73. /SOFTWARE RELEASE PROCESS
/PUSH TO GITHUB
Perform change request
review
/MODIFY HELM CHART
Apply a sprinkle of
YAML
/MERGE TO BRANCH
Triggers ARGO CD
INDEX.HTML
04 TO CLOUD DEPLOYMENTS · CURRENT SOFTWARE RELEASE PROCESS
74. /SOFTWARE RELEASE PROCESS
/PUSH TO GITHUB
Perform change request
review
/DONE
ARGO CD updates K8S and
deploys modif
i
cations
/MODIFY HELM CHART
Apply a sprinkle of
YAML
/MERGE TO BRANCH
Triggers ARGO CD
INDEX.HTML
04 TO CLOUD DEPLOYMENTS · CURRENT SOFTWARE RELEASE PROCESS
75. ~25 mins
From code change to
patch on production
04 TO CLOUD DEPLOYMENTS · CURRENT SOFTWARE RELEASE PROCESS
« no CI/CD (yet) »
76. /FLOPPY SOFTWARE RELEASES
INDEX.HTML
How do we ship software
currently
/CLOUDY DEPLOYMENT
How did “we” ship software
in them ole days?
/WHAT IS THIS TOPIC ABOUT?
YOUR LOGO HERE
78. CopyPro 2000
* 44-230 diskettes per hour
* Unattended operation
* 50 disk capacity option included
* High Speed Drives
* Separate Reject Bin
* Drive Cleaning Mode
* Automatic Batch Mode Processing
using multiple Disk Masters
* Bit
-
by
-
bit verif
i
cation
04 TO CLOUD DEPLOYMENTS · FLOPPY DEPLOYMENTS
80. 300 customers
40 floppy’s / release
200 / hr / machine
12,000
/A BIG NUMBER TO SUPPORT THE CONCEPT
04 TO CLOUD DEPLOYMENTS · FLOPPY DEPLOYMENTS
floppies
81. 300 customers
40 floppy’s / release
200 / hr / machine
60 hrs
/A BIG NUMBER TO SUPPORT THE CONCEPT
04 TO CLOUD DEPLOYMENTS · FLOPPY DEPLOYMENTS
85. JAVA 6 → JAVA 7 → JAVA 8 → JAVA 11 → JAVA 17
Use TAPD for infrastructure, DTAP for code.
Always have a plan B
MIGRATION FROM ORACLE 2 + KUBERNETES
You will not sleep well when there’s
no plan B.
TOMCAT → WILDFLY
REEVALUATE YOUR OLD DECISIONS. What was once
a good decision may no longer be the best.
MIGRATION FROM ORACLE 1
Projectmanagers sleep better with the
security of a plan B
USE SHARED BASE PLATFORM
One parent POM to rule them all and
in the build bind them
.
.
.
/WHAT DID WE DO?
AUTOMATED DEPLOYMENTS
Infrastructure as Code saves your
life.
86. /DTAP FOR CODE, TAPD FOR INFRASTRUCTURE
Always have a plan B
/ONLY DO ONE MIGRATION AT A TIME
And…
Always have a plan B
/REEVALUATE YOUR OLD DECISIONS
What was once a good decision may no
longer be the best.
/PAID SOFTWARE MAY BE WORTH IT
Don’t dismiss paid software that
prevents a problem
/STAY UP TO DATE
Code can’t be preserved, it goes
stale and then bad. Share the load.
/RECOMMENDATIONS
/DON’T USE FLOPPIES
Nobody can use them anymore and they
are expensive as hell
88. CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon, and infographics & images by Freepik
THANKS!
DO YOU HAVE ANY QUESTIONS?
martijn.dashorst@topicus.nl
topicus.nl · topicus.com · martijndashorst.com
@dashorst
89.
90. 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
PRIMARY EDUCATION
SECONDARY EDUCATION
VOCATIONAL EDUCATION
PARNASSYS
SOMTODAY
EDUARTE
2017
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
ORA → PG
SECURITY
KEYHUB
DAYCARE
DIGDAG
PARNASSYS
SUMMER 2018
91. /PARNASSYS
- ~6500 schools
- 1 schema
- 1 database: Oracle
- column tenant identif
i
er
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
92. /WHY MIGRATE FROM ORACLE
/MIGRATION ASSISTENT
Licensing audit
took 3 years,
extortion racket
/PRICE
Our DB server core
count costs €€€
/TOOLING
Requires RAC /
Enterprise license,
€€€
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
93. /BEFORE
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
PRIMARY SECONDARY
standby
ORACLE ORACLE
standard replication
production
95. /ORA2PG
/FREE
Dutchies most favorite
criterium!
/SLOW
1 day for 1 migration attempt, or
unstable (parallel)
/BIG BANG MIGRATION
There’s no PG2ORA, so one
-
way only
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
97. /HVR
/EXPENSIVE
“Call for pricing”
— Dutchies least favourite criterium!
/FAST
Real
-
time sync, low latency
/REVERSIBLE
Bi
-
directional, cross database
synchronization!
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
Cross-database replica
ti
on using redo logs
98. /HVR «BEFORE MIGRATION»
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
PRIMARY
NEW
PRIMARY
ORACLE SECONDARY
failsafe
ORACLE
PROD
PROD
standby
standard replication
99. /HVR «BEFORE MIGRATION»
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
ORACLE
failsafe
ORACLE
PROD
PROD
standby
HVR HVR
standard replication
PRIMARY
NEW
PRIMARY SECONDARY
100. /HVR «AFTER MIGRATION»
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
ORACLE
failsafe
ORACLE
PROD
PROD
standby
standard replication
HVR HVR
OLD
PRIMARY PRIMARY SECONDARY
101. /CURRENT DATABASE
03 ARCHITECTURE UPGRADES · 02 FROM ORACLE TO POSTGRESQL FOR THE 1ST TIME
PROD standby
standard replication
PRIMARY SECONDARY
OLD
PRIMARY
102. /DTAP FOR CODE, TAPD FOR INFRASTRUCTURE
Always have a plan B
/REEVALUATE YOUR OLD DECISIONS
What was once a good decision may no
longer be the best.
/PAID SOFTWARE MAY BE WORTH IT
Don’t dismiss paid software that
prevents a problem
/STAY UP TO DATE
Code can’t be preserved, it goes
stale and then bad. Share the load.
/RECOMMENDATIONS