With Galera 4, MariaDB 10.4 cluster further extends the capabilities of the synchronous Galera replication. The most prominent feature in Galera 4 version, is streaming replication technology, which implements distributed transaction processing within the cluster. With streaming replication, a transaction can be launched to execute in all cluster nodes in parallel. With this, a large transaction can be executed in small fragments due out the transaction life time, and cluster will not choke with the replication of one large transaction write set, as happened in earlier Galera Cluster versions. Streaming replication works as a foundation for many more features, to be released in short term. e.g. XA transaction support will now be possible thanks to streaming replication technology. MariaDB 10.4 will come with new Galera Replication version 4. This presentation will outline the new features of Galera 4 Replication as present in MariaDB 10.4 and share the early user experiences with it. Galera Cluster will be later released supporting MySQL 8.

1. 1
in
MariaDB 10.4
And a little bit in MySQL
Seppo Jaakola
Codership
Galera 4

2. www.galeracluster.com
Seppo Jaakola
CEO Codership
Developer role, 15 yrs
with MySQL engineering
Background:
DBMS Engineering
Data Security

3. C o d e r s h i p
Technology Company
Since 2007
Replication for Open Source
Databases
Galera Cluster
Business through MySQL /
MariaDB Support & Consulting
Technology Company, focus in R&D
Since 2007
Replication for Open Source Databases
Galera Cluster
Business through MySQL / MariaDB
Support & Consulting
Stable continuous growth,
team growing 3 → ~20
C o d e r s h i p

4. 4
www.galeracluster.com
Agenda
●
Galera Cluster Overview
●
Galera 4 in 10.4 Features
●
Upgrading 10.3 Cluster to 10.4
●
Galera 4 Streaming Replication
●
Additional Galera 4 Feature Road Map

5. www.galeracluster.com
galera
●
Generic Replication Plugin for
database servers
●
Uses Replication API to interact with
DBMS (wsrep API project in github)
●
DBMS and Galera plugin must have same
wsrep API version
Galera Replication

6. 6
www.galeracluster.com
Galera Replication Versions
●
Major versions 1..2..3
●
Current production head version 3.26
●
wsrep API versions 1..25
●
Next major version is Galera 4 and using wsrep
API #26
●
wsrep API change requires rolling upgrade path

7. 7
Galera in MariaDB 10.4

8. 8
www.galeracluster.com
Galera 4 in MariaDB 10.4
●
MariaDB 10.4 RC has Galera 4 Replication
●
wsrep API #26
●
Impacts upgrading
●
wsrep patch integration in 10.4 codebase
●
Impacts code stability
●
bug fix & new features turnover

9. 9
www.galeracluster.com
Galera 4 in MariaDB 10.4
●
Streaming Replication
●
Support for large transactions
●
Platform for other new features
●
Group commit support
●
10.4 Backup locks
●
mariabackup SST with “light weight lock”

10. 10
Streaming Replication

11. 11
www.galeracluster.com
Streaming Replication
●
Originally developed for supporting huge
transactions
●
In Galera 3, transaction processes in master node
until commit time
●
For large transactions, the write size will be big,
and is hard to handle
●
There are means to prevent too large transactions
●
wsrep_max_ws_size

12. 12
www.galeracluster.com
Streaming Replication
●
Streaming replication is new technology
developed for Galera Replication 4 to enable
running transaction of unlimited size in cluster
●
Transaction size limits will remain, and cluster
can still reject too large transactions

13. 13
www.galeracluster.com
Streaming Replication
●
Transaction is replicated, gradually in small
fragments, during transaction processing
●
i.e. before actual commit, we replicate a number of small size
fragments
●
Size threshold for fragment replication is configurable
●
Replicated fragments are applied in slave threads
preserving transaction’s state in all cluster nodes
●
Fragments hold locks in all nodes and cannot be conflicted later

14. 14
www.galeracluster.com
Streaming Replication
Huge transaction
Galera Replication
Node A Node B
Update, update, update....
Begin
Trx

15. 15
www.galeracluster.com
Streaming Replication
Huge transaction
Galera Replication
Node A Node B
WS
Update, update, update....
Trx SR Trx

16. 16
www.galeracluster.com
Streaming Replication
Huge transaction
Galera Replication
Node A Node B
Update, update, update....
WS
Trx SR Trx

17. 17
www.galeracluster.com
Streaming Replication
Huge transaction
Galera Replication
Node A Node B
commit
WS
C
Trx SR Trx

18. 18
www.galeracluster.com
Streaming Replication
Huge transaction
Galera Replication
Node A Node B
OK

19. 19
www.galeracluster.com
Fragment Applying
SR transaction pool
SR#1 THD
WS
SR trx :2
CF: 0
applier
applier
certification
applier
apply
SR#2 THD
SR#n THD

20. 20
www.galeracluster.com
Fragment Applying
SR transaction pool
SR#1 THD
SR#2 THD
WS
SR trx :2
CF: 0
applier
applier
certification
applier
SR#n THD
apply Pull THD

21. 21
www.galeracluster.com
applier
Fragment Applying
SR transaction pool
SR#2 THDWS
SR trx :2
CF: 0
ev→apply_event()
…
ev->apply_event()
wsrep_SR_store->append_frag_apply())
SR#1 THD
SR#n THD
applier
applier
WS
SR trx :2
CF: 0
apply

22. 22
www.galeracluster.com
Fragment Applying
SR transaction pool
SR#1 THD
SR#2 THD
WS
SR trx :2
CF: 0
applier
applier
applier
SR#n THD
OK
Push THD

23. 23
www.galeracluster.com
Fragment Committing
SR transaction pool
SR#1 THD
SR#2 THD
WS
SR trx :2
CF: 1
applier
applier
certification
applier SR#n THD
commit Pull THD

24. 24
www.galeracluster.com
applier
Fragment Committing
SR#2 THD
WS
SR trx :2
CF: 1
trans_commit()
wsrep_SR_store->append_frag_commit())
SR transaction pool
SR#1 THD
SR#n THD
applier
applier
WS
SR trx :2
CF: 1
commit

25. 25
www.galeracluster.com
Fragment Committing
SR transaction pool
SR#1 THDapplier
applier
applier
SR#1nTHD
WS
SR trx :2
CF: 1
OK

26. 26
www.galeracluster.com
Configuring Streaming Replication
wsrep_trx_fragment_unit Unit metrics for fragmenting, options are:
●
bytes WS size in bytes
●
rows # of rows modified
●
statements # of SQL statements issued
wsrep_trx_fragment_size ●
Threshold size (in units), when fragment will be
replicated
●
0 = no streaming
Session variables and can be dynamically set

27. 27
www.galeracluster.com
Using Streaming Replication
●
Due to excessive logging and elevated
replication overhead, streaming replication will
cause degraded transaction throughput rate
●
Best use case is to use streaming replication for
cutting large transactions
●
Set fragment size to ~10K rows
●
Fragment variables are session variables and
can be dynamically set
●
Intelligent application can set streaming
replication on/off on need basis

28. 28
New Meta Data

29. 29
www.galeracluster.com
wsrep Tables in mysql database
MariaDB [(none)]> show tables in mysql like 'wsrep%';
+--------------------------+
| Tables_in_mysql (wsrep%) |
+--------------------------+
| wsrep_cluster |
| wsrep_cluster_members |
| wsrep_streaming_log |
+--------------------------+
3 rows in set (0.005 sec)

30. 30
www.galeracluster.com
wsrep Tables in mysql database
MariaDB [(none)]> select * from mysql.wsrep_clusterG
*************************** 1. row ***************************
cluster_uuid: 0be6f4d6-35da-11e9-b8a0-d6501fb08579
view_id: 2
view_seqno: 2
protocol_version: 4
capabilities: 184703
1 row in set (0.005 sec)

31. 31
www.galeracluster.com
wsrep Tables in mysql database
MariaDB [(none)]> select * from mysql.wsrep_cluster_membersG
*************************** 1. row ***************************
node_uuid: ea306990-35b0-11e9-9841-366f30fba24f
cluster_uuid: ea317655-35b0-11e9-9ac4-9f27c3510851
node_name: labrador
node_incoming_address: 127.0.0.1:16000
*************************** 2. row ***************************
node_uuid: eb29a68c-35b0-11e9-ac23-e2418aae257d
cluster_uuid: ea317655-35b0-11e9-9ac4-9f27c3510851
node_name: poodle
node_incoming_address: 127.0.0.1:16001
*************************** 3. row ***************************
node_uuid: eb29e56c-35b0-11e9-a328-fe229f8ae4cb
cluster_uuid: ea317655-35b0-11e9-9ac4-9f27c3510851
node_name: rotweiler
node_incoming_address: 127.0.0.1:16002
3 rows in set (0.006 sec)

32. 32
www.galeracluster.com
wsrep Tables in mysql database
MariaDB [(none)]> show create table mysql.wsrep_streaming_logG
*************************** 1. row ***************************
Table: wsrep_streaming_log
Create Table: CREATE TABLE `wsrep_streaming_log` (
`node_uuid` char(36) NOT NULL,
`trx_id` bigint(20) NOT NULL,
`seqno` bigint(20) NOT NULL,
`flags` int(11) NOT NULL,
`frag` longblob NOT NULL,
PRIMARY KEY (`node_uuid`,`trx_id`,`seqno`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.004 sec)

33. 33
Rolling Upgrade

34. 34
www.galeracluster.com
Galera 3
Galera Rolling Upgrades
Galera Replication
read & write
MariaDB
10.3
Galera 3
read & write
MariaDB
10.3
Galera 3
MariaDB
10.3
Wsrep-API 25
read & write

35. 35
www.galeracluster.com
Galera 3
Galera Rolling Upgrades
Galera Replication
read & write
MariaDB
10.3
Galera 3
read & write
MariaDB
10.3
Galera 3
MariaDB
10.3
Wsrep-API 25
Upgrade for 10.4
Shutdown server
Install 10.4
Start with wsrep_provider
mysql_upgrade
Shutdown server
Start without wsrep_provider
Isolate node
Galera 4
MariaDB
10.4
Allow read only access
read only

36. 36
www.galeracluster.com
Galera 3
Galera Rolling Upgrades
Galera Replication
read & write
MariaDB
10.3
Galera 3
MariaDB
10.3
Wsrep-API 25
Galera 4
MariaDB
10.4
read only
Galera 4
MariaDB
10.4
read only
Upgrade for 10.4
Upgrade node 2

37. 37
www.galeracluster.com
Galera Rolling Upgrades
Galera Replication Wsrep-API 25
Galera 4
MariaDB
10.4
read only
Galera 4
MariaDB
10.4
read only
Wsrep-API 26
Upgrade for 10.4
Shutdown node 1
Allow read-write
Complete node1 upgrade
wsrep API #26 features now
Enabled in replication

38. 38
www.galeracluster.com
Galera Rolling Upgrades
Galera Replication
read & write
Wsrep-API 25
Galera 4
MariaDB
10.4
Galera 4
MariaDB
10.4
Wsrep-API 26
read & write read & write
Galera 4
MariaDB
10.4

39. 39
www.galeracluster.com
Galera Rolling Upgrades
Galera Replication
read & write
Wsrep-API 25
Galera 4
MariaDB
10.4
Galera 4
MariaDB
10.4
Wsrep-API 26
read & write read & write
Galera 4
MariaDB
10.4 join
Galera 3
MariaDB
10.3

40. 40
Galera 4 Road Map

41. 41
www.galeracluster.com
4.0 Release Status
●
Part of Galera 4 feature set was merged in
MariaDB 10.4 beta 2 release and is now in the
10.4 RC and later releases
●
Schedule of remaining Galera 4 features is not
yet confirmed

42. 42
www.galeracluster.com
Features Missed in MariaDB 10.4
●
Gcache encryption
●
Data at rest encryption for full replication pipeline
●
XA transaction support
●
Will be needed in sharding cluster
●
Non blocking DDL
●
Co-incides with other MariaDB non-locking DDL work
●
Cluster error voting
●
Galera library feature, may be in later 10.4.* releases

43. 43
www.galeracluster.com
R
A
M
Data at rest encryption
Client
thread
Client
threadClient
thread
Buffer pool
gcache
ibdataibdata redo logredo log redo logbinlog
D
I
S
K

44. 44
www.galeracluster.com
R
A
M
Data at rest encryption
Client
thread
Client
threadClient
thread
Buffer pool
gcache
D
I
S
K
ibdataibdata redo logredo log redo logbinlog

45. 45
www.galeracluster.com
R
A
M
Data at rest encryption
Client
thread
Client
threadClient
thread
Buffer pool
gcache
D
I
S
K
ibdataibdata redo logredo log redo logbinlog

46. 46
www.galeracluster.com
Features Missed in MariaDB 10.4
●
Gcache encryption
●
Data at rest encryption for full replication pipeline
●
XA transaction support
●
Will be needed in sharding cluster
●
Non blocking DDL
●
Co-incides with other MariaDB non-locking DDL work
●
Cluster error voting
●
Galera library feature, may be in later 10.4.* releases

47. 47
XA Transactions
In Galera 3

48. 48
www.galeracluster.com
XA Transactions with Galera 3
TM
RM RM
XA start ‘foo’ XA start ‘bar’
foo bar
Work on ‘foo’ Work on ‘bar’XA prepare ‘foo’ XA prepare ‘bar’XA commit ‘foo’ XA commit ‘bar’
foo bar
MariaDB

49. 49
www.galeracluster.com
XA Transactions with Galera 3
●
Galera replication is eager to replicate in
MariaDB 2PC prepare stage
●
Prepared XA transaction cannot be rolled back
anymore

50. 50
www.galeracluster.com
XA Transactions with Galera 3
XA Start
Node A Node B
smith smith
XA trans

51. 51
www.galeracluster.com
XA Transactions with Galera 3
XA Insert into persons ‘’jones’
Node A Node B
smith smith
jones
XA trans

52. 52
www.galeracluster.com
XA Transactions with Galera 3
XA Prepare
Node A Node B
smith smith
jones
XA trans

53. 53
www.galeracluster.com
XA Transactions with Galera 3
XA Prepare
Node A Node B
smith smith
jones
XA trans
WS
jones

54. 54
www.galeracluster.com
XA Transactions with Galera 3
XA Prepare
Node A Node B
smith
smith
jones
jones
XA trans apply

55. 55
www.galeracluster.com
XA Transactions with Galera 3
OK
Node A Node B
smith
smith
jones
jones
XA trans

56. 56
www.galeracluster.com
XA Transactions with Galera 3
XA Rollback
Node A Node B
smith
smith
jones
jones
XA trans

57. 57
www.galeracluster.com
XA Transactions with Galera 3
Node A Node B
smith
smith
jonesinconsistency

58. 58
www.galeracluster.com
XA prepare Safety
●
XA prepare should guarantee that transaction
can commit
●
However, galera multi-master conflict resolving
does honor XA transactions

59. 59
www.galeracluster.com
XA Transactions with Galera 3
XA Start
Node A Node B
smith smith
XA trans

60. 60
www.galeracluster.com
XA Transactions with Galera 3
UPDATE smith=kit
Node A Node B
smith smith
XA trans
kit

61. 61
www.galeracluster.com
XA Transactions with Galera 3
Node A Node B
smith smith
XA trans
WS
kit
UPDATE smith=hannibal
WS
hannibal
XA Prepare
kit

62. 62
www.galeracluster.com
XA Transactions with Galera 3
Node A Node B
hannibal hannibal
XA trans
OK

63. 63
www.galeracluster.com
XA Transactions with Galera 3
Node A Node B
hannibal hannibal
XA Commit

64. 64
XA by
Streaming Replication

65. 65
www.galeracluster.com
XA Transaction Support
XA Start
Node A Node B
smith smith
XA trans

66. 66
www.galeracluster.com
XA Transaction Support
XA Insert into persons ‘’jones’
Node A Node B
smith smith
XA transXA transXA transXA trans
jones

67. 67
www.galeracluster.com
XA Transaction Support
XA Prepare
Node A Node B
smith smith
XA trans SR trans
WS
jonesjones

68. 68
www.galeracluster.com
XA Transaction Support
XA Rollback
Node A Node B
smith smith
XA trans SR trans
WS
rollback
jonesjones

69. 69
www.galeracluster.com
XA Transaction Support
Node A Node B
smith smith

70. 70
www.galeracluster.com
XA Transaction Conflict Resolution
Node A Node B
smith smith
XA trans
UPDATE smith=hannibalXA Prepare
kit
SR trans
kit

71. 71
www.galeracluster.com
Features Missed in MariaDB 10.4
●
Gcache encryption
●
Data at rest encryption for full replication pipeline
●
XA transaction support
●
Will be needed in sharding cluster
●
Non blocking DDL
●
Co-incides with other MariaDB non-locking DDL work
●
Cluster error voting
●
Galera library feature, may be in later 10.4.* releases

72. 72
www.galeracluster.com
MariaDB 10.5
●
Galera work for 10.5 is ongoing
●
Feature scoping is not complete, may come
with Galera 4 or new Galera 5

73. 73
www.galeracluster.com
Galera 4 on MySQL
●
Galera 4 is integrated in MySQL 5.6 and 5.7
and 8.0 versions
●
Public releases after MariaDB 10.4 GA is
released

74. 74
www.galeracluster.com
Summary
●
MariaDB 10.4.5 has significant new
Galera 4 features
●
Streaming Replication implements new replication
protocol opening possibilities for many unforeseen
replication features
●
Other new Galera 4 related features to come with
later 10.4 and 10.5 releases
●
MySQL version is coming
●
Rolling cluster upgrade supported and requires care

75. 75
4
Happy Clustering :-)
Thank you for listening!