Severalnines Self-Training: MySQL® Cluster - Part IV

MySQL Cluster Training
presented by severalnines.com

Address:
Contact: SeveralninesAB
Jean-Jérôme Schmidt c/o SICS, Box 1263
Email: services@severalnines.com Isafjordsgatan22
SE-164-29 Kista

Copyright 2011 Severalnines AB Control your database infrastructure 1

Introduction

• At Severalnines, we believe in sharing information
and knowledge; we all come from an open source
background
• We know a lot of things about MySQL Cluster and
think that MySQL Cluster is a great technology
• These free MySQL Cluster Training slides are a
contribution of ours to the knowledge and information
sharing that‟s common practice in the open source
community
• If you have any questions on these slides or would
like to book an actual training class, please contact
us at: services@severalnines.com


Training Slides - Concept

• Over the coming weeks we will be chronologically
releasing slides for the different sections of our
MySQL Cluster Training program on our website.
• The full agenda of the training with all of its modules
is outlined in the next slides so that you can see what
topics will be covered over the coming weeks.
• Particularly specialised topics such as Cluster/J or
NDB API are not fully covered in the slides. We
recommend our instructor-led training classes for
such topics.
• Please contact us for more details:
services@severalnines.com


Full Training Agenda (1/4)
• MySQL Cluster Introduction
– MySQL eco system
– Scale up, scale out, and sharding
– MySQL Cluster Architecture
– Use cases
– Features
– Node types and Roles
• Detailed Concepts
– Data Distribution
– Verifying data distribution
– Access Methods
– Partitioning
– Node failures and failure detection
– Network Partitioning
– Transactions and Locking
– Consistency Model
– Redo logging and Checkpointing
• Internals
– NDB Design Internals


Agenda (2/4)

• Installing MySQL Cluster
– Setting up MySQL Cluster
– Starting/stopping nodes
– Recovery and restarts
– Upgrading configuration
– Upgrading Cluster
• Performance Tuning (instructor-led only; contact us at services@severalnines.com)
– Differences compared to Innodb/MyISAM
– Designing efficient and fast applications
– Identifying bottlenecks
– Tweaking configuration (OS and MySQL Cluster)
– Query Tuning
– Schema Design
– Index Tuning


Agenda (3/4)

• Management and Administration
– Backup and Restore
– Geographical Replication
– Online and offline operations
– Ndbinfo tables
– Reporting
– Single user mode
– Scaling Cluster
• Disk Data
– Use cases
– Limitations
– Best practice configuration
• Designing a Cluster
– Capacity Planning and Dimensioning
– Hardware recommendations
– Best practice Configuration
– Storage calculations


Agenda (4/4)

• Resolving Issues
– Common problems
– Error logs and Tracefiles
– Recovery and Escalation procedures
• Connectivity Overview
– NDBAPI
– Cluster/J
– LDAP
• Severalnines Tools
– Monitoring and Management
– Benchmarking
– Sandboxes
– Configuration and capacity planning
• Conclusion


Agenda: Lab Exercises
(only applicable to instructor-led training classes)

• Lab Exercises
– Installing and Loading data into MySQL Cluster
– Starting/stopping nodes, recovery
– Query tuning
– Backup and Restore
– Configuration Upgrade

• Would you like to try something particular?
– This is possible too, speak with your instructor


Prerequisites
• Readers / Participants have understanding of SQL and basic database concepts.

• Laptops/PCs for hands-on exercises
• Linux: 1GB RAM
• Windows: 2GB RAM
• Approx. 20GB disk space and Virtualbox installed.
• Virtualbox can be downloaded for free at http://www.virtualbox.org/wiki/Downloads

• MySQL Cluster version 7.1 or later


4th Installment of the Severalnines
Cluster Self-Training

Part 2 : Detailed Concepts (ctd…)
Section 4


Topics covered in Section 4

• Recovery


Recovery


Recovery

• There are four types of Recovery in MySQL Cluster
– Node Recovery
• Fastest recovery option
– Initial Node Recovery
– System Restart
– Initial System Restart plus restoring backup
• We will go over how to perform recovery in the
administration part.


Node Recovery

• Recovery in MySQL Cluster is automatic
– In the simplest case – restart the failed node


Node Recovery

• Data Node 2 Crashes during LCP 1

LCP 0 Complete
LCP 1 Complete
LCP 1 Starts
time

DATA NODE 1 1A C 2B C 3F C 4D C 5G C 8H C 1X C
REDOLOG
subid data
subid data 1 A
LCP 1 A 2 B Updated
2 B 3 C 1A 1X!
4 D

DATA NODE 2 1A C 2B C 3F C 4D C
REDOLOG

subid data
DATA NODE 2
1 A
CRASHES JUST AFTER
2 B GCP 8


Node Recovery

• Data Node 2 will:
– Read in its last completed LCP (LCP 0)
– Read it its REDO LOG until GCP 8
• But Data Node 1 continued to live and has more
changes.
– These changes needs to be copied from Data Node 1 to
Data Node 2.

5G C 8H C 1X C

GCP 10 GCP 11 GCP 12


Node Recovery

• Data Node 2 sends a request to Data Node 1
– Send all data from GCP 8 and onwards
– Remember the record header contains when the record was
last modified.
• Data Node 1 starts a scan and scans partition by
partition, table by table.

Start scan
GCP subid data GCP subid data
12 1 AX 2 1 A
4 2 B 4 2 B
6 3 C 6 3 C
8 4 D 8 4 D
10 5 G
11 8 H


Node Recovery

• As soon as the Copying starts, Data node 2 will take
part of the 2PC protocol
– But TC is inactive until recovery is complete.
• This means that data already copied will be changed
to
• UPDATE subsriber SET data=„AAA‟ WHERE subid=3
• INSERT INTO subscriber(subid, data) VALUES(10, „BB‟)

Start scan GCP subid data GCP subid data
COPY
12 1 AX 12 1 A X
4 2 B 4 2 B
2PC
32 3 AAA 32 3 AAA
8 4 D 8 4 D
10 5 G 10 5 G
11 8 H 11 8 H
24 10 BB


Node Recovery

• As soon as the Copying starts, Data node 2 will take
part of the 2PC protocol
– But TC is inactive until recovery is complete.
• This means that data already copied will be changed
too.
• UPDATE subsriber SET data=„AAA‟ WHERE subid=3
• INSERT INTO subscriber(subid, data) VALUES(10, „BB‟)

Start scan GCP subid data GCP subid data
COPY
12 1 AX 12 1 A X
4 2 B 4 2 B
2PC
32 3 AAA 32 3 AAA
8 4 D 8 4 D
COPY
10 5 G 10 5 G
COPY
11 8 H 11 8 H
COPY
24 10 BB 24 10 BB
End scan

Node Recovery

• Scan completed on Node 1 of all partitions for all
tables.
• TC on Node 2 becomes Active

DATA DATA
NODE 1 NODE 2

GCP subid data GCP subid data
12 1 AX 12 1 A X
4 2 B 4 2 B
6 3 AAA 6 3 AAA
8 4 D 8 4 D
10 5 G 10 5 G
11 8 H 11 8 H
24 10 BB 24 10 BB


Node Failures and Error Detection


Node Failures

• What happens if Data Node 2 fails?
• Node failure detected. Fail-over of primary replica F1
to node 2
– Fail-over time dependent on underlying OS and
configuration (e.g. OSE Delta 10 ms, Linux 100 ms)
STORAGE LAYER subid data
1 A
Partition 0
2 B
DATA DATA DATA DATA
NODE 1 NODE 2 NODE 3 NODE 4 3 C
Partition 1
4 D

P0 P1 P2 P3 5 E
Partition 2
6 F
S1 S0 S3 S2
7 G
Partition 3
Node group 0 Node group 1 8 H

Px == PRIMARY Partition x
Sx == SECONDARY Parttionx

Node Failures – Data Node 2 Failed

• Data Node 1 activates S1 and is now the PRIMARY
for Data Node 2‟s partitions.
• TC on Data Node 1 will take over any outstanding
transactions of Node 2.

1 A
Partition 0
2 B
DATA DATA DATA DATA
Partition 1
4 D

P0 P1 P2 P3 5 E
Partition 2
6 F
S1  P1 S0 S3 S2
7 G
Partition 3


Node Failures – Data Node 3 Failed

• Data Node 4 activates S2 and is now the PRIMARY
for Data Node 3‟s partitions.

1 A
Partition 0
2 B
DATA DATA DATA DATA
Partition 1
4 D

P0 P1 P2 P3 5 E
Partition 2
6 F
S1  P1 S0 S3 S2P2
7 G
Partition 3


Node Failures

• After recovery the role of the partitions are toggled
back.

1 A
Partition 0
2 B
DATA DATA DATA DATA
Partition 1
4 D

P0 P1 P2 P3 5 E
Partition 2
6 F
P1S1 S0 S3 P2S2
7 G
Partition 3


If a Nodegroup goes down

• Cluster goes down
• Partition 2 and Partition 3 are lost, and all Partitions
must be present to have a functioning Cluster.

1 A
Partition 0
2 B
DATA DATA DATA DATA
Partition 1
4 D

P0 P1 P2 P3 5 E
Partition 2
6 F
P1S1 S0 S3 P2S2
7 G
Partition 3



• There are two ways the data nodes can detect that
another node has failed.
– Socket to remote host was closed (TCP Disconnect)
– Using Heartbeats



• There are two ways the data nodes can detect that
another node has failed.
– TCP Disconnect
– Heartbeats
• Internally on each node a Watchdog ensures that a
data node does not get stuck:
– Monitors that a node does not end up in an eternal loop
– Protects against software bugs
• Missed Heartbeats are typically because of
– Data node is swapping
– Computer is using CPU for something else
– Unreliable network
– Bug in your NDBAPI application

Heartbeat Circle – Normal Operation
• HearbeatIntervalDbDb regulates how often hearbeat checks are
made. Default: 5000ms, Recommended in Telco: 1500ms
• After 4xHearbeatIntervalDbDb node is declared dead

STORAGE LAYER

DATA DATA
DATA
OK
DATA
OK NODE 3
OK NODE 4
NODE 1 NODE 2

OK


Missed Heartbeats

• Missed Heartbeats are typically because of
– Data node is swapping
– Computer is using CPU for something else
– Unreliable network
– Bug in your NDBAPI application


Heartbeat Circle – Node 3 misses HB
• Node 3 has missed 4 heart beats from Node 2
• Node 2 declares Node 3 as dead.

STORAGE LAYER

DATA DATA DATA
DATA
NODE 1
OK NODE 2 NODE 3 NODE 4

OK


Node 3 Declared Dead
• Node 3 is excluded and a new heartbeat circle is formed.

STORAGE LAYER
DATA
NODE 3

DATA DATA
DATA
NODE 1
OK NODE 2 OK NODE 4

OK


Node 3 Rejoins Cluster
• When Node 3 is restarted on of the first things to happens is that
it rejoins the heartbeat circle.

STORAGE LAYER

DATA DATA
DATA
OK
DATA
OK NODE 3
OK NODE 4
NODE 1 NODE 2

OK


Network Partitioning

• The Data Nodes are separated from each other and
form two “Clusters”.
– Also known as Split Brain.

DATA DATA
NODE 1 NODE 2

P0 P1
S1 S0



• An Abitrator must be asked for advice.

Management
Server

“Am I the chosen one?” “Am I the chosen one?”

DATA DATA
NODE 1 NODE 2

P0 P1
S1 S0


Arbitrator

• By default the Management Servers are arbitrators.
– ArbitrationRank=[0 | 1 | 2 ] regulates this
• 0 – disabled
• 1 and 2 are priority levels
– ArbitrationTimeout=1000ms
• Data nodes wait 1000ms for reply from arbitrator. If exceeded,
data node will shutdown.
– Arbitration = [ Default | Disabled | WaitExternal]
• WaitExternal – your own application can do Arbitration.
• The data nodes agrees on which arbitrator to use at
Startup
– This decision may never change unless the Arbitrator dies
– The data nodes will in that case re-elect another arbitrator.



• The elected Arbitrator is asked for advice.

Management Management
Server 1 Server 2


DATA DATA
NODE 1 NODE 2

P0 P1
S1 S0



• Only one set of nodes will get the positive answer
• The nodes getting “NO” will shutdown.

Server 1 Server 2

Yes
NO!!!

DATA DATA
NODE 1 NODE 2

P0 P1
S1 S0



• Nodes that cannot reach the Arbitrator will shutdown

Server 1 Server 2


DATA DATA
NODE 1 NODE 2

P0 P1
S1 S0


Arbitration

• Arbitration is only used if there are equal splits.
• At least three computers must be present to handle
network partitioning
• The data nodes will handle majority groups by
themselves.
– 2 nodes on one side and 4 nodes on the other.
– The largest side has the majority.


Two computer setup

• Requirements:
– Use redundant switches
– Use bonding
• Eth0 to switch 0
• Eth1 to switch 1
– Set Arbitration=Disabled
• This is a pretty solid setup – chances that two
switches crashes at the same time is small
• In that case you likely have a bigger problem in your
data center.


Coming next in Section 5:

Internals

NDB Design Internals


We hope these training slides are
useful to you!

Please visit our website to view the
next section of this training.

For any questions, comments, feedback or to
book a training class, please contact us at:

services@severalnines.com

Thank you!


Disclaimer

© Copyright 2011 Severalnines AB. All rights reserved.

Severalnines& the Severalnineslogo(s) are trademarks of Severalnines AB.

MySQL is a registered trademark of Oracle and/or its affiliates.

Other names may be trademarks of their respective owners.


Severalnines Self-Training: MySQL® Cluster - Part IV

Recommended

Recommended

More Related Content

More from Severalnines

More from Severalnines (20)

Recently uploaded

Recently uploaded (20)

Severalnines Self-Training: MySQL® Cluster - Part IV

Editor's Notes