Slides from a presentation given at Percona Live MySQL Conference 2013 in Santa Clara, US. Topics include: - How to look for performance bottlenecks - Foreign Key performance in MySQL Cluster 7.3 - Sharding and table partitioning - efficient use of datatypes (e.g. BLOBS vs varbinary)

1. Performance Tuning of MySQL Cluster
April 2013
Johan Andersson
Severalnines AB
johan@severalnines.com

2. Agenda
 7.3 Feature Update
 OS Tuning
 Stability Tuning
 Application design
 Identifying bottlenecks
 Tuning tricks
2
Copyright 2011 Severalnines AB

3. 7.3 Feature Update
 Node.js Connector
 JavaScript (V8 engine) to access data directly in the
Data nodes
 No SQL – bypasses the MySQL Server  lower latency,
high throughput for simple queries (like PK operations,
simple scans from one table)
3
Copyright 2011 Severalnines AB

4. 7.3 Feature Update
 FOREIGN KEYs finally supported!
 Implemented at the Data Node level
 But..
ERROR 1506 (HY000): Foreign key clause is not yet
supported in conjunction with partitioning
 Hopefully fixed for the GA release
 What about the performance penalty?
4
Copyright 2011 Severalnines AB

5. 7.3 Foreign Key Perf
create table users_posts (
uid integer ,
fid integer ,
pid integer auto_increment,
message varchar(1024),
primary key(uid,fid, pid),
constraint fk_forum foreign key(fid) references forum(fid) on delete cascade,
constraint fk_user foreign key(uid) references users(uid) on delete cascade
) engine=ndb;
 Compare INSERT performance with and w/o FKs
 With FK, must check that forum(fid) and users(uid) exists.
 Populate with 1M records
5
Copyright 2011 Severalnines AB

6. 7.3 Foreign Key Perf
 Bencher drivers the load:
 https://github.com/severalnines/bencher
 4 threads, 2 data nodes, 4 cores,
 App  mysqld  data nodes
 FOREIGN KEYs enabled
Summary:
--------------------------
Average Throughput = 1274.58 tps (stdev=59.71)
6
Copyright 2011 Severalnines AB

7. 7.3 Foreign Key Perf
 Bencher drivers the load:
 https://github.com/severalnines/bencher
 4 threads, 2 data nodes, 4 cores,
 App  mysqld  data nodes
 Not using FOREIGN KEYs
Summary:
--------------------------
Average Throughput = 1428.57 tps (stdev=57.10)
 Foreign keys gave ~11% drop in performance.
7
Copyright 2011 Severalnines AB

8. 7.2  7.3 Caveats
 Rolling upgrade from 7.2.10 to 7.3.1 works!
 A little gotcha:
 --engine-condition-pushdown  no longer supported in
MySQL 5.6
 Mysqld will fail to start
 Take it out from my.cnf before upgrading!
8
Copyright 2011 Severalnines AB

9. Facts
 A single query will never run as fast as on Innodb
(served from RAM)
 Network latency is a issue
 More data nodes does not speed up query execution
time.
9
Copyright 2011 Severalnines AB

10. OS Tuning
 Disable NUMA in /etc/grub.conf
 echo ‘0’ > /proc/sys/vm/swappiness
echo ‘vm.swappiness=0’ >> /etc/sysctl.conf
 Bind data node threads to CPUs/cores
 cat /proc/interrupts | grep eth
cpu0 cpu1 cpu2 cpu3
44: 31 49432584 0 0 xen-dyn-event eth0"
45: 1633715292 0 0 0 xen-dyn-event eth1"
10
Copyright 2011 Severalnines AB
Avoid!
OK!
In config.ini [ndbd default]:
ThreadConfig=ldm={count=1,cpubind=1,2},main={cpubind=3} ..

11. Stability Tuning
 Tuning the REDO log is key
 FragmentLogFileSize=256M
 NoOfFragmentLogFiles=<4-6> X DataMemory in MB / 4 x
FragmentLogFileSize
 RedoBuffer=64M for a write busy system
 Disk based data:
 SharedGlobalMemory=4096M
 In the LOGFILE GROUP: undo_buffer_size=128M
 Or higher (max is 600M)
11
Copyright 2011 Severalnines AB

12. Stability Tuning
 Make sure you don’t have more “execution threads” than cores
 You want to have
 Major page faults low
 Involuntary context switches low
mysql> SELECT node_id, thr_no,thr_nm , os_ru_majflt,
os_ru_nivcsw FROM threadstat;
+---------+--------+--------+--------------+--------------+
| node_id | thr_no | thr_nm | os_ru_majflt | os_ru_nivcsw |
+---------+--------+--------+--------------+--------------+
| 3 | 0 | main | 1 | 541719 |
| 4 | 0 | main | 0 | 561769 |
+---------+--------+--------+--------------+--------------+
2 rows in set (0.01 sec)
12
Copyright 2011 Severalnines AB

13. Application Design
 Define the most typical Use Cases
 List all my friends, session management etc etc.
 Optimize everything for the typical use case
 Engineer schema to cater for the Use Cases
 Keep it simple
 Complex access patterns does not scale
 Simple access patterns do ( Primay key and Partitioned Index Scans )
 Note! There is no parameter in config.ini that affects
performance – only availability.
 Everything is about the Schema and the Queries.
 Tune the mysql servers (sort buffers etc) as you would for innodb.
13
Copyright 2011 Severalnines AB

14. Simple Access
 PRIMARY KEY lookups are HASH lookup O(1)
 INDEX searches a T-tree and takes O(log n) time.
 In 7.2 and later JOINs are ok, but in 7.1 you should try
to avoid them.
14
Copyright 2011 Severalnines AB

15. Identifying Bottlenecks
 A lot of CPU is used on the data nodes
 Probably a lot of large index scans and full table scans are used.
 Check Slow query log or a query monitor
 A lot of CPU is used on the mysql servers
 Probably a lot of GROUP BY/DISTINCT or aggregate functions.
 Hardly no CPU is used on either mysql or data nodes
 Probably low load
 Time is spent on network (a lot of “ping pong” to satisfy a request).
 System is running slow in general
 Disks (io util), queries, swap (must never happen), network

16. Need To Add Data Nodes?
 (adding mysql servers is easy)
 top –Hd1
 Is any of data nodes threads at 100%?
 Yes: add more data nodes (online)
 No: do nothing

17. Detecting Query Problems
 Here is a standard method for how to attack the problem.
 Performance tuning is a never-ending loop:
BEGIN
– Capture information – e.g, slow query log
• Change long_query_time if needed
– EXPLAIN the queries
• What indexes are used?
• Are tables JOINed in the correct order (small to big)
– Re-run the optimized typical use cases using bencher/
mysqlslap
GOTO BEGIN;
END;
 Never tune unless you can measure and test!
 Don't optimize unless you have a problem!

18. Enable Logging
 Slow query log
 set global slow_query_log=1;
 set global long_query_time=0.01;
 set global log_queries_not_using_indexes=1;
 General log (if you don’t get enough info in the Slow
Query Log)
 Activate for a very short period of time (30-60seconds) –
intrusive
 Can fill up disk very fast – make sure you turn it off.
 set global general_log=1;
 Use Severalnines ClusterControl
 Includes a Cluster-wide Query Monitor.
 Query frequency, EXPLAINs, lock time etc.
 Performance Monitor and Manager.

19. Setup
19
Copyright 2011 Severalnines AB
subid data
1 A
3 B
2 C
4 D
subscriber
Partition 0
Partition 1
NETWORK!

20. Sharding
 By default, all index scans hit all data nodes
 good if result set is big – you want as many CPUs as possible to
help you.
 For smaller result sets (~a couple of hundred records) Partition
Pruning is key for scalability.
 User-defined partitioning can help to improve equality index
scans on part of a primary key.
 CREATE TABLE t1 (uid,
fid,
somedata,
PRIMARY KEY(uid, fid))
PARTITION BY KEY(userid);
 All data belonging to a particular uid will be on the same
partition.
 Great locality!
 select * from user where uid=1;
 Only one data node will be scanned (no matter how many
nodes you have)

21. Sharding
mysql> show global status like 'ndb_pruned_scan_count’;
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| Ndb_pruned_scan_count | 0 |
+-----------------------+-------+
CREATE TABLE t1( … ) PARTITION BY KEY (userid);
An run query, and verify it works:
select * from user where userid=1;
mysql> show global status like 'ndb_pruned_scan_count’;
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| Ndb_pruned_scan_count | 1 |
+-----------------------+-------+

22. Sharding
mysql> show global status like 'ndb%pruned%';
| Ndb_api_table_scan_count | 264 |
| Ndb_api_range_scan_count | 18 |
| Ndb_api_pruned_scan_count | 3 |

23. Sharding - EXAMPLE
 create table users_posts2 (
uid integer ,
fid integer ,
pid integer auto_increment,
message varchar(1024),
primary key(uid,fid, pid)
) engine=ndb
partition by key(uid);
NO PARTITIONING
create table users_posts2 (
uid integer ,
fid integer ,
pid integer auto_increment,
message varchar(1024),
primary key(uid,fid, pid)
) engine=ndb;
PARTITION BY KEY

24. Sharding – EXPLAIN PARTITIONS
mysql> explain partitions select * from users_posts u where u.uid=1G
id: 1
select_type: SIMPLE
table: u
partitions: p0,p1
type: ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 2699
Extra: NULL
With PARTITION BY KEY (UID)
mysql> explain partitions select * from users_posts2 u where
u.uid=1G
id: 1
select_type: SIMPLE
table: u
partitions: p0
type: ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 2699
Extra: NULL

25. Data Types
BLOBs/TEXTs vs VARBINARY/VARCHAR
 BLOB/TEXT columns are stored in an external hidden table.
 First 256B are stored inline in main table
 Reading a BLOB/TEXT requires two reads
 One for reading the Main table + reading from hidden
table
 Change to VARBINARY/VARCHAR if:
 Your BLOB/TEXTs can fit within an 14000 Bytes record
 (record size is currently 14000 Bytes)
 Reading/writing VARCHAR/VARBINARY is less expensive
Note 1: BLOB/TEXT are also more expensive in Innodb as BLOB/TEXT data is
not inlined with the table. Thus, two disk seeks are needed to read a
BLOB.
Note 2: Store images, movies etc outside the database on the filesystem.

26. Query Tuning
 MySQL Cluster 7.2 and later has pushed down joins  joins
are performed in the data nodes.
 OPTIMIZER in MySQL Cluster 7.1 and earlier is weak
 Statistics gathering is non-existing
 Optimizer thinks there are only 10 rows to examine in each
table!
 FORCE INDEX / STRAIGHt_JOIN to get queries run the way you
want

27. Query Tuning
 if you have two similar indexes:
 index(a)
 index(a,ts)
on the following table
CREATE TABLE `t1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`a` bigint(20) DEFAULT NULL,
`ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_t1_a` (`a`),
KEY `idx_t1_a_ts` (`a`,`ts`)) ENGINE=ndbcluster DEFAULT CHARSET=latin1

28. Query Tuning
mysql> select count(id) from t1 where a=5;
+-----------+
| count(id) |
+-----------+
| 3072000 |
+-----------+
1 row in set (0.02 sec)
mysql> select count(id) from t1 where a=5
and ts>'2013-04-18 14:34:08’;
+-----------+
| count(id) |
+-----------+
| 512 |
+-----------+
1 row in set (0.00 sec)

29. Query Tuning Pre 7.2
mysql> explain select * from t1 where a=2 and ts='2011-10-05 15:32:11';
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref |
rows | Extra |
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
| 1 | SIMPLE | t1 | ref | idx_t1_a,idx_t1_a_ts | idx_t1_a | 9 | const | 10 |
Using where |
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
 Use FORCE INDEX(..) ...
mysql> explain select * from t1 FORCE INDEX (idx_t1_a_ts) where a=2 and ts='2011-10-05
15:32:11;
+| 1 | SIMPLE | t1 | ref | idx_t1_a_ts | idx_t1_a_ts | 13 | const,const | 10 |
Using where |
1 row in set (0.00 sec)
 ..to ensure the correct index is picked!
 The difference can be 1 record read instead of any
number of records!

30. Index Statistics
explain select * from t1 where a=5 and ts>'2013-04-18 14:34:08' G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: t1
type: range
possible_keys: idx_t1_a,idx_t1_a_ts
key: idx_t1_a
key_len: 9
ref: const
Rows: 17
Extra: Using where with pushed condition

31. Index Statistics
mysql> analyze table t1;
+---------+---------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+---------+---------+----------+----------+
| test.t1 | analyze | status | OK |
+---------+---------+----------+----------+
1 row in set (3.40 sec)

32. Index Statistics
Mysql> explain select * from t1 where a=5
and ts>'2013-04-18 14:34:08' G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: t1
type: range
possible_keys: idx_t1_a,idx_t1_a_ts
key: idx_t1_a_ts
key_len: 13
ref: NULL
rows: 253
Extra: Using where with pushed condition; Using MRR
1 row in set (0.00 sec)

33. Ndb_cluster_connection_pool
 Problem:
 A Sendbuffer on the connection between mysqld and the
data nodes is protected by a Mutex.
 Connection threads in MySQL must acquire Mutex and the
put data in SendBuffer.
 Many threads gives more contention on the mutex
 Must scale out with many MySQL Servers.
 Workaround:
 Ndb_cluster_connection_pool (in my.cnf) creates more
connections from one mysqld to the data nodes
 Threads load balance on the connections gives less
contention on mutex which in turn gives increased scalabilty
 Less MySQL Servers needed to drive load!
 www.severalnines.com/cluster-configurator allows you to
specify the connection pool.

34. Ndb_cluster_connection_pool
 Gives atleast 70% better performance and a MySQL Server
that can scale beyond four database connections.
 Set Ndb_cluster_connection_pool=2x<CPU cores>
 It is a good starting point
 One free [mysqld] slot is required in config.ini for each
Ndb_cluster_connection.
 4 mysql servers,each with Ndb_cluster_connection_pool=8
requires 32 [mysqld] in config.ini
 Note that also memcached and node.js, cluster/j etc also has
the concept of the ndb_cluster_connection_pool.

35. Q&A
35
Copyright 2011 Severalnines AB

36. Resources
 MySQL Cluster Configurator
 www.severalnines.com/config
 MySQL Cluster Management + Monitoring
 www.severalnines.com/cmon
 MySQL Cluster Training Slides
 www.severalnines.com/mysql-cluster-training
 My Blog
 johanandersson.blogspot.com

37. Keep in touch…
 Facebook
 www.facebook.com/severalnines
 Twitter
 @severalnines
 Linked in:
 www.linkedin.com/company/severalnines

38. Thank you for your time!
johan@severalnines.com
38
Copyright 2011 Severalnines AB