In this blog post, we have multiple OLTP performance benchmarking scenarios using sysbench 1.0.14 on InnoDB and MyRocks. InnoDB and MyRocks (RocksDB with MySQL) are definitely not to supplement each other, They actually compliment well with respective advantages, Let me quickly explain how InnoDB and MyRocks can benefit you when used wisely, Again this blog post is not to show who (InnoDB or MyRocks) is better ? We regularly benchmark both of these storage engines before recommending to our customers on what is best suited for their database infrastructure operations ? so we would like to share our thoughts on this post.
Table of Contents
How InnoDB and MyRocks are different ?
- MyRocks supports only READ-COMMITTED isolation level, There is no REPEATABLE-READ isolation level like InnoDB so no gap locking like InnoDB, We have written detailed blog on InnoDB transaction isolation levels here
- To get an verbose information about MyRocks instance, the log is located in ” /var/lib/mysql/#rocksdb ” . Much more detailed story about your RocksDB diagnostics report can be generated with the command SHOW ENGINE ROCKSDB STATUS , It really takes good amount of time to understand and interpret MyRocks operations matrices.
- In MyRocks, you have rocksdb_block_cache_size system variable which is somewhat similar to innodb_buffer_pool_size but It’s mainly beneficial for reads. By default it uses buffered reads and OS cache contains cached compressed data and RockDB block cache will contain uncompressed data. You can have two levels of cache or disable buffering by forcing block cache to use direct reads with configuration rocksdb_use_direct_reads=ON.
- LSM Data Structure – MyRocks is not an alternative or advanced version of InnoDB, LSM data structure is great for write-intensive database operations, reads will be slow and full table scans are too expensive. so InnoDB and RocksDB together makes an great combination !
Benchmarking InnoDB and MyRocks performance with sysbench 1.0.14 for OLTP operations
Linux – CentOS Linux release 7.3.1611 (Core)
Database infrastructure – MariaDB 10.3.7
Building database infrastructure for benchmarking
We have used “oltp_common.lua” script to create database infrastructure for benchmarking InnoDB and MyRocks, We have not tuned both InnoDB and MyRocks variables for performance. The script below creates database for benchmarking:
[root@localhost sysbench]# [root@localhost sysbench]# sysbench bulk_insert.lua --threads=1 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Creating table 'sbtest1'... [root@localhost sysbench]#
Benchmarking bulk INSERT performance on InnoDB and MyRocks
Benchmarking OLTP insert on InnoDB using “oltp_insert.lua”
Script to create data (5M records) for benchmarking OLTP INSERT performance:
[root@localhost sysbench]# sysbench oltp_insert.lua --threads=100 --time=180 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest1'... Inserting 5000000 records into 'sbtest1' Creating a secondary index on 'sbtest1'... [root@localhost sysbench]#
MariaDB [test]> show table status like 'sbtest1'\G; *************************** 1. row *************************** Name: sbtest1 Engine: InnoDB Version: 10 Row_format: Dynamic Rows: 5404891 Avg_row_length: 265 Data_length: 1433403392 Max_data_length: 0 Index_length: 157024256 Data_free: 4194304 Auto_increment: 5696281 Create_time: 2018-06-03 12:48:12 Update_time: 2018-06-03 12:52:03 Check_time: NULL Collation: latin1_swedish_ci Checksum: NULL Create_options: Comment: Max_index_length: 0 Temporary: N 1 row in set (0.000 sec) ERROR: No query specified MariaDB [test]>
Script for benchmarking InnoDB OLTP INSERT performance:
[root@localhost sysbench]# sysbench oltp_insert.lua --threads=100 --time=180 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 run sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 100 Initializing random number generator from current time Initializing worker threads... Threads started! SQL statistics: queries performed: read: 0 write: 696280 other: 0 total: 696280 transactions: 696280 (3866.32 per sec.) queries: 696280 (3866.32 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 180.0872s total number of events: 696280 Latency (ms): min: 0.62 avg: 25.85 max: 358.63 95th percentile: 81.48 sum: 17998504.11 Threads fairness: events (avg/stddev): 6962.8000/57.61 execution time (avg/stddev): 179.9850/0.04
What we look for seriously in this benchmarking is QPS (queries per seconds) , In the test above it is 3866 QPS
Benchmarking MyRocks INSERT performance using Sysbench 1.0.14:
The steps are same, except for explicitly mentioning the storage engine RocksDB in sysbench scripts: “–mysql-storage-engine=rocksdb”
Script for benchmarking OLTP insert on MyRocks using “oltp_insert.lua” :
[root@localhost sysbench]# sysbench oltp_insert.lua --threads=100 --time=180 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest1'... Inserting 5000000 records into 'sbtest1' Creating a secondary index on 'sbtest1'... [root@localhost sysbench]#
MariaDB [test]> show table status like 'sbtest1%'\G; *************************** 1. row *************************** Name: sbtest1 Engine: ROCKSDB Version: 10 Row_format: Fixed Rows: 5000000 Avg_row_length: 198 Data_length: 992949774 Max_data_length: 0 Index_length: 38739880 Data_free: 0 Auto_increment: 5000001 Create_time: NULL Update_time: NULL Check_time: NULL Collation: latin1_swedish_ci Checksum: NULL Create_options: Comment: Max_index_length: 0 Temporary: N 1 row in set (0.007 sec) ERROR: No query specified MariaDB [test]>
Script for benchmarking RocksDB OLTP INSERT performance:
[root@localhost sysbench]# sysbench oltp_insert.lua --threads=100 --time=180 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb run sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 100 Initializing random number generator from current time Initializing worker threads... Threads started! SQL statistics: queries performed: read: 0 write: 123049 other: 0 total: 123049 transactions: 123049 (683.37 per sec.) queries: 123049 (683.37 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 180.0618s total number of events: 123049 Latency (ms): min: 1.03 avg: 146.30 max: 1675.09 95th percentile: 308.84 sum: 18001689.44 Threads fairness: events (avg/stddev): 1230.4900/26.26 execution time (avg/stddev): 180.0169/0.02 [root@localhost sysbench]#
Result: OLTP INSERT performance for RocksDB is 683 QPS (queries per second)
The graphical representation of more interesting performance benchmarking results on multiple bulk INSERT transactions scenarios:
Conclusion
OLTP INSERT performance benchmarking clearly proves InnoDB is almost 6X faster than MyRocks. So MyRocks is not recommended for bulk INSERT transactions !
OLTP WRITE only transactions performance benchmarking for InnoDB and MyRocks
OLTP WRITE only performance benchmarking for InnoDB using “oltp_write_only.lua”
The steps remains same for “oltp_write_only.lua” script like “oltp_insert.lua” mentioned above, So we are directly copying the results of benchmarking without explaining details again:
[root@localhost sysbench]# sysbench oltp_write_only.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest1'... Inserting 5000000 records into 'sbtest1' Creating a secondary index on 'sbtest1'...
[root@localhost sysbench]# sysbench oltp_write_only.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 run sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 100 Initializing random number generator from current time Initializing worker threads... Threads started! SQL statistics: queries performed: read: 0 write: 14529 other: 7265 total: 21794 transactions: 3632 (355.03 per sec.) queries: 21794 (2130.37 per sec.) ignored errors: 1 (0.10 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.2285s total number of events: 3632 Latency (ms): min: 1.88 avg: 277.61 max: 2701.56 95th percentile: 977.74 sum: 1008267.12 Threads fairness: events (avg/stddev): 36.3200/4.36 execution time (avg/stddev): 10.0827/0.09
Result : 2130 QPS (queries per second)
OLTP WRITE only performance benchmarking for RocksDB using “oltp_write_only.lua”
[root@localhost sysbench]# sysbench oltp_write_only.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest1'... Inserting 5000000 records into 'sbtest1' Creating a secondary index on 'sbtest1'...
[root@localhost sysbench]# sysbench oltp_write_only.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb run sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 100 Initializing random number generator from current time Initializing worker threads... Threads started! SQL statistics: queries performed: read: 0 write: 25191 other: 12596 total: 37787 transactions: 6296 (625.73 per sec.) queries: 37787 (3755.49 per sec.) ignored errors: 4 (0.40 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0603s total number of events: 6296 Latency (ms): min: 1.39 avg: 159.29 max: 3620.58 95th percentile: 846.57 sum: 1002895.84 Threads fairness: events (avg/stddev): 62.9600/25.26 execution time (avg/stddev): 10.0290/0.02
Result : 3755 QPS (queries per second)
The graphical representation of more interesting performance benchmarking results on multiple WRITE only transactions scenarios:
Conclusion
MyRocks OLTP write only performance is almost 2X compared to InnoDB, So MyRocks is definitely an preferred option for high performance and scalable writes, Thanks to LSM data structure !
OLTP READ-WRITE performance benchmarking using Sysbench lua script “oltp_read_write.lua”
The lua scripts below create data for OLTP READ-WRITE performance benchmarking:
Benchmarking OLTP READ-WRITE performance for InnoDB:
[root@localhost sysbench]# sysbench oltp_read_write.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest1'... Inserting 5000000 records into 'sbtest1' Creating a secondary index on 'sbtest1'...
[root@localhost sysbench]# sysbench oltp_read_write.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 run sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 100 Initializing random number generator from current time Initializing worker threads... Threads started! SQL statistics: queries performed: read: 15652 write: 4472 other: 2236 total: 22360 transactions: 1118 (105.96 per sec.) queries: 22360 (2119.20 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.5301s total number of events: 1118 Latency (ms): min: 15.71 avg: 922.19 max: 4973.09 95th percentile: 2009.23 sum: 1031006.57 Threads fairness: events (avg/stddev): 11.1800/1.68 execution time (avg/stddev): 10.3101/0.13 [root@localhost sysbench]#
Result: 2119 QPS (queries per second)
Benchmarking OLTP READ-WRITE performance for RocksDB:
[root@localhost sysbench]# sysbench oltp_read_write.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest1'... Inserting 5000000 records into 'sbtest1' Creating a secondary index on 'sbtest1'...
[root@localhost sysbench]# sysbench oltp_read_write.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb run sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 100 Initializing random number generator from current time Initializing worker threads... Threads started! SQL statistics: queries performed: read: 26964 write: 7628 other: 3827 total: 38419 transactions: 1901 (182.46 per sec.) queries: 38419 (3687.46 per sec.) ignored errors: 25 (2.40 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.4153s total number of events: 1901 Latency (ms): min: 11.23 avg: 540.87 max: 3480.91 95th percentile: 1352.03 sum: 1028196.02 Threads fairness: events (avg/stddev): 19.0100/2.25 execution time (avg/stddev): 10.2820/0.10 [root@localhost sysbench]#
Result: 3687 QPS (queries per second)
The graphical representation of more interesting performance benchmarking results on multiple READ-WRITE transactions scenarios:
Conclusion
OLTP READ-WRITE I/O operations benchmarking results confirm MyRocks is the definite choice, May be these result vary more if we invest in tuning the InnoDB and MyRocks for performance.
Benchmarking OLTP READ ONLY operations using Sysbench oltp_read_only.lua script
OLTP READ ONLY transactions performance benchmarking for InnoDB:
[root@localhost sysbench]# sysbench oltp_read_only.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest1'... Inserting 5000000 records into 'sbtest1' Creating a secondary index on 'sbtest1'...
[root@localhost sysbench]# sysbench oltp_read_only.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 run sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 100 Initializing random number generator from current time Initializing worker threads... Threads started! SQL statistics: queries performed: read: 51072 write: 0 other: 7296 total: 58368 transactions: 3648 (352.59 per sec.) queries: 58368 (5641.45 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.3436s total number of events: 3648 Latency (ms): min: 1.10 avg: 274.41 max: 2863.46 95th percentile: 733.00 sum: 1001047.53 Threads fairness: events (avg/stddev): 36.4800/6.57 execution time (avg/stddev): 10.0105/0.05
Result: 5641 QPS (queries per second)
OLTP READ ONLY transactions performance benchmarking for RocksDB:
[root@localhost sysbench]# sysbench oltp_read_only.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest1'... Inserting 5000000 records into 'sbtest1' Creating a secondary index on 'sbtest1'...
[root@localhost sysbench]# sysbench oltp_read_only.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb run sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 100 Initializing random number generator from current time Initializing worker threads... Threads started! SQL statistics: queries performed: read: 26362 write: 0 other: 3766 total: 30128 transactions: 1883 (182.77 per sec.) queries: 30128 (2924.32 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.2983s total number of events: 1883 Latency (ms): min: 6.20 avg: 540.16 max: 5258.04 95th percentile: 2045.74 sum: 1017118.10 Threads fairness: events (avg/stddev): 18.8300/2.29 execution time (avg/stddev): 10.1712/0.08
Result: 2924 QPS (queries per second)
The graphical representation of more interesting performance benchmarking results on multiple READ only transactions scenarios:
Conclusion
InnoDB works great if it is OLTP READ only transactions, So we can continue recommending customers to use InnoDB for read intensive database operations.
Benchmarking OLTP DELETE ONLY operations using Sysbench oltp_delete.lua script
OLTP DELETE ONLY transactions performance benchmarking for InnoDB:
[root@localhost sysbench]# sysbench oltp_delete.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest1'... Inserting 5000000 records into 'sbtest1' Creating a secondary index on 'sbtest1'...
[root@localhost sysbench]# sysbench oltp_delete.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 run sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 100 Initializing random number generator from current time Initializing worker threads... Threads started! SQL statistics: queries performed: read: 0 write: 21659 other: 4464 total: 26123 transactions: 26123 (2521.93 per sec.) queries: 26123 (2521.93 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.3568s total number of events: 26123 Latency (ms): min: 0.04 avg: 38.08 max: 2679.06 95th percentile: 116.80 sum: 994654.43 Threads fairness: events (avg/stddev): 261.2300/46.27 execution time (avg/stddev): 9.9465/0.18 [root@localhost sysbench]#
Result: 2521 QPS (queries per second)
OLTP DELETE ONLY transactions performance benchmarking for RocksDB:
[root@localhost sysbench]# sysbench oltp_delete.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb prepare sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest1'... Inserting 5000000 records into 'sbtest1' Creating a secondary index on 'sbtest1'...
[root@localhost sysbench]# sysbench oltp_delete.lua --threads=100 --table-size=5000000 --db-driver=mysql --mysql-db=test --mysql-socket=/var/lib/mysql/mysql.sock --mysql-user=root --mysql-password=MyPassword2018 --mysql-storage-engine=rocksdb run sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 100 Initializing random number generator from current time Initializing worker threads... Threads started! SQL statistics: queries performed: read: 0 write: 7094 other: 421 total: 7515 transactions: 7515 (746.81 per sec.) queries: 7515 (746.81 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0613s total number of events: 7515 Latency (ms): min: 0.43 avg: 133.42 max: 1666.25 95th percentile: 502.20 sum: 1002663.49 Threads fairness: events (avg/stddev): 75.1500/15.50 execution time (avg/stddev): 10.0266/0.02
Result: 746 QPS (queries per second)
The graphical representation of more interesting performance benchmarking results on multiple DELETE only transactions scenarios:
Conclusion
InnoDB is almost 3X faster than MyRocks in OLTP DELETE operations.
InnoDB and MyRocks performance comparison :
- InnoDB is the definite choice if the transaction model is bulk INSERT, READ and DELETE intensive database operations.
- MyRocks performance is much better than InnoDB in WRITE and READ-WRITE intensive database operations.