欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识！
您现在的位置是： 首页  >  IT编程

MySQL 8.0 information_schema系统库的改进

程序员文章站 2022-05-29 10:19:13
[toc] information_schema有何用？ 通过I_S获取mysql的一些元数据信息 获取表的数据文件、索引文件的大小、碎片情况、表行数、自增列增长情况等 获取正在运行的事务有那些，是否有阻塞等 获取当前mysql的连接processlist等等 mysql8.0 之前的查询方式 会在 ......

目录

information_schema有何用？

  • 通过i_s获取mysql的一些元数据信息
  • 获取表的数据文件、索引文件的大小、碎片情况、表行数、自增列增长情况等
  • 获取正在运行的事务有那些，是否有阻塞等
  • 获取当前mysql的连接processlist等等

mysql8.0 之前的查询方式

会在查询information_schema 某个表时创建临时表

  • 来自文件的元数据，扫描文件系统获取frm文件的表定义
  • 存储引擎的详细信息，例如动态表统计信息
  • 来自mysql服务器中全局数据结构的数据

在表的数量很多时，每次查询i_s会从文件系统中读取每个单独的frm文件，使用更多的cpu周期来打开表并准备相关的内存数据结构

mysql8.0 开始的查询方式

  • 引入了基于innodb的本地数据字典表
  • 表中不在有frm表定义文件
  • 所有数据库表的元数据都存储在事务数据字典表中
  • i_s中表被设计为数据字典表上的view（有些还是临时表0_0）

消除了以下成本
查询information_schema时创建的临时表
扫描文件系统目录以查找frm文件

改进
利用mysql优化器的全部功能，使用数据字典表上的索引来更好的查询

MySQL 8.0 information_schema系统库的改进

mysql5.7中表文件
ll test*
jul 10 10:52 testse.frm
jul 10 10:52 testse.ibd

mysql8.0中表文件

ll test*
jul 10 10:25 testse.ibd

mysql5.7 和mysql8.0 i_s 中tables 表的存在方式

mysql5.7.22 
show create table information_schema.tables\g
*************************** 1. row ***************************
       table: tables
create table: create temporary table `tables` (
  `table_catalog` varchar(512) not null default '',
  `table_schema` varchar(64) not null default '',
  `table_name` varchar(64) not null default '',
  `table_type` varchar(64) not null default '',
  `engine` varchar(64) default null,

mysql8.0.15
  show create table information_schema.tables\g
*************************** 1. row ***************************
                view: tables
         create view: create algorithm=undefined definer=`mysql.infoschema`@`localhost` sql security definer view `
  • mysql5.7 中i_s中tables表是以临时表的形式存在的（查询该表就会创建临时表，创建的临时表过多，可能会导致mysql占用的内存暴涨，出现oom）
  • mysql8.0 中i_s中tables表以视图的形式存在（查询该视图，不会创建临时表，会使用到视图中表的索引）

mysql5.7中获取表大小情况

select   table_name,   concat(format(data_length / 1024 / 1024, 2),   'm') as dbdata_size,   concat(format(index_length / 1024 / 1024, 2),   'm') as dbindex_size,   concat(     format((data_length + index_length) / 1024 / 1024 / 1024,2),   'g') as `db_size(g)`,   avg_row_length,   table_rows,   update_time from   information_schema.tables where table_schema = 'test'   and table_name = 'testse';
+------------+-------------+--------------+------------+----------------+------------+---------------------+
| table_name | dbdata_size | dbindex_size | db_size(g) | avg_row_length | table_rows | update_time         |
+------------+-------------+--------------+------------+----------------+------------+---------------------+
| testse     | 0.02m       | 0.02m        | 0.00g      |            862 |         19 | 2019-07-10 10:52:02 |
+------------+-------------+--------------+------------+----------------+------------+---------------------+


执行计划中出现了where，未用到索引（存储引擎检索数据后，server层进行过滤）
desc select   table_name,   concat(format(data_length / 1024 / 1024, 2),   'm') as dbdata_size,   concat(format(index_length / 1024 / 1024, 2),   'm') as dbindex_size,   concat(     format(data_length + index_length / 1024 / 1024 / 1024,     2   ),   'g') as `db_size(g)`,   avg_row_length,   table_rows,   update_time from   information_schema.tables where table_schema = 'test'   and table_name = 'testse';
+----+-------------+--------+------------+------+---------------+-------------------------+---------+------+------+----------+---------------------------------------------------+
| id | select_type | table  | partitions | type | possible_keys | key                     | key_len | ref  | rows | filtered | extra                                             |
+----+-------------+--------+------------+------+---------------+-------------------------+---------+------+------+----------+---------------------------------------------------+
|  1 | simple      | tables | null       | all  | null          | table_schema,table_name | null    | null | null |     null | using where; open_full_table; scanned 0 databases |
+----+-------------+--------+------------+------+---------------+-------------------------+---------+------+------+----------+---------------------------------------------------+

mysql8.0中获取表大小情况

select   table_name,   concat(format(data_length / 1024 / 1024, 2),   'm') as dbdata_size,   concat(format(index_length / 1024 / 1024, 2),   'm') as dbindex_size,   concat(     format((data_length + index_length) / 1024 / 1024 / 1024,2),   'g') as `db_size(g)`,   avg_row_length,   table_rows,   update_time from   information_schema.tables where table_schema = 'test'   and table_name = 'testse';
+------------+-------------+--------------+------------+----------------+------------+---------------------+
| table_name | dbdata_size | dbindex_size | db_size(g) | avg_row_length | table_rows | update_time         |
+------------+-------------+--------------+------------+----------------+------------+---------------------+
| testse     | 0.02m       | 0.02m        | 0.00g      |            862 |         19 | 2019-07-10 10:25:16 |
+------------+-------------+--------------+------------+----------------+------------+---------------------+



能使用到数据字典表的索引
desc select   table_name,   concat(format(data_length / 1024 / 1024, 2),   'm') as dbdata_size,   concat(format(index_length / 1024 / 1024, 2),   'm') as dbindex_size,   concat(     format((data_length + index_length) / 1024 / 1024 / 1024,2),   'g') as `db_size(g)`,   avg_row_length,   table_rows,   update_time from   information_schema.tables where table_schema = 'test'   and table_name = 'testse';
+----+-------------+-------+------------+--------+--------------------+------------+---------+-------------------------+------+----------+-------------+
| id | select_type | table | partitions | type   | possible_keys      | key        | key_len | ref                     | rows | filtered | extra       |
+----+-------------+-------+------------+--------+--------------------+------------+---------+-------------------------+------+----------+-------------+
|  1 | simple      | cat   | null       | index  | primary            | name       | 194     | null                    |    1 |   100.00 | using index |
|  1 | simple      | sch   | null       | eq_ref | primary,catalog_id | catalog_id | 202     | mysql.cat.id,const      |    1 |   100.00 | using index |
|  1 | simple      | tbl   | null       | eq_ref | schema_id          | schema_id  | 202     | mysql.sch.id,const      |    1 |   100.00 | using where |
|  1 | simple      | stat  | null       | const  | primary            | primary    | 388     | const,const             |    1 |   100.00 | null        |
|  1 | simple      | ts    | null       | eq_ref | primary            | primary    | 8       | mysql.tbl.tablespace_id |    1 |   100.00 | null        |
|  1 | simple      | col   | null       | eq_ref | primary            | primary    | 8       | mysql.tbl.collation_id  |    1 |   100.00 | using index |
+----+-------------+-------+------------+--------+--------------------+------------+---------+-------------------------+------+----------+-------------+

测试5.7和8.0不同版本访问i_s库的性能

机器

cat /etc/redhat-release | xargs echo '版本 ' && dmidecode -s system-product-name | xargs echo '是否虚拟化 ' && cat /proc/cpuinfo |grep "processor"|wc -l | xargs echo 'cpu核数 '
版本  centos linux release 7.5.1804 (core)
是否虚拟化  kvm
cpu核数  4

1、分别在mysql5.7和mysql8.0 中创建5个库，每个库中30个表（共1500个表），每个表记录数为10000

user=admin
passwd=admin
port=57222
host=127.0.0.1

#创建5个库,分别在每个库中创建30个表
for i in {1..5};do

mysql -u$user -p$passwd -p$port -h$host<<eof
  create database if not exists test_${i};

eof

sysbench /usr/local/share/sysbench/oltp_read_write.lua --mysql_storage_engine=innodb  --table-size=10000 --tables=30 --mysql-db=test_${i} --mysql-user=$user --mysql-password=$passwd --mysql-port=$port  --mysql-host=$host --threads=8 --time=10 --report-interval=1 --events=0 --db-driver=mysql prepare
done;

2、自定义访问i_s.tables表的sysbench脚本
cat tests/mytest.lua

require("oltp_common")


function thread_init(thread_id)
 drv=sysbench.sql.driver()
 con=drv:connect()
end


local function get_rand_db()
   return sysbench.rand.uniform(1, 5)
end



function event(thread_id)
local vid1
local dbprefix


vid1=get_rand_db()
dbprefix = "test_"

-- 生成5个db中的随机一个db,如 test_1 ,test_5
-- vid2=string.format("'%s%s'",test_,vid1)
vid2="'" .. dbprefix .. vid1 .. "'"

con:query("select   table_name,   concat(format(data_length / 1024 / 1024, 2),   'm') as dbdata_size,   concat(format(index_length / 1024 / 1024, 2),   'm') as dbindex_size,   concat(     format((data_length + index_length) / 1024 / 1024 / 1024,2),   'g') as `db_size(g)`,   avg_row_length,   table_rows,   update_time from   information_schema.tables where table_schema =" .. vid2)

end

function thread_done()
 con:disconnect()
end

3、脚本通过sysbench测试访问i_s.tables的qps
cat test_i_s_access.sh

user=admin
passwd=admin
host=127.0.0.1
#输入mysql端口参数
port=$1
# port=8015
#输入sysbench测试时间参数多少秒
run_time=$2

function get_create_tmp_tables(){
mysql -u$user -p$passwd -p$port -h$host<<eof
select @@version as 'mysql version';

show global  status like '%tmp_table%';
eof
} 2>/dev/null

function begin_test(){
  cd /usr/local/share/sysbench
   sysbench ./tests/mytest.lua --mysql-db=test_1  --mysql-host=$host --mysql-port=$port --mysql-user=$user --mysql-password=$passwd  --threads=40  --report-interval=10 --rand-type=uniform --time=$run_time    run
}


service mysqld${port} restart
get_create_tmp_tables
begin_test
get_create_tmp_tables

4、mysql5.7和mysql8.0 主要配置

mysql5.7.22

5.7.22-log
innodb_buffer_pool_size 128m
innodb_log_buffer_size  64m
innodb_log_file_size    48m
binlog_format   row
innodb_flush_log_at_trx_commit  1
sync_binlog     1

mysql8.0.15

8.0.15
innodb_buffer_pool_size 128m
innodb_log_buffer_size  64m
innodb_log_file_size    48m
binlog_format   row
innodb_flush_log_at_trx_commit  1
sync_binlog     1

5、mysql5.7开启测试
bash test_i_s_acess.sh 57222 60

shutting down mysql.. success!
starting mysql.. success!
mysql version
5.7.22-log
variable_name   value
created_tmp_tables      1
sysbench 1.1.0 (using bundled luajit 2.1.0-beta3)

running the test with following options:
number of threads: 40
report intermediate results every 10 second(s)
initializing random number generator from current time


initializing worker threads...

threads started!

[ 10s ] thds: 40 tps: 4765.73 qps: 4765.73 (r/w/o: 4765.73/0.00/0.00) lat (ms,95%): 18.95 err/s: 0.00 reconn/s:                                                                                                              0.00
[ 20s ] thds: 40 tps: 5409.00 qps: 5409.00 (r/w/o: 5409.00/0.00/0.00) lat (ms,95%): 17.95 err/s: 0.00 reconn/s:                                                                                                              0.00
[ 30s ] thds: 40 tps: 5154.45 qps: 5154.45 (r/w/o: 5154.45/0.00/0.00) lat (ms,95%): 18.61 err/s: 0.00 reconn/s:                                                                                                              0.00
[ 40s ] thds: 40 tps: 5383.50 qps: 5383.50 (r/w/o: 5383.50/0.00/0.00) lat (ms,95%): 17.63 err/s: 0.00 reconn/s:                                                                                                              0.00
[ 50s ] thds: 40 tps: 5456.11 qps: 5456.11 (r/w/o: 5456.11/0.00/0.00) lat (ms,95%): 17.63 err/s: 0.00 reconn/s:                                                                                                              0.00
[ 60s ] thds: 40 tps: 5458.66 qps: 5458.66 (r/w/o: 5458.66/0.00/0.00) lat (ms,95%): 17.63 err/s: 0.00 reconn/s:                                                                                                              0.00
sql statistics:
    queries performed:
        read:                            316322
        write:                           0
        other:                           0
        total:                           316322
    transactions:                        316322 (5270.99 per sec.)
    queries:                             316322 (5270.99 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

throughput:
    events/s (eps):                      5270.9892
    time elapsed:                        60.0119s
    total number of events:              316322

latency (ms):
         min:                                    0.42
         avg:                                    7.59
         max:                                  858.18
         95th percentile:                       17.95
         sum:                              2399738.58

threads fairness:
    events (avg/stddev):           7908.0500/53.41
    execution time (avg/stddev):   59.9935/0.00

mysql version
5.7.22-log
variable_name   value
created_tmp_tables      316327

cpu使用率接近100%
MySQL 8.0 information_schema系统库的改进

  • mysql5.7.22 40个并发线程 压力测试60秒，访问i_s.tables 平均qps为5k，并创建了31w个临时表!

6、mysql8.0开启测试

shutting down mysql.. success!
starting mysql.... success!
mysql version
8.0.15
variable_name   value
created_tmp_tables      1
sysbench 1.1.0 (using bundled luajit 2.1.0-beta3)

running the test with following options:
number of threads: 40
report intermediate results every 10 second(s)
initializing random number generator from current time


initializing worker threads...

threads started!

[ 10s ] thds: 40 tps: 1283.76 qps: 1283.76 (r/w/o: 1283.76/0.00/0.00) lat (ms,95%): 41.10 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 40 tps: 1286.43 qps: 1286.43 (r/w/o: 1286.43/0.00/0.00) lat (ms,95%): 39.65 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: 40 tps: 1333.00 qps: 1333.00 (r/w/o: 1333.00/0.00/0.00) lat (ms,95%): 37.56 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: 40 tps: 1280.79 qps: 1280.79 (r/w/o: 1280.79/0.00/0.00) lat (ms,95%): 40.37 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: 40 tps: 1313.57 qps: 1313.57 (r/w/o: 1313.57/0.00/0.00) lat (ms,95%): 38.94 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: 40 tps: 1222.95 qps: 1222.95 (r/w/o: 1222.95/0.00/0.00) lat (ms,95%): 42.61 err/s: 0.00 reconn/s: 0.00
sql statistics:
    queries performed:
        read:                            77248
        write:                           0
        other:                           0
        total:                           77248
    transactions:                        77248  (1286.95 per sec.)
    queries:                             77248  (1286.95 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

throughput:
    events/s (eps):                      1286.9486
    time elapsed:                        60.0242s
    total number of events:              77248

latency (ms):
         min:                                    1.88
         avg:                                   31.08
         max:                                  692.98
         95th percentile:                       40.37
         sum:                              2400505.33

threads fairness:
    events (avg/stddev):           1931.2000/51.17
    execution time (avg/stddev):   60.0126/0.00

mysql version
8.0.15
variable_name   value
created_tmp_tables      2

cpu使用率接近100%

MySQL 8.0 information_schema系统库的改进

  • mysql8.0.15 40个并发线程，压力测试60秒，访问i_s.tables 平均qps为1.2k，并创建了1个临时表!

结论

  • mysql8.0开始查询i_s中表不会再从文件系统(frm, trg，opt文件)和myisam、innodb系统表读取元数据信息（8.0开始元数据信息统一存放在数据字典表中）
  • mysql8.0 访问i_s.tables不会创建临时表，这减少了内存暴涨的可能，但访问i_s.tables的qps大约是mysql5.7.22的1/5，访问速度没有mysql5.7.22的快
  • mysql8.0 访问i_s.tables 用户空间cpu消耗没有mysql5.7的高（没有创建临时表的缘故吧），但系统空间cpu消耗高于mysql5.7！

参考
mysql 8.0: improvements to information_schema

上一篇： 快速查看表结构 SQL server查看表注释以及字段注释表结构字段说明

下一篇： nodejs进阶（1）——npm使用技巧和最佳实践

推荐阅读