TiDB中TiKV性能参数调整过程

环境信息：
$ ansible --version
ansible 2.6.3
  config file = /home/tidb/tidb-ansible/ansible.cfg
  configured module search path = [u'/home/tidb/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
# ./tikv-server --version
TiKV 
Release Version:   2.0.6
Git Commit Hash:   57c83dc4ebc93d38d77dc8f7d66db224760766cc
Git Commit Branch: release-2.0
UTC Build Time:    2018-08-03 11:28:38
Rust Version:      1.27.0-nightly (48fa6f963 2018-04-05)

# ./tidb-server -V
Release Version:  v2.0.6
Git Commit Hash:  b13bc08462a584a085f377625a7bab0cc0351570
Git Commit Branch:  release-2.0
UTC Build Time:  2018-08-06 10:57:46
GoVersion:  go version go1.10.2 linux/amd64
TiKV Min Version:  2.0.0-rc.4.1

# ./pd-server -V
Release Version: v2.0.5
Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
Git Branch: release-2.0
UTC Build Time:  2018-07-06 10:27:51

1.在配置文件中修改：
$ pwd
/home/tidb/tidb-ansible
$ vim conf/tikv.yml

需要修改的参数如下：
[server]
grpc-concurrency = 4   -->8
grpc-raft-conn-num = 10 -->24
[storage]
scheduler-concurrency = 2048000 
scheduler-worker-pool-size = 8  -->16
[coprocessor]
region-max-size = "144MB"  -->384MB
region-split-size = "96MB"  -->256MB
[raftstore]
region-split-check-diff = "6MB" -->32MB
sync-log = true  
 
[rocksdb]
max-background-jobs = 6   -->32
max-open-files = 40960    -->65535
[rocksdb.defaultcf]
block-cache-size = "48331MB"  -->80G
[rocksdb.writecf]
block-cache-size = "28998MB"   -->30G
[raftdb.defaultcf]
block-cache-size = "2GB"
 
此外参数需要调整为32MB。
target-file-size-base = "8MB" -->32MB


2.启动：使用tidb的身份执行滚动升级。
$ pwd
/home/tidb/tidb-ansible
$ ansible-playbook rolling_update.yml --tags=tikv
3.验证：
...
TASK [tikv : debug] *******************************************************
ok: [10.19.75.102] => {
    "tikv_conf": {
        "coprocessor": {
            "region-max-size": "384MB", 
            "region-split-size": "256MB"
        }, 
        "global": {}, 
        "import": {}, 
        "metric": {
            "address": "10.19.85.149:9091", 
            "interval": "15s", 
            "job": "tikv"
        }, 
        "pd": {}, 
        "raftdb": {
            "defaultcf": {
                "target-file-size-base": "32MB"
            }, 
            "max-open-files": 65535
        }, 
        "raftstore": {
            "raftdb-path": "", 
            "region-split-check-diff": "32MB", 
            "sync-log": true
        }, 
        "readpool": {
            "coprocessor": {}, 
            "storage": {}
        }, 
		"rocksdb": {
            "defaultcf": {
                "block-cache-size": "80GB", 
                "target-file-size-base": "32MB"
            }, 
            "lockcf": {
                "target-file-size-base": "32MB"
            }, 
            "max-background-jobs": 32, 
            "max-open-files": 65535, 
            "wal-dir": "", 
            "writecf": {
                "block-cache-size": "30GB", 
                "target-file-size-base": "32MB"
            }
        }, 
        "security": {
            "ca-path": "", 
            "cert-path": "", 
            "key-path": ""
        }, 
        "server": {
            "grpc-concurrency": 8, 
            "grpc-raft-conn-num": 24, 
            "labels": {}
        }, 
        "storage": {
            "scheduler-concurrency": 2048000, 
            "scheduler-worker-pool-size": 16
        }
    }
}
...
--修改成功的提示信息：
PLAY RECAP *****************************************************************************************

10.19.130.141              : ok=55   changed=4    unreachable=0    failed=0   
10.19.148.186              : ok=55   changed=4    unreachable=0    failed=0   
10.19.15.103               : ok=10   changed=0    unreachable=0    failed=0   
10.19.189.221              : ok=10   changed=0    unreachable=0    failed=0   
10.19.75.102               : ok=55   changed=4    unreachable=0    failed=0   
10.19.85.149               : ok=11   changed=0    unreachable=0    failed=0   
localhost                  : ok=1    changed=0    unreachable=0    failed=0   

Congrats! All goes well. :-)

登录待tikv节点查看配置,随机查看修改过的参数是否生效。
# pwd
/data/deploy/conf
# cat tikv.toml | grep -i target
target-file-size-base = "32MB"
target-file-size-base = "32MB"
target-file-size-base = "32MB"
target-file-size-base = "32MB"

对于既有的sst文件大小还是按照之前默认的8M存储的。
对于新加入的数据sst文件的大小则按照新设置的32MB存储。

-rw-r--r-- 1 tidb tidb  9.6M Aug 23 13:41 797214.sst
-rw-r--r-- 1 tidb tidb  9.5M Aug 23 13:41 797215.sst
-rw-r--r-- 1 tidb tidb  2.8M Aug 23 13:41 797216.sst
-rw-rw-r-- 1 tidb tidb  584M Aug 23 14:09 LOG.old.1535004595111057
-rw-r--r-- 1 tidb tidb    16 Aug 23 14:10 CURRENT
-rw-r--r-- 1 tidb tidb   12K Aug 23 14:10 OPTIONS-797235
-rw-r--r-- 1 tidb tidb   36M Aug 23 14:17 797527.sst
-rw-r--r-- 1 tidb tidb   41M Aug 23 14:17 797528.sst
-rw-r--r-- 1 tidb tidb   38M Aug 23 14:18 797529.sst
-rw-r--r-- 1 tidb tidb   37M Aug 23 14:18 797530.sst
-rw-r--r-- 1 tidb tidb   35M Aug 23 14:18 797531.sst
-rw-r--r-- 1 tidb tidb   35M Aug 23 14:18 797532.sst