SUSE12Sp3-使用Docker导入镜像并安装redis,zookeeper,kafka
程序员文章站
2022-10-04 22:55:01
首先在另外一台联网电脑拉取最新的redis,zookeeper,kafka镜像 然后导出镜像为tar文件 导入镜像到生产服务器 脚本总已经写了。复制到.sh文件中执行即可。 redis的配置和数据已经挂到宿主机了。另外两个没有。 !/bin/bash set x input images 这里要准备 ......
首先在另外一台联网电脑拉取最新的redis,zookeeper,kafka镜像
docker pull redis docker pull zookeeper docker pull wurstmeister/kafka
然后导出镜像为tar文件
docker save -o redis.tar redis ##-o:指定保存的镜像的名字 #另外两个相同
导入镜像到生产服务器
脚本总已经写了。复制到.sh文件中执行即可。
redis的配置和数据已经挂到宿主机了。另外两个没有。
#!/bin/bash set -x #input images 这里要准备好导入的镜像需要提前准备 sudo docker load --input redis.tar && sudo docker load --input zookeeper.tar && sudo docker load --input kafka.tar # 创建文件夹 redis,zookeeper,kafka mkdir -p $pwd/docker/{redis,zookeeper,kafka}/{conf,data}/ # 赋予docker文件夹权限 chmod 777 -r $pwd/docker # 这里是官方的配置,我直接复制过来了。你们可以按需修改。 cat >$pwd/docker/redis/conf/redis.conf <<-eof # redis configuration file example. # # note that in order to read the configuration file, redis must be # started with the file path as first argument: # # ./redis-server /path/to/redis.conf # note on units: when memory size is needed, it is possible to specify # it in the usual form of 1k 5gb 4m and so forth: # # 1k => 1000 bytes # 1kb => 1024 bytes # 1m => 1000000 bytes # 1mb => 1024*1024 bytes # 1g => 1000000000 bytes # 1gb => 1024*1024*1024 bytes # # units are case insensitive so 1gb 1gb 1gb are all the same. ################################## includes ################################### # include one or more other config files here. this is useful if you # have a standard template that goes to all redis servers but also need # to customize a few per-server settings. include files can include # other files, so use this wisely. # # notice option "include" won't be rewritten by command "config rewrite" # from admin or redis sentinel. since redis always uses the last processed # line as value of a configuration directive, you'd better put includes # at the beginning of this file to avoid overwriting config change at runtime. # # if instead you are interested in using includes to override configuration # options, it is better to use include as the last line. # # include /path/to/local.conf # include /path/to/other.conf ################################## modules ##################################### # load modules at startup. if the server is not able to load modules # it will abort. it is possible to use multiple loadmodule directives. # # loadmodule /path/to/my_module.so # loadmodule /path/to/other_module.so ################################## network ##################################### # by default, if no "bind" configuration directive is specified, redis listens # for connections from all the network interfaces available on the server. # it is possible to listen to just one or multiple selected interfaces using # the "bind" configuration directive, followed by one or more ip addresses. # # examples: # # bind 192.168.1.100 10.0.0.1 # bind 127.0.0.1 ::1 # # ~~~ warning ~~~ if the computer running redis is directly exposed to the # internet, binding to all the interfaces is dangerous and will expose the # instance to everybody on the internet. so by default we uncomment the # following bind directive, that will force redis to listen only into # the ipv4 loopback interface address (this means redis will be able to # accept connections only from clients running into the same computer it # is running). # # if you are sure you want your instance to listen to all the interfaces # just comment the following line. # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ bind 127.0.0.1 # protected mode is a layer of security protection, in order to avoid that # redis instances left open on the internet are accessed and exploited. # # when protected mode is on and if: # # 1) the server is not binding explicitly to a set of addresses using the # "bind" directive. # 2) no password is configured. # # the server only accepts connections from clients connecting from the # ipv4 and ipv6 loopback addresses 127.0.0.1 and ::1, and from unix domain # sockets. # # by default protected mode is enabled. you should disable it only if # you are sure you want clients from other hosts to connect to redis # even if no authentication is configured, nor a specific set of interfaces # are explicitly listed using the "bind" directive. protected-mode yes # accept connections on the specified port, default is 6379 (iana #815344). # if port 0 is specified redis will not listen on a tcp socket. port 6379 # tcp listen() backlog. # # in high requests-per-second environments you need an high backlog in order # to avoid slow clients connections issues. note that the linux kernel # will silently truncate it to the value of /proc/sys/net/core/somaxconn so # make sure to raise both the value of somaxconn and tcp_max_syn_backlog # in order to get the desired effect. tcp-backlog 511 # unix socket. # # specify the path for the unix socket that will be used to listen for # incoming connections. there is no default, so redis will not listen # on a unix socket when not specified. # # unixsocket /tmp/redis.sock # unixsocketperm 700 # close the connection after a client is idle for n seconds (0 to disable) timeout 0 # tcp keepalive. # # if non-zero, use so_keepalive to send tcp acks to clients in absence # of communication. this is useful for two reasons: # # 1) detect dead peers. # 2) take the connection alive from the point of view of network # equipment in the middle. # # on linux, the specified value (in seconds) is the period used to send acks. # note that to close the connection the double of the time is needed. # on other kernels the period depends on the kernel configuration. # # a reasonable value for this option is 300 seconds, which is the new # redis default starting with redis 3.2.1. tcp-keepalive 300 ################################# general ##################################### # by default redis does not run as a daemon. use 'yes' if you need it. # note that redis will write a pid file in /var/run/redis.pid when daemonized. daemonize no # if you run redis from upstart or systemd, redis can interact with your # supervision tree. options: # supervised no - no supervision interaction # supervised upstart - signal upstart by putting redis into sigstop mode # supervised systemd - signal systemd by writing ready=1 to $notify_socket # supervised auto - detect upstart or systemd method based on # upstart_job or notify_socket environment variables # note: these supervision methods only signal "process is ready." # they do not enable continuous liveness pings back to your supervisor. supervised no # if a pid file is specified, redis writes it where specified at startup # and removes it at exit. # # when the server runs non daemonized, no pid file is created if none is # specified in the configuration. when the server is daemonized, the pid file # is used even if not specified, defaulting to "/var/run/redis.pid". # # creating a pid file is best effort: if redis is not able to create it # nothing bad happens, the server will start and run normally. pidfile /var/run/redis_6379.pid # specify the server verbosity level. # this can be one of: # debug (a lot of information, useful for development/testing) # verbose (many rarely useful info, but not a mess like the debug level) # notice (moderately verbose, what you want in production probably) # warning (only very important / critical messages are logged) loglevel notice # specify the log file name. also the empty string can be used to force # redis to log on the standard output. note that if you use standard # output for logging but daemonize, logs will be sent to /dev/null logfile "" # to enable logging to the system logger, just set 'syslog-enabled' to yes, # and optionally update the other syslog parameters to suit your needs. # syslog-enabled no # specify the syslog identity. # syslog-ident redis # specify the syslog facility. must be user or between local0-local7. # syslog-facility local0 # set the number of databases. the default database is db 0, you can select # a different one on a per-connection basis using select <dbid> where # dbid is a number between 0 and 'databases'-1 databases 16 # by default redis shows an ascii art logo only when started to log to the # standard output and if the standard output is a tty. basically this means # that normally a logo is displayed only in interactive sessions. # # however it is possible to force the pre-4.0 behavior and always show a # ascii art logo in startup logs by setting the following option to yes. always-show-logo yes ################################ snapshotting ################################ # # save the db on disk: # # save <seconds> <changes> # # will save the db if both the given number of seconds and the given # number of write operations against the db occurred. # # in the example below the behaviour will be to save: # after 900 sec (15 min) if at least 1 key changed # after 300 sec (5 min) if at least 10 keys changed # after 60 sec if at least 10000 keys changed # # note: you can disable saving completely by commenting out all "save" lines. # # it is also possible to remove all the previously configured save # points by adding a save directive with a single empty string argument # like in the following example: # # save "" save 900 1 save 300 10 save 60 10000 # by default redis will stop accepting writes if rdb snapshots are enabled # (at least one save point) and the latest background save failed. # this will make the user aware (in a hard way) that data is not persisting # on disk properly, otherwise chances are that no one will notice and some # disaster will happen. # # if the background saving process will start working again redis will # automatically allow writes again. # # however if you have setup your proper monitoring of the redis server # and persistence, you may want to disable this feature so that redis will # continue to work as usual even if there are problems with disk, # permissions, and so forth. stop-writes-on-bgsave-error yes # compress string objects using lzf when dump .rdb databases? # for default that's set to 'yes' as it's almost always a win. # if you want to save some cpu in the saving child set it to 'no' but # the dataset will likely be bigger if you have compressible values or keys. rdbcompression yes # since version 5 of rdb a crc64 checksum is placed at the end of the file. # this makes the format more resistant to corruption but there is a performance # hit to pay (around 10%) when saving and loading rdb files, so you can disable it # for maximum performances. # # rdb files created with checksum disabled have a checksum of zero that will # tell the loading code to skip the check. rdbchecksum yes # the filename where to dump the db dbfilename dump.rdb # the working directory. # # the db will be written inside this directory, with the filename specified # above using the 'dbfilename' configuration directive. # # the append only file will also be created inside this directory. # # note that you must specify a directory here, not a file name. dir ./ ################################# replication ################################# # master-replica replication. use replicaof to make a redis instance a copy of # another redis server. a few things to understand asap about redis replication. # # +------------------+ +---------------+ # | master | ---> | replica | # | (receive writes) | | (exact copy) | # +------------------+ +---------------+ # # 1) redis replication is asynchronous, but you can configure a master to # stop accepting writes if it appears to be not connected with at least # a given number of replicas. # 2) redis replicas are able to perform a partial resynchronization with the # master if the replication link is lost for a relatively small amount of # time. you may want to configure the replication backlog size (see the next # sections of this file) with a sensible value depending on your needs. # 3) replication is automatic and does not need user intervention. after a # network partition replicas automatically try to reconnect to masters # and resynchronize with them. # # replicaof <masterip> <masterport> # if the master is password protected (using the "requirepass" configuration # directive below) it is possible to tell the replica to authenticate before # starting the replication synchronization process, otherwise the master will # refuse the replica request. # # masterauth <master-password> # when a replica loses its connection with the master, or when the replication # is still in progress, the replica can act in two different ways: # # 1) if replica-serve-stale-data is set to 'yes' (the default) the replica will # still reply to client requests, possibly with out of date data, or the # data set may just be empty if this is the first synchronization. # # 2) if replica-serve-stale-data is set to 'no' the replica will reply with # an error "sync with master in progress" to all the kind of commands # but to info, replicaof, auth, ping, shutdown, replconf, role, config, # subscribe, unsubscribe, psubscribe, punsubscribe, publish, pubsub, # command, post, host: and latency. # replica-serve-stale-data yes # you can configure a replica instance to accept writes or not. writing against # a replica instance may be useful to store some ephemeral data (because data # written on a replica will be easily deleted after resync with the master) but # may also cause problems if clients are writing to it because of a # misconfiguration. # # since redis 2.6 by default replicas are read-only. # # note: read only replicas are not designed to be exposed to untrusted clients # on the internet. it's just a protection layer against misuse of the instance. # still a read only replica exports by default all the administrative commands # such as config, debug, and so forth. to a limited extent you can improve # security of read only replicas using 'rename-command' to shadow all the # administrative / dangerous commands. replica-read-only yes # replication sync strategy: disk or socket. # # ------------------------------------------------------- # warning: diskless replication is experimental currently # ------------------------------------------------------- # # new replicas and reconnecting replicas that are not able to continue the replication # process just receiving differences, need to do what is called a "full # synchronization". an rdb file is transmitted from the master to the replicas. # the transmission can happen in two different ways: # # 1) disk-backed: the redis master creates a new process that writes the rdb # file on disk. later the file is transferred by the parent # process to the replicas incrementally. # 2) diskless: the redis master creates a new process that directly writes the # rdb file to replica sockets, without touching the disk at all. # # with disk-backed replication, while the rdb file is generated, more replicas # can be queued and served with the rdb file as soon as the current child producing # the rdb file finishes its work. with diskless replication instead once # the transfer starts, new replicas arriving will be queued and a new transfer # will start when the current one terminates. # # when diskless replication is used, the master waits a configurable amount of # time (in seconds) before starting the transfer in the hope that multiple replicas # will arrive and the transfer can be parallelized. # # with slow disks and fast (large bandwidth) networks, diskless replication # works better. repl-diskless-sync no # when diskless replication is enabled, it is possible to configure the delay # the server waits in order to spawn the child that transfers the rdb via socket # to the replicas. # # this is important since once the transfer starts, it is not possible to serve # new replicas arriving, that will be queued for the next rdb transfer, so the server # waits a delay in order to let more replicas arrive. # # the delay is specified in seconds, and by default is 5 seconds. to disable # it entirely just set it to 0 seconds and the transfer will start asap. repl-diskless-sync-delay 5 # replicas send pings to server in a predefined interval. it's possible to change # this interval with the repl_ping_replica_period option. the default value is 10 # seconds. # # repl-ping-replica-period 10 # the following option sets the replication timeout for: # # 1) bulk transfer i/o during sync, from the point of view of replica. # 2) master timeout from the point of view of replicas (data, pings). # 3) replica timeout from the point of view of masters (replconf ack pings). # # it is important to make sure that this value is greater than the value # specified for repl-ping-replica-period otherwise a timeout will be detected # every time there is low traffic between the master and the replica. # # repl-timeout 60 # disable tcp_nodelay on the replica socket after sync? # # if you select "yes" redis will use a smaller number of tcp packets and # less bandwidth to send data to replicas. but this can add a delay for # the data to appear on the replica side, up to 40 milliseconds with # linux kernels using a default configuration. # # if you select "no" the delay for data to appear on the replica side will # be reduced but more bandwidth will be used for replication. # # by default we optimize for low latency, but in very high traffic conditions # or when the master and replicas are many hops away, turning this to "yes" may # be a good idea. repl-disable-tcp-nodelay no # set the replication backlog size. the backlog is a buffer that accumulates # replica data when replicas are disconnected for some time, so that when a replica # wants to reconnect again, often a full resync is not needed, but a partial # resync is enough, just passing the portion of data the replica missed while # disconnected. # # the bigger the replication backlog, the longer the time the replica can be # disconnected and later be able to perform a partial resynchronization. # # the backlog is only allocated once there is at least a replica connected. # # repl-backlog-size 1mb # after a master has no longer connected replicas for some time, the backlog # will be freed. the following option configures the amount of seconds that # need to elapse, starting from the time the last replica disconnected, for # the backlog buffer to be freed. # # note that replicas never free the backlog for timeout, since they may be # promoted to masters later, and should be able to correctly "partially # resynchronize" with the replicas: hence they should always accumulate backlog. # # a value of 0 means to never release the backlog. # # repl-backlog-ttl 3600 # the replica priority is an integer number published by redis in the info output. # it is used by redis sentinel in order to select a replica to promote into a # master if the master is no longer working correctly. # # a replica with a low priority number is considered better for promotion, so # for instance if there are three replicas with priority 10, 100, 25 sentinel will # pick the one with priority 10, that is the lowest. # # however a special priority of 0 marks the replica as not able to perform the # role of master, so a replica with priority of 0 will never be selected by # redis sentinel for promotion. # # by default the priority is 100. replica-priority 100 # it is possible for a master to stop accepting writes if there are less than # n replicas connected, having a lag less or equal than m seconds. # # the n replicas need to be in "online" state. # # the lag in seconds, that must be <= the specified value, is calculated from # the last ping received from the replica, that is usually sent every second. # # this option does not guarantee that n replicas will accept the write, but # will limit the window of exposure for lost writes in case not enough replicas # are available, to the specified number of seconds. # # for example to require at least 3 replicas with a lag <= 10 seconds use: # # min-replicas-to-write 3 # min-replicas-max-lag 10 # # setting one or the other to 0 disables the feature. # # by default min-replicas-to-write is set to 0 (feature disabled) and # min-replicas-max-lag is set to 10. # a redis master is able to list the address and port of the attached # replicas in different ways. for example the "info replication" section # offers this information, which is used, among other tools, by # redis sentinel in order to discover replica instances. # another place where this info is available is in the output of the # "role" command of a master. # # the listed ip and address normally reported by a replica is obtained # in the following way: # # ip: the address is auto detected by checking the peer address # of the socket used by the replica to connect with the master. # # port: the port is communicated by the replica during the replication # handshake, and is normally the port that the replica is using to # listen for connections. # # however when port forwarding or network address translation (nat) is # used, the replica may be actually reachable via different ip and port # pairs. the following two options can be used by a replica in order to # report to its master a specific set of ip and port, so that both info # and role will report those values. # # there is no need to use both the options if you need to override just # the port or the ip address. # # replica-announce-ip 5.5.5.5 # replica-announce-port 1234 ################################## security ################################### # require clients to issue auth <password> before processing any other # commands. this might be useful in environments in which you do not trust # others with access to the host running redis-server. # # this should stay commented out for backward compatibility and because most # people do not need auth (e.g. they run their own servers). # # warning: since redis is pretty fast an outside user can try up to # 150k passwords per second against a good box. this means that you should # use a very strong password otherwise it will be very easy to break. # # requirepass foobared # command renaming. # # it is possible to change the name of dangerous commands in a shared # environment. for instance the config command may be renamed into something # hard to guess so that it will still be available for internal-use tools # but not available for general clients. # # example: # # rename-command config b840fc02d524045429941cc15f59e41cb7be6c52 # # it is also possible to completely kill a command by renaming it into # an empty string: # # rename-command config "" # # please note that changing the name of commands that are logged into the # aof file or transmitted to replicas may cause problems. ################################### clients #################################### # set the max number of connected clients at the same time. by default # this limit is set to 10000 clients, however if the redis server is not # able to configure the process file limit to allow for the specified limit # the max number of allowed clients is set to the current file limit # minus 32 (as redis reserves a few file descriptors for internal uses). # # once the limit is reached redis will close all the new connections sending # an error 'max number of clients reached'. # # maxclients 10000 ############################## memory management ################################ # set a memory usage limit to the specified amount of bytes. # when the memory limit is reached redis will try to remove keys # according to the eviction policy selected (see maxmemory-policy). # # if redis can't remove keys according to the policy, or if the policy is # set to 'noeviction', redis will start to reply with errors to commands # that would use more memory, like set, lpush, and so on, and will continue # to reply to read-only commands like get. # # this option is usually useful when using redis as an lru or lfu cache, or to # set a hard memory limit for an instance (using the 'noeviction' policy). # # warning: if you have replicas attached to an instance with maxmemory on, # the size of the output buffers needed to feed the replicas are subtracted # from the used memory count, so that network problems / resyncs will # not trigger a loop where keys are evicted, and in turn the output # buffer of replicas is full with dels of keys evicted triggering the deletion # of more keys, and so forth until the database is completely emptied. # # in short... if you have replicas attached it is suggested that you set a lower # limit for maxmemory so that there is some free ram on the system for replica # output buffers (but this is not needed if the policy is 'noeviction'). # # maxmemory <bytes> # maxmemory policy: how redis will select what to remove when maxmemory # is reached. you can select among five behaviors: # # volatile-lru -> evict using approximated lru among the keys with an expire set. # allkeys-lru -> evict any key using approximated lru. # volatile-lfu -> evict using approximated lfu among the keys with an expire set. # allkeys-lfu -> evict any key using approximated lfu. # volatile-random -> remove a random key among the ones with an expire set. # allkeys-random -> remove a random key, any key. # volatile-ttl -> remove the key with the nearest expire time (minor ttl) # noeviction -> don't evict anything, just return an error on write operations. # # lru means least recently used # lfu means least frequently used # # both lru, lfu and volatile-ttl are implemented using approximated # randomized algorithms. # # note: with any of the above policies, redis will return an error on write # operations, when there are no suitable keys for eviction. # # at the date of writing these commands are: set setnx setex append # incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd # sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby # zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby # getset mset msetnx exec sort # # the default is: # # maxmemory-policy noeviction # lru, lfu and minimal ttl algorithms are not precise algorithms but approximated # algorithms (in order to save memory), so you can tune it for speed or # accuracy. for default redis will check five keys and pick the one that was # used less recently, you can change the sample size using the following # configuration directive. # # the default of 5 produces good enough results. 10 approximates very closely # true lru but costs more cpu. 3 is faster but not very accurate. # # maxmemory-samples 5 # starting from redis 5, by default a replica will ignore its maxmemory setting # (unless it is promoted to master after a failover or manually). it means # that the eviction of keys will be just handled by the master, sending the # del commands to the replica as keys evict in the master side. # # this behavior ensures that masters and replicas stay consistent, and is usually # what you want, however if your replica is writable, or you want the replica to have # a different memory setting, and you are sure all the writes performed to the # replica are idempotent, then you may change this default (but be sure to understand # what you are doing). # # note that since the replica by default does not evict, it may end using more # memory than the one set via maxmemory (there are certain buffers that may # be larger on the replica, or data structures may sometimes take more memory and so # forth). so make sure you monitor your replicas and make sure they have enough # memory to never hit a real out-of-memory condition before the master hits # the configured maxmemory setting. # # replica-ignore-maxmemory yes ############################# lazy freeing #################################### # redis has two primitives to delete keys. one is called del and is a blocking # deletion of the object. it means that the server stops processing new commands # in order to reclaim all the memory associated with an object in a synchronous # way. if the key deleted is associated with a small object, the time needed # in order to execute the del command is very small and comparable to most other # o(1) or o(log_n) commands in redis. however if the key is associated with an # aggregated value containing millions of elements, the server can block for # a long time (even seconds) in order to complete the operation. # # for the above reasons redis also offers non blocking deletion primitives # such as unlink (non blocking del) and the async option of flushall and # flushdb commands, in order to reclaim memory in background. those commands # are executed in constant time. another thread will incrementally free the # object in the background as fast as possible. # # del, unlink and async option of flushall and flushdb are user-controlled. # it's up to the design of the application to understand when it is a good # idea to use one or the other. however the redis server sometimes has to # delete keys or flush the whole database as a side effect of other operations. # specifically redis deletes objects independently of a user call in the # following scenarios: # # 1) on eviction, because of the maxmemory and maxmemory policy configurations, # in order to make room for new data, without going over the specified # memory limit. # 2) because of expire: when a key with an associated time to live (see the # expire command) must be deleted from memory. # 3) because of a side effect of a command that stores data on a key that may # already exist. for example the rename command may delete the old key # content when it is replaced with another one. similarly sunionstore # or sort with store option may delete existing keys. the set command # itself removes any old content of the specified key in order to replace # it with the specified string. # 4) during replication, when a replica performs a full resynchronization with # its master, the content of the whole database is removed in order to # load the rdb file just transferred. # # in all the above cases the default is to delete objects in a blocking way, # like if del was called. however you can configure each case specifically # in order to instead release memory in a non-blocking way like if unlink # was called, using the following configuration directives: lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no replica-lazy-flush no ############################## append only mode ############################### # by default redis asynchronously dumps the dataset on disk. this mode is # good enough in many applications, but an issue with the redis process or # a power outage may result into a few minutes of writes lost (depending on # the configured save points). # # the append only file is an alternative persistence mode that provides # much better durability. for instance using the default data fsync policy # (see later in the config file) redis can lose just one second of writes in a # dramatic event like a server power outage, or a single write if something # wrong with the redis process itself happens, but the operating system is # still running correctly. # # aof and rdb persistence can be enabled at the same time without problems. # if the aof is enabled on startup redis will load the aof, that is the file # with the better durability guarantees. # # please check http://redis.io/topics/persistence for more information. appendonly no # the name of the append only file (default: "appendonly.aof") appendfilename "appendonly.aof" # the fsync() call tells the operating system to actually write data on disk # instead of waiting for more data in the output buffer. some os will really flush # data on disk, some other os will just try to do it asap. # # redis supports three different modes: # # no: don't fsync, just let the os flush the data when it wants. faster. # always: fsync after every write to the append only log. slow, safest. # everysec: fsync only one time every second. compromise. # # the default is "everysec", as that's usually the right compromise between # speed and data safety. it's up to you to understand if you can relax this to # "no" that will let the operating system flush the output buffer when # it wants, for better performances (but if you can live with the idea of # some data loss consider the default persistence mode that's snapshotting), # or on the contrary, use "always" that's very slow but a bit safer than # everysec. # # more details please check the following article: # http://antirez.com/post/redis-persistence-demystified.html # # if unsure, use "everysec". # appendfsync always appendfsync everysec # appendfsync no # when the aof fsync policy is set to always or everysec, and a background # saving process (a background save or aof log background rewriting) is # performing a lot of i/o against the disk, in some linux configurations # redis may block too long on the fsync() call. note that there is no fix for # this currently, as even performing fsync in a different thread will block # our synchronous write(2) call. # # in order to mitigate this problem it's possible to use the following option # that will prevent fsync() from being called in the main process while a # bgsave or bgrewriteaof is in progress. # # this means that while another child is saving, the durability of redis is # the same as "appendfsync none". in practical terms, this means that it is # possible to lose up to 30 seconds of log in the worst scenario (with the # default linux settings). # # if you have latency problems turn this to "yes". otherwise leave it as # "no" that is the safest pick from the point of view of durability. no-appendfsync-on-rewrite no # automatic rewrite of the append only file. # redis is able to automatically rewrite the log file implicitly calling # bgrewriteaof when the aof log size grows by the specified percentage. # # this is how it works: redis remembers the size of the aof file after the # latest rewrite (if no rewrite has happened since the restart, the size of # the aof at startup is used). # # this base size is compared to the current size. if the current size is # bigger than the specified percentage, the rewrite is triggered. also # you need to specify a minimal size for the aof file to be rewritten, this # is useful to avoid rewriting the aof file even if the percentage increase # is reached but it is still pretty small. # # specify a percentage of zero in order to disable the automatic aof # rewrite feature. auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb # an aof file may be found to be truncated at the end during the redis # startup process, when the aof data gets loaded back into memory. # this may happen when the system where redis is running # crashes, especially when an ext4 filesystem is mounted without the # data=ordered option (however this can't happen when redis itself # crashes or aborts but the operating system still works correctly). # # redis can either exit with an error when this happens, or load as much # data as possible (the default now) and start if the aof file is found # to be truncated at the end. the following option controls this behavior. # # if aof-load-truncated is set to yes, a truncated aof file is loaded and # the redis server starts emitting a log to inform the user of the event. # otherwise if the option is set to no, the server aborts with an error # and refuses to start. when the option is set to no, the user requires # to fix the aof file using the "redis-check-aof" utility before to restart # the server. # # note that if the aof file will be found to be corrupted in the middle # the server will still exit with an error. this option only applies when # redis will try to read more data from the aof file but not enough bytes # will be found. aof-load-truncated yes # when rewriting the aof file, redis is able to use an rdb preamble in the # aof file for faster rewrites and recoveries. when this option is turned # on the rewritten aof file is composed of two different stanzas: # # [rdb file][aof tail] # # when loading redis recognizes that the aof file starts with the "redis" # string and loads the prefixed rdb file, and continues loading the aof # tail. aof-use-rdb-preamble yes ################################ lua scripting ############################### # max execution time of a lua script in milliseconds. # # if the maximum execution time is reached redis will log that a script is # still in execution after the maximum allowed time and will start to # reply to queries with an error. # # when a long running script exceeds the maximum execution time only the # script kill and shutdown nosave commands are available. the first can be # used to stop a script that did not yet called write commands. the second # is the only way to shut down the server in the case a write command was # already issued by the script but the user doesn't want to wait for the natural # termination of the script. # # set it to 0 or a negative value for unlimited execution without warnings. lua-time-limit 5000 ################################ redis cluster ############################### # # ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # warning experimental: redis cluster is considered to be stable code, however # in order to mark it as "mature" we need to wait for a non trivial percentage # of users to deploy it in production. # ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # # normal redis instances can't be part of a redis cluster; only nodes that are # started as cluster nodes can. in order to start a redis instance as a # cluster node enable the cluster support uncommenting the following: # # cluster-enabled yes # every cluster node has a cluster configuration file. this file is not # intended to be edited by hand. it is created and updated by redis nodes. # every redis cluster node requires a different cluster configuration file. # make sure that instances running in the same system do not have # overlapping cluster configuration file names. # # cluster-config-file nodes-6379.conf # cluster node timeout is the amount of milliseconds a node must be unreachable # for it to be considered in failure state. # most other internal time limits are multiple of the node timeout. # # cluster-node-timeout 15000 # a replica of a failing master will avoid to start a failover if its data # looks too old. # # there is no simple way for a replica to actually have an exact measure of # its "data age", so the following two checks are performed: # # 1) if there are multiple replicas able to failover, they exchange messages # in order to try to give an advantage to the replica with the best # replication offset (more data from the master processed). # replicas will try to get their rank by offset, and apply to the start # of the failover a delay proportional to their rank. # # 2) every single replica computes the time of the last interaction with # its master. this can be the last ping or command received (if the master # is still in the "connected" state), or the time that elapsed since the # disconnection with the master (if the replication link is currently down). # if the last interaction is too old, the replica will not try to failover # at all. # # the point "2" can be tuned by user. specifically a replica will not perform # the failover if, since the last interaction with the master, the time # elapsed is greater than: # # (node-timeout * replica-validity-factor) + repl-ping-replica-period # # so for example if node-timeout is 30 seconds, and the replica-validity-factor # is 10, and assuming a default repl-ping-replica-period of 10 seconds, the # replica will not try to failover if it was not able to talk with the master # for longer than 310 seconds. # # a large replica-validity-factor may allow replicas with too old data to failover # a master, while a too small value may prevent the cluster from being able to # elect a replica at all. # # for maximum availability, it is possible to set the replica-validity-factor # to a value of 0, which means, that replicas will always try to failover the # master regardless of the last time they interacted with the master. # (however they'll always try to apply a delay proportional to their # offset rank). # # zero is the only value able to guarantee that when all the partitions heal # the cluster will always be able to continue. # # cluster-replica-validity-factor 10 # cluster replicas are able to migrate to orphaned masters, that are masters # that are left without working replicas. this improves the cluster ability # to resist to failures as otherwise an orphaned master can't be failed over # in case of failure if it has no working replicas. # # replicas migrate to orphaned masters only if there are still at least a # given number of other working replicas for their old master. this number # is the "migration barrier". a migration barrier of 1 means that a replica # will migrate only if there is at least 1 other working replica for its master # and so forth. it usually reflects the number of replicas you want for every # master in your cluster. # # default is 1 (replicas migrate only if their masters remain with at least # one replica). to disable migration just set it to a very large value. # a value of 0 can be set but is useful only for debugging and dangerous # in production. # # cluster-migration-barrier 1 # by default redis cluster nodes stop accepting queries if they detect there # is at least an hash slot uncovered (no available node is serving it). # this way if the cluster is partially down (for example a range of hash slots # are no longer covered) all the cluster becomes, eventually, unavailable. # it automatically returns available as soon as all the slots are covered again. # # however sometimes you want the subset of the cluster which is working, # to continue to accept queries for the part of the key space that is still # covered. in order to do so, just set the cluster-require-full-coverage # option to no. # # cluster-require-full-coverage yes # this option, when set to yes, prevents replicas from trying to failover its # master during master failures. however the master can still perform a # manual failover, if forced to do so. # # this is useful in different scenarios, especially in the case of multiple # data center operations, where we want one side to never be promoted if not # in the case of a total dc failure. # # cluster-replica-no-failover no # in order to setup your cluster make sure to read the documentation # available at http://redis.io web site. ########################## cluster docker/nat support ######################## # in certain deployments, redis cluster nodes address discovery fails, because # addresses are nat-ted or because ports are forwarded (the typical case is # docker and other containers). # # in order to make redis cluster working in such environments, a static # configuration where each node knows its public address is needed. the # following two options are used for this scope, and are: # # * cluster-announce-ip # * cluster-announce-port # * cluster-announce-bus-port # # each instruct the node about its address, client port, and cluster message # bus port. the information is then published in the header of the bus packets # so that other nodes will be able to correctly map the address of the node # publishing the information. # # if the above options are not used, the normal redis cluster auto-detection # will be used instead. # # note that when remapped, the bus port may not be at the fixed offset of # clients port + 10000, so you can specify any port and bus-port depending # on how they get remapped. if the bus-port is not set, a fixed offset of # 10000 will be used as usually. # # example: # # cluster-announce-ip 10.1.1.5 # cluster-announce-port 6379 # cluster-announce-bus-port 6380 ################################## slow log ################################### # the redis slow log is a system to log queries that exceeded a specified # execution time. the execution time does not include the i/o operations # like talking with the client, sending the reply and so forth, # but just the time needed to actually execute the command (this is the only # stage of command execution where the thread is blocked and can not serve # other requests in the meantime). # # you can configure the slow log with two parameters: one tells redis # what is the execution time, in microseconds, to exceed in order for the # command to get logged, and the other parameter is the length of the # slow log. when a new command is logged the oldest one is removed from the # queue of logged commands. # the following time is expressed in microseconds, so 1000000 is equivalent # to one second. note that a negative number disables the slow log, while # a value of zero forces the logging of every command. slowlog-log-slower-than 10000 # there is no limit to this length. just be aware that it will consume memory. # you can reclaim memory used by the slow log with slowlog reset. slowlog-max-len 128 ################################ latency monitor ############################## # the redis latency monitoring subsystem samples different operations # at runtime in order to collect data related to possible sources of # latency of a redis instance. # # via the latency command this information is available to the user that can # print graphs and obtain reports. # # the system only logs operations that were performed in a time equal or # greater than the amount of milliseconds specified via the # latency-monitor-threshold configuration directive. when its value is set # to zero, the latency monitor is turned off. # # by default latency monitoring is disabled since it is mostly not needed # if you don't have latency issues, and collecting data has a performance # impact, that while very small, can be measured under big load. latency # monitoring can easily be enabled at runtime using the command # "config set latency-monitor-threshold <milliseconds>" if needed. latency-monitor-threshold 0 ############################# event notification ############################## # redis can notify pub/sub clients about events happening in the key space. # this feature is documented at http://redis.io/topics/notifications # # for instance if keyspace events notification is enabled, and a client # performs a del operation on key "foo" stored in the database 0, two # messages will be published via pub/sub: # # publish __keyspace@0__:foo del # publish __keyevent@0__:del foo # # it is possible to select the events that redis will notify among a set # of classes. every class is identified by a single character: # # k keyspace events, published with __keyspace@<db>__ prefix. # e keyevent events, published with __keyevent@<db>__ prefix. # g generic commands (non-type specific) like del, expire, rename, ... # $ string commands # l list commands # s set commands # h hash commands # z sorted set commands # x expired events (events generated every time a key expires) # e evicted events (events generated when a key is evicted for maxmemory) # a alias for g$lshzxe, so that the "ake" string means all the events. # # the "notify-keyspace-events" takes as argument a string that is composed # of zero or multiple characters. the empty string means that notifications # are disabled. # # example: to enable list and generic events, from the point of view of the # event name, use: # # notify-keyspace-events elg # # example 2: to get the stream of the expired keys subscribing to channel # name __keyevent@0__:expired use: # # notify-keyspace-events ex # # by default all notifications are disabled because most users don't need # this feature and the feature has some overhead. note that if you don't # specify at least one of k or e, no events will be delivered. notify-keyspace-events "" ############################### advanced config ############################### # hashes are encoded using a memory efficient data structure when they have a # small number of entries, and the biggest entry does not exceed a given # threshold. these thresholds can be configured using the following directives. hash-max-ziplist-entries 512 hash-max-ziplist-value 64 # lists are also encoded in a special way to save a lot of space. # the number of entries allowed per internal list node can be specified # as a fixed maximum size or a maximum number of elements. # for a fixed maximum size, use -5 through -1, meaning: # -5: max size: 64 kb <-- not recommended for normal workloads # -4: max size: 32 kb <-- not recommended # -3: max size: 16 kb <-- probably not recommended # -2: max size: 8 kb <-- good # -1: max size: 4 kb <-- good # positive numbers mean store up to _exactly_ that number of elements # per list node. # the highest performing option is usually -2 (8 kb size) or -1 (4 kb size), # but if your use case is unique, adjust the settings as necessary. list-max-ziplist-size -2 # lists may also be compressed. # compress depth is the number of quicklist ziplist nodes from *each* side of # the list to *exclude* from compression. the head and tail of the list # are always uncompressed for fast push/pop operations. settings are: # 0: disable all list compression # 1: depth 1 means "don't start compressing until after 1 node into the list, # going from either the head or tail" # so: [head]->node->node->...->node->[tail] # [head], [tail] will always be uncompressed; inner nodes will compress. # 2: [head]->[next]->node->node->...->node->[prev]->[tail] # 2 here means: don't compress head or head->next or tail->prev or tail, # but compress all nodes between them. # 3: [head]->[next]->[next]->node->node->...->node->[prev]->[prev]->[tail] # etc. list-compress-depth 0 # sets have a special encoding in just one case: when a set is composed # of just strings that happen to be integers in radix 10 in the range # of 64 bit signed integers. # the following configuration setting sets the limit in the size of the # set in order to use this special memory saving encoding. set-max-intset-entries 512 # similarly to hashes and lists, sorted sets are also specially encoded in # order to save a lot of space. this encoding is only used when the length and # elements of a sorted set are below the following limits: zset-max-ziplist-entries 128 zset-max-ziplist-value 64 # hyperloglog sparse representation bytes limit. the limit includes the # 16 bytes header. when an hyperloglog using the sparse representation crosses # this limit, it is converted into the dense representation. # # a value greater than 16000 is totally useless, since at that point the # dense representation is more memory efficient. # # the suggested value is ~ 3000 in order to have the benefits of # the space efficient encoding without slowing down too much pfadd, # which is o(n) with the sparse encoding. the value can be raised to # ~ 10000 when cpu is not a concern, but space is, and the data set is # composed of many hyperloglogs with cardinality in the 0 - 15000 range. hll-sparse-max-bytes 3000 # streams macro node max size / items. the stream data structure is a radix # tree of big nodes that encode multiple items inside. using this configuration # it is possible to configure how big a single node can be in bytes, and the # maximum number of items it may contain before switching to a new node when # appending new stream entries. if any of the following settings are set to # zero, the limit is ignored, so for instance it is possible to set just a # max entires limit by setting max-bytes to 0 and max-entries to the desired # value. stream-node-max-bytes 4096 stream-node-max-entries 100 # active rehashing uses 1 millisecond every 100 milliseconds of cpu time in # order to help rehashing the main redis hash table (the one mapping top-level # keys to values). the hash table implementation redis uses (see dict.c) # performs a lazy rehashing: the more operation you run into a hash table # that is rehashing, the more rehashing "steps" are performed, so if the # server is idle the rehashing is never complete and some more memory is used # by the hash table. # # the default is to use this millisecond 10 times every second in order to # actively rehash the main dictionaries, freeing memory when possible. # # if unsure: # use "activerehashing no" if you have hard latency requirements and it is # not a good thing in your environment that redis can reply from time to time # to queries with 2 milliseconds delay. # # use "activerehashing yes" if you don't have such hard requirements but # want to free memory asap when possible. activerehashing yes # the client output buffer limits can be used to force disconnection of clients # that are not reading data from the server fast enough for some reason (a # common reason is that a pub/sub client can't consume messages as fast as the # publisher can produce them). # # the limit can be set differently for the three different classes of clients: # # normal -> normal clients including monitor clients # replica -> replica clients # pubsub -> clients subscribed to at least one pubsub channel or pattern # # the syntax of every client-output-buffer-limit directive is the following: # # client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds> # # a client is immediately disconnected once the hard limit is reached, or if # the soft limit is reached and remains reached for the specified number of # seconds (continuously). # so for instance if the hard limit is 32 megabytes and the soft limit is # 16 megabytes / 10 seconds, the client will get disconnected immediately # if the size of the output buffers reach 32 megabytes, but will also get # disconnected if the client reaches 16 megabytes and continuously overcomes # the limit for 10 seconds. # # by default normal clients are not limited because they don't receive data # without asking (in a push way), but just after a request, so only # asynchronous clients may create a scenario where data is requested faster # than it can read. # # instead there is a default limit for pubsub and replica clients, since # subscribers and replicas receive data in a push fashion. # # both the hard or the soft limit can be disabled by setting them to zero. client-output-buffer-limit normal 0 0 0 client-output-buffer-limit replica 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 # client query buffers accumulate new commands. they are limited to a fixed # amount by default in order to avoid that a protocol desynchronization (for # instance due to a bug in the client) will lead to unbound memory usage in # the query buffer. however you can configure it here if you have very special # needs, such us huge multi/exec requests or alike. # # client-query-buffer-limit 1gb # in the redis protocol, bulk requests, that are, elements representing single # strings, are normally limited ot 512 mb. however you can change this limit # here. # # proto-max-bulk-len 512mb # redis calls an internal function to perform many background tasks, like # closing connections of clients in timeout, purging expired keys that are # never requested, and so forth. # # not all tasks are performed with the same frequency, but redis checks for # tasks to perform according to the specified "hz" value. # # by default "hz" is set to 10. raising the value will use more cpu when # redis is idle, but at the same time will make redis more responsive when # there are many keys expiring at the same time, and timeouts may be # handled with more precision. # # the range is between 1 and 500, however a value over 100 is usually not # a good idea. most users should use the default of 10 and raise this up to # 100 only in environments where very low latency is required. hz 10 # normally it is useful to have an hz value which is proportional to the # number of clients connected. this is useful in order, for instance, to # avoid too many clients are processed for each background task invocation # in order to avoid latency spikes. # # since the default hz value by default is conservatively set to 10, redis # offers, and enables by default, the ability to use an adaptive hz value # which will temporary raise when there are many connected clients. # # when dynamic hz is enabled, the actual configured hz will be used as # as a baseline, but multiples of the configured hz value will be actually # used as needed once more clients are connected. in this way an idle # instance will use very little cpu time while a busy instance will be # more responsive. dynamic-hz yes # when a child rewrites the aof file, if the following option is enabled # the file will be fsync-ed every 32 mb of data generated. this is useful # in order to commit the file to the disk more incrementally and avoid # big latency spikes. aof-rewrite-incremental-fsync yes # when redis saves rdb file, if the following option is enabled # the file will be fsync-ed every 32 mb of data generated. this is useful # in order to commit the file to the disk more incrementally and avoid # big latency spikes. rdb-save-incremental-fsync yes # redis lfu eviction (see maxmemory setting) can be tuned. however it is a good # idea to start with the default settings and only change them after investigating # how to improve the performances and how the keys lfu change over time, which # is possible to inspect via the object freq command. # # there are two tunable parameters in the redis lfu implementation: the # counter logarithm factor and the counter decay time. it is important to # understand what the two parameters mean before changing them. # # the lfu counter is just 8 bits per key, it's maximum value is 255, so redis # uses a probabilistic increment with logarithmic behavior. given the value # of the old counter, when a key is accessed, the counter is incremented in # this way: # # 1. a random number r between 0 and 1 is extracted. # 2. a probability p is calculated as 1/(old_value*lfu_log_factor+1). # 3. the counter is incremented only if r < p. # # the default lfu-log-factor is 10. this is a table of how the frequency # counter changes with a different number of accesses with different # logarithmic factors: # # +--------+------------+------------+------------+------------+------------+ # | factor | 100 hits | 1000 hits | 100k hits | 1m hits | 10m hits | # +--------+------------+------------+------------+------------+------------+ # | 0 | 104 | 255 | 255 | 255 | 255 | # +--------+------------+------------+------------+------------+------------+ # | 1 | 18 | 49 | 255 | 255 | 255 | # +--------+------------+------------+------------+------------+------------+ # | 10 | 10 | 18 | 142 | 255 | 255 | # +--------+------------+------------+------------+------------+------------+ # | 100 | 8 | 11 | 49 | 143 | 255 | # +--------+------------+------------+------------+------------+------------+ # # note: the above table was obtained by running the following commands: # # redis-benchmark -n 1000000 incr foo # redis-cli object freq foo # # note 2: the counter initial value is 5 in order to give new objects a chance # to accumulate hits. # # the counter decay time is the time, in minutes, that must elapse in order # for the key counter to be divided by two (or decremented if it has a value # less <= 10). # # the default value for the lfu-decay-time is 1. a special value of 0 means to # decay the counter every time it happens to be scanned. # # lfu-log-factor 10 # lfu-decay-time 1 ########################### active defragmentation ####################### # # warning this feature is experimental. however it was stress tested # even in production and manually tested by multiple engineers for some # time. # # what is active defragmentation? # ------------------------------- # # active (online) defragmentation allows a redis server to compact the # spaces left between small allocations and deallocations of data in memory, # thus allowing to reclaim back memory. # # fragmentation is a natural process that happens with every allocator (but # less so with jemalloc, fortunately) and certain workloads. normally a server # restart is needed in order to lower the fragmentation, or at least to flush # away all the data and create it again. however thanks to this feature # implemented by oran agra for redis 4.0 this process can happen at runtime # in an "hot" way, while the server is running. # # basically when the fragmentation is over a certain level (see the # configuration options below) redis will start to create new copies of the # values in contiguous memory regions by exploiting certain specific jemalloc # features (in order to understand if an allocation is causing fragmentation # and to allocate it in a better place), and at the same time, will release the # old copies of the data. this process, repeated incrementally for all the keys # will cause the fragmentation to drop back to normal values. # # important things to understand: # # 1. this feature is disabled by default, and only works if you compiled redis # to use the copy of jemalloc we ship with the source code of redis. # this is the default with linux builds. # # 2. you never need to enable this feature if you don't have fragmentation # issues. # # 3. once you experience fragmentation, you can enable this feature when # needed with the command "config set activedefrag yes". # # the configuration parameters are able to fine tune the behavior of the # defragmentation process. if you are not sure about what they mean it is # a good idea to leave the defaults untouched. # enabled active defragmentation # activedefrag yes # minimum amount of fragmentation waste to start active defrag # active-defrag-ignore-bytes 100mb # minimum percentage of fragmentation to start active defrag # active-defrag-threshold-lower 10 # maximum percentage of fragmentation at which we use maximum effort # active-defrag-threshold-upper 100 # minimal effort for defrag in cpu percentage # active-defrag-cycle-min 5 # maximal effort for defrag in cpu percentage # active-defrag-cycle-max 75 # maximum number of set/hash/zset/list fields that will be processed from # the main dictionary scan # active-defrag-max-scan-fields 1000 eof docker run \ -p 6379:6379 \ -v $pwd/docker/redis/data:/data:rw \ -v $pwd/docker/redis/conf/redis.conf:/etc/redis/redis.conf:ro \ --privileged=true \ --name redis \ -d redis docker run -d --name zookeeper --publish 2181:2181 zookeeper docker run -d --name kafka --publish 9092:9092 \ --link zookeeper \ --env kafka_zookeeper_connect=zookeeper:2181 \ --env kafka_advertised_host_name=127.0.0.1 \ --env kafka_advertised_port=9092 wurstmeister/kafka:latest
suse12sp3 安装配置. net core 生产环境 - 总汇