InnoDB Architecture （InnoDB In-Memory Structures 转载）

转载、节选于 

innodb architecture

the following diagram shows in-memory and on-disk structures that comprise the innodb storage engine architecture. 

 

2.innodb in-memory structures

2.1 buffer pool

the buffer pool is an area in main memory where caches table and index data as it is accessed. the buffer pool permits frequently used data to be processed directly from memory, which speeds up processing. on dedicated servers, up to 80% of physical memory is often assigned to the buffer pool.

for efficiency of high-volume read operations, the buffer pool is divided into pages that can potentially hold multiple rows. for efficiency of cache management, the buffer pool is implemented as a linked list of pages; data that is rarely used is aged out of the cache using a variation of the lru algorithm.

buffer pool lru algorithm

the buffer pool is managed as a list using a variation of the least recently used (lru) algorithm. when room is needed to add a new page to the buffer pool, the least recently used page is evicted and a new page is added to the middle of the list. 

his midpoint insertion strategy treats the list as two sublists:

• at the head, a sublist of new (“young”) pages that were accessed recently

• at the tail, a sublist of old pages that were accessed less recently

the algorithm keeps pages that are heavily used by queries in the new sublist. the old sublist contains less-used pages; these pages are candidates for .

by default, the algorithm operates as follows:

• 3/8 of the buffer pool is devoted to the old sublist.

• the midpoint of the list is the boundary where the tail of the new sublist meets the head of the old sublist.

• when innodb reads a page into the buffer pool, it initially inserts it at the midpoint (the head of the old sublist). a page can be read because it is required for a user-specified operation such as an sql query, or as part of a read-ahead operation performed automatically by innodb.

• accessing a page in the old sublist makes it “young”, moving it to the head of the buffer pool (the head of the new sublist). if the page was read because it was required, the first access occurs immediately and the page is made young. if the page was read due to read-ahead, the first access does not occur immediately (and might not occur at all before the page is evicted).

• as the database operates, pages in the buffer pool that are not accessed “age” by moving toward the tail of the list. pages in both the new and old sublists age as other pages are made new. pages in the old sublist also age as pages are inserted at the midpoint. eventually, a page that remains unused reaches the tail of the old sublist and is evicted.

by default, pages read by queries immediately move into the new sublist, meaning they stay in the buffer pool longer. a table scan (such as performed for a mysqldump operation, or a select statement with no where clause) can bring a large amount of data into the buffer pool and evict an equivalent amount of older data, even if the new data is never used again. similarly, pages that are loaded by the read-ahead background thread and then accessed only once move to the head of the new list. these situations can push frequently used pages to the old sublist where they become subject to eviction.

2.2 change buffer

the change buffer is a special data structure that caches changes to secondary index pages when those pages are not in the buffer pool. the buffered changes, which may result from insert, update, or delete operations (dml), are merged later when the pages are loaded into the buffer pool by other read operations.

unlike clustered indexes, secondary indexes are usually nonunique, and inserts into secondary indexes happen in a relatively random order. similarly, deletes and updates may affect secondary index pages that are not adjacently located in an index tree. merging cached changes at a later time, when affected pages are read into the buffer pool by other operations, avoids substantial random access i/o that would be required to read secondary index pages into the buffer pool from disk.merging cached changes at a later time, when affected pages are read into the buffer pool by other operations, avoids substantial random access i/o that would be required to read secondary index pages into the buffer pool from disk.

 periodically, the purge operation that runs when the system is mostly idle, or during a slow shutdown, writes the updated index pages to disk. the purge operation can write disk blocks for a series of index values more efficiently than if each value were written to disk immediately.

change buffer merging may take several hours when there are many affected rows and numerous secondary indexes to update. during this time, disk i/o is increased, which can cause a significant slowdown for disk-bound queries. change buffer merging may also continue to occur after a transaction is committed, and even after a server shutdown and restart

the type of data cached in the change buffer is governed by the innodb_change_buffering variable. 

change buffering is not supported for a secondary index if the index contains a descending index column or if the primary key includes a descending index column.

when insert, update, and delete operations are performed on a table, the values of indexed columns (particularly the values of secondary keys) are often in an unsorted order, requiring substantial i/o to bring secondary indexes up to date. the  caches changes to secondary index entries when the relevant page is not in the buffer pool, thus avoiding expensive i/o operations by not immediately reading in the page from disk. the buffered changes are merged when the page is loaded into the buffer pool, and the updated page is later flushed to disk. the innodb main thread merges buffered changes when the server is nearly idle, and during a slow shutdown.

because it can result in fewer disk reads and writes, the change buffer feature is most valuable for workloads that are i/o-bound, for example applications with a high volume of dml operations such as bulk inserts.

however, the change buffer occupies a part of the buffer pool, reducing the memory available to cache data pages. if the working set almost fits in the buffer pool, or if your tables have relatively few secondary indexes, it may be useful to disable change buffering. if the working data set fits entirely within the buffer pool, change buffering does not impose extra overhead, because it only applies to pages that are not in the buffer pool.

you can control the extent to which innodb performs change buffering using the innodb_change_buffering configuration parameter. you can enable or disable buffering for inserts, delete operations (when index records are initially marked for deletion) and purge operations (when index records are physically deleted). an update operation is a combination of an insert and a delete. the default innodb_change_buffering value is all.

the innodb_change_buffer_max_size variable permits configuring the maximum size of the change buffer as a percentage of the total size of the buffer pool. by default,innodb_change_buffer_max_size is set to 25. the maximum setting is 50.

test different settings with a representative workload to determine an optimal configuration. the innodb_change_buffer_max_size setting is dynamic, which permits modifying the setting without restarting the server.

2.3 adaptive hash index

the adaptive hash index feature enables innodb to perform more like an in-memory database on systems with appropriate combinations of workload and sufficient memory for the buffer pool without sacrificing transactional features or reliability. the adaptive hash index feature is enabled by the innodb_adaptive_hash_indexvariable, or turned off at server startup by --skip-innodb-adaptive-hash-index.

based on the observed pattern of searches, a hash index is built using a prefix of the index key. the prefix can be any length, and it may be that only some values in the b-tree appear in the hash index. hash indexes are built on demand for the pages of the index that are accessed often.

if a table fits almost entirely in main memory, a hash index can speed up queries by enabling direct lookup of any element, turning the index value into a sort of pointer. innodb has a mechanism that monitors index searches. if innodb notices that queries could benefit from building a hash index, it does so automatically.

with some workloads, the speedup from hash index lookups greatly outweighs the extra work to monitor index lookups and maintain the hash index structure. access to the adaptive hash index can sometimes become a source of contention under heavy workloads, such as multiple concurrent joins. queries with like operators and %wildcards also tend not to benefit. for workloads that do not benefit from the adaptive hash index feature, turning it off reduces unnecessary performance overhead. because it is difficult to predict in advance whether the adaptive hash index feature is appropriate for a particular system and workload, consider running benchmarks with it enabled and disabled. architectural changes in mysql 5.6 make it more suitable to disable the adaptive hash index feature than in earlier releases.

the adaptive hash index feature is partitioned. each index is bound to a specific partition, and each partition is protected by a separate latch. partitioning is controlled by the innodb_adaptive_hash_index_parts variable. the innodb_adaptive_hash_index_parts variable is set to 8 by default. the maximum setting is 512.

you can monitor adaptive hash index use and contention in the semaphores section of show engine innodb status output. if there are numerous threads waiting on rw-latches created in btr0sea.c, consider increasing the number of adaptive hash index partitions or disabling the adaptive hash index feature.

2.4 log buffer

the log buffer is the memory area that holds data to be written to the log files on disk. log buffer size is defined by the innodb_log_buffer_size variable. the default size is 16mb. the contents of the log buffer are periodically flushed to disk. a large log buffer enables large transactions to run without the need to write redo log data to disk before the transactions commit. thus, if you have transactions that update, insert, or delete many rows, increasing the size of the log buffer saves disk i/o.

the innodb_flush_log_at_trx_commit variable controls how the contents of the log buffer are written and flushed to disk. the innodb_flush_log_at_timeout variable controls log flushing frequency.

 

转载、节选于