Lecture
In-memory database (IMDB) — a database stored in random-access memory (RAM). In-memory DBMS — a database management system for in-memory databases, one of the types of software systems operating in the paradigm of in-memory computing.
Thanks to the optimizations possible when storing and processing data in byte-addressable RAM, in-memory DBMSs deliver better performance than DBMSs that work with databases on persistent storage devices, which typically have a block-based organization and are connected via bus or network interfaces. At the same time, the size of an in-memory database is limited by the RAM capacity of the node. A number of in-memory DBMSs implement replication and sharding techniques, allowing a single in-memory database to run across multiple nodes. Since RAM is volatile, write-ahead logging to a non-volatile device is used to ensure database integrity in the event of a sudden restart — that is, working with an in-memory database does not eliminate dependence on I/O subsystem performance (though it does reduce it).
In-memory databases are widely used for applications where response time is critical, in particular for telecommunications equipment management tasks and real-time trading. Real-time analytics and hybrid transactional/analytical processing (HTAP) are noted as effective use cases for in-memory databases.
The first relational in-memory DBMSs appeared in the 1990s, including SolidDB[en] (1992) and Timesten[en] (1997, acquired by Oracle in 2005). The field gained popularity as the per-unit cost of server RAM modules fell and with the development of the SAP HANA in-memory DBMS (2010). By the mid-2010s, the major commercial relational DBMSs (Oracle Database — in-memory option, DB2 — BLU, MS SQL Server) had been equipped with additional mechanisms for working with in-memory databases. Another direction that developed in the 2010s, within the NoSQL movement, was in-memory «key-value» DBMSs, providing basic key-based access capabilities and high performance under horizontal scalability.
A potential technical obstacle to storing data in memory is the volatility of RAM. In particular, in the event of a power loss, whether intentional or otherwise, data stored in volatile RAM is lost. With the introduction of non-volatile random-access memory technology, in-memory databases will be able to run at full speed while also preserving data in the event of a power failure.
In their simplest form, main-memory databases store data on volatile memory devices. These devices lose all stored information when the device loses power or restarts. In this case, it can be said that IMDBs do not support the «durability» property of ACID (atomicity, consistency, isolation, durability). IMDBs based on volatile memory can, and often do, support the other three ACID properties: atomicity, consistency, and isolation.
Many IMDBs have increased reliability through the following mechanisms:
Some IMDBs allow the database schema to specify different durability requirements for selected areas of the database — so rapidly changing data that can easily be regenerated, or that is meaningless after a system shutdown, does not need to be logged for durability (although it will still need to be replicated for high availability), while configuration information is marked as requiring persistence.
Although storing data in memory delivers performance advantages, it is an expensive way to store data. One approach to realizing the benefits of in-memory storage while limiting costs is to keep the most frequently used data in memory and the rest on disk. Since there is no hard-and-fast rule about which data should be kept in memory and which on disk, some systems dynamically update the storage location of data depending on usage patterns.[10] This approach differs slightly from caching, in which data that was most recently accessed is cached, as opposed to data that is most frequently accessed, which is kept in memory.
The flexibility of hybrid approaches allows a balance to be struck between:
In the cloud computing industry, the terms «data temperature», or «hot data» and «cold data», have emerged to describe how data is stored in this respect.[11] Hot data is used to describe business-critical data that requires frequent access, while cold data describes data that is needed less often and less urgently — for example, data kept for archival or audit purposes. Hot data should be stored in a way that offers fast retrieval and modification, which is often, but not always, achieved through in-memory storage. Cold data, on the other hand, can be stored more cost-effectively, and access to it is generally expected to be slower compared to hot data. While these descriptions are useful, no precise definition of the terms «hot» and «cold» exists.
Production efficiency is another reason for choosing a combined in-memory and on-disk database system. Some device lines, especially in consumer electronics, include some devices with persistent storage while others use memory for storage (for example, set-top boxes). If such devices require a database system, the manufacturer may adopt a hybrid database system at lower cost and with less code customization, rather than using separate in-memory and disk-based databases for diskless and disk-based products, respectively.
The first database engine to support both in-memory and on-disk tables within a single database, WebDNA, was released in 1995.
Another option involves large volumes of non-volatile memory on a server — for example, flash memory chips used as addressable memory rather than structured as disk arrays. A database in this form of memory combines very high access speed with resilience across restarts and power loss.
There are a number of commercial and open-source in-memory database implementations. These include (in alphabetical order):
The term «in-memory database» is not entirely clear. Conventional databases also typically use main memory to avoid slow I/O operations, such as those on a hard disk. Depending on the system configuration, the entire database may also be stored in main memory. Moreover, the data or index structure is usually more important to a database system's performance than the access speed of the storage medium. Only optimizing these data structures in main memory, without structures for I/O access, represents a genuine difference from conventional databases.
The high speed of «in-memory databases» is achieved partly by limiting durability and consistency guarantees. This is entirely legitimate for databases that are primarily intended for reading, but problematic for other use cases. To compensate for these issues, some products, such as Redis and MySQL, are also often combined, with the «in-memory database» then serving as a cache.[8
Comments