Lecture
Redis (which stands for Remote Dictionary Server) – is a fast
in-memory «key-value» data store with open
source code for use as a database, cache,
message broker or queue.
Redis delivers sub-millisecond response times, enabling
real-time applications to perform
millions of requests per second. Such applications are in demand in the fields of gaming,
ad technology, financial services, and healthcare.
Redis is widely used for caching, session management,
game development, building leaderboards, real-time
analytics, working with geospatial data, powering ride-hailing
services, chats and messaging services, and streaming.
of multimedia and applications that send messages using the
«publisher – subscriber» (Pub/Sub) model.
Essentially, Redis is a large distributed hash table, where an arbitrary string is used as the key, and one of the supported data structures (strings, lists, hash tables, sets, sorted sets, etc.) is used as the value. Let's briefly run through its characteristics:
Persistence: there are two storage modes — RDB and AOF.
Fault-tolerance support in various topologies for every occasion and any task.
Information security: access control and data encryption.
Configuration management and monitoring.
Persistence — is an important aspect that deserves a closer look. Redis supports two storage modes — RDB and AOF.
AOF (append only file) implements an operation log, where all new operations are simply appended to the end of the file. This file is human-readable, meaning it can be read and even edited. Unlike RDB, AOF does not block Redis.
This mode also has its drawbacks: for example, the gradual growth of the log. But to solve this problem, its compaction is provided — AOF Rewrite. When the file grows too large, Redis runs AOF Rewrite and collapses some commands. For example, if we incremented a counter five times, then after the algorithm runs, instead of five commands only one will remain.
RDB (Redis database) periodically saves a snapshot of the entire dataset. To do this, the main process is forked and the snapshot-writing procedure — BGSAVE — is launched. For this reason, saving a snapshot during intensive data modification will, in the most general case, require twice the amount of RAM relative to the size of the dataset.
Of course, the modern OS kernel has a Copy-on-Write mode, but it only helps when the intensity of data changes in RAM is low. And if we are talking about hundreds, millions, and billions of keys, then these are very substantial volumes.
RDB, unlike AOF, leads to blocking and is not recommended for use on master nodes.
Single node or stand-alone. The simplest topology. It has a right to exist, but exclusively for development and testing environments, since it is not fault-tolerant.
Master-Replica (Secondary). In such a topology, continuous asynchronous replication takes place between the master and the replica. For forced synchronization, that is, for switching to synchronous mode, Redis has a special command WAIT.
Sentinel. This deployment topology was widely used in early versions of Redis, before full-fledged cluster support. It consists of separate special nodes that monitor the operation of the main Redis nodes, track Master failures, and initiate the recovery process. It works on top of the Master-Replica topology.
A full-fledged cluster, which consists of a set of master nodes and a set of replicas (secondary nodes). The Gossip protocol is used for replication: information spreads through it in a way similar to an epidemic — each node passes information to the «neighbors» it knows. Clients can work with both the master and the replica, but only reads go to the replica.
Data in the cluster is sharded, that is, divided into segments. The cluster does not use consistent hashing; instead, so-called hash slots are used.
In total, the cluster has 16384 slots. To compute the hash slot for a key, the formula crc16(key) % 16384 is used.
Each Redis node is responsible for a specific subset of hash slots. For example:
Node A contains hash slots from 0 to 5500.
Node B contains hash slots from 5501 to 11001.
Node C contains hash slots from 11001 to 16383.
This makes it easy to add and remove cluster nodes, that is, to perform fast resharding.
Redis supports a huge number of data structures. The main ones are, of course, string, list, hash, table and set. For working with each of them there is a set of operations: with strings we work using get, mget, set, append; with lists we use lpush, lpop, ltrim, llen. There are so many operations and data structures that it is simply impossible to describe them all in this article. Therefore, I propose to consider only the specific data types: Sorted Set, Bitmap, HyperLogLog, Stream.
Sorted Set. Based on this structure, you can build the already familiar «sliding window» algorithm (Rate Limiter):
For scoring, you can use the timestamp of the incoming request.
Outdated elements that have fallen outside the window can be removed with the ZREMRANGEBYSCORE command.
The number of requests in the window is computed using the ZCARD command.
If the number of requests does not exceed the limit, then a new request can be added to the window with the ZADD command.
Comments