Lecture
Kappa Architecture is a software architecture used for processing streaming data. The core premise of the Kappa architecture is that you can perform both real-time processing and batch processing, especially for analytics, using a single technology stack. It is based on a streaming architecture in which an incoming series of data is first stored in a messaging engine such as Apache Kafka. From there, the stream processing engine reads the data and transforms it into an analyzable format, then stores it in an analytical database for end-user queries.
The Kappa architecture supports (near) real-time analytics, where data is read and transformed immediately after being inserted into the messaging engine. This makes the latest data quickly available for end-user queries. It also supports historical analytics by reading the stored streaming data from the messaging engine at a later time in batch mode, to produce additional analyzable output for more types of analysis.
The Kappa architecture is considered a simpler alternative to the Lambda Architecture, since it uses the same technology stack to handle both real-time stream processing and historical batch processing. Both architectures involve storing historical data to perform large-scale analytics. Both architectures are also useful for addressing the «human fault tolerance» problem, where issues in the processing code (whether bugs or simply known limitations) can be resolved by updating the code and rerunning it on historical data. The main difference in the Kappa architecture is that all data is processed as a stream, so the stream processing engine acts as the sole data transformation mechanism.
The Kappa architecture is usually built on Apache Kafka® together with a high-speed stream processing engine.
Despite all the merits of the Lambda architecture, the main drawback of this approach to designing Big Data systems is considered to be its complexity, caused by duplicating the data processing logic across the cold and hot paths. That is why, in 2014, Kappa was proposed as an alternative model that consumes fewer resources while being well suited to processing events in real time.
Unlike Lambda, in the Kappa architecture streaming data flows along a single path. All data is ingested as a stream of events into a distributed, fault-tolerant, single log – the event log. There, events are ordered, and the current state of an event changes only when a new event is appended. Similar to the speed layer of the Lambda architecture, all event processing is performed on the input stream and stored as a real-time view. If the entire dataset needs to be recomputed, as with the batch layer in the Lambda architecture, the stream is replayed from the beginning. Parallelism is used to complete the computation in a timely manner.
The functional equation that defines a Big Data query in the Kappa architecture looks like this:
Query = K (New Data) = K (Live streaming data)
This equation means that all queries can be handled by applying the Kappa function (K) to real-time data streams at the speed layer.
Thus, it can be said that the Kappa architecture is a simplification of the Lambda approach to designing Big Data systems, in which the batch processing layer has been removed from the model. Here, the canonical data store is an immutable, append-only log. From the log, data is passed directly into the stream computation system, being enriched as needed with data from the non-canonical store (the serving layer). The purpose of the serving layer is to provide optimized responses to queries. However, these stores are not canonical (immutable): they can be wiped and rebuilt from the canonical data store at any time.
The following Big Data technologies are used to implement the Kappa architecture:
Kappa architecture for Big Data systems
So, the Kappa architecture is well suited to enterprise data processing models where:
All of these scenarios are excellently covered by the Apache Kafka message broker – a fast, fault-tolerant, and horizontally scalable system for collecting and aggregating big data. That is why the Kafka-based Kappa architecture is actively used by LinkedIn and other Big Data projects where it is necessary to retain a large volume of data to serve queries that are essentially simple copies of one another.
Streaming architecture is a specific set of technologies that work together to handle stream processing, i.e., the practice of acting on a series of data as it is generated. In many modern deployments, Apache Kafka acts as the store for streaming data, after which several stream processors can act on the data stored in Kafka to produce multiple outputs. Some streaming architectures include workflows for both stream processing and batch processing, which either involves other technologies for handling large-scale batch processing, or uses Kafka as the central store, as specified in the Kappa architecture.
First of all, it should be noted that, given the shared goal of building a reliable and fast big data processing system, the Lambda and Kappa approaches do not compete with each other but can be used together for different cases. In particular, for reliable work with a Data Lake based on Apache Hadoop and machine learning models for predicting future events based on historical data, the Lambda approach should be chosen. On the other hand, if you need to inexpensively deploy a Big Data system for efficiently processing unique events in real time without historical analysis, the Kappa architecture handles that task excellently. Kappa is suited to machine learning algorithms that train online and do not need a batch layer. Thus, Kappa is characterized by the following advantages:
Nevertheless, the absence of a batch layer can lead to errors when processing information or updating the database. That is why the Kappa architecture requires an exception manager for reprocessing data or performing reconciliation.
Both architectures handle real-time analytics and historical analytics within a single environment. However, one of the main advantages of the Kappa architecture over the Lambda architecture is that it lets you build a stream and batch processing system on a single technology. This means you can create a stream processing application to handle real-time data, and if you need to change the output, you update your code and then rerun it over the data in the messaging engine in batch mode. There is no separate technology for handling batch processing, as is proposed in the Lambda Architecture.
Differences between the Kappa and Lambda architectures
With a sufficiently fast stream processing engine (for example, Hazelcast Jet), you may not need a separate technology optimized for batch processing. You simply read the stored streaming data in parallel (assuming the data in Kafka is properly split into separate channels, or «partitions») and transform the data as if it were coming from a streaming source. In some environments, you can potentially generate analyzable output on demand, so when a new query is sent by the end user, the data can be transformed specifically to provide the optimal response to that query. Again, this requires a high-speed stream processing engine to ensure low processing latency.
Although the Lambda Architecture does not specify which technologies must be used, the batch processing component is often run on a large-scale data platform such as Apache Hadoop. The Hadoop Distributed File System (HDFS) can economically store raw data, which can then be transformed into an analyzable format using Hadoop tools. While Hadoop is used for the batch processing component of the system, a separate engine designed for stream processing is used for the real-time analytics component. However, one advantage of the Lambda architecture is that much larger datasets (in the petabyte range) can be stored and processed more efficiently in Hadoop for large-scale historical analysis.
Comments