Lecture
An idempotent function is a function that returns itself when applied to itself. In messaging, an idempotent receiver is able to process a duplicate message so that it leads to the same final system state as if the message had been processed only once.
First, is the message inherently idempotent? That is, does it contain an operation that holds a fixed value at a particular point in time, such as a stock quote, rather than a transformation instruction such as «add $5 to the balance». If the message is naturally idempotent, then much of the following discussion can be ignored, because reprocessing the message multiple times results in the same value, for example f(x) = f(f(x)).
If the message is not inherently idempotent, we will want to take steps to mitigate or eliminate the effects of processing the same message multiple times. In other words, we want artificial idempotency. In this case we need to consider the computational cost of reprocessing a particular message. Understanding this cost can be important, because it helps us determine whether we should handle message idempotency at the application level proactively or reactively.
One of the requirements for creating idempotency in messages that are not idempotent is creating a message store. This message store is associated with the identifier of the message being processed at the application level. It contains a list of the messages that were sent as a result of processing the received message. From here on we will refer to this simply as the message store .
When the computational cost of reprocessing a message is high compared to a query to determine whether the message has been processed, and the likelihood that the message will be received more than once, we will want to proactively filter out messages that have already been processed. Remember that the reasons for receiving the same message twice can vary and range from an unexpected process termination to other applications (including our own) resending messages.
When proactively filtering messages, we simply query our message store to determine whether the received message has already been processed. If it has, resend the messages loaded from the query and stop processing the current message. Note that the way messages are stored in our data store depends heavily on the transactional characteristics of the data store itself.
When we reactively filter a message, we know that the computational cost of processing is very low and/or the likelihood of a duplicate message is relatively low. In this case we simply process as usual. Then, when we attempt to save any outbound messages resulting from the processing to our data store, we catch the unique key violations (or similar exceptions) raised by our data store. If we catch an exception, we abort our current unit of work and resend the messages stored in the data store.
All of the above scenarios are well and good, but what do we do if we are not working with a transactional resource? What if we are performing an operation against a web service or some other RPC-based call using an unreliable protocol? How can we know whether the operation succeeded? This requires a bit more work, but still falls within the same general set of patterns described here. Generally, we want to follow the proactive approach described above, but with a few notable exceptions.
The main theme of this last «pattern» is that we cannot truly wrap all the work into a single unit and send it as a batch, because we are working with a non-transactional resource. One possible solution would be to simply query the non-transactional resource to determine the state of a given message. One of the problems is the latency of querying remote resources, such as web services, over HTTP. Other considerations might include the fact that some HTTP calls charge a fee per call, making it prohibitively expensive to run a status-check query for every single message. Finally, some resources might not even support a query operation.
At this point we should probably clarify a bit more what a non-transactional resource means. Although a resource can be transactional in the sense that it either performed or did not perform an operation, this is effectively all-or-nothing only for a single call, such as a web service. Moreover, our application might not even know whether the remote call succeeded or failed, because the response never made it back to our application. Or our process might have been terminated after the «request» packet was sent over the network, but before any «response» packet was received.
Additionally, the fact that we are working with a non-transactional resource does not mean that we cannot effectively model transactional behavior by recording application status in our own durable resource.
In cases where the resource supports queries, we can implement several methods to avoid excessive, and sometimes costly, calls. The first approach is to record the status of the call at each step in a durable resource. For example, when we are about to call a web service, we record that fact. When we receive the results of the web service call, we also record that fact. Finally, when we are about to send a message containing the results of the web service call, we record that message in our message store. When we receive a message from our own queue and process it, we should check the log to determine what work, if any, has been done for that particular message. If we previously recorded that we called the web service but no corresponding log entry was written for the results of the web service call,
Let's look at the above using the example of a call to the Meest express delivery web service, or perhaps processing a credit card transaction. If our application recorded that we called the web service, but the log has no entry containing the results, we can query the web service and ask what was done for credit card ABC or order 1010. Depending on the results, we can take appropriate action, for example, authorizing the credit card if the transaction was not received, or resuming from where the previous attempt failed.
The last «pattern» worth our consideration is when the resource does not support query operations. Here things get a bit tricky, because we have no way to know whether the work was actually done. The canonical example is sending an email while connected to an SMTP server. As in the previous Meest express call scenario, we can perform virtually all the same actions. The key difference is that we cannot run a query to determine the state of the resource involved. In this case we may have to accept a small level of inconsistency for a very specific failure condition.
The inconsistency mentioned is the possibility of sending an email twice (continuing the SMTP example). If our process terminates after we record that we are about to call the SMTP server (and possibly after the SMTP server was called), but before we can record that the SMTP server was called, we have absolutely no way to know what work was done. In this case we have no choice but to perform the operation again. At the same time, is it really so bad if the end user receives an email twice?
If clients can make the same call multiple times while producing the same result, for example: - Avoid processing a transaction multiple times.
An idempotent HTTP method is an HTTP method that can be called many times without different results. It does not matter whether the method is called only once or ten times. The result should be the same.
Examples
GET
The method is idempotent , as multiple GET calls to a resource will always return the same response.
PUT
The method is idempotent , as calling the PUT method multiple times will update the same resource and will not change the result.
POST
This is not idempotent , and calling the POST method multiple times may have different results and will lead to the creation of new resources.
DELETE
It is idempotent , because once a resource has been deleted it is gone, and calling the method multiple times will not change the result.
Conclusion
I hope you find some of these patterns useful when building applications using messaging patterns with guaranteed at-least-once delivery provided by the messaging infrastructure.
Of course, this is only a brief summary of some of the various «patterns» for creating idempotent messages. The examples are a bit arbitrary and contrived, and we can easily poke holes in them, but the principles still hold.
An idempotent receiver is useful for preventing out-of-order or duplicate message processing. In CPI, idempotency depends on the message itself and the receiver.
Comments