Lecture
Testing distributed systems differs substantially from testing centralized ones. Few testers can boast serious knowledge and experience in this area.
The fault injection approach is very popular. While the system is running, we use special programs and mechanisms to introduce faults: disk failures or failures of entire machines, possibly network failures, failures of internal components of the system under test. Since the vast majority of distributed systems must be resilient to this kind of failure, at least on some limited scale, the system must not stop working or exhibit any anomalies in its behavior. In essence this is fault-tolerance testing, since it is one of the most important non-functional requirements for distributed systems. The more machines are running, the higher the probability of an individual problem on one of them. For example, if a thousand machines are involved, then, roughly speaking, disks will fail about once a week. The system must survive such situations without noticing them in any way.
There are more academic approaches, for example formal verification. Distributed systems have internal algorithms and protocols that allow them to work. These are themselves fairly complex, but they guarantee certain invariants that must always hold, regardless of any failures in the system, reordering of packets on the network, or anything else. The essence of the approach is that the correctness of an algorithm is checked based solely on its description in a special language. This gives confidence that the algorithm being used, provided it is correctly implemented, will work.
In 2015, Microsoft Research published an academic paper «Proving Practical Distributed Systems Correct», in which they described a model of a distributed storage system, then used special tools to verify the correctness of this model, and then generated code that worked right away.
For example, nosql databases are popular nowadays; they can be more high-performance, but they do not support transactions. That is, their consistency level is lower than that of classical ones (MySql, PostgreSQL, Oracle). And when such a distributed system, like a nosql database, is being tested, it is important to understand exactly which invariants it supports. The anomalies that will be observed in the tests depend on this. In complex tests, for example when there are several concurrent writers and readers, you can see many different states. In other words, you need to understand which effects can be observed in the system, and which — cannot.
– Do we need to create special clusters for testing, or can we use «live» clusters that are in production? How do you determine the optimal size of a test cluster?
– Test clusters are what is used most often. If we are talking about testing on production servers, then the most widely known example is Netflix, which actively promotes its approach called «simian army», that is, the «army of monkeys». It consists in doing fault injection in production. They kill nodes right during working hours, and the developers watch to make sure the system does not degrade in any way as a result. But here it must be understood that this kind of possibility only appears starting from a certain scale. If the system runs on 10-20 nodes, then testing in this way means there will be 5-10% degradation. Not everyone in production is ready to make such sacrifices. In addition, there may be some kind of service level agreement (SLA), and such testing can be expensive because of its violation. In any case, even where the practice of testing in production is found, there exists, before it, an enormous test infrastructure that catches most of the defects. The advantage of testing in production is only that there is no need to replicate the production environment.
Regarding the size of the test cluster. If the system is distributed, it must be greater than one — that is the lower bound. On the subject of upper bounds there is an article «Simple Testing Can Prevent Most Critical Failures», which investigates the question of what errors exist in distributed systems. According to the article, the researchers concluded that 98% of all defects can be reproduced on just 3 nodes. Specifically, in our own work we use more than that; our test cluster usually consists of 8 nodes, but this is related to the internal design of our system.
– How do you deal with complete or partial failures of a distributed system during testing?
– There's probably no special way to deal with this, because the scale in the test environment is much smaller. If faulty hardware is really getting in the way, it can simply be excluded from the test environment. We had a case where a failure occurred on test hardware, but we were actually glad about it, since it allowed us to find some unusual defects. Since a distributed system must be resilient to failures, this shouldn't cause any problems even in tests.
– What specific technologies and tools are used to build the test environment? For test automation?
– The test environment depends on the technologies used for development, and on the technologies the team is familiar with. We, for example, actively use Python, because it is well suited to such tasks and our testers know it. It is simple for writing tests, high-level enough that you can write clearly with it. In my view, it has a bit of "trouble" with concurrency, but that problem is solvable. The system itself is developed in C++, but using it for high-level tests is quite hard, since you can't develop quickly and easily with it, and development speed sometimes matters in tests.
Regarding test automation. Usually a repository of tests is built, which are run automatically on a special server. For this we use TeamCity and some of our own internal tools.
– Is there anything else you would like to add on the topic?
– I would like to add that there is a huge amount of material on the topic of testing distributed systems, both academic and close to industry, and a huge variety of approaches and testing methods. The search for methods and their improvement never stops for a single day. This topic is constantly developing – and that is exactly what makes it interesting.
Chaos Monkey, a tool that randomly disables our production instances, to make sure we can survive this common type of failure without any impact on the customer. The name comes from the idea of unleashing a wild monkey with a weapon in your data center (or cloud region) to randomly knock down instances and chew through cables - all the while we continue to serve our customers without interruption. By running Chaos Monkey in the middle of the working day in a carefully controlled environment with engineers ready to solve any problems, we can still learn lessons about the weak points of our system and build automatic recovery mechanisms to fix them. So the next time an instance failure happens at 3 a.m. on a Sunday, we won't even notice it.
Inspired by the success of Chaos Monkey, we started creating new monkeys that cause various kinds of failures or detect abnormal conditions and test our ability to survive them; a virtual monkey army ensuring the security, reliability, and high availability of our cloud.
Latency Monkey introduces artificial delays at our client-server RESTful communication layer, to simulate degraded quality of service, and measures whether upstream services respond appropriately. In addition, by making very large delays, we can simulate the downtime of a node or even an entire service (and test our ability to survive that) without physically shutting down those instances. This can be especially useful when testing the fault tolerance of a new service by simulating the failure of its dependencies, without making those dependencies unavailable to the rest of the system.
Conformity Monkey finds instances that do not conform to best practices, and shuts them down. For example, we know that if we find instances that don't belong to an auto-scaling group, that spells trouble. We shut them down, to give the service owner the opportunity to relaunch them properly.
Doctor Monkey taps into health checks that run on each instance, as well as monitoring other external signs of health (such as CPU load) to detect unhealthy instances. Once unhealthy instances are detected, they are removed from service, and after giving the service owners time to fix the root cause of the problem, they are eventually terminated.
Janitor Monkey ensures that our cloud environment is free of clutter and waste. It searches for unused resources and gets rid of them.
Security Monkey - is an extension of Conformity Monkey. It detects security violations or vulnerabilities, such as improperly configured AWS security groups, and terminates the offending instances. It also ensures that all our SSL and DRM certificates are valid and not subject to renewal.
10–18 Monkey (short for Localization-Internationalization, or l10n-i18n) detects configuration and runtime issues in cases where customers are served across multiple geographic regions, using different languages and character sets.
Chaos Gorilla is similar to Chaos Monkey, but simulates the failure of an entire Amazon availability zone. We want to make sure that our services automatically reconfigure into functional availability zones without any user-visible impact or manual intervention.
Comments