Lecture
In software testing, monkey testing is a method in which the user tests applications or systems by providing random inputs and checking the behavior, or seeing whether the application or system will crash. Monkey testing is usually implemented as random automated unit tests . Monkey test ( eng. Monkey Test ) - in computer science, an automated test that is executed without a specific, clearly defined test scenario . The name is metaphorical, meaning that the data input operations are random and meaningless, as if performed by a monkey. For example, a monkey test can enter arbitrary strings into input fields to check the correctness of the system's operation when any possible data is entered. The input data can be set in advance, randomly generated, or generated taking into account user statistics
While the source of the name «monkey» is uncertain, some believe that the name is related to the infinite monkey theorem , which states that a monkey hitting keys at random on a typewriter keyboard for an infinite amount of time will almost surely type a given text, such as the complete works of William Shakespeare . Some believe that this name comes from the classic Mac OS application «Monkey», developed by Steve Capps before 1983. It used logging traps to feed random events into Mac programs and was used to test bugs in MacPaint .
Monkey Testing is also included in Android Studio as part of the standard testing tools for stress testing .
Monkey testing can be divided into smart monkey tests or dumb monkey tests .
Smart monkeys are usually identified by the following characteristics:
Some smart monkeys are also called genius monkeys , , which perform testing according to user behavior and indicate certain probabilities of errors , to be damaged.
Dumb monkeys, also known as «ignorant monkeys», are usually identified by the following characteristics
Monkey testing is an effective way to identify some non-standard bugs. Since the scenarios tested are usually ad hoc , monkey testing can also be a good way to perform load and stress testing. The inherent randomness of monkey testing also makes it a good way to find serious bugs that can disrupt the operation of the entire system. Setting up monkey testing is simple, so it is suitable for any application. Smart monkeys, if properly configured using an accurate state model, can be really good at finding various kinds of bugs.
The randomness of monkey testing often makes the bugs found difficult or impossible to reproduce. Unexpected bugs found during monkey testing can also take a lot of time to analyze. In some systems, monkey testing can continue for a long time before a bug is discovered. For smart monkeys, the ability strongly depends on the state model provided, and developing a good state model can be expensive.
Although monkey testing is sometimes treated the same as fuzz testing , and the two terms are usually used together, some believe that they differ, arguing that monkey testing is more about random actions, while fuzz testing is more about random data input . Monkey testing also differs from ad hoc testing in that ad hoc testing is performed without planning and documentation, and the goal of ad hoc testing is to randomly divide the system into parts and check their functionality, which is not the case in monkey testing.
This article argues that conventional unit testing environments do not offer
adequate support for the systematic testing of object-oriented units. A model based on a finite state machine and a more structured system is presented as the proposed
improvement.
1 What's wrong with the traditional approach?
When considering how to test OO units 3,
two aspects are important: state and behavior. Therefore, exhaustive testing of an object-oriented
unit essentially means that for all possible states of the unit, the object's behavior
is asserted, ensuring that the unit handles all possible input data
correctly. Of course, exhaustive testing of any non-trivial unit would in most cases
take too much time to be practical. Therefore, the tester should select
a number of important states and a number of important input sequences,
using, for example, domain analysis or equivalence class partitioning .
Usually automated unit testing is carried out by writing scripts,
where the implementation under test (IUT) is brought into a certain state,
while the functionality of the IUT is checked against the technical specifications.
Unfortunately, conventional test scenarios are often written on an ad hoc
basis, without analyzing the true nature of the IUT. Thus, it is easy
to miss a state whose functionality needs to be tested, or to fail to check some aspect
of the functionality in a certain state.
In addition, the conventional way of unit testing is relatively static,
which means that the computer is not involved in testing in any other
1 http://www.xprogramming.com
2 http://www.junit.org
3 In the object-oriented domain, a unit is essentially an object.
1 way than checking the unit's output against some set of predefined
values.
2 Finite state machine - model-based unit testing
The finite state machine model-based approach to unit testing requires the tester
to develop a finite state machine model of the unit. The model must contain
the states important for testing, and the transitions between these states.
The transitions must effectively check all messages that can be used to get
from one state to another.
2.1 Example test model
Consider a simple example: a stack with a limited maximum capacity.
(assume that the stack can contain no more than ten objects). A simple model
for this case would include three states: empty (the stack contains no objects),
full (the stack contains the maximum number of objects), and loaded (the stack is
neither empty nor full). The empty and full categories contain only one state,
but there is a range of real states that fall into the loaded category. For
simplicity, one real state is chosen to represent all loaded category-states.
In this example, the stack in the loaded state will contain four objects.
In this example the stack has three methods that change the state of
the stack: push (adds an object to the top of the stack), pop (removes the topmost
object of the stack), and clear (removes all objects from the stack). Given
these methods and the set of states identified earlier, six transitions
are identified:
1. From empty to loaded: four push calls.
2. From loaded to full: six push calls.
3. From loaded to empty: four pop calls.
4. From loaded to empty: one clear call.
5. From full to loaded: six pop calls.
6. From full to empty: one clear call.
2.2 Corresponding Oracles
An oracle gives the expected results for a test case (Binder, 2000). This
section describes several oracle patterns relevant to finite state machine
testing.
2.2.1 Resolved-example Oracle
The resolved-example oracle relies on a set of predefined assertions, usually
chosen with the help of equivalence partitioning and boundary value analysis. In
the case of finite state machine-based testing this means that for each modeled
state there is a set of tests that guarantee the correct operation of the IUT. In
essence, these tests guarantee that the state invariant holds for each state. To
avoid redundant test code, a class invariant test should also be written. The
class invariant will be asserted after each transition.
2.2.2 A different, but equivalent Oracle
A different, but equivalent oracle uses the characteristics of the state
machine model. When two separate instances of the IUT are brought into
the same state, they must be logically equal, even if the state is reached via
different transition paths.
Since the IUT is usually transitioned from one state to another very quickly,
it is easy to explore a huge number of transitions and see which
sequences break the implementation.
2.2.3 Smoke test
A somewhat trivial, but nevertheless important, aspect of model-based testing is the smoke
test. A smoke test means that the IUT is exercised with
a number of presumably legitimate message sequences, to see whether a crash actually occurs at some point. A smoke test simply passes when no
abnormal state is detected.
Again, a smoke test performed with a finite state machine-based tester only requires specifying how the IUT is transitioned from one state to another.
(although applying methods that are not involved in generating the state
transitions of course helps)
If the stack that was used as an example uses an array to store its
contents, a smoke test might, for example, reveal logic errors when accessing
table indices (assuming the implementation of the programming language used can determine when an array index is out of bounds), or if
3
the last possible push operation does not add an item to the stack (the last
pop operation will probably fail).
This kind of smoke test can also detect memory leaks if a very long
sequence of transitions is performed for a single instance of the system under
test.
2.2.4 Built-in Oracle
Some languages, such as Eiffel and Java (starting with version 1.4), have direct support
for an assertion facility that can be used to write built-in assertions into the code
to ensure that invariants are true.
However, using built-in assertions does not guarantee that errors are actually
found - the code must first be executed with input(s) that reveal
the errors. A well-written finite state machine-based tester gives the code good
exercise, giving built-in assertions the opportunity to reveal possible errors.
2.3 Selecting transition paths
There are several ways to select transition paths for executing
finite state machine-based tests. This section describes several options.
2.3.1 All transitions
The computer selects paths so that all transitions are executed at least
once. If all transitions cannot be executed with a single instance of the unit
under test, the computer creates as many instances as necessary.
This method can be used in situations where other methods would require
an unnecessarily long time to execute.
2.3.2 Random paths
Given a seed for the random generator, a maximum path length, and
a number of repetitions, the computer randomly switches the instance
of the unit under test according to the finite state machine model. When a dead-end
state is reached or the maximum path length is reached, the computer creates a new
instance and starts again.
2.3.3 Transition tree
Given a maximum depth, the computer generates a transition tree and
executes all transitions up to the terminal nodes of that tree. In other
4
words, the computer executes all the different paths that are at most
the given length (the depth of the tree).
2.3.4 Cyclic test
When traversing the finite state machine model, the computer selects (if possible)
transitions that do not lead to a dead end. In other words, the computer
executes only cyclic subpaths. The goal is to earn a very large number of
transitions with a single instance of the unit under test. This is efficient
for smoke testing.
3 monkeys
Monkey4 Unit is a finite state machine-based unit testing framework for Java,
currently under construction. It will be an extension of the popular JUnit framework.
To define a Monkey Unit test suite, the test author performs the following
steps:
1. For each transition, a method is written that performs calls to the methods
of the unit under test to bring it to the target state of the transition
2. The finite state machine model is described by defining the source and
target states for each of the transitions
3. The Factory Method pattern is used to define how to create a new
instance of the unit under test. A factory is bound to its target state,
that is, the state in which the unit under test will be after it has been
created. There can be several factories.
4. Typically, class- and state-invariant resolved-example tests are written
and bound to their states.
To put it all together, the user provides a reference to the finite state machine
model that is used for testing. The user also provides information about which
oracle types (the initial version of Monkey Unit will
at a minimum include explicit support for state/class invariant testing and other
but-equivalent oracles).
The Monkey Unit suite is run through one of the standard JUnit
TestRunners. In the event of a failure, the monkey unit must generate a test
4
report. A monkey is a metaphor for algorithms that select how the model is traversed and
checked.
5
describing (at least) which transition path was executed, which
test revealed the failure, and on which line.
One possible future application of Monkey Unit is to use it for testing multithreaded
units. This would probably just mean running several Monkey Unit tests in separate threads.
Comments