Lecture
Imagine a typical tourist who visits some tourist city. There is something to see here. There is something to do here. Let the hard «work» begin! The same is true for test engineers, who aim to explore their city – the software product – as fully as possible.
Many questions arise. Which transport should you take? How do you make it in time? How do you plan a through route? How should you behave when there is no good spot for a selfie? Do you need to sort it out yourself, or call a policeman?
Strategy and goals. Without them, you'd better stay home. Goals largely determine the tourist's plan: the time and the places. And they will be quite different, for example, for sailors who have spent six months at sea, and for students… Well, maybe that's not the best example.
Because here's how it can turn out. Without a plan and a strategy you can still come across interesting places. Sometimes interesting places are hard to miss. But what these places are, what significance they have, what their history is – that can be missed. Riding the subway without a map, you may not realize at all what distance was covered: is 5 stops a lot or a little? Cognitive psychologists would say: «Your cognitive map is inadequate», and neurophysiologists would invite you to study LCK (a safe method of studying the brain). For a test engineer, such a map (of coverage) is dangerous.
For Whittaker, this is a typical description of how testing happens. It's freestyle. And there may not be another chance to visit these wonderful places. This will be the first and last time. That is exactly why it's necessary to avoid aimlessness in exploration and risk missing essential functionality or an important bug.
But don't write off freestyle testing. It gains considerable advantages when paired with the scripted approach, since scripts don't always describe all the interesting «streets and alleys» (which is very relevant for sailors and students).
In this sense, tourism – is a combination of Kantian freedom and Tatlin-style (or, for some, Fuller-style) structure. The same can be said about testing.
There are many tourist metaphors that help us in exploratory testing and complement freestyle. And this lets us make testing faster and more consistent. Here they are:
No, not them yet. It's important to understand that functionality rarely works independently of other functionality. Features share resources, process common input, and operate on the same internal data. This means that testing functionalities separately can miss the errors of their interaction.
Fortunately for us, tourist metaphors don't perform this kind of decomposition. They do decompose, but differently. A real tourist will choose a mix of landmarks that must be seen and places that must be visited. A test engineer will do the same: draw up a plan for visiting various features with the intention of performing some operation relevant to this product. And, of course, this can involve multiple functional blocks.
The tourist metaphor in testing – is districts. For software, this is a logical division, these are paths through the application. Here are these tours.
These are places where «business is done». They begin when the code launches and end when it finishes running. They house the functions for the sake of which users use this application. This is the «back of the box» (read: of the application), which is the main thing when demonstrating the commercial value of the product, as well as the code that supports it.
These are historic places beloved by tourists, places surrounded by history. This includes «legacy» code and functions and features that have historically contained a large number of bugs. «Legacy» code is often poorly understood. Using and changing it involves a large number of assumptions. This tour focuses precisely on this component.
Many cities have places that are of interest only to tourists and novice users. The locals avoid such places because they feel cramped there. That is (note: for software), users who already have experience interacting with the application almost never use such functions.
After exploring the historic and tourist places, it is sometimes necessary to give yourself a rest, to have a little fun. Software also has something similar – additional functionality. And it needs to be checked.
In every tourist city there should be places where you can rest, recover your strength, wait out bad weather. But software doesn't rest at all during the user's moments of rest. And it would be good to check what it does and how.
These are unsafe districts which, nevertheless, are of some interest to someone. Bad things happen there, and it's apparently best not to go there. Everyone except test engineers, that is. These places must be tested, since they may contain vulnerabilities of the product.
From morning until late evening, work is in full swing here. This is precisely where «business is done». And if you ask why users use your application, this is the part they will tell you about.
Guidebooks often contain a lot of information about places worth seeing. They tell you about the best hotels, shopping complexes and attractive places without going into detail. Experts who have visited these places will kindly tell tourists exactly how to enjoy them. It should be beautiful, clean and safe here, so that tourists can calmly spend their money. These are very important places, and they must be included in the exploratory testing strategy. We must make sure that users get benefit and enjoyment from working here.
A variant of this tour is the F1 tour. Namely, strictly following the user manual provided (if provided) by the manufacturer, which gives descriptions of key functions. The goal is to go through all the scenarios described in the user manual as fully as possible. Here you can often find various tips on input data and interface navigation while using the functions. In addition to the functionality itself, this tour lets you check how precisely and carefully the user manual was put together.
Variations of the tour also include the blogger's tour, which contains third-party advice, as well as the expert's tour, which examines the feedback of dissatisfied users. Various forums, news, communities or even books (if the application is large) can help with this. Another useful tour – the competitor's tour, which lets you look at the application from the point of view of a competing system.
The above variants of the guidebook tour let you evaluate the application in terms of its advertised functions. These are straightforward tests, and any deviation from the manual should be treated as a bug. This tour lets us use functions in exactly the same sequence and relationship in which the users themselves will do it.
Every application has functionality that, in the eyes of the user (the customer), will earn them money. «This is the most attractive thing to me» — the user will say in this case. «Money» functionality carries «money power» with it. This includes various nuances of using the product, since it is precisely these that let people earn money. Sales specialists may steer the product demo towards these scenarios, touching on things that were not clearly captured in the product documentation. Such a «sales vision» is an excellent basis for a money tour.
In this tour it's important to hear how the sales specialists present the product, and to go through exactly what is emphasized. During this kind of testing you not only find bugs, but also preserve the vision of the application's «money vein».
A variation of the money tour is the skeptical customer's tour. Going through the tour, you suddenly stop and ask: «What if I do this?» or «How would I do this?». «What if I now supplement the walkthrough with scenarios that go beyond what was originally intended? What if I just throw in a couple more features!».
Once again: watching product demos in terms of «money value», as well as good relations with sales specialists, provide an excellent opportunity to run money tours much more effectively.
We pick out some landmarks and make a leap to one of them through the entire application. For example, we choose a key function as a landmark that we identified in the guidebook tour or the money tour. Next we compile a set of such landmarks, determine their order, and explore the application from landmark to landmark until we reach each one. After that a landmark coverage map can be created. Different order, different landmarks – that's the field for variability of this kind of tour.
Ask the application difficult, intellectual questions. How can you make the application work as hard as possible? Which features push it closest to its own limit? What input and input data will make it strain? What data can trick the error-checking procedures? Obviously, this depends directly on the specifics of a particular application.
A variant – the obnoxious tourist's tour, which plays on stereotypes about foreigners. Ask silly questions, try to be annoying, draw attention to yourself. Invent obstacles and see how the application handles them. Does this make sense? Maybe not. But we can still do it.
The intellectual tour and its variants can help find both truly difficult, high-priority bugs and silly bugs. You need to be able to tell them apart. To do this, you need to build real cases so that it will be easier in the future to argue that something is important and needs to be fixed.
Think about the data that has to pass through the application. Data begins its journey with input, then is stored in internal storage and variables, where it is often processed and changed, and then it may undergo calculations. In the end, most of this data is presented to the user or somewhere else.
You need to concentrate on the data. You need to identify the stored data and «trace» its path through the application. Say you entered your email address. Where does it show up? In which places? Try to find all the areas that the data touches. Determine its entire life cycle.
After the money activity of the application, it still keeps working. This is task maintenance, archiving of data and files. This can be an automatic process, but it can also be run manually. This tour reminds us to do this.
A variation of the tour is the morning start-up. The task is testing the startup procedures and scripts.
Janitors are the kind of people who know the neighborhood even better than the people who live there or the local police officers. Day after day, street after street.
We can mark out places for a methodical check of the interface: screen by screen, dialog by dialog (mostly by the shortest path) without detailed testing. Just the obvious places.
This tour involves choosing a target (all menus, errors, dialog boxes, for example) and visiting each of them by the shortest possible path.
Such districts represent legacy code, features present in old versions, and fixed bugs that still need to be checked.
Bad neighborhoods are sections where a large number of bugs are found. How can you identify these sections in advance? You can't. But through experience this is done based on already-discovered bugs and their number. This makes it possible to form some assumption about where they appear. Since bugs can cluster locally, this tour is worth taking. After dealing with such bugs, it's recommended to run the janitor's tour through the nearby functions to make sure the fixes didn't affect them.
The museum antiquities here are the legacy code. It can be identified by the dates given in the repository. Such code, when used in a new environment, tends to work poorly. This code is hard to investigate: the programmer may no longer be at the company, and the documentation may be scarce. The tester must identify such code and pay attention to how it works.
After updating from a previous version, it's advisable to run the tests that were written for that earlier version. This lets you make sure that the functionality users used before can still be used usefully in the new version of the product. If functionality was removed or updated in the new version, testing should be carried out accordingly.
This tour focuses on several points:
— short, quick test cases;
— longer scenario checklists;
— the point isn't to make the application work, but to quickly check functionality… and say, «I was there!».
Collect output data, and the more the better. The idea is to go wherever you can and document everything you see. Make sure you see all the output data the application can generate. For a text editor: printing, spell-checking, formatting, various structures, tables, graphics. For a store: the ability to buy from any valid department, successful and failed card transactions. You need to chase down every possible piece of output data until you can claim that you've been everywhere, seen everything, and completed your collection.
This is a long tour and it's better done as a group activity.
The idea of the tour is to check the farthest corners of the application, as far as possible. Which feature requires the largest number of clicks? Pick one, «click your way» to it, and test it. Which feature requires the largest number of screens to be usable? Pick one and test it. The point is to travel for as long as possible before reaching the final destination.
For this tour, think superficially, don't go any deeper than what's on the surface. Take the lessons that modern times teach us. This tour isn't about functions or anything substantial, but about how the application looks and what first impression it makes.
Focus on the interface. Is it good? Well executed? If I make changes, is that reflected on it? Does it happen correctly, and are there no artifacts left on the screen? Is everything in its place? Does it violate any usability requirements or standards?
Maybe it's not even worth touching, but what a damn good-looking view.
This tour is meant to test running the application multiple times simultaneously. Launch your application, and then again, and again. Now use features related to memory or disk space. Make all the copies of the application do something with the same file or with the transfer of some data.
This is especially relevant for large applications. They have places that are quite hard to find without someone's help. They aren't hard to use, they're just hard to find. There can be quite a lot of such places. The problem is finding out about them. How? Talk to users, browse various blogs, and simply immerse yourself in the application.
This is a chance to have a little fun with the application: make it more attractive, play with templates and colors. It gives you some break from the hard work. The connection between the mainline and supporting side of the application matters: it should be useful and meaningful.
Whenever sales specialists demonstrate a product, the user tends to be tempted by the features next to the ones being highlighted. The Supporting Actor Tour focuses on the features that share the screen with the main ones. Their proximity to the main ones increases their visibility, and we should give them due attention. Turn your head – right and left – and see whether due attention was given to the actors supporting the main ones.
The «behind the scenes» tour. Everything that gets the least attention, that is used least often, that is used most rarely. The very bottom, the farthest shelves, whatever is behind the facade.
An interesting variation – the mixed-purpose tour. Mix the most popular and the least popular features, visit them. It's like a landmark tour mixing the biggest and the smallest landmarks. It may turn out that the features interact in a way the developers never intended.
How do you determine that features may interact with each other? Whittaker suggests three questions:
— The input question. Is there input data that is processed by two features in the same stream?
— The output question. Do the functions operate on the same parts of the graphical interface? Do they generate or update the same output data?
— Do the functions operate on the same internal data? Do they use or modify the same internal data?
The clubbing tour – never stop, one more club, one more cocktail. Can you survive the whole night?
You can, but can the application? How long can it keep working and processing data before it crashes? Accumulating data in memory, writing (overwriting), repeatedly reading variables – this can lead to application failures: memory leaks, data corruption, race conditions, and more. The main condition – never close the application. Open files and don't close them. Don't save files. Connect to remote resources and don't disconnect. And while the above conditions are in effect, you can take other tours to exercise the application and its data processing. Restarting the application clears some of its areas – don't let that happen.
This kind of tour is often run automatically, by looping a script. Long-running operation is especially important for mobile devices – that's their normal condition of use.
This is a place located far from the noise and bustle, a place for easy rest and relaxation. This is a place where you can step away from the main functions of the application, the popular features, and test the secondary, supporting functions that might not be reflected in the test plans.
The idea is to start an operation and then abruptly stop it, as if it had suddenly started raining right then and you don't feel like going anywhere at that time. Start entering purchase information only to immediately finish the procedure and exit; start printing and cancel it before it prints. If the functionality has a cancel button, or if it takes longer than a few seconds to run – use this tour.
Look for time-consuming operations – there's room to play here. Search engines are a great example: enter words that load the search, making it take a long time. Is there a «Back» button — press it. No? Press ESC, or even the browser's «Back» button. Start an operation, and then start it again without stopping the first one.
The causes of possible errors lie in the application being unable to clean up its storage after itself. Something might be left open – files, data, variables. A state may arise that contradicts the application's normal flow of operation. So after various cancellations, go through it and see whether everything is in order. Make sure that all canceled procedures can still be started correctly.
There's always someone who crosses their arms. They're bored and show no activity at all. Yet it often happens that the guide tries harder than usual to interest such a person.
The same goes for applications: if a person shows no high activity, it doesn't mean the application is working just as inactively. On the contrary, it often loads the application more, since the else branch of the if-then-else construct is used. The application has to figure out what to do if the user leaves fields blank. At the moments when the user doesn't take the initiative, the «default logic» runs.
The idea of the tour is to make as little effort as possible. Leave fields blank, accept default values, fill out forms with the least amount of data, don't click on anything extra, skip through forms without clicking or feed in whatever comes to hand. If there's a choice, pick the path of least resistance.
Again, lazy interaction on the user's part doesn't mean the application works lazily: it still has to plug in default values, handle empty forms, and so on.
Input data should break the application.
Try to break the application by any means possible. Ask the application to read some data from disk, and then sabotage that attempt, damaging the operation and the file. Ask it to perform some memory-intensive operation while the application is installed on a machine with little memory, or while another application consuming a large amount of resources is running at the same time.
The concept is this:
— force the application to perform an action
— determine what resources are required for that action
— remove or restrict those resources.
You'll find that there are many ways to weaken the application: adding and removing files, changing access permissions, unplugging the network cable, running other applications in the background, deploying the application on a machine with known problems, and so on. The concept of fault injection is also useful here (this is where Whittaker refers to his other book — How to Break Software).
In this district, attempts are made to destroy the application. Being nice, polite, and following the crowd isn't the best way to do it.
This tour involves requiring the entry of the least attractive data and/or otherwise negative input. If a typical user does «A», this tour assumes the test engineer will never do «A», something more senseless has to be found.
Three ways can be identified to implement such antisocial behavior:
— The Contrarian Tour. Enter the most unlikely data, things that are out of context, silly or completely meaningless. This tests the ability to handle errors. If it helps, you can test the «patience of the application».
— The Crime Tour. The idea is to supply data that shouldn't appear at all… shouldn't appear at all, period. You wouldn't expect a tourist to steal a pint of beer at a pub, would you? That's not just antisocial behavior, that's a crime.
By breaking the law, a tourist gets into trouble. By breaking the law, a test engineer may run into error messages. The data in this tour triggers error messages, and if it doesn't – you've found a bug. Enter incorrect data, data of the wrong format, too long, too short, and so on. Think «What constraints are tied to this input?», and then break the constraints. «Want positive values? Here, have negative ones». And pay attention to the errors, they may come in handy for other tours.
— The «Wrong Turn» Tour. The idea is to do something in the wrong order. Take a set of correct actions and perform them in an incorrect order. Try checking out before putting anything in the cart. Try returning an order before you've bought it. Change the delivery settings before the purchase is complete.
In this tour you need to enter the same data over and over again, perform the same action again and again. Copy, paste, cut, replace, and then do it again. This is usually called repetition. Select an item, and then select the same item again, to check whether discounts appear.
Developers often think that users will use the application in the intended order, purposefully. But users make mistakes and go back, they don't always understand the «correct» path and forge their own.
Tours bring structure to testing. They help test engineers discover more relevant and interesting scenarios than freestyle testing does. By giving test engineers goals, tours push them onto a certain path, which is more sophisticated than classical testing based on decomposing functionality.
Tours are a tool for exploring features in combination. Moreover, tours are something that is fixed, that can be repeated. «You get this tour, and you get that one» — this alone rules out many problems of test engineers overlapping with each other.
Some tours will turn out better, some worse. This can be determined empirically. They can then be ranked in various ways (by bugs found, by execution time, by code/feature/interface coverage). Tours can be managed, and that matters. You can learn from them. They can be distributed across a team. Their effectiveness can be measured. They help give structure to testing and to knowledge.
Comments