Lecture
Testing in large companies, in the enterprise, is most often a complex and thankless task. The gap between business units and IT is huge: a developer has a vision at the code level, and verification – at the level of unit tests, while the customer thinks in terms of services that work or don't work, or even entire processes that go beyond the scope of a single development team, or even an entire department\company. And asks to organize business testing, or end-to-end testing, or testing based on scenarios from beginning to end (end 2 end).
Quite often our clients regard tests as a fifth wheel when it comes to development. You know the consequences: an astronomical number of project anomalies, damaging errors in production and, worse still, software that gradually turns into something ossified.
Sometimes we decide to carry out testing, and we manage to convince the «higher echelons» of the benefit and necessity of the time (and, consequently, the money) that testing requires, but:
And then we bring out the heavy artillery: end-to-end tests, usually GUI tests of the Selenium type:
But after a few weeks or months we start to realize that this may not be the best idea:
Then we invest even more, because we tell ourselves that we're not too far off, and it would now be a shame to throw it all away. And yes, everything can be improved, but at what cost? And for how long? Overall, the «black box» testing strategy is neither the most effective nor the most cost-effective.
I can see you smiling more than once as you read this, but rest assured: you are not alone.
Before I tell you more about the test pyramid, let's recall some criteria that are important to consider when thinking about a testing strategy. And, to clear up a much too common misunderstanding, in this article we will talk exclusively about automated tests.
A test, whatever it may be, has no other purpose than to give you feedback: «Does my program do what it's supposed to?» We can judge the quality of this feedback by three criteria:
Source: Code of Culture, OCTO technology
«Stop calling your build unreliable. Have you ever gone to production when, say, your search feature «sometimes works»? »
Pavan Sudarshan, Though Works
Source: Code Complete, 2nd edition, Steve McConnell
Even when tests are fast, there is even a virtuous cycle: they are easy to run and build confidence in your code, encouraging you to write even more.
That's why a good testing strategy aims to maximize the number of tests that meet these three criteria: accuracy, speed and reliability.
Knowing that you are unlikely to have an unlimited budget, your testing strategy will necessarily depend on it. You need to weigh the cost of the different types of tests against the value they provide, in other words, assess their ROI.
The following table shows the main criteria for choosing which type of test to use:
From this point of view, we might say: «Great! I only need to run unit tests », but other types of tests exist for a good reason: unit tests cannot check everything.
Now we have arrived at the well-known testing pyramid, first described by Mike Cohn in his book «Succeeding with Agile» , which is very helpful when defining a testing strategy.
This concludes our discussion of the theory. In the next article we will take a closer look at the foundations of the pyramid: unit testing and its application in practice in a Java / Spring project.
Let's start from the very beginning – with the two pillars from which this notorious «end-to-end business testing» arose, namely the testing pyramid and the ISO9000 standard.
Any tester who is well versed in their profession and has taken some knocks while dealing with related departments is surely familiar with the testing pyramid. It is especially often invoked when justifying test automation. Which tests are cheaper and more important to develop? And to run?
fig. testing pyramid
|
End-to-end testing |
System testing |
|
Verifies the software system as well as interconnected subsystems. |
Verifies only the software system according to the requirements specifications. |
|
Verifies the entire end-to-end process flow. |
Verifies the functional capabilities and features of the system. |
|
All interfaces and back-end systems are considered for testing. |
Functional and non-functional testing are considered |
|
Performed after system testing is complete. |
Performed after integration testing. |
|
End-to-end testing includes checking external interfaces, which is difficult to automate. Therefore, manual testing is preferable. |
Both manual and automated testing can be performed for system testing. |
The essence of the testing pyramid is not tricky: the foundation of testing should use the simplest and fastest tests to write and run – unit tests. Of course, checking the interfaces of classes and functions is hardly something you can show to the customer, but without this solid, monolithic, fail-safe foundation, it is unlikely that anything can be built above it. As a rule, a few dozen functions, methods and classes implement some functionality for the customer, and in essence a dozen unit tests can be reduced to a few higher-level tests. The customer needs a beautiful, finished apartment, but at the same time he is hardly going to be satisfied if the crooked windows in his apartment stop opening, and the floor and ceiling crack from the first gust of wind. However, having the customer himself walk into the apartment and check its quality may not be the best idea. You'll agree, it's hard for a user to check the quality of the concrete in the foundation, just as it's hard to reproduce all weather conditions. The same is true in testing, of course: higher-level testing is needed, but only once we have run unit tests, and then tests of an even higher level.
More complex to develop and longer to run are higher-level tests – integration tests, which check that modules worked on simultaneously by the whole team, releasing its product (system), work correctly together. That is, code integration is verified, and the system is tested without regard to interaction with external systems. Such tests already imply higher-level verification, most likely through accessing the system via a system API or even a GUI (front end). Working with this type of test is more difficult – to cover all branches and nuances of the code, it is most likely necessary to use a large number of heavily overlapping checks on various test data, and when automating, it is often necessary to develop a whole heap of conditions and branches in scripts. That is, on the one hand we have already gotten closer to the user, making life harder for ourselves, but on the other hand, it is still difficult for us to find common ground, it costs us more, and the quality of the checks is still insufficient. That is, we can let the customer into the new apartment, he can check everything, but without regard to interaction with other tenants, weather conditions and utility services. You'll agree, an ideal world, a model, is generally of little use in real life.
If we add these conditions as well – let's see how our system interacts with external systems – suppliers and consumers, with our environment, that is, if we carry out system testing, we will easily see that the complexity of testing also increases. We will need to achieve simultaneous operability of all interacting systems, though without involving specialists on them. For now, it is enough for us simply to accept some data from our suppliers and pass our data to our consumers. In the correct sequence and format. We are not concerned with the further fate of the data. The main thing – our system works correctly in the correct environment. And all would be well here – for our customer we can already give a full-scale demonstration, except that in real life this is still not all the criteria for success for our development. Of course it is good that the customer has gotten his apartment in a solid building, but if to get to it he has to climb over barbed wire, then canoe across a lake with crocodiles to huts swarming with snakes, then perhaps we did something wrong, and in the wrong place?
That's why the first idea for end-to-end testing – is to check not only our environment, but also all interconnected systems through which data received or sent by our system passes. And this, in turn, means that we will have to combine several such «testing pyramids» with each other. Building a fragile bridge, across which we will lead by the hand the data that is valuable to the user.
The only question is how to do this? Who should do it? How do we bring it all together?
A series of standards describing quality management systems, among other things, states that any process in an organization must be described, documented, even if it is the process of issuing rakes to the janitor in autumn. And if that's so, then not a single process taking place inside software used and developed within the organization can go undescribed. The question is, how to do this? Of course, the best description, from a BDD point of view — is a description of behavior through tests, underneath which will lie the testing pyramid. But we immediately return to our dilemma of joining several pyramids with thin ropes from apex to apex, along which our tightrope-walking customer and his users will walk without a safety net.
Process approach is a management strategy that requires organisations to manage its processes and the interactions between them. Thus you need to consider each major process of the company and their supporting processes.
That's why it's easiest to use abstractions – to create at least a diagram of the process, and indicate its inputs and outputs, make the process controllable and measurable, ensure interconnection with parent and child processes, as ISO9000 requires
All processes have:
• inputs;
• outputs;
• operational control;
• appropriate measurement & monitoring.
Each process will have support processes that underpin and enable the process to become realised
Business diagrams are best suited for this purpose, and standards such as UML, BPMN, ARIS, etc. are most often used. The processes themselves become flowcharts with «blocks» strung on them. Interaction takes place between the «blocks»; in the BPMN standard — this is the flow of actions and the flow of messages. And this is exactly what we need!
Any company that wants to have a certificate and follows the ISO9000 standard has most likely acquired such diagrams, and they are an integral part of the top-level requirements. If a company employs good analysts, then reference-requirements to individual actions from the diagrams will most likely trickle down to the low-level requirements. And those are exactly what we need.
In fact, in the diagrams we can see the whole process, and understand what scenario we need to build, and which system\team to run to with what data at what moment.
I'm not belittling here the work of the developers who write competent code that passes messages between different parts of software-and-hardware complexes, but it's impossible to keep everything in your head. And when a process is used within many other processes, it's better to have such a «map» on hand for conducting competent testing, and even more so for building a test model.
So, we have two premises – each team\system must have a pyramid of tests prepared – from the smallest, unit tests, to complex system tests, as well as the fact that within the organization we are required to have requirements described in the form of business processes. This fact will allow us to quickly tell the customer how a given business process works, and at what point and why it breaks, and, for ourselves, upon receiving defects from production, to quickly perform root cause analysis. In theory.
But in practice, everything again falls on the tester – how do you pick the ones you need out of a pile of tests, especially other people's tests, arrange them into a chain, and feed the correct data to the input of each of the systems and compare it against a correctly defined expected result?
The simplest option – is to originally develop tests based on business models, and split teams by projects that implement one business process or another. For this purpose, some test management tools already have the ability to import BPMN diagrams (for example, for HPE ALM – importing in the XPDL format is supported). HP ALM will itself break the diagram down into a set of requirements (actions), and, if desired, create a hierarchy of requirements (the Requirements->Business Models module). From there, our job is to cover the requirements with tests, and then to arrange the requirements, and hence the tests, into chains that cover our business process. These chains are called «paths» in HPE ALM, and let you see all combinations of sequences. If desired, requirements and chains can be converted directly into tests.
But even if you don't use testing tools, you will still have to build chains out of the business process. Especially given the imperfection of the tools (not everything is so rosy), as well as the fact that, most likely, the test model will need to be assembled after the fact, and executed altogether as a regression by a «general team» not attached to the new projects.
How many paths can the little rodent take to reach the pine cone?
In this case we will need to open the tests of each of the teams, find the ones tied to the requirements appearing in the business model, and build chains out of them, saving them in a «shared space». Creating a shared space – is some kind of surrogate, but in any case it must exist, if only in the form of a ledger book, Excel, or a project area in the test management tool. If we again talk about HPE ALM, then this functionality is handled by the BPT (Business Process Testing) module, which also allows passing the results of one test into the parameters of another. However, with desire and persistent effort, this can also be achieved in HPE ALM by restructuring test sets (Test set) into an execution flow (Execution flow). Then, when the full set is run, the testers responsible for passing each of the components of the end-to-end scenario will be called in turn.
And, alas, a test management tool alone is not enough. From my experience, almost all tools have some fatal shortcomings, and so, if you get as far as the stage of automating testing along a business process – you will arrive at creating a script that will trigger the tests in the required sequence.
As a result, two conclusions can be drawn:
1) for end-to-end scenarios, tests already developed earlier for each of the systems included in the chain (scenario) of the business process are used with a high degree of probability
The whole set of the company's test suites can be represented as a sparse matrix, where the columns hold tests for each system (for simplicity – system-level tests), and the rows – business processes. That is, for a given business process, tests covering that business process need to be selected\created, and relationships need to be established. If there is no coverage – this is a reason to fill the gaps in the test model, or to make sure that quality is ensured by other levels of testing (integration testing, unit testing, code review and running it through analyzers).
2) A tool for observing, tracing and keeping the business process up to date is needed, for the purpose of synchronization with the test model.
And while testing tools cope more or less tolerably with creating the test model, in reality things are very bad when it comes to keeping it up to date – it is often easier to recreate the model from scratch than to try to spot changes in the process and the test model. And the experience of real teams shows that it's better to create a living visualization of the architecture. The simplest way to do this is in a shared area, using a plain whiteboard and sticky notes. Then the teams involved in the business process can visually see how the process changes (links are removed and added, actions are removed and added). The main thing – is that everyone has access to the board. Also, note that if the process involves messages between systems, then, as a rule, there should be at least two tests from each system – for sending and for receiving data. That said, instead of sticky notes you can use a whole lego city (of large blocks), or something even more creative. The main thing here – is a single language and a single information space, which is exactly what's lacking in the enterprise.
Organizing clear and correct testing along business processes – is a complex and very expensive thing. Note that E2E testing – is not just acceptance testing, the user testing that the customer will perform, it is building a small bridge, taking into account all possible situations, along which the customer will walk and lead the users behind him, in step.
Once again – E2E – is not a stroll across a bridge in a Lada Kalina, and not even a crossing by two KamAZ trucks. It is complex engineering work, hanging bridges with sensors and running all possible checks and situations — or at least describing these scenarios.
Whether or not your company needs such an ideal final clean run – is entirely a matter of your goals and needs. As always, with any testing, you should weigh the potential risks from defects missed at this stage against the cost of preparing and running end-to-end testing. Assess which of the two would cost you more, and only then act. But in the case of end-to-end testing along business processes, remember that it makes no sense without a solid foundation in the form of a 100% passrate of unit tests (~90-100% coverage), integration tests (~60-80% coverage, 90-100% passrate), and system tests (20-40% coverage, 80-100% passrate). Setting success criteria (quality gates) – is more a requirement for the quality of the released product; the main thing to remember here is that the volume of E2E tests – is only the tip of the pyramid (1-2% coverage, ~99% passrate), which should not be larger than its base, nor should it plug holes left by the previous stages. It – is an addition, which is a priori considered already closed at the previous stages.
Organizing such testing – is mainly the work of preparing and synchronizing test cases and data (test analysis), as well as a set of organizational measures, synchronizing teams in one place at one time on a working test range. Keeping this in mind, you should not try to show the customer «end-to-end testing» ahead of schedule, so as not to waste the time of a large number of people at once without all the working components assembled together.
P.S. the tools described, as well as the practices – are strictly for example purposes, the author did not set himself the goal of advertising products or proclaiming this approach to end-to-end testing as the only correct one.
What for:
to be sure that nothing is broken in the process
to reduce regression testing time to a few minutes
to test without changing the workflow/code
For WEB programming
as an application of behavior-driven tests
Test example
Installation
npm install cypress
Running it
./node_modules/.bin/cypress open
Choosing the test
We write tests in the integrations folder
cd cypress/integration
We write a test in BDD style using cypress
describe('Guest test checkout', function(){
it('has to open PDP and pass checkout', function(){
cy.visit('https://example.com/paget1'); // <-- will open the page and wait for it to load
cy.get('#add-to-cart').click(); // <-- will find the button and click it once it becomes visible
cy.get('.mini-cart-link-cart').click();
End-to-End Test Automation for Behavior-Driven Development
we discussed the theory of the testing pyramid - a testing strategy that ensures the quality of our application at a reasonable cost. Notably, we discussed the concept of feedback and the importance of having fast, accurate and reliable feedback. Unit tests usually meet these criteria for a modest investment. In this article we will develop a concrete example to study the use of automated unit tests and try to answer some of our readers' recurring questions.
«The difference between theory and practice is that in theory there is no difference between theory and practice, but in practice there is».
Jan van de Snepscheut
Let's move on to the practical part. For this, and to complete our overview of tests, we will take a microservices example. Of course, this choice is not entirely random: microservices should be as autonomous as possible (team, connection, deployment, etc.), and this autonomy is enabled through testing: integration and end-to-end tests are not entirely suitable if we want to continuously deploy our service independently of the others.
The following diagram concisely describes the architecture of our example:
We decided to create a set of services for searching and booking train trips, but instead of using the API from the French national railway company, SNCF, we chose the Swiss Open API, available at: https://transport.opendata.ch/. The latter will provide us with routes and schedules.
The connection search service is a facade over this API, which allows us to decouple from this external service. Our interest in this article is more educational, but we will come back to it.
And finally, the heart of the system, the journey booking service, is responsible for finding routes and recording them in the database.
Endpoints:
GET /journeys/search?from=...&to=...
allows searching for available routes, but not booked trips (this is the entry point for the search service).
GET /journeys
gives a list of all reserved trips
GET /journeys/{id}
gives the trip whose identifier is passed in the request
POST /journeys
allows booking a trip
PUT /journeys/{id}
allows modifying a trip booking
DELETE /journeys/{id}
deletes the trip whose identifier is passed in the request
The last 5 endpoints will interact with the database (in our case, Postgres).
Our booking microservice is structured as shown in the following diagram. This is very standard, the example is simple, and the business logic is minimal. It would be reasonable to do everything in the controller, but for the sake of the example we will keep our service layer and see where it leads us.
From a technology standpoint, we will use the standard: Spring and its ecosystem. There are many tools for testing Spring, and it's good to know what to use and when. The full project is available on gitlab.
We will start at the base of the pyramid with unit tests. A unit test aims to verify a single piece of behavior (i.e., a method or a subset of a method), arising from business use, in isolation from the rest of the world:
Some will say that it's not necessary to isolate everything. In Working Effectively with Unit Tests, Jay Fields introduces the notions of sociable tests and solitary tests. Personally, I am in favor of isolating as much as possible, to avoid any interference. For simplicity, our unit tests do not depend on any external input/output, that is, databases, file systems, networks, etc.
To do this, we use what some call stubs, others fakes, mocks or doubles - what is called a Test Double in the literature. This is an object that we fully control and that replaces the object under test. This allows us to check different behavior variants depending on the values returned by the double, for example, the happy path, edge and corner cases, and errors.
Although test doubles can be created manually, there are also many libraries available that simplify their implementation: Mockito, EasyMock or JMockit are the best known in the Java world.
If we look at our previous diagram, we will unit test each of the objects that make up our component:
To be honest, since the Client is implemented with the Feign library, there is no real code to test:
The same applies to the Repository part, which is based on Spring Data and therefore has no code:
We will come back to these two elements in our integration tests, since our goal is not to test the underlying frameworks, which are already well tested elsewhere.
So, now we have the following diagram:
As we already said, the Controller is almost a simple utility layer, almost.
Utility layers
A common question many readers ask us: « Is it worth testing the utility layer? «To which we respond with another question: « Is it worth having this utility layer? » , Often these layers exist only to provide a multi-layered structure and serve no purpose other than being «just in case».
In general, the practice of TDD (Test Driven Development) helps us avoid this. Without going into the details of the practice, which would deserve a whole article, TDD aims to define the expected behavior through a test before implementing it. Thus, we first write the test, and then the simplest code that makes the test pass and, therefore, satisfies the specified behavior. This allows us to avoid over-engineering and creating «just in case» layers, and focuses on the simplest code that delivers value quickly.
In our example, although the controller seems to have little code, it still has two responsibilities: exposing data transfer objects (DTOs) instead of entities, and exposing the API through annotations. The code (though minimal) will be tested individually, and we will check the exposure (URL mapping, error code handling, etc.) in the component tests.
Private methods
Another recurring question among our clients: « should you / how do you test private methods? «.
100% coverage or nothing
With tools such as Jacoco, Cobertura or Clover, we can determine what portion of our code is reached/covered when running the tests. Beyond this simple indicator, these tools let us see where the tests passed and, especially, where they failed. We can then check whether the critical paths of our application are tested or not.
We should be careful about relying on code coverage as an indicator, because it can be misleading: it is certainly possible to execute 100% of the code while testing nothing (for example, by asserting nothing). Don't aim for 100%; instead, start by focusing on the critical parts of the application, and track the trend of your code coverage. Is it increasing? Decreasing? If you want to go further, you can apply mutation testing (also known as chaos-monkey testing), which more or less randomly alters the business code and checks that the test fails. If the tests keep passing, the code is probably not being effectively verified. The Pitest framework can automate this in Java.
For example, the following report indicates that JourneyService (after removing all assertions) is fully covered by tests, but these tests score rather poorly on mutation coverage.
Example of an «incomplete» test:
And the associated report:
We will use JUnit, AssertJ and Mockito to implement our tests; note that at this level of the pyramid there is no Spring. Here is an excerpt from our tests for JourneyService (link to Gitlab):
A few things to note about this code:
unitUnderTest_ShouldExpectedBehavior_WhenInitialState
We may not follow this naming convention, but test code should be as readable as possible, if not more so than business code. As long as it is clear, test code documents what your application actually does better than any documentation.
Personally, I use a few comments from Behavior Driven Development (BDD) syntax: given, when, then, to structure the test. Others use the 3A rule: arrange, act, assert . The key is to have well-structured and readable code.
It goes without saying that these tests should run continuously in your build pipeline after every commit, in order to detect regressions as quickly as possible. Unit tests verify the business aspects of your application, that is, the business logic and algorithms. They form a protective shell for any code modification - that is, adding features, refactoring and fixing bugs - and I cannot stress enough that they are necessary.
They are necessary, but not sufficient. Later on we will discuss component tests, which complement the set of tests that are useful to have in your toolbox.
Comments