Quality assurance metrics and testing metrics

Lecture

According to the international standard ISO 14598:

Metric is a quantitative scale and method that can be used for measurement.

We would add that the introduction and use of metrics is necessary to improve control over the development process, and in particular over the testing process, which we will discuss further.

The goal of testing control is to obtain feedback and visualize the testing process. The information needed for control is collected (both manually and automatically) and used to assess status and make decisions, such as coverage (for example, requirements or code coverage by tests) or exit criteria (for example, criteria for ending testing). Metrics can also be used to assess the progress of planned work and budget utilization

Creating, using and analyzing metrics

In our view, for greater clarity it makes sense to group metrics by the types of entities involved in software quality assurance and testing, namely:

Metrics for test cases (Test Cases)
Metrics for bugs / defects
Metrics for tasks

Let's look at each of them in more detail:

Test case metrics

Name	Description
Passed/Failed Test Cases	The metric shows the results of test case execution, namely the ratio of the number of successfully passed to those that ended with errors. Ideally, by the end of the project, the number of failed tests tends toward zero
Not Run Test Cases	The metric shows the number of test cases that still need to be executed in the given testing phase. Having this information, we can analyze and identify the reasons why the tests were not run

Bug metrics

Name	Description
Open/Closed Bugs	The metric shows the ratio of the number of open bugs to closed ones (fixed and re-verified)
Reopened/Closed Bugs	The metric shows the ratio of the number of reopened bugs to closed ones (fixed and re-verified)
Rejected/Opened Bugs	The metric shows the ratio of the number of rejected bugs to opened ones
Bugs by Severity	Number of bugs by severity
Bugs by Priority	Number of bugs by priority

We would like to note that the metrics "Open/Closed Bugs", "Bugs by Severity" and "Bugs by Priority" clearly visualize how close the product is to meeting the quality criteria for bugs. Having requirements for the number of open bugs, after each testing iteration we compare them with the actual data, thereby seeing where we need to add effort in order to reach the goal sooner.

The metrics "Reopened/Closed Bugs" and "Rejected/Opened Bugs" are aimed at tracking the work of individual members of the development and testing teams.

First example:
Suppose we have a situation where the number of bugs reopened after being fixed does not decrease, or even grows. This is a signal that an analysis of the causes needs to be carried out, since such a situation may show that:

The requirements for the feature can be interpreted in different ways
The tester did not describe the problem accurately
Poor, superficial resolution of the problem (bug fix)

The second example will show why the "Rejected/Opened Bugs" metric is needed:
We observe that the percentage of rejected bugs is very high. This may mean:

The requirements for the feature can be interpreted in different ways
The tester did not describe the problem accurately
The developer does not want to fix the mistake they made, or does not consider it to actually be a mistake. (This problem is a direct consequence of the 2nd one, caused by an inaccurate description)

All these problems noticeably destabilize the situation on the project. Therefore, when they arise, it is recommended to hold a short conversation with the project team leads, in order to subsequently reduce the number of reopened and rejected defects.

Task metrics

Name	Description
Deployment tasks	The metric shows the number and results of application deployments. The application deployment procedure was described in the article Procedure for Deploying a New Software Version (Deployment WorkFlow). If the number of versions rejected by the testing team is critically high, it is recommended to urgently analyze and identify the causes, and resolve the existing problem as soon as possible.
Still Opened Tasks	The metric shows the number of tasks that are still open. By the end of the project all tasks should be closed. By tasks we mean the following kinds of work: writing documentation (architecture, requirements, plans), implementing new modules or modifying existing ones per change requests, environment setup work, various research, and much more

Task metrics can vary; we have given only two of them. A metric on task completion time, among many others, may also be of interest.

1.1 Missed defects

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
1	Number of defects leaked into production	Bugs Leakage	Defects registered in the prod. environment Total defects registered× 100% To collect this metric, a field must be added to the bug tracker «in which environment the defect was found»		• As one of the testing KPIs • To analyze the reasons defects were not found in the test build
2	% of defects found by users	Bugs Reported by Users	Defects found by users Total defects registered× 100%		• As one of the testing KPIs • To analyze which defects are critical for users
3	% of defects found not by testers	Bugs Reported not by Testers	Defects found not by testers Total defects registered × 100%	• As one of the testing KPIs • To analyze why other process participants find errors missed by testers
4	Missed defects by category	Bugs Leakage by Category	Defects from the production environment by category Total defects by category× 100%		To analyze in which areas testing needs to be developed further • Categories can include testing levels, testing types, areas of functionality, etc.

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
5	Requirements coverage by tests	Requirements Coverage	Requirements with tests Total requirements× 100% To calculate this metric, it is necessary to define the criterion for «requirements with tests». This could be «at least 1 test», «at least 1 test per boundary, etc.». It is calculated in the requirements management system, based on status or the presence of a link to tests.		• For planning the expansion of test coverage • For assessing the risk of missed defects
6	Confirmed requirements coverage by tests	Approved Requirements Coverage	Requirements with approved tests Total requirements× 100% To calculate this metric, the test suites for each requirement must be agreed upon. Most often, this agreement takes place with the analyst and the developer responsible for implementing the requirement. Following the discussion, the status «tests approved» is set in the requirements management system
7	Requirements coverage by exploratory tests	ET Requirements Coverage	Setting exploratory testing tasks based on requirements or user scenarios, and calculating the % of requirements for which testing was performed.		• To assess release readiness • To plan the costs of testing
8	Coverage of user scenarios by tests	User Stories Test Coverage
9	Code coverage by function	Code Coverage	• Choice of assessment tool, based on the needs and architecture of the project • Creating an instrumented build • Running tests on this build to analyze coverage Code coverage assessment is possible both for automated and for manual tests!		• To assess the level of coverage • To investigate parameters affecting execution that is not specified in the documentation but does affect the product's code • To expand test coverage
10	Code coverage by condition	Alternatives Coverage
11	Code coverage by decision	Decision Coverage
12	Coverage of lines of code	Path Coverage

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
13	GUI Coverage	GUI Coverage	Objects covered by tests Total objects× 100% Where the objects can be: • Screen forms • Graphical interface elements • API functions • Integration interfaces • Etc.		• Identify risk zones and «bottlenecks» • Assess the status of testing
14	API Coverage	API Coverage
15	Coverage of integrations	Integration Coverage
16	Coverage of performance requirements	Performance Coverage	Requirements in the group covered by tests Total requirements in the group× 100% Where the group can be: • Different types of requirements • Different modules • Different functional areas • Etc.		• For planning the expansion of test coverage • For assessing the risk of missed defects
17	Coverage of load requirements	Load Coverage
18	Coverage of supported data formats	Data Coverage
19	Coverage of user scenarios by usability tests	Use-Cases Covered by Usability Tests	• Conducting a usability review involving the target audience, with analysis of scenario feasibility • Conducting an internal UX review of the convenience of performing user scenarios		• Assess usability risks • Decide which tests need to be run
20	Coverage of environments by unique tests	Environmental Unique Tests Coverage	• Analysis of risks related to environments • Preparation of environment-dependent tests • Agreement on the sufficiency of environment-dependent tests		Platform	Windows	Mac	Android	Unix	• Assess the quality of test planning that depends on the environment • Determine the need for additional analysis • Plan the expansion of test coverage
Common tests applicable	1350	1350	340	940
Can only be run on this platform	42	60	160	36
Unique risks on this platform	28	n/a	112	n/a
Unique tests agreed with the development team	X	X	V	X
21	Running tests on supported environments	Environments Coverage	Tests run on the platform Total tests selected for the platform× 100%		• Assess risks related to environments • Plan additional environment tests

1.3 Quality of test design

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
22	Average expert evaluation of tests	Expert test cases evaluation	AVG (Test ratings in the test repository) Tests can be rated: • With a single rating • On different scales Ratings can be categorized: • By employee • By group • By test type or area • Etc.		• To identify weak areas in test coverage • To detect lack of qualification and make decisions about additional training • To identify tests that need to be improved
23	Test relevance	Test cases relevance	AVG (Latest documentation version - Latest documentation version used in the tests) or Tests current as of the latest version Total number of tests× 100%		• Assess the risks of using current tests • Determine the need to allocate resources for updating the tests
24	Rating of defect detection by tests	TC bugs detecting ratio	Tests run Defects registered Calculated by version, iteration, or time period Statistics can be categorized by: • By employee (test designer) • By functional area • By test type • Etc.		• Identify tests subject to the pesticide effect (see Product 1) • Assess the effectiveness of introducing new techniques and approaches (see Product 2)
25	Speed of defect detection per test case	TC bugs detection speed	Defects registered Time spent testing per TC		Project	Project 1	Project 2	• Identify effective tests, whose error detection speed is higher than in ET • Identify ineffective tests, which find few defects
Exploratory testing costs	140 man-hours	166 man-hours
Bugs found in ET	73	91
Cost of finding 1 bug in ET	1,9 man-hours	1,8 man-hours
Cost of reviewing test cases	97 man-hours	130 man-hours
Bugs found via TC	132	48
Cost of finding 1 bug via TC	0,7 man-hours	2,7 man-hours
26	Test case efficacy compared to exploratory testing	Exploratory / Scripted Testing Efficacy Comparison	Detection speed via TC Detection speed in ET

2. Project planning

2.1 Adherence to the work plan

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
27	Deadline slippage on tasks	Schedule slippage	Actual finish date – Planned finish date The metric can be collected by: • Task • Iteration • Release And categorized by: • Employee • Task type • Etc.		• To identify the average slippage in deadlines and include it in work plans • To analyze the causes of deadline slippage for each task, or for the largest shifts • To include in planning risks based on slippage statistics in percent
28	Deviation from the work plan	Schedule Variance	When assessing groups of tasks, iterations, and projects, it is measured as a percentage:
29	Overrun of labor costs	Estimation Changes	Data can be categorized by: • Task • Iteration • Team • Employee • Etc.		• To analyze the causes of deviations in estimates • As a possible KPI when labor costs need to be reduced • To find overruns in labor costs by category
30	Total downtime	Team Idling	If a specialist cannot perform tasks in the current iteration (build blocked, test environment not ready, etc.), they charge the time spent in this period to the «Downtime» category		• To analyze the causes of downtime • To factor downtime into work plans • To monitor the dynamics of combating downtime
31	Share of downtime	Team Idling %

2.2 Accounting for project risks

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
32	Risk management on the project	Risk management	Presence of a risk management process on the project		• To assess the maturity level of projects • To analyze where the introduction of risk management is needed for more competent planning
33	Team-based risk approval	Risk management approval	Approval of risks and the mitigation strategy by the entire project team
34	Accuracy of risk forecasting	Risk predicting correctness	Forecasted risks that materialized All risks× 100% Where «All risks» is the sum of: • Forecasted risks that materialized • Forecasted risks that did not materialize • Risks that materialized but were not forecasted		• Identifying excessive risk forecasting (see Iteration 3) • Identifying insufficient forecasting (see Iteration 2)
35	Level of risk mitigation	Risk avoidance efficacy	Successfully mitigated All risks in the strategy× 100% Calculated based on the mitigation strategy		• Assess how correct the decisions we choose for mitigating risks are • Find more effective tools for mitigation

2.3 Metrics for forecasting testing labor costs

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
36	Ratio of labor costs between developers and testers	Test/Dev Effort Rate	Testing costs Development costs Calculated per change request, iteration, module, etc.
37	Effort (man-hours) per KLOC	Test Effort per KLOC	Cost of testing the change Code volume of the change Calculated based on the code volume in lines (KLOC) or accounting for code complexity (Cyclomatic Complexity)
38	Effort (man-hours) per verifying a requirement	Test Effort per Req	Cost of testing the change Number of requirements in the change
39	Cost per 1 test environment	Tet Effort per 1 Environment	Cost of compatibility testing Environments tested
40	Time spent creating a TC/checklist item	Test Effort for 1 TC creation	Cost of creating tests Number of tests created
41	Cost of reviewing 1 incident	Test Effort for 1 incident submit	Cost of reviewing incidents for the period Number of incidents processed
42	Cost of validating 1 defect	Test Effort for 1 bug validation	Cost of validating defects Number of defects validated

How to use:

• Rough estimation metrics (36, 37, 38) are used to quickly obtain the planned effort based on previously collected

statistics. If a change containing 150 requirements comes in for testing, we can quickly estimate the cost of testing it as

150x1.36 = 204 man-hours.

• Precise estimation metrics are used when planning effort according to the test strategy. In the strategy we specify that we need to

create around 120 tests, run testing on 4 test environments, validate 40 defects, etc. In the estimate we sum up the costs

for each of these items, taking the costs from previously collected statistics.

• By analyzing the cost of recurring activities, we can find that some types of tasks take too long to perform. In this case we

carry out improvements and process optimization aimed at reducing costs, and turn such indicators into KPIs (for example, the goal is to spend

no more than 10 minutes validating 1 defect, the strategy is automating the creation of test data and environments, the KPI is metric #42)

3. Product quality

3.1 User satisfaction

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
43	Average rating from users	Average Client Rating	AVG (User ratings for the period) Can be collected in: • Direct surveys • App stores Used to assess: • Static data per release • Dynamics across iterations		• Project KPI • Identifying the sources of low ratings (see the chart – issues in the iOS version) • Tracking the dynamics of change rollout
44	Distribution of user ratings	Clients Grades Evaluation	Collecting «raw» data on the number of each rating type		• Analysis of extremely high and extremely low ratings
45	New requests from users for the period	New Improvement Requests	• Implementing a «customer improvement requests» queue in the task tracker • Tracking request statuses: o «on hold» o «in progress» o «implemented» o «accepted by the customer»		• Assess the level of customer satisfaction with the rollout of changes • Plan the work for implementing requests from users • Filter out long-standing on-hold changes and make decisions on them
46	Implemented user requests for the period	Improvement Requests Implemented
47	Open user requests	Opened Improvement Requests
48	Rate of implementation of user requests	Improvement Ratio	Improvements implemented Improvements requested × 100%
49	Passing customer acceptance	Customer Acceptance Ratio	Builds accepted by the customer Total builds for the period× 100%		• Project team KPI • Analysis of reasons for build rejection • A tool for justifying decisions on deepening test coverage

3.2 Defects in the product

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
50	Defects in the product by status	Defects by Status	Defects in category Total defects × 100% It is also possible to collect more complex metrics, for example: defect statuses by priority, defect severity by area, etc.		• Assess the need to fix defects • Plan the costs of achieving the required quality level for defects
51	Defects in the product by severity	Defects by Severity	• Assess the priority of logging bugs by testers • Assess the number of hidden defects (when only high-severity ones are logged)
52	Defects in the product by area	Defects by Functional Area	• Assess the quality of individual functional areas • Make decisions on expanding development and/or testing subteams
53	Defects in the product by test type	Defects by Test Type	• Assess product quality against compliance with non-functional requirements
54	Defects in the product by platform	Defects by Environment	• Assess build quality by environment • Decide whether to reduce or increase testing volume by environment
55	Defect dynamics	Defects Dynamics	Defect growth = Defects logged – Defects fixed Number of open defects = Number of defects open at the start of the period + Defect growth		• Track the dynamics of product quality by defects • Forecast the product's readiness for release (detecting the point of «defect convergence»)

3.3 Testing results

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
56	Passed tests	Passed Test Cases	Tests passed Tests executed× 100%		• Shows the percentage of working functionality (in combination with test coverage assessment!)
57	Failed tests	Failed Test Cases	Tests failed Tests executed× 100%	• Shows the quality and stability of the software
58	Blocked tests	Blocked Test Cases	Tests blocked Tests executed× 100%	• Shows the amount of work needed after receiving a new build for testing
59	Tests executed	Executed Test Cases	Tests executed Total tests× 100%	• To estimate remaining testing work • To assess the reliability of the data for metrics 56-58
60	Test results by category	Test Results by Category	If tests have categories, used to assess various aspects of quality: • Test types (functional, performance, load, etc.) • By test priority • By functional area		• Assessing the quality of different software categories • Allocating resources to the most problematic areas
61	Readiness of requirements by tests	Requirements Readiness by Tests	1. The requirements management system contains links to the tests covering this requirement 2. Based on the test execution results, the requirement's readiness status is set: • Done - all tests for the requirement passed successfully • Errors - some tests for the requirement failed • Not working - all tests for the requirement failed • Not verified - tests for the requirement have not been run • Unknown - no tests are linked to the requirement		• To estimate the remaining testing volume • To assess risks when making the release decision • To prioritize testing and defect-fixing tasks

3.4 Software quality characteristics – 1/4

Metric (RU)

Metric (EN)

How to calculate

Metric visualization

How to use

Performance over time

Dynamical

performance

Assessment of the speed of key business functions (loading key pages, executing key operations or requests).

•Change in the speed of the main elements

of the application.

•The dynamics of the change in the speed of the main elements,

relative to the target

level.

Quality assurance metrics and testing metrics

• To detect performance regressions

• To assess improvements

• To assess changes in the

speed of

key functionality

Performance compared to

competitors

Performance

compared to

competitors

1. Assessment of the speed of

the main business scenarios compared to the figures

of competitors.

•By the minimum number of user actions.

•By execution speed

2.Comparison of non-speed

key

performance parameters, such as • the compression ratio during

archiving

•Errors made by users

•The number of concurrent

connections, etc.

Quality assurance metrics and testing metrics

• For marketing

promotion

• To find areas for improvement

• To justify the presence of

performance defects

• By identifying the best

values among

competitors, we

determine what

to strive for

3.4 Software quality characteristics – 2/4

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
64	Performance under load	Performance under load	Response time at different loads, in the vast majority of cases is measured automatically (JMeter, Grinder, HP Performance Center, etc.)		• Assess compliance with load requirements • To assess the possibility of scaling the application by number of users
65	Stability under load	Stability under load	1. Automating a large number of concurrent requests to the server 2. Collecting statistics on what % of requests succeed, and what % causes errors		• To determine the fault tolerance of the system under stress conditions • SLA compliance (Service Level Agree ment - the agreement on the level of service provision) • Determine at what number of concurrent connections errors start to appear
66	Errors under load	Failures during load tests	Erroneous responses Total requests× 100%

3.4 Software quality characteristics – 3/4

Metric (RU)

Metric (EN)

How to calculate

Metric visualization

How to use

Compatibility

Percentage of supported

platforms, environments, browsers, OS versions, screen resolutions, etc.

The list of possible statuses per environment may vary; the minimum set of statuses: • Supported

• Not supported

For each status the calculation is:

Testing status

Total environments × 100%

Quality assurance metrics and testing metrics

• To assess risks for

untested

environments

• To identify

unsupported

environments

Modifiability

Quality assurance metrics and testing metrics

3.4 Software quality characteristics – 4/4

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
69	Usability of individual forms according to GOMS	Usability measured by GOMS	GOMS calculation models t – the time spent performing the action H – moving the hand to the mouse K – pressing a keyboard or mouse key P – moving the cursor M – thinking through the next step		• To find unsuccessful (slow to perform) implementations in the interface • To choose an option for the implementation of the screen form • For comparative analysis of the product against competitors
70	Usability of scenarios: the success rate of completion by users	Scenarios completion by users	• The user performs the given scenarios, their errors and questions are counted, and the success of completing the scenarios is assessed • A scenario is considered successful if the actual result of the user completing the scenario matches the expected one Completion status Users tested× 100%	and	• To prioritize usability improvements by scenario • To identify confusing areas and usability errors • To justify the need to fix errors
71	Breakdown of user support requests	Users complains by product modules	Collecting statistics on support requests with the status «user error» (requests where the cause of the error was user misunderstanding rather than product defects) Requests per module Total requests× 100%		• Identify areas where a usability review is needed • Discover errors and inconsistencies that go unnoticed by experienced project participants who are used to the product

4. Testing efficiency

4.1 Testing speed

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
72	Time to test the build	Time to test the build	AVG (Testing completion date − Testing start date) Measured in working days		• Statistics for subsequent release planning • To identify areas of improvement / increasing speed • To find deviations and identify causes (downtime and missed deadlines) • To assess the dynamics and impact of implemented changes
73	Time before testing starts	Time to start testing	AVG (Testing start date − Build readiness date ) Measured in working days
74	Time to detect a critical defect	Time to start testing	AVG (Defect logging date − Date the defect was introduced into the code ) To implement this metric, you need to add a «defect introduction date» field to the bug tracker. The developer fills in this field when changing the defect's status to «fixed» (at the time of the fix, they already know exactly which commit caused this defect and can check when it was made). Since collecting this metric involves additional costs, it is usually advisable to collect this statistic only for critical defects.		• To find out average values (for planning and risk assessment) • To detect critical deviations and find out the reasons why it took so long to detect a critical defect
Defect	Days	Reason for the long detection time	Resolution
17382	9	Was blocked from being detected	Remove from statistics
17315	11	Did not know about the parameter's impact in test automation (TA)	Conduct a review of the module's tests with the architect, possibly something else was missed

4.2 Financial indicators

#	Metric (RU)	Metric	How to calculate	Metric visualization	How to use
75	Payroll Fund (Fund for Remuneration of Labor) in testing	Total Cost of Testing Labor	The payroll fund of the testing team, including all costs necessary to support staff activities. The calculation requires interaction with the finance department and accounting, to account for taxes, vacations, sick leave, office payments, purchases of equipment and services, and social benefits. Typically, the amount of additional costs for the team ranges from 50 to 120% of the total salary amount.		Total cost of testing	• For budgeting and calculations when requesting an expansion • For control when there is a need for a reduction
Cost Cost Cost Types of expenses per employee per month of employee of employee of employee 1 (RUB) 2 (RUB) 3 (RUB)	Total per team/project (RUB)
Salary (gross) 80 000 85 000 83 000 245 000
Wages (net) 69 600 73 950 72 210	215 760
Payroll taxes.
PFR - 22% 17 600 18 700 18 260 FMS - 2.9% 2 320 2 465 2 407 FFOMS - 5.1% 4 080 4 435 4 233 NDFL - 13% 10 400 11 050 10 790	54 560 7 192 12 648 32 240
Employee vacation (1/12 of salary) 6 667 7 083 6 917	20 667
Organizational expenses
office rent (per sanitary norms, each employee must have at least 4.5 sq. m of office space), based on an average 4 500 4 500 4 500 rental rate of 1000 RUB/sq. m. PCs and office equipment, furniture. On average 1/20 of salary. employee4 000 4 250 4 150	13 500 12 400
Administrative expenses
HR staff salary: recruiting, hiring, dismissal, training, onboarding Accounting staff salary: hiring, dismissal, support, sick leave 2 000 2 000 2 000 Manager's (head's) salary: support, control, management	6 000
Employee professional development (courses, trainings, etc.)
On average 1/12 of the average course price for a manual 1 250 1 250 1 250 tester (15 000 rub.), assuming 1 course is taken per year	3 250
TOTAL per month: 122 417 129 583 126 717 378 717
Cost of the testing team per month: 378 717
76	Testing costs	Total Cost of Testing	Team payroll + test equipment + test environments + tooling used + external contractors		• For budgeting the project and the testing team • For justifying an expansion
77	Cost per bug find	Cost per Bug Find	Testing costs Number of defects Calculated over a period		• For assessing the economic justification of testing
78	Cost per bug fix	Cost per Bug Fix	Cost of fixing defects Number of defects to be fixed Calculated over a period	• For finding costly-to-fix defects and analyzing causes
79	Testing budget variance	Testing Budget Variance	Actual costs − Planned costs Planned costs× 100%		• For planning the budget going forward • For analyzing the reasons for variances

4.3 Working with defects

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
80	% of fixed defects	Bugs Fix Ratio	Defects fixed Defects logged × 100% Calculated over a period or per release/iteration		• For identifying the causes of UNfixed defects. • For assessing the status of testing by module, area, and test type
81	% of rejected defects	Bugs Reject Ratio	Defects rejected Defects logged × 100% Calculated over a period or per release/iteration
82	Reasons for defect rejection by group	Rejection Causes	Collecting this metric requires a drop-down list in the bug tracker that the developer must fill in when rejecting a defect, for example: • not reproducible • expected result misunderstood • cannot be fixed • requirements error Rejection reason Total rejected× 100%		• To find a way to reduce the number of rejected defects (solve the problem causing the largest number of rejections)
83	Defect logging quality	Bugs Submission Quality	A subjective assessment in the bug tracker, broken down by developers, testers, defect types, etc.		• Assess the quality of defect logging on the project, in the team, for an individual employee • Find out the reasons for abnormally low scores in order to fix them • Account for developers who may over- or under-rate scores

4.4 Automated testing

#	Metric (RU)	Metric (EN)	How to calculate	Metric visualization	How to use
84	Autotest execution speed	Autotests execution speed	Autotest execution time: AVG (Autotest report ready time − Autotest start time) Average time per test: Autotest execution time Number of autotests		• For planning test cycles • For deciding on the effectiveness of autotests and their expansion • For detecting problems with autotest speed • For timely detection of performance regressions
85	Autotest development speed	Autotests implementation speed	Automation costs for a period Number of new autotests for a period Can be broken down by employee, team, autotest type, etc.
86	Autotest stability	Autotests Stability	Autotests passed successfully Total autotests× 100% Measured as an average over a period or release. Can be grouped by autotest developer, product functional area, etc.		• For planning labor costs for autotest maintenance and analysis • For a comparative assessment of autotest stability by category (developer, test type, functional area, etc.)
87	Reasons for test instability by category	Autotests Failures Causes	Filled in from a drop-down list each time changes are made to autotests, to record the reason for the change, with a further calculation of the share of each reason: Reason for autotest failure Total autotest failures× 100%		• To find the key cause of autotest instability • To solve the problem causing the largest number of failures • In rare cases – to decide on changing the automation strategy (choosing other interfaces, tools, or giving up automating new functionality, etc.)

Metric (RU)

Metric (EN)

How to calculate

Metric visualization

How to use

False

alarms

Autotesting

False Alarms

When analyzing the causes of autotest failures, identify all results that were not caused by errors in the product

Quality assurance metrics and testing metrics

• For assessing the confidence level

of autotests

• For assessing the

possibility

of making product release decisions based on autotest results

(how

trustworthy

the autotest

results are before

conducting a manual analysis)

False

test

passes

Autotesting

False Positives

When registering defects in areas covered by autotests, clarify why the defect was not registered based on the autotest run results. If

autotests are found that miss errors (missing checks, logic errors), such cases are marked as false test passes.

!Important

It is impossible to identify all false passes, just like gaps in manual testing, and this will be an approximate estimate.

Economic justification

of autotests

Autotests

Return of

Investment

Benefit from implementing autotests Cost of implementing autotests × 100%

Where the benefit of implementation accounts for all manual testing costs, and the automation costs include

• Autotest development

• Maintenance and updating

of autotests

• Runs and analysis of automated testing

results

Quality assurance metrics and testing metrics

• For making

decisions about implementing automated testing

• For selecting and

prioritizing

autotests based on their economic

justification

Coverage

of product

functionality

by autotests

Features

Coverage by

Autotests

Functions tested by automation Total functions in the product× 100%

Functions here can mean functional requirements,

user scenarios, product areas and modules, etc.

Quality assurance metrics and testing metrics

• For monitoring

plan compliance

• For assessing the

possibility

of reducing manual testing

• For finding areas where test

automation needs to be

expanded

Coverage

of product code by autotests

Code Coverage by Autotests

See metrics #9-12

Automated regression

tests

Regression

Coverage by

Autotests

Automated regression tests −

Total regression tests× 100%

Execution

of the

test automation

plan

Automation

Plan Fulfilment

Tests automated

Tests planned for automation× 100%

Conclusion

In conclusion, we would like to note that having the necessary metrics and charts reflecting how the project's state changes over time will let you improve not only the testing process but development as a whole, and will also make it easier to analyze a completed project, so that past mistakes can be avoided in the future.

Quality assurance metrics and testing metrics

Creating, using and analyzing metrics

Test case metrics

Bug metrics

Task metrics

1.1 Missed defects

2. Project planning

2.1 Adherence to the work plan

3. Product quality

3.1 User satisfaction

Conclusion

See also

See also

Comments

To leave a comment

Lectures and tutorial on "Quality Assurance"

Terms: Quality Assurance