1. Reliability
Why measure reliability?
To reduce and prevent severe malfunctions or outages, and errors that can affect users and decrease user satisfaction. Software is better if it fails less often, and easily recovers from failure when it happens.
How can you measure reliability?
Production incidents – A good measure of a system’s reliability is the number of high priority bugs identified in production.Reliability testing – Common types of reliability testing are load testing, which checks how the software functions under high loads, and regression testing, which checks how many new defects are introduced when software undergoes changes. The aggregate results of these tests over time can be a measure of software resilience.Reliability evaluation – An in-depth test conducted by experts who construct an operational environment simulating the real environment in which the software will be run. In this simulated environment, they test how the software works in a steady state, and with certain expected growth (e.g. more users or higher throughput).Average failure rate – Measures the average number of failures per period per deployed unit or user of the software.
2. Performance
Why measure performance?
To understand the level of performance experienced by users and how it impacts their usage of the software. Software is better if it meets or exceeds the level of performance users expect.
How can you measure performance?
Load testing – Conducted to understand the behavior of the system under a certain load, for example, with 1,000 concurrent users.Stress testing – Understanding the upper limit of capacity of the system.Soak testing – Checking if the system can handle a certain load for a prolonged period of time, and when performance starts to degrade.Application performance monitoring (APM) – This is a new category of software that can provide detailed metrics of performance from the user’s perspective.
3. Maintainability and code Quality
Why measure maintainability and code quality?
This is an aspect of software quality that is more significant to the organization developing the software, but it also indirectly affects users. Software is better if it is maintainable because it will take less time and cost to adapt it to users’ changing requirements. Software which is maintainable and has high quality code is also more likely to have improved reliability, performance and security.
How to measure maintainability and code quality?
Lines of code – A very simple metric that has an impact on the maintainability of a system. Software with more lines of code tends to be more difficult to maintain and more prone to code quality issues. The image below shows lines of code on several popular PHP frameworks, using several measurement techniques.Static code analysis – Automatic examination of code to identify problems and ensure the code adheres to industry standards. Static analysis is done directly on the code without actually executing the software.Software complexity metrics – There are several ways to measure how complex software is, such as cyclomatic complexity and N-node complexity. Code that is more complex is likely to be less maintainable.
4. Rate of Delivery
Rate of software delivery is related to quality, because a new version of a software system will typically contain improvements that can impact the user. A higher frequency of releases that are delivered to the user should, in theory, mean that the user gets better software faster.
How to measure rate of software delivery?
Number of software releases – This is the basic measurement of how frequently new software is delivered to users.Agile stories which are “done” in a certain time period – Counting the number of “stories,” or user requirements, which are actually shipped to the user, provides a more granular measure of the rate of delivery.User consumption of releases – For example, measuring the number of users who download or install a new patch or software update
5. Security
Why measure security?
Increasingly, users rely on software to perform sensitive operations related to their personal lives and businesses. Software is better if it is less vulnerable to security breaches.
How can you measure software security?
Number of vulnerabilities – It is possible to scan software applications to identify known vulnerabilities. The number of vulnerabilities found is a good (negative) measure of security.Time to resolution – How long does it take from the time a vulnerability was introduced in the software until a fix or patch was released?Deployment of security updates – For software deployed on users equipment, how many users have actually installed a patch or security update?Actual security incidents, severity and total time of attacks – How many times was a system actually breached, how badly did the breach affect users, and for how long?
What do we need/Goal?
A central dashboard that can show how many tests passed, now or in comparison to previous versions, across all software quality dimensions.
Examples of testing metrics that may be of use: https://www.qasymphony.com/blog/64-test-metrics/
Comments