The race among companies to reduce the time it takes to bring high-quality software to market is driving the widespread adoption of DevOps. In this article, I will show how developers and testers can leverage a test management tool and a few best practices to help your organization participate in and even win that race.
The Whole Team Does Testing
If your software team is using agile development and testing methodologies, then it's important to remember that testing isn't a role on agile teams, it's an activity that involves the all members of a cross-functional agile team. On agile teams, everyone is equally responsible for the quality of the product or the success of the project. This means testing is done by the whole team, not just designated testers or quality assurance professionals, including team members whose primary expertise may be in programming, business analysis, database or system administration. Team members whose expertise is in software testing or using a particular testing tool aren't limited to only doing that activity--they can also collaborate with customers or business owners on product requirements and work with other members of the team to develop high-quality code that fulfills those requirements.
DevOps continuous delivery is an agile development strategy that optimizes your delivery process to get high-quality software into the hands of your customers as quickly as possible. To do this, a majority of agile development teams use the popular Scrum agile process framework. Scrum takes a time-boxed, incremental approach to software development and project management by advocating frequent interaction with the business during what are known as Sprints (which are called iterations in other agile frameworks). The simplest Scrum project team (as shown in Figure 1) is made up of a customer/ business unit stakeholder (known as a Product Owner), the team facilitator (called a ScrumMaster) and the rest of the agile development team. Team members interact frequently with business users, write software based on requirements that they pull from a product backlog (a prioritized list maintained by the Product Owner) that they then integrate frequently with software written by other team members.
Overview of the Scrum Agile Process Framework
Image Source: Dr ian mitchell
As you can see from Figure 1, Scrum doesn’t have a testing or test manager role. On a simple Scrum project, there is not even a Project Manager role. Unlike traditional software teams, Scrum teams are not directed and controlled from the top; instead the team is expected to manage itself. Because Scrum teams are self-organizing, no one (not even the Scrum Master) will tell other team members how to turn items on the Product Backlog into releasable code. Large-scale agile projects, such as those using the Scaled Agile Framework (SAFe) will have project, program and portfolio managers--but even the largest agile project won't have team members with a role solely dedicated to software testing. The reason for this is that software testing on an agile project is not the job of one person. Agile testing involves teamwork since team members are expected to actively and collectively participate in all Scrum practices and events.
This doesn't necessarily mean that developers must find errors in their own code. Self-organizing agile teams may very well decide to make different people responsible for different development and testing activities. Scrum works via a pull system, where tasks that must be done are stored in a queue on the product and sprint backlogs. This prioritized task list contains activities such as user stories, bug fixes and other testing activities that need to be done. A team member who is currently not working on anything will go to the queue and take off the highest priority item that they are able to do and work on it.
The testing process is broken down into the following levels in order to test different aspects of the system:
This is the level where individual units/components of a software/system are tested, with the goal of validating that each unit of the software performs as designed. Unit testing is typically carried out by the developer.
At the integration level, individual units are combined and tested as a group. The purpose of this level of testing is to expose faults in the interaction between integrated units.
At the system level, a complete, integrated system/software is tested. System testing seeks to detect defects both within modules that have passed integration tests and also within the system as a whole.
At the acceptance test level, the system is tested to determine if it meets mutually agreed-upon requirements and is acceptable for delivery.
As noted above, unit testing is typically carried out by an agile developer as part of a test-driven development (TDD) practice. In TDD, unit tests are written to define the desired functionality before the code is written. This allows developers to discover defects immediately upon making a change. Agile projects always strive to find defects and other problems earlier, where they are much less expensive to fix, following the popular maxim "Test early and often." Because large and slow software releases make for buggy and unreliable code, a DevOps continuous delivery pipeline depends on frequent releases of smaller amounts of functionality. A typical DevOps pipeline can be broken down into the following sequences:
Build automation is the first stage in moving toward implementing a culture of Continuous Delivery and DevOps. If your developers are practicing test-driven development, they'll have already written unit tests for each piece of code they write, even before the code itself is written. TDD helps developers think through the desired behavior of each unit of software they're building, including inputs, outputs, and error conditions. New features implemented by developers are then checked into a central code base prior to the software build, which compiles the source code into binary code. With build automation, the software build happens automatically, using tools such as Makefiles or Ant, rather than when a developer manually invokes the complier.
Each check-in is then verified by an automated build, allowing teams to detect errors and conflicts as soon as possible. Automation framework and Continuous Integration tools such as Jenkins and Bamboo are useful in helping build, test and deploy applications automatically when requirements change, which speeds up the release process.
The next stage in implementing a DevOps deployment pipeline is test automation. Manual testing is a time-consuming and labor-intensive process and, in most cases, also a non-value added activity since you're only trying to verify that a piece of software does what it’s supposed to do.
If developers are integrating code into a shared repository several times a day, testing needs to be done continuously as well. This means running unit tests, component tests (unit tests that touch the filesystem or database), and a variety of acceptance and integration tests on every check-in.
In the last stage of the pipeline, once an application passes all the required tests, it's then released into production. For all intents and purposes, this means releasing every good build to users. A fully automated deployment pipeline requires the ability to deploy and release any version of a software application to any environment. Doing this effectively requires infrastructure automation, where environments (machines, network devices, operating systems, middleware, etc. ) can be configured and specified in a fully automatable format. This means using cloud resources and virtual infrastructure to setup and manage your deployment process.
Shift Left Testing
DevOps shift-left testing is an approach to software and system testing in which testing is performed earlier in the software lifecycle, with the goal of increasing quality, shortening long test cycles and reducing the possibility of software defects making their way into production code. The following V-model (Verification and Validation model) is useful in showing the relationships between each phase of the software development life cycle and its associated phase of testing.
Levels of software testing (V-Model)
Image source: Software Testing Lecture
One way that DevOps teams do Shift Left testing is by shifting integration testing to the left of its usual position in the delivery pipeline so that it occurs as close as possible to the build process. Since integration testing is where many disruptive, significant defects are often detected, this allows teams to receive feedback on code quality faster, with more accurate results.
Shift Left testing takes advantage of two DevOps practices mentioned earlier: Continuous Testing and continuous deployment. Continuous Testing involves automating tests and running those tests as early and often as possible, many times using virtualized components and environments. Continuous deployment automates the provisioning and deployment of new builds, enabling Continuous Testing to happen quickly and efficiently.
Continuous Verification and Validation
Software quality control often relies on the related notions of verification and validation (V&V) that check to see that a software system meets specifications and that it fulfills its intended purpose. The terms verification and validation are often used interchangeably but have different meanings. A simple way of looking at the difference is:
- Verification: Checks the software with respect to specifications. ("Is our team building the product right?")
- Validation: Checks the software with respect to customer's expectations ("Are we building the right product?")
In traditional waterfall testing and development, verification is done during the development phases of the software lifecycle (Requirements, Specification, Design and Coding on the V-Model above) and validation is done during the testing phases (Unit Testing, Integration Testing, System Testing, and Acceptance Testing on the V-Model.)
For all intents and purposes, an agile DevOps pipeline involves almost Continuous Validation and Verification. Indeed, some proponents of Agile/DevOps Shift Left Testing are now using multiple small V&V loops to model the continual user/customer feedback needed on DevOps projects.
Caption: Figure 4: Using multiple V-models to model DevOps Shift Left Testing
Image source: SEI Insights
On an agile project, team members engage with customers and other stakeholders via the Product Owner throughout the project-- to do things like prioritizing bug fixes and enhancements in the team's backlog-- and not just at the end of the project. It follows that an agile team that uses a test management tool that allows them to both solicit feedback and work collaboratively in real-time will have a leg up on the competition when it comes to doing all the activities (Build Automation, Continuous Integration, Continuous Testing and Deployment Automation) necessary to enable effective DevOps shift-left testing.