Identifying software testing metrics in a DevOps environment

Insight Main Image


As development teams face more pressure to deliver quality software releases in an accelerated timetable, the need for effective tester performance increases.


More organizations are pursuing DevOps and test management software solutions to improve their IT development efforts. As IT departments and operations become fragmented across different units and as critical duties are outsourced to teams located in other countries, the potential for ensuing communication issues and bottlenecks to create costly release delays increases dramatically. When development and operations teams are out of synch, there is a tendency for software flaws and bugs to go through the quality assurance process unscathed, resulting in performance issues and the need for further corrections to be made after the fact. To address these concerns, many businesses have employed a DevOps model to bridge these elements through a common interface. According to a recent RightScale survey, 54 percent of respondents stated that their companies have leveraged DevOps solutions.

In a recent blog entry, Forrester analyst Glenn O’Donnell confirmed the growing upswell of DevOps support among IT departments as companies seek to break down the barriers separating business units.

“After years of development and operations standing on either side of a wall lobbing things at one another, the walls are breaking down as Dev and Ops are discovering that they are (or need to be) on the same side,” O’Donnell wrote. “There is no ‘us’ and ‘them’ anymore, just us!”

Ensuring software tester effectiveness efficiency

A sophisticated agile test management strategy can help bring DevOps units together, but ultimately, all groups need to take responsibility for catching bugs before software reaches the production stage. To ensure that these departments are running at the highest level, organizations need to have some way to measure their performance. BNA Software QA Director and Sticky Minds contributor Elfriede Dustin noted that this is not always an easy process, as identifying the exact qualities that make up an effective tester can be subjective. Despite the challenges presented by trying to attain an accurate view of tester ability, it is still a worthwhile pursuit, as the performance of these business units can have far-reaching effects for an organization.

“A tester’s effectiveness can also make a big difference in the interrelationships with other groups,” Dustin wrote. “A tester who always finds bogus errors or reports ‘user’ errors (based on the application working as expected) will lose credibility with other team members and groups. This can tarnish the reputation of an entire test program. Overlooked, nonreported defects can have the same effect. It is important to evaluate the tester’s effectiveness on an ongoing basis, not only to determine training needs, but most importantly to ensure the success of the testing program.”

Some aspects of tester evaluation may be subjective, but there are key testing metrics QA management officials can use to assess employee performance.

  • Testcase Creation – Before a QA engineer can analyze the validity of a product they have to understand what the potential threats are. Creating test cases allow engineers to specify which features require verification as well as certify requirements of the software/product as a deliverable item.
  • Testcase Execution – Executing the defined parameters of a test case is key to ensuring minimal QA error and reducing the loss of productivity. As most steps in test case execution require complete precision in order to proceed, it’s imperative that each step is followed and executed in the precise manner it was defined in order to reach a concise and accurate verdict.
  • Defect Backlog – As bugs tend to reappear in later versions of the same software, having a defect backlog database supplies the QA engineer with in depth details and visuals of the recurring defects on hand. Having the ability to understand and quickly fix a previous bug defect provides the ability to meet rigorous deadlines and reduce product to market delays.

Related Articles: