London-based BDQ is a software services company that uses the Atlassian technology stack and Zephyr test management products in its work with a wide range of customers such as Danish Rail, Freshfields, Elexon, Clarks, Betway, and so forth. In his CAB presentation, BDQ CEO Chris Bland provided insights based on his background in end-to-end product development ("starting with a blank sheet of paper, we are used to shipping finished product to actual customers"). Bland emphasized that his “insights” were anecdotal, rather than statistical, and culled by his consultants from real-world experiences in a number of industry verticals.
In discussions with his customers, Bland said that testing sometimes feels like a poor relation or afterthought. "They often think that being first makes you money, but I tell them the ability to deliver quality product, rapidly, is their real competitive advantage." Indeed, Bland said, poor quality costs money since it requires developers to do rework instead of concentrating on adding new features. "Which, of course, results in reduced customer satisfaction and competitive losses."
Bland qualified his presentation by saying that the 3 or 4 real world examples he was going to discuss were not drawn from clients using the latest DevOps framework because, from his experience, "customers aren't tending to do that. A lot of customers actually are just trying to get things working." While not all the examples were directly related to testing, Bland told the other CAB members, "you'll see there are some common themes that run throughout."
Ensure Your Waterfall to Agile Transition is Seamless
The first example Bland talked about involved problems caused by process change on a multi-project team at a large national infrastructure company. "In the move from waterfall to agile, the ability to do structured testing and all the reporting around it had somehow basically disappeared," Bland said. The client had been creating test cases in HP's ALM (Application Lifecycle Management) product that were subsequently discarded in the move to agile. According to Bland, their testing process had come down to "putting notes in Jira issues and hoping for the best." BDQ reintroduced test management by showing the development and testing teams how to use Zephyr for Jira in their agile sprints, which improved transparency and communication in their agile testing process.
Another software company called BDQ in to assist with JIRA reporting because they said their products were taking too long to ship. The client said they couldn't predict how long their products took to build and thought additional reports would help them gauge their status transition times. Bland said that because the client had no traceability through to requirements, they were building the wrong products. "On two occasions, the client had actually built and shipped software that the end users rejected because they literally had built the wrong products."
Discover Which DevOps Pipeline Tool is Needed to Make Your Culture Change Stick
A fellow CAB member agreed finding the right DevOps tool is a common problem for software teams: "They end up building the wrong solution because they misinterpret a feature and then they align what they're seeing with their development expertise (rather than what the requirement says)."
Bland said that this illustrates a point he often makes on software projects, that modern agile development is about more than culture change and that teams need DevOps pipeline tools that promote transparency and communication. "(The right tools) can help you make culture change stick because, otherwise, you are running around with Post-It notes, which don't work. People say DevOps is not about tools, but without tools you're not going to get very far."
Another project Bland described was a multibillion dollar nationwide project in the UK where incorrect incentives were reducing QA and test coverage. Because there are a lot of third-party contractors on the project, testing had turned into a contractual step rather than a quality check. "The result is that testing had becomes a gate check with the emphasis on passing the gate check to meet timelines," Bland said. "To get a product through the test phase, (team leaders would say things) like 'we're going to increase our appetite for risk'... but actually it's all about reducing test coverage. Here, too, transparency and collaboration will help prevent this problem from occurring."
Gain Visibility into Automated and Manual Tests Through DevOps Dashboard Tools
Bland also talked about his clients who are moving to DevOps and want to automate all their testing practices. The problem there, he said, is that " some tests are difficult or impossible to automate, so there is an incentive to reduce test coverage in order to achieve automation." Fellow CAB members agreed that companies adopting DevOps processes will still likely need to do manual testing in areas like UI (user interface) testing for the foreseeable future. Bland said that while the role of manual testing is reduced on DevOps projects, "it's not going away. That's why you need ... (test management tools) that gives you visibility over both."
He underscored the fact that you need to be wary of projects where the test phase is outsourced and becomes something that must be passed for a contractor to get paid. In those situations, "you're likely to hear phrases such as 'descoping' or 'our team now has more appetite for risk', which can just be synonyms for reduced testing. It's important that teams own the outcome of their tests and don't try to outsource everything away."
Clarify DevOps Requirements Across Departments
In conclusion, Bland stressed that requirements should be available in a form that everyone understands – "not opaque one-liners in an Excel sheet." Calling requirements "the root of all evil", he said they must be understandable to three different audiences: business managers (at a high-level), developers and testers. Bland suggested that teams can leverage techniques like rapid prototyping and wireframing to ensure that the correct product is being shipped. He also talked about various techniques in Jira to make requirements explicit and help teams take ownership of them, including linking Jira issues to development stories and product versions within a build chain, which can be linked into Zephyr for full transparency. Bland praised Zephyr's Vortex tool, saying it provides a great way to do fast, simple automation of all the links in your build chain.
The Zephyr Client Advisory Board (CAB) is made up of a cross-section of Zephyr clients from around the world. The CAB provides ongoing feedback and advisory to Zephyr executive team which helps guide the path for all facets of the business including product, support, account management and communications.