Why a Robust DevOps Monitoring Dashboard is Critical to your DevOps Adoption Strategy
In 2011, Internet pioneer Marc Andreessen famously wrote in The Wall Street Journal that software is eating the world and that "every company needs to become a software company." In the seven years since Andreessen made that pronouncement, software’s appetite in areas as diverse as artificial intelligence, cloud computing, the Internet of Things, mobile computing and big data analytics is more insatiable than ever. As enterprise development environments have grown exponentially more complex over the last half dozen years, they've also been tasked with delivering applications faster and with higher quality. To handle this complexity, many organizations are turning to agile DevOps pipelines to increase the speed at which they develop and deliver software.
DevOps is not a methodology or a suite of tools but a cultural shift to remove the barriers between Dev and Ops in order to meet the need for shorter and more frequent software deliveries. This allows an organization to respond in a more agile manner to changing business requirements. The DevOps cultural shift depends on continuously optimizing workflow, architecture, and infrastructure in order to deliver high-quality applications.
There are DevOps tools in the market that can help you practice DevOps at your company. These tools automate manual tasks, help teams manage complex environments at scale, and allow engineers to maintain the release velocity of applications within your DevOps pipeline. But as both speed and quality are at the heart of the DevOps approach, QA testing needs to be a key part of any DevOps pipeline. Zephyr fulfills this critical need by providing the right set of QA tools to perform functional, integration and performance testing, that are essential to validating DevOps quality, feature and infrastructure changes, and ensure that every code check-in is production ready.
Metrics are a way of measuring or validating that the software you're building meets your customers expectations. QA metrics provide insight into the quality of in-development products as well as the effectiveness of current test methods. All members on a DevOps team from developers to testers to executives can view these metrics and gain a greater understanding of how quality assurance efforts are operating and to what extent they are actually succeeding.
These insights are not just important to the person doing the testing. QA test metrics can provide a complete visibility up and down the chain of command about the effectiveness of your software development efforts. This way, everyone from QA managers to C-level executives can get a glimpse into how operations in the DevOps continuous delivery pipeline are proceeding and if anything needs to be done to improve the development process.
Understanding which high-quality metrics are available to each persona and how they can support QA processes is essential to ensuring that developers, testers and executives are effectively managing a company’s overall quality standards.
Although agile teams have many options for QA metrics at their disposal, here are four of the most widely used project-level key performance indicators (KPIs) in use today:
1. Requirements Coverage
The quality of a piece of software quality is often defined by its ability to meet the detailed project requirements defined by business and project team members at the beginning of the development process. Metrics for requirements coverage measure your organization's testing effort and help answer the question, “How much of the application was tested?” Determining what requirements have test coverage is a straightforward calculation: you just have to divide the number of requirements covered by the total number of scoped requirements for a sprint, release or project. In addition to having a clear, prioritized set of requirements, your project should have a WIP (work in progress) limit, which is a strategy for preventing bottlenecks in software development. Work in progress limits are agreed upon by the development team before a project begins and are enforced by the team's facilitator.
2. Defect Distribution
The goal of agile QA test management on DevOps projects is to find and fix as many bugs as early in the process as possible. The defect distributions metric provides visibility into areas where defects are being found. The number of identified defects should gradually decline as the project progresses; areas that don't follow this trend need greater attention. Defect distribution metrics can be used identify hotspots such as problematic requirements that are causing bottlenecks in the DevOps pipeline.
3. Defect Open and Close Rate
It's important that DevOps teams keep accurate records of the number of spotted defects to make sure they don't slip through the cracks and show up in the final release. The Defect Open and Close Rate metric is a ratio of the defects found after delivery divided by the defects found before delivery. In addition to helping ensure that projects run quickly and without major problems in the final version, this is a good metric for determining how quickly developers and testers are collaborating toward resolving each issue.
4. Execution trends
These QA metrics identify which tests have been executed by a given member of the QA team as well as trends related to the status of defects. QA managers can use these metrics to quantify the effectiveness of individual team members and the project team as a whole. Trends across a single development cycle or multiple projects offer insight into the ongoing ability of a given team to deliver on its promises.
Customize Your Team’s DevOps Monitoring Dashboard
There are many different approaches to visualizing the QA metrics that matter most to your DevOps project. One of the most popular and effective ways is to add gadgets to a Team Dashboard where gadgets are configured from different data sources/DevOps tools used in managing your Continuous Integration/Continuous Delivery (CI/CD) pipeline. The Zephyr DevOps Dashboard tool allows you to add the following gadgets:
A “Feature” gadget, showing the user stories, epic, or requirements in Jira for your team. It breaks down the stories and shows which are done and which are still in progress. From this team view, you can drill down deeper to get more insightful data, or you can click on your feature number to be redirected to Jira for greater detail.
Each Feature gadget is arranged by project-, version- and sprint- level, and you can also select multiple sprint levels per Issue type: Epics, Stories, etc.
A "Build” gadget, showing the build status – either success or failure – and the average duration of the builds from individual projects in Jenkins and Bamboo. The gadget also shows what changes were made for each build and lets you drill down deeper into Bamboo or Jenkins for greater detail.
A “Code repo” gadget shows you the previous 14 days data in the code repository. You can see how many commits were made on each day broken down by contributor. You can drill down to get information on each commit or get fuller details by being redirected to GitHub or Stash.
A “Quality” gadget shows static code analysis and code coverage of your unit tests. It also shows security analysis of your code as well as Technical Debt, which is a gauge of the extra development work needed to code the best overall solution. This gives you a good indication on how well your code is written and how well it is being maintained. The gadget allows you to be redirected to SonarQube for more detail.
A "Zephyr" gadget pulls together the execution data of your releases for the past 14 days broken down by test status. It also provides a snapshot of your defect counts based on statuses and priorities. This gives a great overview of your testing progress as well as the status of each release. You can drill down on each of the gadget graphs to get more details or be redirected to Zephyr Enterprise for full details of your executions and defects.
View of Zephyr’s DevOps Dashboard tool after all Gadgets are configured.
Configuring a Visual Software Delivery Pipeline Dashboard
In addition to project-level dashboards described above, it's possible to create a dashboard for products that involve multiple projects or teams. This pipeline view helps visualize the status of the entire software delivery pipeline and each team's progression within the development and testing lifecycle. A product-level pipeline dashboard gives a holistic, rolled-up view of the entire product release cycle that can include the ability to drill down into each team activities: feature management, code repository, build status, etc.
Clicking on “Add a new Team” allows you to add projects to the pipeline view on the Zephyr DevOps Dashboard tool. This give you the high-level view (above) of each project with end-to-end summary information of the Defect System, Code Coverage system, Source Control, Build System and Test Management System.
(For more details on QA metrics for DevOps projects, get the free Zephyr whitepaper here: QA Metrics: The Value Of Testing Metrics Within Software Development).
View of the Software Delivery Pipeline
Flexibility to Report on the DevOps Metrics Unique to Your Team
Building a successful Continuous Delivery pipeline means creating a DevOps culture of collaboration among the various teams involved in software delivery (developers, operations, quality assurance, business analysts, management, etc.), as well as reducing the cost, time, and risk of delivering software changes by allowing for more incremental updates to applications in production. DevOps metrics chosen to measure improvement need to take into account unique aspects of the organization, market, or environment they are being used in. There is no set of "universal metrics" that will work in every capacity all of the time.
In addition to collecting the right data, you also need a way to view your data in real-time to make sure that important alerts and key trends are being monitored. A DevOps monitoring dashboard is a powerful tool that allows agile teams to track different DevOps metrics and get deeper end-to-end visibility into the entire software delivery pipeline. A flexible, customizable DevOps monitoring dashboard makes it easier for team members to identify bottlenecks in the pipeline and drill down to specifics within the dashboard to gain granular insights.
Empower Your Reporting with an Agile DevOps Dashboard Tool
The Zephyr DevOps Dashboard tool can be configured with an array of customizable gadgets that provide quality metrics you can use to quickly identify necessary corrective actions. In addition, the dashboard integrates with Zephyr's predictive analytics product to enable enterprises to identify problems with the release quality of products in the software delivery pipeline before they occur. This helps organizations building DevOps continuous delivery pipelines to quickly identify what QA tests can be automated versus what should stay manual. The combination of a dashboard and a predictive analytics tool can also help DevOps teams predict test failure rates, figure out security and coverage gaps, and create quality confidence indexes to make better data-driven decisions.
Robust DevOps monitoring dashboards are invaluable in nurturing the adoption of a DevOps culture of faster planning, decision making and software delivery, which will help your company break silos, collaborate better and innovate faster. If software is eating the world as Marc Andreessen says, being able to deliver software rapidly and reliably via a comprehensive DevOps pipeline will ensure that your organization has a seat at the table.