Greg stated that one notable fact from this year’s survey is that a third of the people are still using spreadsheets and Word documents for test automation instead of a proprietary tool or integration with JIRA.
Perhaps the most surprising result of the survey is that respondents using a standalone test management tool are more likely to automate their testing across a majority of testing types, including integration testing, unit testing, UI testing, API testing, and performance testing, as compared to those who use spreadsheets and Word documents. Greg added emphasis to this point by using a comic book voice— “the kind I use when I’m reading to my kids”—to read the following: “Those who invested in a test management tool automated more for every type of testing than those using spreadsheets only!
Milan began the next part of the presentation with an overview of the two products that make up the Zephyr test management suite: Zephyr for Jira and Zephyr Enterprise. Both products provide scalable test environments, he said, via the data center inside Zephyr for Jira and also via Zephyr Enterprise's scalable SOA- and microservices-based architecture. Zephyr has an architecture that is continuously “bashed” or tested, using APIs, virtual users, physical users, and various configurations to ensure that that the application holds up from a load testing, security and safety perspective and can perform at scale.
He said there are quite a few approaches to reporting in Zephyr, including widgets within the Zephyr dashboard as well as live charts that are clickable to allow you to drill down to view data points in more detail. In addition to a range of traceability and custom reports, he stated that it’s also possible to link up automation frameworks within Zephyr Enterprise so that you can have a consolidated view of semi-automated, fully automated and manual testing in one place.
Since most of the webinar’s audience considered themselves heavy users of JIRA, Milan said Zephyr offers multi-JIRA, multi-project support: meaning you can simultaneously connect multiple instances of JIRA and multiple projects, which can be set up within both Zephyr Enterprise and Zephyr for JIRA. It’s also possible to report against all of these instances and projects by leveraging widgets available in the dashboard.
Using Zephyr and TestComplete to do Test Automation
Milan said Zephyr offers a couple of different ways to do test automation. Zephyr Enterprise is a live HTML5 environment that connects with JIRA and it has a whole host of applications inside it. For example, there is an artificial intelligence (AI) engine add-on that looks at all of your requirements and testing data and helps figure out whether you're going to finish your sprint on time and what steps you can take if you're running short on time like adding additional users. It also uses a probabilistic model to ascertain whether a test case is likely to fail and flags it as a predicted fail. There's also a DevOps add-on and an add-on that allows you to work with automation frameworks using a full set of REST-based APIs to design your desired automation scripts. “Since we use these APIs to develop the UI for Zephyr in the first place, you can look at network traffic and see which APIs are being used,” Milan said. You can also use the Vortex application to trigger automation jobs directly from Zephyr Enterprise.
TestComplete + Zephyr
Using Vortex, it’s relatively straightforward to configure one-click automation in TestComplete and Zephyr. Vortex features include control of test executions, auto-creation of test cases, the ability to track and manage automation runs, centralized location of results for easy reporting, and analytics on testing activities. “You can trigger TestComplete suites directly from Zephyr,” Milan said. “You can also automatically create test cases that have been executed using a build automation solution, a CI engine, or a CI/CD framework like Jenkins, Bamboo or TeamCity. The scripts will be auto created inside of Zephyr Enterprise for you. For instance, if you have a job or a Java class that's being executed that runs a Selenium script, Vortex will auto create a test case and place that into the test repository. It also places the executed version of that test case into the test case execution folder.”
Milan explained that Vortex is an automation platform that uses automation agents called Zephyr Bots or ZBots, which can be installed on laptops, machines, servers, or any consolidated results location on either Linux- or Windows-based machines. Once you've installed an agent on a given machine, you have the option of setting up a job that triggers different automation frameworks directly from the Zephyr server or setting up a folder watcher that listens in on a given results location.
“In this case,” he said, “the ZBot is looking at a particular location where TestComplete results are pumped in from a Jenkins build. As the Jenkins build happens, it pushes the results into the folder and the ZBot checks every 60 seconds to see whether there are new results in this folder. As soon as it finds new results, it then populates the Zephyr test planning application with new details.”
“If we look at the execution results updated by Vortex, we see all the passes and indeed a failure (red circle above) as well. The results of any automation run can be viewed in live reports (blue circle), but also in a dashboards area (green circle). You can also have a sprint-oriented view, which shows all of your manual, semi-automated and fully automated test results in one single place.”
There are a number of dashboard widgets for showing automation results including one that shows the automated status of a particular release and another one that displays what percent of tests are automated with respect to every phase or by tags in a release. This information can be published by the Daily Pulse widget that offers a running view on testing activity and tracks the progress of tests created, tests executed, and defects found.
“When you observe a test case execution failure,” Milan said, “you can immediately click on a button and file a defect against any failure you've observed. These defects will go straight into JIRA for you. And then you can come back into Zephyr and build a comprehensive traceability matrix for the defect, as well.”
Questions and Answers
In response to a question about Vortex in the Q&A section of the presentation, Milan clarified that Vortex is an application within Zephyr Enterprise. “It’s there on the bottom left hand side (orange circle, above) within Zephyr Enterprise. Vortex doesn’t work directly with Zephyr for JIRA. In order to bring TestComplete results over into Zephyr for JIRA, you need to leverage ZAPI (or Zephyr API).” (ZAPI is an add-on to Zephyr for JIRA that allows access to its testing data including the ability to view and upload information programmatically.) Zephyr for JIRA users can integrate with test automation tools using ZAPI capabilities, he explained. “You can also leverage the Zephyr for Jira, Jenkins or Bamboo add-ons. In this case the Zephyr for Jira Jenkins add-on is the better one to use since it’s open source, so you'll be able to configure it and tweak the data sent over.”
Greg answered in the affirmative when asked whether the Zephyr/TestComplete integration would work “headless” on a device without a graphical user interface. He referred the questioner to SmartBear’s TestExecute product, which runs TestComplete tests on computers where TestComplete is not installed. TestExecute is a lightweight version of TestComplete that supports all the testing functionality provided by the Desktop, Web, and Mobile modules, but uses fewer resources than TestComplete. “This is the most common use case in a CI/CD pipeline where people will have TestExecute set up on all of their test runner machines and Jenkins on their other CI/CD platforms, which will call out to TestExecute to run the tests,” he said. “All of that backend information can then be pushed into JIRA (where you can create JIRA issues for failed tests.)
Asked whether Vortex works with Selenium, Milan said Vortex has automation triggering capabilities that let you can run Selenium scripts directly from Vortex within Zephyr Enterprise. “You can also have Jenkins trigger your Selenium scripts and put the output in a folder you’ve set up to listen for Selenium results.”
In response to another question about Zephyr and TestComplete, Greg said that on the SmartBear product roadmap going into 2019, the company was committed to integrating Zephyr with SmartBear’s TestComplete and SoapUI automation tools. “We're really taking a very thoughtful approach to how we want to integrate all of our tools into Zephyr to make sure that it’s the best for all of our users,” he said.
Asked about TestComplete’s ease-of-use, Greg recommended that the questioner download a full feature trial of TestComplete at SmartBear.com or TestComplete.com and take it for a spin. “It’s designed to be easy to use: you just have to click, record and navigate through your application. Once you're done with that, you click play and it will run the script back so you’ll know how well your app runs on different browsers and different machines.” SmartBear also has a plethora of video tutorials, trainings and documentation available freely online without any login necessary.
He concluded: “If your question is whether X, Y or Z tool can be integrated with Vortex and Zephyr, the answer is generally ‘Yes.’”
Zephyr, a SmartBear company, provides the world’s most widely used software test management solutions, powering more than 18,000 customers and 5 million users across 100 countries. Zephyr is leading the global transformation toward DevOps and Continuous Testing Agility through widely adopted advanced quality management, automation and analytics tools. Leading product and IT teams in finance, healthcare, media, mobile, IT services and enterprise leverage the Zephyr family of products to keep pace with accelerating software delivery lifecycles. Dedicated to helping organizations spend less time testing and more time building, Zephyr launched the first testing solution natively inside Jira nearly a decade ago. Today, companies and teams of all sizes rely on Zephyr’s end-to-end solutions, unmatched scalability and support to move from ideas to impact with increasing velocity and ease. Zephyr is headquartered in San Jose, Calif., with regional offices in Philadelphia, Europe and India.