Bookmark and Share

Webinar recap: Implementing open source automation tools & frameworks for success

Insight image

Thank you to everyone who registered and joined us for our webinar on Implementing Open Source Automation Tools & Frameworks for Success! As many you know, test automation is no longer just a supplement to testing, but an increasingly important factor for quality assurance success. That’s why we can’t say we’re surprised with the overwhelming response of attendees who signed up and we really do appreciate your participation and feedback!

As more organizations make great strides in their development processes and make the transition to agile methodology from older practices such as waterfall, project teams are very much dependent on automation tools.  With many factors to consider such as shorter release cycles, less financial resources, higher software project expectations and more frequent demand of delivery, these tools are not a luxury but a necessity.

In this 60 minute webinar, we were lucky enough to have Bob Crews, President of Checkpoint Technologies and Joe Colantonio, Founder of TestTalks, as well as Zephyr’s VP of Marketing, Francis Adanza, tackle the topic of automation tools and frameworks.

The webinar kicked off with Bob Crews commenting on how automation is not only still gaining traction to this day, but is more critical now more than ever with the speed of mobile application testing and the Internet of Things. Joe Colantonio agrees that test automation is very critical, especially as organizations move towards continuous integration and deployment, ensuring that software is developed faster.  

A quick poll found that a majority of the audience did not have interest or experience with Serenity, Robot Framework, Gauge or Sahi. It was no surprise with our panelists, whom know that there so many tools available. Next, our panelists compared open source solutions and commercial solutions. Open source solutions have a lower cost of entry, greater access to open community of users, and can be modified to your own needs and customization. Commercial solutions however, are perceived to have greater maturity, have more tech support, and are less fragmented with end-to-end integration.

Our second poll found that 40% of the audience have a framework implemented and 32% are in the process of implementing one. Frameworks actually make the whole test automation process easier. There are four types of frameworks which are business process, keyword driven, data driven, and hybrid. The benefits of frameworks are countless, such as the ability to provide clear automation methodology and decreases the need for technical expertise.

The last part of our webinar focused on best practices for implementing automation tools and frameworks. Joe Colantonio touched upon his idea of best practices, which are to have main goals which needs to be a collaborative approach, to have tests quickly, and to have reliable testers.

We would like to thank Bob Crews and Joe Colantonio for being on our panel! Their expertise and insights were invaluable to this discussion.


Here are some questions from the audience:

  • Any suggestions on open source solutions for testing GUI (buttons, menu items) that minimize maintenance due to changes in GUI layout?

      There aren’t as many open source tools for GUI applications as there are for Browsers. The only one i have experience with is White framework and Sikuli

  • Any tips/tools for testing highly dynamic UI’s? E.g. Radar images?

      If you have to deal with image verification than a tool like Applitools could help. We use it to help test x-ray images.

  • Are there any reference frameworks you can recommend?

 Depends on your teams needs and skill sets.

  • Can you apply Lean/Six Sigma in the software development cycle, especially in testing and can you give an example of how this applies?


  • Is it best to create the first test before building the framework?

Yes - you can use a test driven development to help drive your framework creation so that you are only creating what you need.

  • What would you prefer for cross-platform mobile applications testing: Appium or Calabash?


  • Is automation a good idea for testing COTS?


  • If both manual and automation testers are required as we develop software, how do you determine the ratio of manual to automation testers within your scrum team?

I really don’t like focusing on ratios like this they often time lead to bad practices. As a team you should decided what works and come to an agreement not management. Even 100% automation does not guarantee a quality product (See my TestTalks with Todd Garden

  • Is it a good idea to reuse Unit Tetss written by Dev as part of the E2E Test Automation Suite?

I would have to see an example. Usually the unit test are at such a low level that they are hard to integrated into an end-to-end approach. I would make sure that if you have something tested already at a unit level to not create a end-to-end test for it since it is already covered.

  • How do you know what to automate and what not to automate?

If something is deterministic and the effort to automate and maintain it is less than the effort it  would take to manual test over time AND and when it fails people will care about it because it adds value- that to me would be a good place to start automating.

  • At what stage of a product is the best time to start automation?

I would like to make sure when we say automate we are talking about the same thing. Automation should not be just a functional end-to-end test. In my perfect world automation would start with developers with unit tests. If automation is no thought of before development begins with developers thinking how they can make their code more automateable than its too late.

To learn even more about open source automation tools and frameworks, the benefits of frameworks and best practices for implementing open source tools and frameworks, watch the entirety of the webinar here