A mobile test strategy for DevOps

Insight Main Image

Mobile is Taking Over

According to the November 2016 edition of the Ericsson Mobility Report, global smartphone subscriptions are expected to almost double from 3.9 billion in 2016 to approximately 6.3 billion by 2022.  By 2022, the same report projects there will be a ten-fold increase in total smartphone traffic due to both the rising number of smartphone subscriptions and increasing data consumption per subscriber.

In a world that is increasingly mobile and social, it's not surprising that enterprise IT teams are looking for ways to speed the pace of their mobile application development.  In many cases, this involves using cloud-based tools, technologies, components and services in a mobile app development platform (MADP), which make it easier for enterprises to develop and deploy a portfolio of cross-platform mobile apps.  Cloud-based MADP tools allow companies to deliver mobile and responsive apps that run on all major device platforms (including iOS and Android) from a single codebase.  In addition to helping with source code management, many MADP tools can also leverage cloud-based Web analytics and testing tools that permit mobile apps to be optimized for performance as part of an integrated DevOps solution.

Mobile development and cloud-based technologies are a good match for organizations that use DevOps practices.  The cloud allows mobile developers to practice continuous delivery, a development approach where software is be released in constant, small increments.  Enterprise IT shops have several options when it comes to developing the mobile front-ends and cloud back-ends that make up the majority of modern mobile applications.

On the back-end, developers can use services, infrastructures and platforms hosted in by cloud-based vendors like Google, Amazon and Microsoft to reduce the time and money needed to test, host and manage applications.  Using a cloud computing architecture known as mobile backend as a service (MBaaS),  mobile applications can get access to the servers, storage, databases and other resources they need to run, which is especially helpful for enterprises who want to extend legacy databases to mobile devices or Web apps through APIs and connectors. Both Google Cloud Platform, (which gives developers access to services like Google Compute Engine, Google App Engine, Cloud Datastore and Google Cloud Storage) and AWS Mobile Hub, which offers similar back-end functionality for Amazon Web Services, are examples of MBaaS popular with mobile developers.

Native, HTML5 or Hybrid?

Platform-independent MADP tool vendors generally support one or more of the following mobile device architectures: native, HTML5, or hybrid:

  • Native apps are specific to a given mobile platform (iOS or Android) using the development tools and language that each platform supports (such as Swift and Objective-C for iOS, Eclipse and Java for Android). Xamarin, which is owned by Microsoft, lets developers build native iOS, Android, and Windows apps using C#, which is a language skill set many enterprise IT shops may not have.  Native apps are installed through an application store (such as Google Play or Apple’s App Store) and live on the end-user's device. Because they're developed specifically for each platform, they can take full advantage of all the device features — such as the camera, GPS,  accelerometer, contact list, and so forth.  Building native apps offers several business advantages if your company has the necessary in-house mobile development expertise.
  • HTML5 apps.   An HTML5 mobile app is basically a web page, or series of web pages, designed to be opened with any modern mobile browser, which means they can scale and adapt to the dimensions of almost any device.  Web apps have several advantages:  they're quicker and cheaper to develop since they don't require the skill set required to do native app development;  the back-end content on the web can be searchable, which is a big benefit for shopping carts and similar apps.  Some disadvantages of HTML5 apps include session management, lack of access to native device functionality and insecure offline storage (because of this limitation, a best practice for most mobile web apps that talk to a database is to require them to use a restricted set of permissions).
  • Hybrid apps.  Hybrid apps, like native apps, run on the device itself (as opposed to inside a browser), but are written with web technologies (HTML5, CSS and JavaScript).  Hybrid apps built with the Adobe Cordova tool run inside a thin native container known as a Webview and leverage the device’s browser engine (but not the browser) to render the HTML and process the JavaScript locally. The Telerik tool uses a NativeScript runtime that allows the JavaScript to be compiled to native code for each supported platform, which gives it some performance advantages over tools that use WebView.  Hybrid apps can be sold through the Google and Apple app stores, which offers a way for developers to monetize their apps.  Using plug-ins, hybrid apps can get access to device features like the camera GPS and contact lists, which in some cases may require additional security measures.

One Size Doesn't Fit All

A one-size-fits-all strategy doesn't work for testing mobile apps.  While Android is the largest operating system in the world, it is also fragmented.  There are thousands of devices and Android versions that your app needs to be compatible with.  Fragmentation isn't as much of an issue on the iOS side, but newer devices like the Apple Watch and other wearables create problems for testers when it comes to cross-device compatibility issues.  Automating the testing of mobile applications is also challenging in a DevOps environment as mobile devices are less accessible and less open than standard desktop environments or web-based applications.

To deal with these problems, DevOps developers and testers need a flexible mobile testing strategy to support a wide range of devices and operating systems.   One way to get a handle on the amount of work needed is to analyze the market and give priority to the devices that are most widely used.  On agile DevOps teams, these decisions are usually made by a product owner or customer representative, who is someone with a solid understanding of the market place, the competition and future trends for the type of system being developed.  For mobile products, this can be a demanding exercise since the popular of certain kinds of handsets (such as the Blackberry) can rise and fall quickly.  

Because of the different capabilities of each targeted mobile platform, many DevOps teams maintain separate build and integration areas for each platform-specific version of a mobile app.  Given the speed of development and the fragmentation of platforms and form factors that need to be supported, most DevOps shops also try to test as many mobile apps as possible with automated testing tools, using cloud-based simulators that are provided by the SDKs, in addition to doing manual testing on actual physical devices.

Priority for fixing defects in the different mobile builds should be based on the risk potential of the defect, which should also include market risk.  On fast-paced agile DevOps projects, bug fixes for low severity bugs on less-popular platforms will often get low priority and are usually only scheduled when time is available. 

Risk-based software testing looks at two factors – the probability of a bug occurring and the impact of the bug when it occurs.  High impact/high probability bugs fixes should be scheduled first:  for example, a bug in a module of code that supports a particularly popular feature in a widely-used handset.  On the other hand, a bug in software for a less-popular device should have a lower priority.

Use a risk matrix to determine the priority of multiple bug fixes.

Image Source

A good shorthand way to determine the priority of multiple bug fixes on multiple mobile builds is to multiply the two vectors (Impact and Probability) using a standard risk matrix (see figure 2 above).  The product of the two is the PriorityTest management software is also useful in keeping tabs on bug activity in the different builds, identifying the length of time to find and address defects, as well as reporting on test case progress and bug status by priority.

Mobile Test Strategies

Depending on how popular one or more mobile apps are with different segments of your customer base, a DevOps team might want to build and test a version of the mobile app to target that segment. For example, creating mobile app versions #1,#2 and# 3, with each version having a different priority, such as low, medium, and high.  Since HTML5 mobile apps are quicker and cheaper to develop than other mobile apps, an organization that wants to establish a presence in a large number of market sectors might use a HTML5 app to "put a toe in the water."  Based on positive feedback from customers, the DevOps team might then devote more resources to a newer hybrid version of the app, which could then to upgraded to a native app for customers who want or need to full advantage of all the features specific to a particular device.  Similarly, support can be degraded in the DevOps pipeline for mobile platforms that fall out of favor with customers.

The only effective way to quickly meet your end-users expectations of mobile app quality is for your development and testing teams to embrace DevOps processes like Continuous Integration, Continuous Delivery and Test Automation.  Continuous Integration is a process where developers check code into a shared repository several times a day.  Each check-in is then verified by an automated build, allowing teams to detect errors and conflicts as soon as possible. The software build, which compiles the source code into binary code, is done automatically using tools such as Makefiles or Ant, rather than when a developer manually invokes the complier.  This allows immediate validation of successful changes to your application, which saves time and reduces the need for rework.

By automating the deployment process, you are, in effect, releasing every good build to testers and users. This allows for the delivery of new functionality to users within minutes whenever it's needed, as well as instant feedback to your DevOps teams that, in turn, allows them to respond more rapidly to customer demand.

When it comes to mobile testing, automation is essential.  In addition to doing automated regression testing in a development tool like JIRA, many agile DevOps teams also practice test-first approaches such as test-driven development (TDD), acceptance test driven development (ATDD) and behavior-driven development (BDD).  While there is a role for ad hoc exploratory testing of mobile apps on actual physical devices, test automation is needed to speed up the testing process and to capture quality metrics needed to increase transparency and accountability.  Selecting, implementing, and updating a test automation framework--especially for testing mobile apps-- should be done with the same care and thought that goes into writing production code.

Because DevOps is a cultural shift that promotes collaboration between development, quality assurance and operations , there is no such thing as a separate DevOps for mobile apps.  DevOps is an approach that works for all applications and components — from front-end mobile apps, to middleware, to backend server components and data stores.  Using DevOps for mobile apps means pushing your mobile apps and updates through your organization's DevOps pipeline to the various endpoint devices used by the customers, internal employees, or business partners that make up your mobile ecosystem.  DevOps provides a way to streamline this complex ecosystem and iteratively improve all of the development and testing steps involved in high quality mobile app delivery.

Related Articles: