With over 10.97 billion mobile connections globally, the demand for sophisticated, high performance B2B and B2C mobile apps has been growing. The global mobile app market has been growing at over 11.5% annually, valued at over $154.06 billion – fueled even further by the COVID-19 shift to remote work and the rise in the amount of time spent online.
In this mobile-first world, expectations are high – and there is zero tolerance for bugs or performance issues in mobile apps, whether an e-Commerce app, a loyalty app, or a SaaS product. In fact, 25% of apps are used only once before being abandoned.
Every time a company designs a rich, useful, or new experience, it raises the bar for what consumers care about. On the flip side, if an experience is frustrating or annoying, a consumer may not give that company a second chance.Jason Spero, Google
The top reason for mobile app abandonment rates is a poor user experience: a confusing UI/UX, too many bugs, slow load time, or an app crash. The need to create a strong mobile app that is seamless to both iOS and Android requires continuous testing to ensure performance. However, mobile app testing can be quite a complex process across platforms, operating systems, network connections, carriers, and situations – which is where automation steps in.
In this article, we will look at the best practices for mobile app testing, including the best test cases to automate, and how to select the best framework for mobile automation testing.
Step 1. Setup Mobile Automation Testing Goals
Software testing is an important step in app development to check that the app works as expected (positive testing) and that it can handle unexpected conditions (negative testing).
How do you test mobile applications? Mobile application testing mirrors your development process, with your unit tests being your base – something you do often during development and throughout maintenance to ensure performance. As you go up the pyramid, you get closer to the user experience, where testing takes the form of more specific user feedback. The closer you get to the end user, the harder it is to automate your testing.
Manual vs Automated Mobile App Testing
Manual mobile app testing involves developers or test engineers who check on the features, uses, and functionalities of the app against various test cases and across various mobile devices and emulators. However, manual testing is time-consuming, labor-intensive, prone to human error, and is difficult to scale as the functions of the app become more complex.
Automated mobile app testing leverages specialized tools to conduct and control test cases, reducing the time spent testing. Automated testing is best suited for large projects that require continuous or repeated testing of pre-written scripts, with the benefit that many simultaneous tests can be run at the same time across different components or mobile frameworks. Automated mobile testing is well suited for Agile development, which approaches testing with the same focus on early, iterative, and continuous testing to find and repair bugs and performance issues early in the development phase – before bugs become “baked in” and harder to repair.
The goal of automation is to increase both the efficiency (time and cost) and quality of your mobile app testing. As you decide which types of test cases are candidates for automation, you must always refer to these two main goals:
- Will automating this test case save me time?
- Will automating this test case increase the quality or performance of my app?
Step 2: Plan Test Cases
The Agile testing matrix / testing quadrants lay out the different kinds of test cases you will implement throughout your project development cycle. Here it is important to note that testing is not sequential or reserved for the end of your product development, but rather an integrated part of each Agile sprint.
As you can see, in line with our pyramid, business facing / customer facing tests in Q3 rely heavily on manual testing, while Q1 and Q2 tests are open to automation. For automation to be appropriate, the test case should be something that is:
- Repeated often or is repetitive
- Involves time consuming data entry
- Subject to human error
- Low risk
- Easily measured / objective
If the test case relies on subjective feedback (e.g. UI / UX), is exploratory, or requires many steps, it’s likely not a good candidate for automation. Equally, if you only need to run the test once, it is not worth writing a test automation for. Since setting up automation takes time, you want it to be worth your effort!
The three most common mobile app test cases to automate are unit tests, functional tests, and integration tests.
3 Types of Test Cases to Automate
1. Unit Testing
Unit testing is an Agile Testing process for checking the quality and efficiency of individual user stories, i.e., for a specific feature (or unit) built by the developers. This white box testing is your smallest unit of testing, making sure each feature / user story works from a design and technical perspective. Identifying and fixing issues at the unit level is more cost effective than finding these issues later when many dependencies are present. As your application is made up of many user stories, unit testing helps ensure a solid foundation of code that translates into a better end user experience in the real world.
For mobile app development, unit tests will need to be run locally and also instrumented against each mobile OS (iOS / Android) to check for dependencies.
Example: When the authentication and login user story is ready, the unit test will be run to check if the login works as per the expectations. This may look at field length, character minimums, error messages, and the login button only works when fields are filled out.
Why automate? Unit tests focus on isolated code with many frameworks available to concurrently test units across multiple mobile frameworks. These unit tests are easy to run hourly or daily (or any logical timeframe) to validate behavior against any changes, helping surface fixes quickly.
2. Integration Testing
Just like oil and vinegar do not mix, in mobile app development you can have validated units that fail to work when combined. Your integration tests look at system wide performance, ensuring the functionality between your units.
Example: Continuing with login, your integration test would combine your login user story with the database authentication. The integration test does not verify that you get the right next step (welcome page or login error screen, for example), but simply that the process to connect to the database to verify authentication works.
Why automate? While there can be dependencies in integration testing (such as databases), integration tests are still very simple and focused. Continually running the test (regression testing) ensures that new code / features do not create new bugs or performance issues.
3. Functional Testing
Functional testing is a type of black-box testing where the software system is tested to check whether it adheres to the specified functional requirements, often requiring checks of the database, client/server communication, user interface (UI) and any other functional component or call (API). The functional test will look at both the output of the test as well as its performance, examining the basic functionality, usability, accessibility, and error conditions of the application.
Example: The functional test looks at the desired expectation: did a successful login (the input) trigger the welcome message (the correct output)? Is the user prompted to recover a forgotten password? If your fields are case sensitive, does your error message reflect that? Each scenario of the login process should be tested.
Why automate? Functional testing compares performance against a known requirement, making the testing very open to automation as the mobile app continues to evolve. In the case of more complex functions, you may not capture all the scenarios, leaving the potential for automated functional testing to miss some issues.
Step 3: How to Select a Test Automation Framework
Now that you have identified test cases you’d like to automate, the next step is to select the right automation framework – an integrated system that sets the rules of automation for your test. Think of the test automation framework as the best practices to create and review your tests. Let’s take a look at the top six frameworks for automated mobile app testing.
Mobile Automation Frameworks
1. Linear Automation Framework
Known as a “record and playback model,” this framework is very linear and incremental, perfect for unit tests or simple applications.
2. Modular-Based Testing Framework
In a modular testing framework, we create tests for scenarios (small tests). Modules can be combined into larger scenarios for testing. A module is said to hide information by leveraging an abstraction layer so that changes made in the units of the application do not affect the module.
3.Library Architecture Testing Framework
This framework is similar to the modular testing framework, but instead of modules, we group common tasks into functions; functions are then stored in a library. This library is referenced to create new test cases and makes it very easy to create reusable test cases.
4. Data-Driven Framework
A data-driven framework recognizes that the test may stay the same but the data may differ. This framework pulls data from an external system (or systems, if we are comparing input against expected data), testing a functionality (say login) against this data.
5. Keyword-Driven Framework
Sometimes called table-driven, this framework pairs external test data, like in the data-driven framework, alongside keywords (actions) stored in a table (such as Excel).. Different test scripts can access the same keywords, but these frameworks can be time consuming to set up.
6. Hybrid Testing Framework
Hybrid frameworks reference two or more of the above frameworks, allowing teams to create the ideal test environment.
Step 4: How to Select the Right Automation Testing Tool
Mobile automation testing tools help you create test scripts following one or more of the above test automation frameworks. Although it is not necessary to understand the ins and outs of these frameworks to choose the right mobile testing tool, a basic understanding of the frameworks can help ensure you’re getting the right tool for the job at hand.
Mobile Application Testing Checklist
The factors you may want to consider when selecting a mobile app testing tool are:
- OS support (iOS / Android / Windows)
- Type of tests supported (unit tests, regression tests, functional tests, etc)
- Easy to use, including script less test creation, simple tutorials, clear reporting
- Integration with existing continuous integration / continuous delivery (CI/CD) tools
- Cost and scalability
Top Mobile Testing Tools to Choose
Appium is a flexible, open-sourced tool that leverages multiple languages, frameworks, and is ideally suited for black box testing of native iOS, Android or Windows apps, as well as hybrid and mobile web apps. Appium makes it easy to reuse test cases across platforms, but testing may be slower or lack the precision of native testing tools.
2. Google Espresso
Made by Google, Espresso is designed specifically for Android, Java, and white box testing and UI tests.
3.XCTest & XCUITest
Apple’s XCTest and XCUITest leverage libraries, Swift/Objective C for iOS testing and are ideally suited for white box testing.
An open-sourced, cross-platform framework from Perfecto, Quantum is Java based tool aimed at code-less automation.
Step 5: Virtual Devices vs Real Devices
Robust mobile app testing will extend to the operating system (OS) as well as the hardware (device). However, the variety in number of mobile devices and configurations makes device testing a logistical nightmare. While it may be nice to test on real devices, even for Apple this includes 14 generations of devices and several models per generation.
In order to be realistic, best practices would suggest testing on at least one of each target device (most recent iOS device, top Android phone, etc), with the rest of your testing taking place with virtual devices (known as simulators or emulators). Virtual devices mimic many aspects of real devices, improving the speed and cost of testing, with a slight loss to accuracy.
Before you even begin developing your mobile app, you can begin setting up your test cases. With this plan in mind, you can begin testing your mobile app early and often to help improve your time to market and your ultimate performance. The more focused and organized your mobile app testing, the faster you can integrate feedback (of course testing to avoid integration issues!) and work on perfecting your app.
As we mentioned earlier, great mobile apps are one part design and one part testing. Net Solutions is a full service design agency with experts who can help you at every stage. Contact us today to see how we can help accelerate your app development.