As mobile apps have overtaken PC internet usage, there is a growing shift towards mobile-first development, and consequently, Mobile testing. This has triggered new sets of challenges for developers & testers that didn’t exist up until recently.
Increased mobile usage (and development) has changed user behavior wherein mobile users are increasingly unforgiving. If an app fails or is buggy with a new release, the users can easily give up and never look back. This low tolerance for error puts the pressure on mobile app testing, because your Mobile App Doesn’t Get a Second Chance to Make a First Impression.
At 3Qi Labs we have been at the forefront of Mobile Testing – focused on both Mobile Browser & Mobile App Testing across Native & Hybrid Android & iOS platforms. We have conducted over 1 Million+ Tests across a plethora of Mobile Device-SDK-Form Factor combinations and many of the challenges we encountered (and addressed) are listed below.
Fragmentation
The most difficult aspect with mobile testing would be the vast amount of different testing variations when running against different screen sizes and operating system versions. This matrix is especially detailed when it comes to Android fragmentation, where in 2014 the market saw nearly 19,000 separate Android device models. This problem will continue to intensify (especially for the Android platform) as the frequency of new hardware releases continues to grow in an increasingly competitive device/hardware environment – The number of android device manufacturers has almost sextupled since 2012!
OS Versions
Aside from the extremely varying screen sizes of devices, there are also compatibility issues as mobile applications can be run across different operating systems – iOS, Android, Blackberry, Windows, etc.
Both iOS and Android average 1-2 major OS releases per year (e.g. Android Lollipop 5.0, iOS 8), and 2-4 major point releases during that period (e.g. Android Lollipop 5.1, iOS 8.4). While all releases claim to be backward compatible, sometimes applications, unfortunately, end up with unexpected behavior (i.e. bugs) when deployed on a new OS. Case in point – Apple’s most recent iOS 8 release triggered a 67% increase in crash rates for popular apps like Facebook and Dropbox.
On the other hand the OS Fragmentation, while not as acute as the device fragmentation, makes it imperative to test on older OS versions as app developers need to support older versions & APIs. This is especially important for Android applications since > 40% of Android users are still using 3+ year old Android Jellybean (4.3) or lower version.
Different Mobile App Types
Mobile apps come in a few different flavors. Apps can be native, web, or hybrid- and the testing implementation changes drastically with each one. Aside from having a different architecture, apps can have significantly different functionality, performance, and capabilities. E.g. Native apps tend to have richer UI experience and better performance & responsiveness – a big reason why game developers will typically go the native route.
As a result, the app behavior differs from installation to functionality and the combination of process, methodology, and tools is very different for the different types of mobile applications being tested.
With enterprises actively moving to Responsive Web Applications (https://3qilabs.com/what-to-know-before-implementing-responsive-design/), the need to test across different phone, tablet, break-point combinations becomes crucial in ensuring a positive and consistent user experience.
Usability
Usability is one of the key drivers of a mobile application’s success (or failure) and is one of the hardest to test. “Fat fingering” is the rule, rather than the exception for mobile apps and apps need to be designed as such. When converting a desktop application into a mobile one the biggest problem is usually trying to fit all features and information into a much smaller space. Minute details down to font size, readability, and click area size are all things that need to be tested for. This is another reason responsive design has been taking center stage in web application development; the type of information displayed & available interaction on the limited mobile real estate is as, if not more, important than the aesthetics and look & feel.
It is also extremely important to understand the different user personas and the context and environment in which users will be accessing the applications on their devices. A typical test case we encounter for FinServ engagements involves how the application behaves in a slow or intermittent network and the corresponding messaging/alerts the user sees. When & how does the application provide a success/fail notification for a payment initiated on the application just as the user was walking into the elevator?
These are only 4 of a long list of challenges we face with Mobile App Testing. After running millions of automated tests across native, hybrid, responsive/web applications across 100s of Device-OS-Browser combinations, we have come to appreciate (and address!) the many challenges this rapidly growing mobile ecosystem has spawned. We see the daily struggles of mobile app testing because we have real world experience testing native AND hybrid applications on both iOS & Android. We have continually fine tuned the Awetest Mobile offering to ensure that the challenges mentioned above are being tackled in a comprehensive and scalable manner so that bugs are caught as early in the SDLC as possible.
