State of QA Testing 2017: Where Are We Now?

SmartBear’s 2017 State of QA Testing report contains responses from over 3,400 employees in the QA industry, including software developers, manual/automation tester, consultants, etc. As always, seeing these types of reports provides an enlightening look into the direction of the QA industry, and allows us to examine our own methodologies and approach to testing. Lets analyze some key stats from the survey, and reflect on their implications for the upcoming years.

API and UI Testing make up the majority of testing responsibilities.

63% of testers were responsible for both frontend and backend testing. Blurring the line between simply being only a UI or API tester means most will now need to understand how their DevOps/Agile methodologies are informing their testing; effectively handling how you write and execute your test cases across both fields will determine your efficiency. We see this number continuing to grow in the future – more technologies and tools are providing more opportunities to understand and develop adopt an automated process for both UI & API testing. A backend developer who may have been purely testing API’s can now use a tool like Galen to create scaleable/reusable UI tests to run every time they merge their code, ensuring nothing was broken on the frontend.

Web apps are the most commonly tested.

85% of those surveyed were testing Web applications. This is to be expected – web apps constitute the majority of testing and are only continuing to grow. However, what we find interesting is the low number of mobile native and mobile web testing. As both mobile devices and apps have increased exponentially over the past few years, there needs be an equivalent rise in the volume of mobile testing. This tells us that not enough teams have adopted a committed mobile testing process into their SDLC –  which is a huge red flag. US Smartphone adoption reached higher than 80% as of December 2016 and will only continue to rise – meaning you will need to be compensating for the proliferation of users accessing your native and web apps.

More than half of surveyed testing teams claimed to be agile.

57% of teams claimed they were agile – however, many of these teams are likely not understanding agile processes correctly. Stats from those surveyed showed that the “Agile teams” were releasing software just as often as those that replied as using Waterfall or another methodology. This is not consistent with a methodology that emphasizes rapid and continuous delivery; if these teams were Agile by standard definition, we would see higher numbers across the Once a day and Daily categories. As we’ll see below, teams stated they had less than 50% of tests automated – which may help to explain the discrepancy between teams claiming to be agile and the release interval statistics.

Automation coverage sits at less than half.

As a whole, majority of responders only had less than half of tests all automated. Even more fascinating was the spread of test automation % by company size – the larger the company, the more tests were automated. We see the highest category at 10k+ having 56% coverage, and the lowest at 1-25 at 19% coverage. Why is there such a large gap in coverage between small and large companies? Smaller companies may have less resources to invest in their initial test automation build out. However, investing those initial resources into developing a test automation process will manifest long term savings in cost & time; these test coverage numbers will only continue rise as more teams develop solidified automation processes.

Frequent application changes are the top challenges to testers.

Modifying and creating new test scripts to account for application changes was the biggest issue for testers. Constant changes will require modification to existing test scripts, and creation for new scripts – as well as regression to ensure no old features are being broken by the new application changes. This means that this maintenance + development process should be a focus for automation engineers. Emphasis on not only the test case definition/parameters, but also the framework development. By developing a framework that is scaleable and dynamic, you can reduce the pain points of testing each release.

API functionality tops the list of  backend testing concerns.

Functionality validation topped the list of priority for those surveyed. This doesn’t come as a huge surprise – your application is broken if your API functionality is failing. This went for testing against both internal and external API’s.

The majority testing on mobile devices.

80% of the surveyed reported testing with mobile devices – with Android taking taking the lead. We’ve previously discussed the challenges faced with mobile testing; specifically, device fragmentation leading to an immense amount of different models/versions of Android phones. With mobile test automation at less than 50% according to the report, resources allocated to manual testing across different app versions/models could be easily cut down with an increase in attention to mobile automation development. 64% of these mobile testers are not testing on the latest device/version combination – meaning easily omitting important test scenarios and missing bugs that their customers will experience. Mobile is only going to grow, and the growth will be addressed with an equal increase in automation/test coverage.

Majority of web and mobile testers are running in Parallel.

Majority of respondents ran device tests in parallel – meaning they had multiple devices running tests simultaneously. Increase in parallel testing will mirror increase in automation – you can’t test 5 devices at the same time by yourself manually. Testing in parallel expands your testing coverage and reduces test times, while simultaneously helping you test across multiple browsers/environments. Again, growth in parallel testing will increase with growth in automation and mobile device coverage.

Takeaway?

In regards to the future, all signs point to automation. Not to say manual will become extinct (in fact, we think manual testing is extremely important), but adoption of automation is tied to to many rising trends to be denied. Above, we can see a graph of what respondents predicted automation percentages would be in 2 years from now. Simply put, QA teams that are not adopting automation (or understanding why they need to) are only holding themselves back. We see the daily struggles of clients’ web & mobile app testing prior to implementing any form of automation. In addition to Awetest, our Automation platform that supports Open Source Mobile, API and Browser Automated Testing & Reporting, 3Qi has been providing a variety of QA & Test Automation Services for several years now.  Our Cross-Border team of 100+ DevOps Automation Engineers specialize in a variety of technologies and can help you achieve your automation goals in a timely and cost-effective manner (resulting in 80+% cost savings for some customers). Contact us for more info!