Here is a list of challenges and pain points every Dev or QA manager should take into account when considering whether or not to implement test automation into their quality assurance cycle. After reading these, head to our How To section to understand how to properly implement Watir Webdriver, Selenium, and Ruby for consistent & efficient automation scripts.
1. Automate Everything?
Knowing when you have enough test automation coverage can be a tough realization. The test engineers we speak to on a day to day basis seem to be constantly grappling with these questions:
- What constitutes an adequate degree of test automation coverage?
- Does every single function in the application, big or small, need a corresponding test associated with it, or just the core functionality?
- Does every application need a full end to end automation suite built out or just enough to satisfy the quality assurance and business development departments?
The answer differs from every team to every project, to every organization, but should always be derived from a clear, continuous, communication between Dev, Biz Dev, QA, and Management about the size, scope, time-frame, budget, and end-user experience.
2. Understanding Company, Client, and End-User Expectations
Developers can always help build automated test cases from their technical expectations, but are those really going to address all the expectations of the end user? Surprisingly, the expectations of End-Users can often differ greatly from those of the developers, so understanding customer feedback can often be one of the most essential things a software developer can do. Testers can also learn a great deal from this customer feedback, and thus build better scripts to map actual user interactions with greater precision.
3. Clear Communication Between Testers and Development, QA, Biz Dev, Management, and End-User-Client
This sounds like a straight forward challenge but proves time and time again to be a real sore point in the testing process, whether manual or automated. Whatever the cause of this eternal struggle to keep a clear line of communication between testers and the rest of the development process, there are things you can do about it.
Scheduling regular meetings between everyone involved to discuss the important details about the project is essential. A disconnect in communication between stakeholders can result in missing coverage or lack of adequate preparation. For example, a sizable new feature is recently scheduled for release and needs automation built around it, but the testers are not told about it until the last minute. Producing a solid automated test for that feature is going to be very difficult to do with a constrained timeframe. Now, multiply that scenario by 100 features/releases and you can see how the lack of clear communication can quickly turn into a production nightmare.
4. Keeping Automation Test Cases Up-To-Date
With more and more development cycles adopting an Agile approach, the automation suite has to always have as up-to-date test cases as possible. Unfortunately for those who think automation means that you can set and forget your software testing, there will always be some amount of maintenance needed on your automation scripts. The simple fact that any change to the DOM elements can halt an automation script in its tracks means your testers need to have an intimate understanding of how the scripts are interacting with the application under test so they can quickly identify issues, correct them, and re-validate the test. It is always a good practice to create detailed documentation for all test cases in the automation suite to help facilitate the debugging process.
5. Proper Use of Testing Resources
Let’s face it. Test Automation is a technical skill. It takes someone with a technical mind to properly build, maintain, and understand the reporting of test automation. Developers can manage but they aren’t seasoned testers, just as testers can write simple programs but aren’t really developers. Keep this in mind when you consider how to utilize your companies resources with the automation process. More times than not, you might find yourself asking the question, “Do I need to hire an automation specialist for this project?”
6. Understanding the Testing Process
With automation (costing time, money, and resources) being relied on more and more in the software testing process, it is crucial that every team member involved has a good understanding of the steps taken to automate. This can often be mischaracterized as record and playback testing by those who don’t understand it at a deeper level. Be sure that every team member understands how automation scripts are built, maintained, executed, and how to understand the results.
7. I Have An End to End Automation Suite and A Release Coming Up, What Needs to be Focused on in this Short Time Frame?
Talk with the developers. They might have some insight into trouble areas of the application. Talk with biz dev. They might have greater concerns with some areas of the applications than others. If there has been a clear line of communication between these departments. the testers should have a good idea of the trouble points of the application. As long as the release is not going to production then a quicker smoke test of the application can suffice in a time crunch. Just remember to run your automation end to end at regular intervals to catch small bugs hiding in the corners of your application.
8. When Can You Stop Testing? How Much Coverage is Enough?
The truth of today’s modern cloud-based applications is that testing can go on forever. Even if development on the initial application ceases updates to underlying technologies or to the network or database could and probably will change something. Testing should never really be thought of as something that will end. Think of it more like eating. You might be full now but dinner is right around the corner. You’re always going to have to eat.
9. Making Scripts Reusable Means a Fair Amount of Labor, Don’t Be Discouraged
Continuous Migration or Refactoring code might sound like a ton of extra upfront work. However, it will end up saving your testing team countless hours when new test cases need to be drafted on a moment’s notice; if the coded has been migrated to a central utility file it can then be called in a single line in the test script. A test case might end up looking as simple as
Test Something New *** not in util.
10. Don’t Only Focus on the Easy Bugs
Not taking the time to seek out and script for tougher bugs can defeat the whole purpose of testing your software in the first place but taking too long to test can really slow down today’s fast paced development cycles. It will likely come down to a balancing act between scripting for difficult features or extending coverage. Be prepared to budget the time spent on test automation accordingly.