DevOps and Test Automation

Traditional IT Ops teams require nearly 60% more time per week to handle support cases as compared to their DevOps Oriented Counterparts! This is just one of many statistics legitimizing DevOps as a software development methodology and it continues to be one of the big areas of focus for companies of all shapes and sizes.


In a nutshell, DevOps is a method of software development that delivers efficiency though communication, collaboration, and integration between Development, Tech Operations, and QA. It has its roots in the Agile methodology and has 3 main benefits (among many others):

  • Increased Velocity for faster delivery of features
  • Increased Stability of operating environments
  • Increased Utilization of Infrastructure & Resources: More Time to focus on value-add activities (rather than fixing problems)

Now that you’ve got an idea of what it is, lets delve into the relationship between QA and DevOps.

Pre-DevOps Testing:

QA and Testing in a Pre-Agile, Pre-DevOps (i.e. Waterfall) world was a sequential process involving gathering requirements, test planning, test documentation, execution & reporting.  This meant, requirements were set in stone right from the onset and introducing changes once the cycle was underway meant a delay in the release OR changes would need to wait until a future release.

Given the business requirements and desired velocity in the past,  this process worked fine, and is the desired process in the Waterfall context.  Unfortunately, it doesn’t lend itself well to the Agile approach organizations are adopting to to keep up with constantly changing requirements and market conditions.

Testing With DevOps:

As mentioned earlier, one of the big benefits of DevOps is increased Software Velocity – which essentially translates to an increased level of feature-functionality development & deployment in a compressed timeframe. Case in point – Netflix, the popular movie streaming site, deploys code to its production environment hundred times per day!

There are 2 main concepts of DevOps:

  • Continuous Integration (CI): dictates the submittal and management of developed code to a shared repository where Developers will test it.
  • Continuous Delivery (CD): ensures that any point during the lifecycle, your software is deployable

The CI aspect of DevOps focuses on the Build activities and the CD aspect of DevOps focuses on the Deploy activities.  Open Source tools like Jenkins help automated and facilitate the continuous Build & Deploy process and companies like Electric-Cloud have built enterprise grade platforms that support CI and CD processes across distributed products & teams for large enterprises.

In order to realize the true potential of DevOps (in addition to CI & CD) the testing should also be “Continuous” in nature.  Testing should be initiated immediately upon a successful build completion (using a CI Tool like Jenkins).  Also, different environments (e.g. DEV, SIT, UAT, etc.) will have different types and scope of Testing associated with it and the Testing Activities should be planned accordingly.  A typical 2-4 week Agile Sprint cycle makes the testers job a lot harder because he/she has to test the new features & fixes in addition to the (ideally) entire existing functionality (i.e. regression test) during the course of the sprint.  One way to accelerate the test cycle is through the successful adoption & implementation of Test Automation tools & technologies.

Automating Your Tests

Regression Testing ensures that something that worked in the past has not been broken because of newer changes in the product. Since regression tests are often part of a growing test suite that needs to be run regularly to monitor existing functionality, they have become a great target for test automation over time.  We have been involved extensively with implementing Open Source Automation Tests for numerous Agile projects and we typically break down the Automation into 3 Categories:

  • Happy/Critical Path Testing:  The core functionality/flows in a given application.  This is what we typically aim to automate first since these are required to be executed most frequently.
  • Regression Testing:  This the next area of focus where we expand the critical path automated tests to adjacent scenarios to incrementally automate the Regression Tests.  This (regression) coverage expands over time but the ROI is the highest due to the sheer volume of test coverage.
  • Release Testing: This relates to the features & fixes in a given sprint.  We automate these tests using a TDD approach.

Once built, Automation will allow you to rapidly execute your test cases across a variety of devices and environments. Much like any part of DevOps, its strength lies in its flexibility; once built, one can easily tailor a script to address a specific issue in a test case. It allows you to increase coverage at a lower cost, and without the fatigue of manual testing. Most importantly, it frees up time for Testers to focus their energies on more strategic aspects of testing (e.g. Exploratory Testing).

We @ 3Qi Labs continue to expand both our Awetest Platform to support DevOps initiatives.  Most recently we have partnered with Electric-Cloud to provide an integrated offering that leverages the Electric Cloud’s enterprise Electric Flow platform and Awetest’s Mobile, Responsive & API Test Automation capabilities to facilitate an Enterprise-Ready Continuous Delivery Platform.  We also integrate with other DevOps & Agile Tools likes Jenkins, Anthill Pro, Jira and continue to expand our APIs to increase the interoperability in an increasingly Agile & DevOps environment.

We’d love to hear your thoughts & experiences; contact us to get more details.