Integrating sikuli Visual Automation with Awetest.

Awetest now supports Visual Automation using sikuli Script, a revolutionary new technology from MIT’s Artificial Intelligence Research Labs. Visual Automation complements Awetest’s existing regression testing capabilities by allowing users to integrate complex interface scenarios – think Drag/Drop, Scroll, Swipe, and any other complex Browser-to-Desktop-to-Browser interactions.

You may be asking yourself, “why is visual based automation considered so revolutionary and how it differs from existing testing methods?” Traditional automation tools use an external COM component or a browser plug‐in to drive the browser by manipulating DOM elements. Sikuli offers a new way of adding interactions & tests that were previously hard/impossible to automate. Visual Automation delivers keyboard and mouse events at specified locations using image patterns defined during the scripting process. This revolutionary approach gives us the ability to create automated tests using visual cues (screenshots, images, etc.) coupled with simple commands (e.g. click, drag & drop, double click, etc.) that can be written & executed in a matter of minutes without requiring any technical expertise. Consider the following example.

In the example above, Sikuli is used to automate the sending of a tweet.

Sikuli IDE takes actual screen shots (.png’s) of the sections on the page you highlight and displays them right in the script. In this case, Sikuli looks first for an image of the text box and clicks on the graphic that matches. It then types in to this text field “this is a tweet from Sikuli!!” and then clicks on the update button by again matching the graphic. Incredibly simple, right? Extremely powerful, yes.

Sikuli Script is a Jython and Java library that automates GUI interaction using image patterns to direct keyboard/mouse events. The core of Sikuli Script is a Java library that consists of two parts: java.awt.Robot, which delivers keyboard and mouse events to appropriate locations, and a C++ engine based on OpenCV, which searches given image patterns on the screen. The C++ engine is connected to Java via JNI and needs to be compiled for each platform. On top of the Java library, a thin Jython layer is provided for end‐users as a set of simple and clear commands. Therefore, it should be easy to add more thin layers for other languages running on JVM, e.g. JRuby, Scala, Javascript, etc.

Here is an example of Sikuli opening up a browser,

in this case firefox, navigating to google maps, fining the zoom scroll bar, dragging the zoom bar upwards 1000 px. Sikuli then finds the images in sequence below until it reaches the “Boston” image and then double-clicks. Finally Sikuli displays a pop-up that we have arrived at our destination, “Here we are!”

Sikuli Scripts are incredibly simple to understand

yet powerful enough to control most user actions within the browser and OS. Sikuli Scripts allow for greater enhancement of existing GUI testing by adding interactions & tests that were previously hard/impossible to automate, the ease of developing very comprehensive tests with little/no technical expertise, support for most desktop interactions, test iOS and Android applications (including Native Applications) through desktop emulators. This makes the Sikuli Technology a brilliant addition to the Awetest test automation framework.