PhantomJS:Tighten the testing loop

A common theme in testing browser-based applications is the browser matrix. Most projects are targeted at a small set of browsers and versions. This is intended to align to a majority of users that will use the application. The browser matrix is an import consideration in web applications. Unfortunately, testing a browser matrix indroduces several problems into the teams quality automation.

First, multiple browsers increase the complexity of test automation. The developer building automation must handle the matrix and browser expectations in code. This makes the code more costly to produce and maintain. This reduction in implementation speed can lean a project toward reduces the quantity of browser based tests.

Second, the time to run a full test suite increases dramatically. The tests must be run for every combination in the matrix. While this can be parallelized, the test must run within an active browser. This is increases the time to get feedback. The faster a set of tests run, the quicker the team can respond to breakages. One key goal of agile is reducing the feedback times to make changes less costly to implement.

So, how do you reduce the cost of building tests and the time for feedback? The test pyramid is a common metaphor for placing efforts on cheaper, faster tests before more expensive, integrated tests. While the test pyramid has the UI tests on the top, it is possible to break the UI tests into a simple pyramid of their own.

In order to have a high value suite of tests that run quickly, a new tool is required. This tool is PhantomJS. PhantomJS runs a headless browser and is completely automatable. It is based on WebKit and runs from the command line. It is also very fast.

The testing pyramid for UI automation is based on a broad suite of tests targeted at a single browser: PhantomJS. The PhantomJS tests are the unit tests of browser automation. The cover the logic and flows deeply. The also cover it quickly. This suite is run as often as possible. Ideally this should be on every project commit or code release.

On the server side, we write unit tests as quick-executing tests that mock most things, and integration tests run against the whole stack. You can think of PhantomJS as mocking out the browser. Most of your logic and dom-manipulation tests are run through PhantomJS. This gives you confidence that your logic is correct. When you want to test the website integrated with the rest of your application stack, that's an integration test. In the UI world, your browser matrix tests are your integration tests. They expose unexpected conditions raised by working against other processes and environments. However, because you've already tested the logic of the Javascript code, they should be a much smaller set of tests that cross the entire browser matrix and give a degree of confidence in the full platform.