We talk to a lot of people about their mobile testing strategy. Most clients seem to start with a very small mobile pilot project. That pilot is tested by hand, usually in the emulator first and then on a physical handset. You can pretty much guess the story from there - the mobile app becomes very popular, very quickly. The team starts adding features, and then the regression bugs start popping up. The team is then faced with two options:
- Slow down delivery of new features, adding more and more manual regression testing
- Figure out how to tackle automation to keep velocity up
The problem with #1 is that you lose speed. The problem with #2 is that most teams have never done this before.
You've got lot of options for #2, but they all have significant pros and cons. At the most basic level, I would recommend starting with a clear model for how the application is to be tested from an architectural perspective - put another way, the lead developer should be able to clearly explain how the application is expected to be testable.
The actual strategy varies tremendously depending on the target mobile platform. This is one reason why mobile web applications are increasingly popular - they are much easier to test!
Unfortunately, a complete description of all of the different test strategies is outside the scope of this post - each platform could easily take a book to cover.
Here are a few pointers to get you started:
That said, the actual strategy will vary tremendously depending on the application. A business application that provides an interface to a web service is completely different from an OpenGL-based game. For multiple devices, you may want to use SauceLabs Mobile.
Depending on the application, you may want to consider more exotic testing strategies. For example, you can actually configure a VNC server on your Android device, and then drive it via a continuous integration server and Sikuli.
Getting started with all of this can be overwhelming - that's why it can be a good plan to bring in someone to help.