Thoughts on Mobile Application QA and Testing

Mar 18th, 2011 by flingsoft

As we wait for the imminent release of Movie Vault 1.2 on the iPhone and 1.3 on the iPad (which both add AirPlay!!), I thought I would offer up some information on the devices and testing we do for the various applications. We are also avidly working on Android, OS X, and Windows Phone builds which should be showing their faces very soon!

Most development teams do not have access to a wide array of devices, so they are forced to depend on the simulators that are included with the SDKs or great third party services like DeviceAnywhere. While most of these are really good ways of testing (and we use them in addition to the devices), we do notice that you can miss a lot of if you rely of them solely. So we have a nice stable of devices thanks to our partners and we are growing it rapidly. Here’s a shot of some of them:

We test on the following for iOS:

  • iPhone 4
  • iPhone 3GS
  • iPhone 3G
  • iPhone Classic
  • iPad
  • iPad 2 (coming soon, still in line!)

For Android, we test on:

  • Motorola Droid 2
  • Motorola Cliq
  • Motorola Backflip
  • Adding more soon!

For Windows Phone 7 we test on:

  • Samsung Focus
  • HTC coming soon!

For IP TV boxes, we test on:

  • Boxee
  • Roku HD
  • Roku XD|S

Some issues:

To start, it is very difficult to test memory leaks and memory related crashes through only the simulator. Whatever the reason is, the simulators seem to not exhibit similar leak behaviors. I can run an app for hours in the simulator and then put it on the device and kill it in 10 seconds. 

Next, UI interaction… you know the jumpiness/smoothness factor. Because the simulator takes into account system processing power things always feel crisp. Not so on the device and especially when you go between versions (iPhone 3G to 3Gs to 4 for example). DeviceAnywhere also has issues here because it is network based so sometimes it can feel jumpier than it might actually be. Two heavy extremes.

Video and this is obviously where we want to shine. We would never use a hosted service to test this and the simulator in most cases can’t even run video well. We can test our streams outside of the app for general functionality, but we always like to try them on device to make sure it looks the best it can given our original source.

General usability is the best use of these tools and where they accelerate development. If you want to see how things look and how interaction functions, then they can drastically speed things up. 

I won’t delve into the actual QA process, but we are very engineering oriented so we go through QA cycles, bug review, fixes, re-test, etc, etc. I can always talk to our process here in detail as well if anyone is really truly interested. I hope everyone enjoyed this and we always welcome comments and questions. Always happy to help and share our experiences.

Filed under:

development, iphone, ipad, boxee, roku, qa,

View Archive