Managing mobile resources with limited budget in testing

Credit image to https://www.360logica.com

Mobile testing is expansive especially for startups. Many small teams lack budgets to purchase mobile devices , it is common to use their personal devices to develop and test. Many developers and testers use emulators (or simulators for iOS) to run their applications.

Running in emulators or simulators for pre-alpha tests are alright , but they do not represent customer’s perspective. It is useful to see the layout and conduct some basic functionality tests in emulator / simulators. Emulators (especially Androids) testings are useful if you are testing across multiple platform versions in several emulated devices , but I caution using them as a primary testing platform as they can add a false sense of confidence how AUT behave in real devices. Hence , my recommendation is test on real devices as much as possible.

However, startups typically have 2–6 mobile devices or worse use their personal devices for testing and development due to budget constraints. This is not healthy in the mid term. They use their own personal devices for development and testings.

Using personal real devices have several disadvantages.

How I planned resources for a mobile project

I recommend the team to choose the devices base on the business current market needs — the most popular device (iOS / Android) for tier one / tier two customer. They should also include a high end device in both platforms. In this way , we may cover approximately 10–20% of the market — this may varies according to your business market need.

These 6 devices are use extensively for manual and automation testing only. For automation , it will do a CI for P1 (sanity test) and an over night regression runs. Developers will commit in the morning. The sanity scripts on the real devices will kick in before lunch and complete in an hour. The functional regression tests will kick in after 8pm and the I will have the report before I come in the next morning. The manual testers / product owners will have 3–4 hours to use the 6 devices in the afternoon. I encourage the manual testers to explore the hardware related with the AUT such as using hardware related to AUT (example camera capture sending images to the app) or other AUT testing which is customer persona (example as usability). Some testings are done with other system testing such as wifi / bluetooth / incoming calls. This is something which is not worthwhile to do in automation.

I have also evaluated using mobile crowd resources in AWS , Sauce Labs and Browserstack to expand our mobile resources. I have chosen one of the services. We have now got an access to more than 200 devices with 500 minutes per month (which we can upgrade). Since the release is twice a month (if there are no showstopper / critical patches) , these mobile resources are sufficient for now.

Managing cloud resources

Base on the capture customer profiles , we select the next top 7–10 real devices from the services. These devices will run P2 AUT features (high level features) which will take approximately 10 minutes per devices every Wednesday and Friday. New features are also slot in automation iteration to run on the cloud. Our manual testers will also use the services to do exploratory testing on selected cloud devices after they have complete the local real devices.

Start with three to four devices on premise , use cloud testing to compliment your coverage. Then scale up your local devices as your operation grows.

In this way , mobile resources are better managed with the cost kept down. We do not need to continuously buy an array of 20–30 devices but we balance it with a limited local grid and expanded mobile cloud devices. This helps us to cast our testing across wider area yet keep the cost down in a small startup team. There are still much more to improve in managing the resources and we are working towards that.

Let me know how you manage your mobile testing resources with a limited budget.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store