It was 10 pm and after waiting patiently for a build to test the latest release delivery for a complex app – this was to be the 4th of 5th delivery drops for this app – one with 3 new functionality elements but a fairly complete and functional app. Client delivery weeks are generally crazy but I was getting tired of the waiting game. I have always appreciated the hard work my development counterparts put in to create, troubleshoot and refine code but after 3 years of being a test engineer and having gone through these wait periods multiple times, I have concluded that waiting is boring.
So instead of being frustrated and getting on the development teams’ nerves, I thought of doing something creative with what was available to me – test scenarios, scripts, use case documentation, functional spec and an app with 3 builds already functioning well with <10 bugs. I decided to automate a suite of build validation tests (BVT). I always was of the opinion that automating testing for a mobile app is a bit more complex than for a non-mobile application; but I decided to try and prove myself wrong.
Here is what I researched…
In pursuit of automation, I had to benchmark my current efforts and compare them with efforts needed for test automation. For manual testing, the efforts usually multiply in linear manner with every build and new features. However, for automation it was obvious that there will be a more upfront cost which will provide cumulative benefits with every subsequent release. Here’s how I put the automation efforts and payback.
While there will be initial high effort for scripting the automation tests, most of my test automation effort will curve down and will benefit for future regression. I was looking forward to the day when I would be able to conduct unmanned test execution (CUTE 😀).
Once deployed, this solution will drive adoption of the mobile app, thereby allowing Utility companies to gain a better handle on usage patterns, generate customer and regional usage profile, and ultimately offer valuable input into generation.
I had the following expectations from my automation activities:
Here are the reasons for my selection
If my project budget had a dedicated budget for Test Automation, I would have gone for SeeTestAutomation from Experitest which provides extensive documentation and support. As my endeavor for automation was a bootstrap project, I decided to forego the license spend and experiment with something which was available for free or at a price-point which could be justified as ‘throwaway’
So onto the drawing board, I had the possible tools and the requirements: organized as below. I tried to play devil’s advocate here, to stress on the demerits and their impact
While building a case for optimum tool, one has to know that if there is an Agent to be embedded in the app then it needs to be removed before submitted to Apple App Store, which means to assure the app is Ready-to-submit (RTS) one has to perform a final manual testing. So considering this and other offerings, Appium and Calabash seemed to impress me, as they were the ones that fit within my constraints and yet fulfilled my objectives. At the onset, I chose Appium as it offered support for multiple languages rather than Ruby being the only choice in case of Calabash. This was a personal choice as I know Calabash to be an extremely capable tool.
With the project delivery drop number 5 three weeks away, I collaborated with another teammate and decided to spend a couple of hours each day to work on the BVT automation. We selected test cases and ran them by the project’s business analyst. It turned out that she too was in a similar waiting game and appreciated the extra effort we were putting in. A week later we were ‘testing’ our automation suite of 30 test cases and here is what we found:
I had the following expectations from my automation activities:
Our initiative had turned out really well. I showed this to my manager and was even more encouraged to see his excitement and support. He was able to get me some much needed help in a couple of areas which I struggled with.
These were:
As it turned out, 6 months from when I automated the first BVT suite for my project, this practice of automation has become the norm for every project in the company. Our team has grown and we are now adding mobile test automation as a service line to our portfolio. Here is a generic architecture of how we have tackled mobile test automation.
With these benefits, I would say that test automation is a necessity for every development project. Our experience with automation in mobile testing has been so effective that we now consider this as a key contributor to increased productivity, better quality and effort reduction.
As a company, the test team now works even more closely with the development team to be future-ready for new OS updates and development frameworks. My experiment has blossomed into a full blown practice area and we have incorporated mobile app testing into all our engagements.
The author is a Sr. Test Engineer at Accellor. His passion and initiative has led the company to develop a practice and we now have capability with a wide range of test tools available in the market place.
Are you losing out on productivity and quality?
To know more about our services, please fill out the information and you will get a call back from one of our associates.
