3 best practices when automating testing of firmware
As a technical director for a Test Center of Excellence (CoE), I work for a manufacturer of phones and other Internet of Things (IoT) devices. My team and I face an escalating cascade of challenges. Device firmware projects have become increasingly complex, with an ever-increasing spectrum of real-world use cases. At some point, you simply can’t assemble enough resources to ensure effective, 100% test coverage.
That was our experience while testing software for a Symbian device S60/S40 firmware project. A diverse team of 20-plus senior quality assurance engineers were involved, who specialized in everything from load testing to manual regression testing.
Fortunately, we had begun to adopt test automation to significantly reduce the mundane, repetitive tasks that quality assurance engineers typically execute manually. Automation became important to our Symbian project, as you’ll see in the following best practices based on our experiences.
1. Select an open source-based test automation framework.
Our project leveraged a number of free and paid test automation solutions based on open-source languages. This allowed us to automate the vast majority of our manual regression test cases and save loads of man hours – all while expanding our test coverage over hundreds of devices. With automation, we can run tests simultaneously on multiple devices and test everything from Wi-Fi connectivity to native applications, including default email and text messaging.
Automation engineers are able to write one set of automated test scripts and reuse them on several devices or platforms. This is important given the large numbers of devices those test cases ultimately need to executed on. By automating as much of our test plan as possible, and we were able to run tests on the largest number of devices we could get our hands on. This allowed us to expand our test coverage to confirm that the default functionality worked as expected. We could do that without expanding the QA team for the project, which lowered costs.
Now having said all that, automation does add a bit of overhead. Popular open-source tools can be a perfect choice since they cover most testing needs and are supported by a wide and active user community. It is critical, though, to make open-source tools enterprise ready. You’ll need a single, integrated test platform for performance, functional and API testing that is compatible with your existing test infrastructure.
2. Extend test coverage at enterprise scale.
Often, QA is faulted for testing in isolation rather than under real-world conditions. While it is great to test whether a specific feature meets your original requirements and works as designed, you need to understand how it will stack up in production. Our Test CoE needs the scalability to test loads of over 10 million virtual users on demand for any given test project – and to be able to do so across multiple test projects simultaneously. We also need centralized visibility into the dedicated workspaces used by multiple test projects so we can create comprehensive drill-down reports.
Since more daily test tasks are transitioning to an online environment, there now are zero platform dependencies. As a result, it becomes simple for our Test CoE to extend test coverage at scale to distributed developer teams working on software releases across any phone or device program. A self-service test creation portal makes everything easy to use. You simply “load and go,” with no proprietary languages to learn.
3. Accelerate test automation for your CI/CD pipeline.
Automated testing is great, but it means very little if you lack a proper continuous integration pipeline that includes reporting, analytics and everything else required. In our case, we use Jenkins to manage the deployment and testing of many different projects – automating test runs and establishing a continuous integration/continuous delivery pipeline.
When ensuring continuous quality controls across diverse test projects, though, it’s important to recognize that automated test flows rely on the test cases you generate and on the suite of regression test scripts you use to validate them.
With multiple software development teams operating remotely and driving agile testing within every sprint, test cases guide effective test script creation and execution. Test automation helps you meet those needs by automatically generating the optimal set of test cases right from your requirements model, and then automatically updating them when requirements change.
Take the next step
These are only a few of the challenges our Test CoE has experienced over the years, and they are not unique. In fact, they are likely to be similar to those experienced by many Test CoEs in other industries. That’s why many companies are adopting CA BlazeMeter and automating their software quality assurance testing. This SaaS-based, open-source platform is powerful, easy to use and supports each of the best practices described above.
You can run massively scalable, open source-based performance tests against all your mainframe apps – web, mobile, microservices and APIs – from the cloud or behind your firewall. CA Technologies even offers testing experts to help you create, execute and analyze tests, resolve issues and support peak-event readiness. Start your continuous testing journey today by requesting a free BlazeMeter demo, and visit www.blazemeter.com/shiftleft to learn more.
You might also find these useful:
Interested in writing for our Blog? Send us a pitch!