Customer
How Deckers Brands Automated Their Testing Process with BlazeMeter
BLAZEMETER BENEFITS FOR DECKERS BRANDS:
- Efficiency
- Reliability
- Improved Customer Experience
“BlazeMeter helps us separate the wheat from the chaff so we can test the most relevant performance requirements, efficiently and accurately. With BlazeMeter, we can make sure we are better today than we were yesterday.”
— Ryan O’Neill, Senior Engineering Manager of QA, Automation and DevOps, Deckers Brands
ABOUT DECKERS BRANDS
Deckers Brands is a shoes and fashion brand based in California. Its products are sold in brick-and-mortar stores as well as online. In addition to the Deckers brand, the company is also the parent company of UGG, Teva, Sanuk, Hoka One One and Koolaburra brands.
THE CHALLENGE - A TEDIOUS AND MANUAL TESTING PROCESSESS
Deckers shoes can be purchased online. Following the pandemic, their online sales surged. To ensure a good customer experience and operational stability, the QA and DevOps teams of the ecommerce group are tasked with ensuring online orders are properly tracked and that the ecommerce website is performing as expected. They are also in charge of new website releases.
In addition to these ongoing requirements, the team also has to make sure that the ecommerce website is performing optimally during peak seasons. The website has to be able to handle heavy traffic loads of holiday shoppers without crashing. These loads are expected to be 10x the amount of their normal website traffic, with specific order peak times at certain hours, like lunch breaks.
Before using BlazeMeter, the QA team was testing their systems manually. They had hundreds of manual tests dispersed in Confluence, spreadsheets and word documents. Each new release or code update required scouring through different documents to find the relevant testing process. The tests were then run manually and an email update was sent with the tests’ results.
This inefficient process resulted in a severe waste of time. Testers were running bloated tests, they were spending time searching through documents instead of testing and it was difficult for them to determine which tests were relevant for each release.
In addition, it was hard for the engineering organization to effectively track test results and determine which features needed to be refactored. Running the test themselves also took a long time. Finally, the QA team found it challenging to ensure there were no manual errors in tests, which could impede delivery and impact code quality.
On top of the QA testing challenges, the DevOps team was struggling with their testing environment. The company uses Salesforce-based websites and environments. However, they were suspicious that their Salesforce testing environment was different from their production environment, which was impairing their ability to rely on test results and ensure production-ready code.
THE SOLUTION - AN AUTOMATION-FIRST STRATEGY WITH BLAZEMETER
Ryan O’Neill, Senior Engineering Manager of QA, Automation and DevOps at Deckers Brands, set out to find a more efficient and less painful solution. He was looking for a way to modernize the teams’ testing systems and methods through an automation-first and agile strategy.
He started out by incorporating the scrum methodology and implementing CI/CD practices into the testing process. Then, the team began converting their manual tests to automated tests, a handful at a time.
Once the team had a substantial number of automated tests, they began categorizing them into types, like performance tests, load tests, component tests, UI tests, etc. They added their performance and load tests to BlazeMeter and began running their JMeter scripts in BlazeMeter automatically, as part of their continuous testing methodology. Currently, they are running approximately ten different tests against three different websites through BlazeMeter.
In addition, to prepare for the big 5 peak and ensure the website is ready for the holiday season, the team began running BlazeMeter performance and load tests months in advance.
Now, this process starts out in July of each year, when Ryan begins to plan the load testing strategy. Planning includes determining which tests will be run, the test cases, which metrics and KPIs that will be targeted and how to monitor them to validate success. Testing itself begins three months before the holiday season, in September, to ensure that by November the website and environments are ready for the expected peaks.
THE RESULTS - EFFICIENCY AND A BETTER CUSTOMER EXPERIENCE
By using BlazeMeter, the QA team is now able to continuously test and monitor their e-commerce website regularly and automatically. BlazeMeter load and performance tests are run every weekend to evaluate website performance. They ensure that the user flow - from the landing page to filling the cart to checkout - is working properly and that the user experience is positive.
In addition, BlazeMeter testing is ramped up before the holiday season to ensure the website is ready for the peak shopper demands. Finally, when the website is updated or improved, BlazeMeter tests are run to evaluate their impact on functionality and performance. The team uses BlazeMeter’s Baseline Comparison feature to track metrics and determine if they improved how long pages take to load.
As a result, Deckers Brands is now able to support more users and release new updates much faster than before. In addition, the testing process is much more efficient and scalable than before, and less prone to manual errors - at all times and especially before peak shopping season.
BlazeMeter also helped the Deckers Brands team with their Salesforce challenges. To test their suspicions, they ran a performance test against their production environment late at night, when there was no real website traffic. By displaying the production environment results compared to the testing environment results side by side, they were able to guide Salesforce on how to fix the testing environment and ensure the testing environment was both reliable and accurate.
“Our testing process is 100% automated. To rely solely on automated testing you need to have a huge amount of confidence in your tests, and we have that confidence in BlazeMeter.”
— Ryan O’Neill, Senior Engineering Manager of QA, Automation and DevOps, Deckers Brands
FUTURE PLANS WITH BLAZEMETER
Deckers Brands plans to expand their use of BlazeMeter by:
- Incorporating BlazeMeter tests in their CI/CD pipeline
- Adding BlazeMeter’s virutual services - to validate their shipping addresses flow
- Using test data - for testing data that can’t be randomized, like product data, credit card numbers, addresses, and more. Test data will be mainly used to safeguard consumer privacy.
- To dive deeper into their approach, watch the Webinar - Deckers Testing Strategy and learn how Deckers successfully scales testing with BlazeMeter.
Watch the entire webinar here.