Kraft Foods Group is one of the largest consumer packaged food and beverage companies in North America — where at least one of its brands can be found in nearly every household. Complex supply and distribution networks support the production and distribution of two billion-dollar brands — Kraft and Oscar Mayer — and several million-dollar brands, and disruption to supply chain or distribution capabilities could adversely affect this market penetration. Because Kraft Foods increasingly relies on its IT infrastructure to manage business operations and comply with various regulatory, legal, and tax requirements, it can’t afford shutdowns or other failures during software upgrades or enhancements.
The importance of keeping defects out of production took on added importance for Kraft about five years ago when it began a major SAP implementation for its North America operations. Its manual regression testing processes struggled to keep up with the volume of regression scripts needed to be executed for all the new releases going into production.
By implementing some test management software by HP, Kraft was able to automate 50% of its regression test script inventory, focusing primarily on regression testing for its SAP applications by partnering with Infosys Limited. Because of the success with these tools, Kraft decided to implement a second application to increase its automated inventory with a goal of reducing the number of defects that were making their way into production. Additionally, Kraft decided to automate the highly integrated non-SAP applications. This decision came about in September 2012 — shortly before Kraft split off its confections and snack business.
"Once you automate and execute a
few of their business processes and
everyone sees how simple it is, you
get that buy-in and they come to
your door wanting more."
- Lori Etelamaki, Associate Director of Quality and Testing, Kraft Foods
It’s rare for a company with a history dating back a century to adopt a startup mentality, but with the business split, the rebranded Kraft started thinking like a smaller company — one that was keenly focused on avoiding any major production errors. Now, with a single instance of SAP ERP, a production error in a new release could potentially affect the entire business. Also, the business consolidation meant the company could operate with a leaner IT department. And by significantly automating the regression test inventory, this leaner IT organization could manage more frequent regression cycles to support the system and configuration changes.
Lori Etelamaki, Kraft’s Associate Director of Quality and Testing, had two goals for the organization’s new automation tool: automate 80% of its regression inventory, and do so without disrupting the business. She says, "Minimizing the impact to the project teams working on implementing the applications is important, because if they are busy testing, then they are not focused on innovation, financials, or providing business value."
The overriding goal, of course, is minimizing disruption to the customer — keeping Kraft brands in refrigerators and pantries. "There are certain things you can’t afford to mess up," Etelamaki says. "Quality always has to be there to make sure we’re not stopping production lines or trucks aren’t waiting for inventory."
Worksoft Fills the Gap
Knowing that the volume of regression test scripts would increase, one requirement for the new automation tool was that it be able to maintain and manage a heavy volume of processes. Another was ease of use. Partly
because of these factors, Kraft decided to implement Worksoft Certify, the flagship product from SAP partner Worksoft that promises end-to-end business process validation across the entire SAP ecosystem.
"We previously had some challenges around testing SAP NetWeaver Portal and SAP GUI applications, as well as some non-SAP applications like web apps connecting to our servers," Etelamaki says. "We believed that Worksoft would work well on an end-to-end scale."
Because testers don’t need programming skills to be trained to use Worksoft Certify, adding the new application was easy, especially because Kraft already had an automated testing framework in place that testers were familiar with. To further help ease the transition, in conjunction with automating its initial regression inventory, Kraft established a testing center of excellence (TCoE), using Infosys Limited as its partner for testing services. This TCoE provided test planning, scripting, execution, and automation across various test cycles.
With the addition of the Worksoft application, this team of testers was then responsible for building up the automation portfolio for all critical SAP and SAP-integrated applications. In the manual regression testing environment, regression testing was the domain of the particular project or business team — a labor-intensive approach. The TCoE allowed the flexibility of executing the automated regression inventory for major releases, minor releases, and monthly break/fix releases, significantly increasing the test coverage and reducing defects introduced into production.
"One benefit of a TCoE was a continuity of resources," explains Etelamaki. "We didn’t want to have situations where a project team would implement an application, and then when a new update would come along, those team members were no longer available. A centralized team for executing the automation process knowing the scenarios and exactly what needs to be done is the glue between all the multiple releases."
Increasing Automation Inventory
In building its automation inventory, Etelamaki’s project team channeled its focus on identifying and executing automated regression testing for core end-to-end business processes. The team looked at integration scripts in use for prior releases as a starting point, and built its inventory based on whether the scripts were for core processes. Every time an application went into production, the key question was whether new functionality was added and, if so, whether that functionality touches a core process or introduces a new business scenario. The new Worksoft application needed to extend out to Kraft’s roughly 80 non-SAP integrated applications to reach the stated goal of automating 80% of regression testing.
Another decision the team made was to target automation only for stabilized applications, to help build a run and re-use inventory, and then proactively update those test cases to reflect upcoming release changes. This is where the new application’s ease of use came into play. "I could have my Infosys functional testers learn Worksoft, update the test, and put it back into the queue," Etelamaki says. "That was key for efficiency. And because of that simplicity, those testers would often approach me with new ideas for how to use the automation application in other areas."
Changing the Business Mindset
The approach for automated testing was to start with a key functional area, and automate enough scenarios to significantly reduce the impact on the business. Kraft started with a very critical and highly integrated order-to-cash process. "The business teams would say that it would be quicker for them to just do it themselves," says Etelamaki. "It’s difficult to explain that it would end up saving them several days of work every time there’s a new release, and we need a little time up front to ensure the process is appropriately documented for testing — so we took a "show-me" approach. We demonstrated the execution of automated scripts took a fraction of the time that the manual execution took. Also, the business was only required to validate the exceptions."
After automation occurs, however, the inventory queue grows quickly. "Once you automate and execute a few of their business processes and they see how simple it is, you get that buy-in and everyone comes to your door wanting more," says Etelamaki. This user acceptance helped make increasing the automation inventory from 50% to 80% a relatively straightforward endeavor, especially because Etelamaki’s team was able to show rather quickly that the upfront effort, which was initially met with resistance by business teams, was actually paying dividends in higher quality and shortened timelines.
This success was reflected in break fixes and monthly production cycles. "Normally, teams don’t have a large window for testing, and the testing time for monthly cycles is very short," says Etelamaki. "Now, our teams are a lot more confident knowing that it will only take us three days to run these tests before go-live. And because the testing does find defects, we’ve been able to prove that it’s worth our time and effort."
Mobile, the Next Wave
Because Kraft reached its goal of automating more than 80% of its core business process tests with little disruption to business teams, the organization can now plan its release cycles differently because testing cycle times have been reduced. "All the manual, labor-intensive work that used to take weeks to perform is now done in days and, in some cases, even hours," Etelamaki says. "A previous four-week regression cycle now can be done in half the time, with 80% completed in the first three days. So not only do we have time to identify and fix errors, but since we have several releases a year, we have a lot more available time to devote instead to blueprinting, development, or other phases. We can actually have more releases and innovate faster."
With the success of Worksoft Certify, Kraft started implementing Worksoft Certify for Mobile to its testing suite in November. "That is the next big frontier," says Etelamaki.