For any organization that’s planning a new system implementation or upgrade, load tests are vital. These tests validate whether or not a system can handle the expected future load required to support the company’s evolving business needs. While there are many resources that detail how to perform load tests,1 we’ve found that certain aspects of load testing are too often overlooked, causing project delays, uncertain results, and increased costs.
Most people think that the most important criteria for a successful load test project is whether targeted key performance indicators (KPIs) are met. But another crucial criterion is how well you handle the series of system and coding optimization recommendations that the load test yields — a factor which can also influence how well you meet your KPIs. If properly understood and implemented, these recommendations are powerful instruments to ensure solid, long-term performance within the production system.
So how can you make sure that your company meets its target KPIs during load testing — and, more importantly, once your system is in production? We recommend a two-pronged strategy: Properly prepare your test system data and conduct extensive unit or single user tests before you even begin your full load tests, and understand how to sustainably handle the performance-optimizing recommendations that your load test will generate.
To help illustrate the benefits of this dual approach, we’ll follow the example of one utilities company, Electricité de France (EDF), that has executed extremely successful and productive load tests as a result of following this approach (see sidebar). Throughout this article, you’ll find tips a nd best practices from EDF to help others follow its lead.
Set Up Your Load Tests for Success: Planning and Preparation Tips
To get the best results from your load test, planning and preparation are key. To start, you’ll need to build a team of business owners, SAP functional experts, SAP technology experts, and performance experts. This team will then be responsible for laying the groundwork for the load tests by, for example:
- Determining the scope of the tests
- Determining which business processes need testing
- Determining the timeline of the project, the KPIs, and whether additional experts will be needed
- Managing interdependence with the go-live and functional development cycle
- Identifying the type of data to load into the test system
- Setting up unit testing or single user testing
What follows are key considerations for each phase.
Increase Testing Efficiency Through Progressive Scoping
One precondition for successful load testing is defining what you want to test. To do this, you must first determine and set the scope of the test. This involves:
- Inventorying available resources — including hardware, software, teams, and support
- Defining test objectives and monitoring indicators
- Assessing what is required to achieve your objectives
- Documenting the decision process for determining which processes should be tested
In EDF’s case, the team needed to test many different aspects of performance and scalability to ensure that the system could handle the final volume targets. That’s why EDF decided to run a series of load tests, each with a different scope.
- The goal of the first load test (conducted in 2009), which represented 10 million business partners, was to identify potential application bottlenecks for the target volume. This test focused on the company’s most critical business processes and tested the majority of custom-developed coding. In addition, the tests helped to analyze the layout of the architecture to see if it could serve as a basis for the target architecture.
- The goal of the second load test (to be conducted this year), which represents 20 million business partners, is to check the infrastructure’s efficiency — this includes testing how it interacts with SAP and non-SAP systems, the height of its availability, and its fail-over limit, for example. EDF will also test a few functional modifications — including implemented recommendations from the previous test phase — to analyze their performance impact.
- The goal of the third load test (to be conducted in 2011), which represents 28 million business partners, is to verify whether the production system will be able to handle the final target load.
When dealing with such large volumes of users or with such a large-scale implementation, we recommend splitting your load tests in such a way that you can more effectively address each set of challenges. This approach allows you to progressively implement improvement recommendations and then validate them in the next round of tests.
Run Your Tests on Representative Data
Because the goal of load tests is to provide solid recommendations for how to improve and optimize the production system, the load test system itself should reflect the data layout of the production system as closely as possible. In addition, matching the testing system and the production system will improve your precision when sizing the production system. To determine what kind of data needs to be loaded into your system, the project team will need to work closely with the functional experts, who will contribute valuable insight into the kind of data that the production system will contain.
EDF took great efforts to load real, representative master data and transactional data, and then “age” this data in the performance test system, so that its database would be similar in all aspects to the future production system.
Focus on Testing Critical Business Processes
Another way to limit the complexity of your load tests is to focus only on critical business processes. In some cases, these processes will be clear — a bank would likely want to focus on account balancing calculation jobs, for example. But in other cases, determining these processes might require further analysis.
For instance, dominating EDF’s business load are multiple batch processes that run at night, some of which can be run in parallel and some of which depend on the completion of a previous job — that is, they’re on the “critical path.” To determine exactly which processes are on the critical path, EDF analyzed its current production system to find the execution time of the processes in relation to the amount of data processed and the parallelization parameters (the number of parallel jobs or the time intervals, for example). This analysis helped them identify which processes needed special attention at the load testing stage, as well as which processes would need special attention in the new system.
Run Unit or Single User Tests Before Load Testing
Once you have identified which processes you want to test, prepared the test system, and set up the scripts, you can begin the testing phase. To improve the results of your load tests, SAP recommends running initial tests, either as unit tests — that is, one process step at a time — or as single user tests, in which one user or click stream is monitored for performance KPIs prior to the load tests. Why? Well, for your load test to produce meaningful optimization suggestions, you’ll need to re-run your test cases several times to identify the root cause of the performance problem, and then validate the impact of the suggested solution.
These initial tests reveal areas of improvement, many of which — such as minor changes to system settings, customizations, or coding — can be put into practice quickly. Other recommendations may reveal functionality-related problems, architectural issues, or severe changes to the coding, which the company can address and then validate during load testing.
Let’s revisit our EDF example. Before EDF started its “stage one” load tests, the individual process steps were tested as separate unit tests. The team was then able to analyze and o ptimize the performance of each process.
Some of the improvements that a unit test identifies might take weeks or even months to address, so their benefit will not be immediately visible during the load test.
Then, using the data from these unit tests, EDF could determine the optimal system settings and the precise combination of parallelization parameters — including number of jobs, processing interval, and packet size — to use in the load tests to render the highest amount of throughput for each process (see Figure 1).
||Finding the optimal number of parallel processes to achieve maximum throughput
Thanks to unit testing, EDF identified several bottlenecks before even reaching the load testing phase and was able to optimize its coding, thus cutting the batch night process duration in half (from 30 hours down to only 13). EDF reduced the duration of the batch night process even further after the load testing phase (down to 9 hours), a feat that may not have been possible without the changes made during the unit testing phase.
Use Your Load Test Results Wisely: Handling Recommendations
After your load test, the experts who ran the tests, monitored the system, and analyzed the processes will generate a list of recommendations to improve the system and optimize performance. A very simple recommendation could be to increase the table buffer by 500MB, a parameter modification that can be done on the fly and will only impact available memory. More complex recommendations are those that affect custom coding or modifications to the SAP standard — these require further testing once implemented.
To best manage these recommendations, you’ll need to establish an effective procedure for evaluating the impact of each recommendation, as well as its potential requirements for further coding and test efforts. This procedure must include methods for qualifying and prioritizing the recommendations, as well as a mechanism to integrate resulting improvements into the production rollout.
Prioritize Your Recommendations
Before you can implement the recommendations from the performance test team, you’ll need to qualify and prioritize them. An easy way to do this is to set up a matrix (see Figure 2). This matrix would help you easily see how your recommendations are ranked in terms of ease of implementation, pervasiveness, functional impact, additional research required, and so on.
Problem Description and Impact
Runtime of the job too long.
Impact of data storage: very high.
Error message number: 123456
| : )
Modify standard coding — responsibility of partner A.
Evaluate impacts of tests planned for calendar week X:
- Runtime (-10%)
- Error log files written (-90%)
Parallelization not possible.
Impact on business need: high.
Error message number: 654321
| : (
SAP suggests a custom development.
Business to decide if they want it.
Contact person: Jane Doe
| Within two weeks
||Example of a decision and follow-up matrix
At EDF, the different experts (business application, system, storage, database, and SAP application experts) sat in the same room so that everyone was always on the same page. EDF also appointed an independent test coordinator, whose job was to mitigate a common agreement on the applicability of the recommendations issued by the different parties involved. This coordinator was empowered to decide how critical recommendations should be dealt with, for example, by demanding additional tests and validations.
Consolidate and Integrate Recommendations
Some of the recommendations will require additional functional development work before they can be implemented. The performance test team, therefore, must work closely with the application development team to clearly communicate what the recommendations entail and to test any completed development work. To facilitate this and stay organized, the performance test team should categorize the recommendations — as functional or performance-related, for example.
A close relationship with the application development team can also help if you come across a problem and cannot easily find a solution. In this case, you’ll need to run proof-of-concept tests to help determine viable recommendations.
The true objective of any load test is to ensure that your production system will run smoothly at go-live. As demonstrated by EDF’s successful load testing strategy, the keys to reaching this objective are twofold: Rigorously prepare for load tests by setting up truly representative test scenarios on a production-like test system, and take great care to ensure that the recommendations obtained during the tests are not only of high quality, but are sensibly addressed.
To learn more about SAP’s services, which can help guide your company through successful load tests, visit service.sap.com/tmc.
Susanne Janssen (firstname.lastname@example.org) has been a member of the SAP Performance and Scalability team at SAP AG since 1998. She specializes in performance methodologies and procedures, as well as hardware capacity planning. In this role, she spent time at EDF during the first stage of its load test series.
Fayçal Hadjiat (email@example.com) is project manager at EDF, where he’s worked for 10 years. He leads mass-market performance initiatives to ensure that EDF will meet its goal of accommodating 28 million business partners by 2013. He is currently in the planning phase for this project’s next major test stage, the technical infrastructure tests.
1 See www.service.sap.com/performance and www.service.sap.com/tmc for more information. [back]