Benchmarking is critical to understanding how the performance of your technology assets compares to the industry average, so you can make smarter decisions about the system configuration that supports your business-critical solutions. This information empowers you to increase efficiency with a properly configured landscape, explore new options by comparing different platforms, make a solid business case for potential purchases with proof-of-concept scenarios, and ensure the continual improvements to your underlying infrastructure that are necessary to stay competitive.
For decades, SAP has been helping its customers and partners determine the best configuration for their software solutions using the SAP standard application benchmarks, maintained by the SAP benchmark council, which test the performance and scalability of SAP solutions and the hardware on which they run. (To learn more about the origins of the SAP standard application benchmarks and the benchmark council, see the sidebar “What Are the SAP Standard Application Benchmarks?”) For example, let’s say an SAP customer is interested in implementing SAP Business Suite software. To help calculate the hardware requirements for the software, the customer can use the Quick Sizer tool (available at http://service.sap.com/quicksizer), which provides, among other information, an SAPS1 value that customers can map to the SAPS results listed in the certified benchmark results from various partners. Finding the right hardware is made easy with this approach.
System infrastructure needs have changed over time, however. To support the demands of a modern business environment, which requires unprecedented levels of flexibility and responsiveness, hardware continues to become increasingly powerful, and running several SAP systems on the same infrastructure is a common scenario. While benchmarks have traditionally been run on a single SAP system or server, there is clearly a need to support scenarios in which multiple applications are running simultaneously on the same infrastructure.
To meet this need, SAP has created a new type of benchmark — the concurrent benchmark.
Introducing the Concurrent Benchmark
The concurrent benchmark is the most recent addition to the SAP standard application benchmarks. It aims to show the performance and effectiveness of a shared resource — which is typically a physical server, but could also be a storage system, operating system, database, or network component — when running multiple SAP applications simultaneously. Scenarios that were difficult to measure or not possible to measure with other benchmarks — such as server consolidation scenarios and storage system or network equipment vendor scenarios — are now possible with this new benchmark type.
For example, it was difficult to measure server consolidation scenarios with previous benchmarks because you could run just one benchmark against one SAP system at a time on a single hardware component. The concurrent benchmark makes it possible to run several benchmarks against different SAP systems on the same hardware component, enabling server consolidation proofs of concept and performance showcases that demonstrate the benefits of powerful hardware and the ability to reduce the number of servers and the complexity of hardware landscapes. It was also previously not possible for storage system vendors and network equipment vendors to run benchmarks for their solutions because the benchmarks focused only on stressing servers. With support included in the concurrent benchmark for additional types of shared resources, these vendors can now transparently demonstrate the capacity of their solutions by stressing shared storage and network resources.
So what exactly is a concurrent benchmark? Let’s compare it to a single benchmark, which the SAP benchmark council defines as a benchmark that runs against one database or system with one SAP system ID. A concurrent benchmark, on the other hand, is defined as the parallel execution of multiple single benchmarks at the same time. So, one concurrent benchmark consists of several single benchmarks, which can be of the same type (for example, several benchmarks measuring OLTP workload) or of different types (for example, some benchmarks measuring OLTP and some measuring OLAP workloads) — as long as the benchmarks are dialog benchmarks, which are benchmarks that simulate user interaction steps. The technology partner running the benchmark decides which benchmark types to choose and how to mix them to simulate typical customer configuration scenarios.
Each single benchmark that constitutes a concurrent benchmark must comply with all the certification rules specified by the SAP benchmark council, which validate that the benchmark complies with certain configurations, parameter settings, and constraints. These specifications include maximum response times, error-free runs, and allowed tunings, to name just a few. In addition to these, there are two new rules specific to the concurrent benchmark with which each single benchmark within a concurrent benchmark must comply — a common high load phase and a named shared resource.
Certification Rules Specific to the Concurrent Benchmark
All single benchmarks of a concurrent benchmark must have a common high load phase, which is the time frame during which all benchmark users are active on the system and running the scenario while the performance measure is being taken. For dialog benchmarks — benchmarks that simulate user interaction steps — this means that the high load phases of all single benchmarks within the concurrent benchmark must overlap by at least 15 minutes.2
Let’s look at an example. Suppose a vendor wants to run a concurrent benchmark that includes three single sales and distribution (SD) benchmarks to measure a typical customer scenario. Figure 1 shows the common high load interval required for the concurrent benchmark run. First, all users configured for each single benchmark log on to their respective systems. Once all users are logged on, the high load phase starts, during which all configured users are active in the system. As you can see in the example, the high load phases do not start synchronously. So the trick is to configure the single benchmarks so that the overlap of the individual high load phases is at least 15 minutes, which technology partners can modify using the configuration options available in the benchmark tool scripts.
The second certification rule specific to the concurrent benchmark is that the technology partner must name a shared resource — an operating system, server, storage system, database, or network, for example — that will be listed on the published benchmark certificate. The shared resource that is chosen determines the infrastructure setup that the benchmark will measure for feasibility and performance. For instance, if a network is chosen as the shared resource, the number of concurrent benchmarks and used servers are then chosen according to the specific network infrastructure, and the network is the component that will be fully loaded during the high load phase.
A Look at the First Concurrent Benchmark Results
Cisco was the first vendor to publish results for the concurrent benchmark, which was performed in December 2014 and certified by SAP in January 2015 (see Figure 2).3 Cisco ran two SD benchmarks concurrently to show how organizations can run multiple SAP applications with very little overhead on modern servers using virtualization technologies.
Previously, Cisco had published a single SD benchmark result (see Figure 3), which was performed and certified in November 2014, for the server that was used to run the concurrent benchmark.4 Let’s compare the two benchmark certificates to gain a better understanding of the main differences between concurrent benchmarks and other types of benchmarks.
As you can see, both certificates list the number of benchmark users, the average dialog response time, the SAPS unit of measure, the average database request time, CPU utilization, operating system and server details, RDBMS, software solution information, and configuration details. The main differences are that the concurrent benchmark certificate includes:
- The number of concurrent benchmarks (the number of single benchmarks included in the concurrent benchmark, which is two in this case)
- The shared resource that was defined, which is a server in this certificate
- A results table format to display the standard benchmark KPIs for each single benchmark of the concurrent benchmark, including identifier information for the single benchmarks
- An additional paragraph describing the details of the configuration, such as the landscape setup or any background information on the benchmark scenario
Using the Concurrent Benchmark
What does it look like to use the concurrent benchmark to properly configure your landscape to support an SAP application? Let’s take a look at an example based on Cisco’s concurrent benchmark.
Cisco’s scenario uses a server as the shared resource, putting the emphasis of the benchmark on the evaluation of the server’s capabilites. From Cisco’s benchmark, a customer can gain insight into the (non-)impact of the two benchmarks on two systems running simultaneously on one server, and also use the results as a proof of concept that having several SAP systems on one server is feasible. It is also possible to get an idea of virtualization overhead — and to take that understanding a step further by comparing the user and throughput numbers of both the concurrent and single benchmark results from Cisco.
As you can see, using the information in a concurrent benchmark certification, customers can get an idea of the processing power of shared resources — and not just modern servers, but other components, such as storage systems and networks — as well as gather information on additional factors, such as the effects of virtualization, depending on the scenario the technology partner chooses. Then, based on this information, customers can determine the right hardware infrastructure to optimally support their software implementation.
Getting Your Applications on Solid Ground
Knowing where you stand is the key to putting your organization in the best position to succeed, and the same rule of thumb applies to your technology infrastructure. Benchmarks are valuable tools for assessing your infrastructure to identify opportunities to unlock value, and SAP’s new concurrent benchmark brings these tools into modern SAP landscapes by allowing the benchmarking of multiple SAP applications running simultaneously on shared resources.
The beginning of 2015 brought with it the first published results for the new concurrent benchmark, delivered by Cisco, enabling SAP customers to start evaluating their infrastructure options for supporting business-critical applications. SAP expects to publish additional concurrent benchmarks in the near future to continue helping customers get the most out of their SAP solutions.
Learn more about SAP standard application benchmarks at www.sap.com/benchmark.
1 The SAP Application Performance Standard (SAPS) is a hardware-independent unit that describes the performance of a system. [back]
2 Note that batch scenarios are currently not in the scope of the concurrent benchmark definition. [back]
3 Cisco’s concurrent benchmark certification (number 2015002) is available at http://global.sap.com/campaigns/benchmark/assets/Cert15002.pdf. [back]
4 Cisco’s single SD benchmark certification (number 2014045) is available at http://download.sap.com/download.epd?context=40E2D9D5E00EEF7CAB5B77150020882CC9D2E4188791D9F00BEE76AB1F13DBC3. [back]