GRC
HR
SCM
CRM
BI


Article

 

Benchmarking Modern SAP System Landscapes

How SAP’s New Concurrent Benchmark Ensures Your Applications Are on Solid Ground

by Tobias Kutning | SAPinsider, Volume 16, Issue 3

July 1, 2015

Learn how to better support your SAP applications and increase the efficiency of your company’s technological landscape with SAP’s new concurrent benchmark. Concurrent benchmarks show the performance and effectiveness of a shared resource, such as a physical server, storage system, or database, when running multiple SAP applications simultaneously.

 

Benchmarking is critical to understanding how the performance of your technology assets compares to the industry average, so you can make smarter decisions about the system configuration that supports your business-critical solutions. This information empowers you to increase efficiency with a properly configured landscape, explore new options by comparing different platforms, make a solid business case for potential purchases with proof-of-concept scenarios, and ensure the continual improvements to your underlying infrastructure that are necessary to stay competitive.
 
For decades, SAP has been helping its customers and partners determine the best configuration for their software solutions using the SAP standard application benchmarks, maintained by the SAP benchmark council, which test the performance and scalability of SAP solutions and the hardware on which they run. (To learn more about the origins of the SAP standard application benchmarks and the benchmark council, see the sidebar “What Are the SAP Standard Application Benchmarks?”) For example, let’s say an SAP customer is interested in implementing SAP Business Suite software. To help calculate the hardware requirements for the software, the customer can use the Quick Sizer tool (available at http://service.sap.com/quicksizer), which provides, among other information, an SAPS1 value that customers can map to the SAPS results listed in the certified benchmark results from various partners. Finding the right hardware is made easy with this approach.
 
System infrastructure needs have changed over time, however. To support the demands of a modern business environment, which requires unprecedented levels of flexibility and responsiveness, hardware continues to become increasingly powerful, and running several SAP systems on the same infrastructure is a common scenario. While benchmarks have traditionally been run on a single SAP system or server, there is clearly a need to support scenarios in which multiple applications are running simultaneously on the same infrastructure.
 
To meet this need, SAP has created a new type of benchmark — the concurrent benchmark.

What Are the SAP Standard Application Benchmarks?

The SAP standard application benchmarks are developed by SAP and its technology partners, and are monitored and maintained by the SAP benchmark council, which was established in 1995 and consists of representatives from SAP and partners such as hardware and database vendors.* In addition to providing quality assurance, these benchmarks are used to verify the scalability, power efficiency, and multi-user behavior, for instance, of system software components, relational database management systems (RDBMS), and business applications.
 
The SAP benchmark council helps define the content of the benchmarks, and establishes the rules for the testing procedures used by hardware and database vendors during a benchmarking run. During the run, a substantial load is placed on the hardware, software, and database system to measure performance, including CPU utilization, memory consumption, network load, functional errors, and system availability. The vendor then submits the results of the benchmark run to SAP for certification. There are a number of rules and guidelines — such as specifications for maximum response times, error-free runs, and allowed tunings — that must be followed to obtain a certification. Once the benchmark is certified by SAP on behalf of the SAP benchmark council, it is published at www.sap.com/benchmark and available for customers to use to properly size their systems for SAP applications.**
 
SAP maintains an array of benchmarks to meet customer and partner needs, including benchmarks for financial accounting, human resources, sales and distribution, materials management, business intelligence, customer relationship management, supply chain management, banking, retail, and many others, as well as benchmarks for cloud environments and power consumption. The latest addition is the concurrent benchmark, which measures the performance of shared resources by enabling several benchmarks to run concurrently on a common server, database, or other resource.
 
While the SAP benchmark council first began discussing the need for a concurrent benchmark back in 2003 — to address large servers using server partitioning and virtualization, which led to the introduction of virtualized benchmarks — it took some time to achieve a common understanding of what it should look like. In 2012, with hardware becoming increasingly powerful and the need for a benchmark to address this growing stronger, a workgroup was created to finalize the new benchmark type. The workgroup delivered the first draft for the concurrent benchmarking rules in September 2012, and the rules were finalized and available for use by technology partners in April 2013.

 

* For additional background information on the SAP standard application benchmarks, see my article “The Benchmarks of Success” in the January-March 2013 issue of SAPinsider.

** More details about the benchmark publication process are available at http://global.sap.com/solutions/benchmark/PDF/Benchm_Publ_Proc_3_7.pdf.

Introducing the Concurrent Benchmark

The concurrent benchmark is the most recent addition to the SAP standard application benchmarks. It aims to show the performance and effectiveness of a shared resource — which is typically a physical server, but could also be a storage system, operating system, database, or network component — when running multiple SAP applications simultaneously. Scenarios that were difficult to measure or not possible to measure with other benchmarks — such as server consolidation scenarios and storage system or network equipment vendor scenarios — are now possible with this new benchmark type.
 
For example, it was difficult to measure server consolidation scenarios with previous benchmarks because you could run just one benchmark against one SAP system at a time on a single hardware component. The concurrent benchmark makes it possible to run several benchmarks against different SAP systems on the same hardware component, enabling server consolidation proofs of concept and performance showcases that demonstrate the benefits of powerful hardware and the ability to reduce the number of servers and the complexity of hardware landscapes. It was also previously not possible for storage system vendors and network equipment vendors to run benchmarks for their solutions because the benchmarks focused only on stressing servers. With support included in the concurrent benchmark for additional types of shared resources, these vendors can now transparently demonstrate the capacity of their solutions by stressing shared storage and network resources.
 
So what exactly is a concurrent benchmark? Let’s compare it to a single benchmark, which the SAP benchmark council defines as a benchmark that runs against one database or system with one SAP system ID. A concurrent benchmark, on the other hand, is defined as the parallel execution of multiple single benchmarks at the same time. So, one concurrent benchmark consists of several single benchmarks, which can be of the same type (for example, several benchmarks measuring OLTP workload) or of different types (for example, some benchmarks measuring OLTP and some measuring OLAP workloads) — as long as the benchmarks are dialog benchmarks, which are benchmarks that simulate user interaction steps. The technology partner running the benchmark decides which benchmark types to choose and how to mix them to simulate typical customer configuration scenarios.
 
Each single benchmark that constitutes a concurrent benchmark must comply with all the certification rules specified by the SAP benchmark council, which validate that the benchmark complies with certain configurations, parameter settings, and constraints. These specifications include maximum response times, error-free runs, and allowed tunings, to name just a few. In addition to these, there are two new rules specific to the concurrent benchmark with which each single benchmark within a concurrent benchmark must comply — a common high load phase and a named shared resource.

Certification Rules Specific to the Concurrent Benchmark

All single benchmarks of a concurrent benchmark must have a common high load phase, which is the time frame during which all benchmark users are active on the system and running the scenario while the performance measure is being taken. For dialog benchmarks — benchmarks that simulate user interaction steps — this means that the high load phases of all single benchmarks within the concurrent benchmark must overlap by at least 15 minutes.2 
 
Let’s look at an example. Suppose a vendor wants to run a concurrent benchmark that includes three single sales and distribution (SD) benchmarks to measure a typical customer scenario. Figure 1 shows the common high load interval required for the concurrent benchmark run. First, all users configured for each single benchmark log on to their respective systems. Once all users are logged on, the high load phase starts, during which all configured users are active in the system. As you can see in the example, the high load phases do not start synchronously. So the trick is to configure the single benchmarks so that the overlap of the individual high load phases is at least 15 minutes, which technology partners can modify using the configuration options available in the benchmark tool scripts.
 
The second certification rule specific to the concurrent benchmark is that the technology partner must name a shared resource — an operating system, server, storage system, database, or network, for example — that will be listed on the published benchmark certificate. The shared resource that is chosen determines the infrastructure setup that the benchmark will measure for feasibility and performance. For instance, if a network is chosen as the shared resource, the number of concurrent benchmarks and used servers are then chosen according to the specific network infrastructure, and the network is the component that will be fully loaded during the high load phase.

Figure 1 — The high load phases of the single benchmarks within the concurrent benchmark must overlap by at least 15 minutes

 

A Look at the First Concurrent Benchmark Results

Cisco was the first vendor to publish results for the concurrent benchmark, which was performed in December 2014 and certified by SAP in January 2015 (see Figure 2).3 Cisco ran two SD benchmarks concurrently to show how organizations can run multiple SAP applications with very little overhead on modern servers using virtualization technologies. 

Cisco’s concurrent benchmark

Figure 2 — Cisco’s concurrent benchmark

 

Previously, Cisco had published a single SD benchmark result (see Figure 3), which was performed and certified in November 2014, for the server that was used to run the concurrent benchmark.4 Let’s compare the two benchmark certificates to gain a better understanding of the main differences between concurrent benchmarks and other types of benchmarks.

Cisco’s SD single

Figure 3 — Cisco’s SD single benchmark

 

As you can see, both certificates list the number of benchmark users, the average dialog response time, the SAPS unit of measure, the average database request time, CPU utilization, operating system and server details, RDBMS, software solution information, and configuration details. The main differences are that the concurrent benchmark certificate includes:

  • The number of concurrent benchmarks (the number of single benchmarks included in the concurrent benchmark, which is two in this case)
  • The shared resource that was defined, which is a server in this certificate
  • A results table format to display the standard benchmark KPIs for each single benchmark of the concurrent benchmark, including identifier information for the single benchmarks
  • An additional paragraph describing the details of the configuration, such as the landscape setup or any background information on the benchmark scenario 
Using the Concurrent Benchmark

What does it look like to use the concurrent benchmark to properly configure your landscape to support an SAP application? Let’s take a look at an example based on Cisco’s concurrent benchmark.
 
Cisco’s scenario uses a server as the shared resource, putting the emphasis of the benchmark on the evaluation of the server’s capabilites. From Cisco’s benchmark, a customer can gain insight into the (non-)impact of the two benchmarks on two systems running simultaneously on one server, and also use the results as a proof of concept that having several SAP systems on one server is feasible. It is also possible to get an idea of virtualization overhead — and to take that understanding a step further by comparing the user and throughput numbers of both the concurrent and single benchmark results from Cisco.
 
As you can see, using the information in a concurrent benchmark certification, customers can get an idea of the processing power of shared resources — and not just modern servers, but other components, such as storage systems and networks — as well as gather information on additional factors, such as the effects of virtualization, depending on the scenario the technology partner chooses. Then, based on this information, customers can determine the right hardware infrastructure to optimally support their software implementation.

Getting Your Applications on Solid Ground

Knowing where you stand is the key to putting your organization in the best position to succeed, and the same rule of thumb applies to your technology infrastructure. Benchmarks are valuable tools for assessing your infrastructure to identify opportunities to unlock value, and SAP’s new concurrent benchmark brings these tools into modern SAP landscapes by allowing the benchmarking of multiple SAP applications running simultaneously on shared resources.
 
The beginning of 2015 brought with it the first published results for the new concurrent benchmark, delivered by Cisco, enabling SAP customers to start evaluating their infrastructure options for supporting business-critical applications. SAP expects to publish additional concurrent benchmarks in the near future to continue helping customers get the most out of their SAP solutions.
 
Learn more about SAP standard application benchmarks at www.sap.com/benchmark.

 

 

1 The SAP Application Performance Standard (SAPS) is a hardware-independent unit that describes the performance of a system. [back]

2 Note that batch scenarios are currently not in the scope of the concurrent benchmark definition. [back]

3 Cisco’s concurrent benchmark certification (number 2015002) is available at http://global.sap.com/campaigns/benchmark/assets/Cert15002.pdf. [back]

4 Cisco’s single SD benchmark certification (number 2014045) is available at http://download.sap.com/download.epd?context=40E2D9D5E00EEF7CAB5B77150020882CC9D2E4188791D9F00BEE76AB1F13DBC3. [back]

 

An email has been sent to:





 

Tobias Kutning
Tobias Kutning

Tobias Kutning (tobias.kutning@sap.com) joined SAP in 2001. Until 2007, he worked as a database support consultant in SAP Service and Support with a focus on IBM DB2 LUW. In 2008, he joined SAP IT as team lead of the database administration team. Since 2010, he has worked on the Performance, Architecture, and Scalability team, responsible for product management for SAP standard application benchmarks.



More from SAPinsider



COMMENTS

Please log in to post a comment.

SAPinsider
FAQ