GRC
HR
SCM
CRM
BI


Article

 

The Record-Breaking Performance Benchmarks of 2011

by Tobias Kutning and Bernd F. Lober | SAPinsider

January 1, 2012

Originally introduced to boost quality assurance, SAP standard application benchmarks are also used to test and verify the scalability, concurrency, power efficiency, and multi-user behavior of system software components, relational database management systems (RDBMSs), and business applications. This article takes a closer look at these benchmarks and explores some of the record-breaking results achieved this year.
 

For those companies looking to make major technology investments, SAP standard application benchmarks are a huge boon. Originally introduced to boost quality assurance, these benchmarks are also used to test and verify the scalability, concurrency, power efficiency, and multi-user behavior of system software components, relational database management systems (RDBMSs), and business applications.

Customers can use the results of these tests to find the appropriate hardware configurations for their IT solutions and determine if certain hardware technologies will be able to handle their estimated workloads. Another benefit of the benchmarks is the competitive spirit they create, which drives technology providers to work harder to beat their competition. As a result, customers are seeing more and more improvements in the products being benchmarked. 

About 50 SAP standard application benchmarks are performed on average each year. In 2011, SAP saw major investments from its technology partners in the area of benchmarking — and several companies overtook previous SAP benchmarking records.1 Also in 2011, a partner recorded the first results in SAP’s newest power benchmarks category (see sidebar).

About SAP’s Benchmarking Program

SAP, working together with its hardware and technology partners, developed the SAP standard application benchmarks to test the hardware and database performance of partners’ applications and components. The benchmarking procedure is standardized, defined, and monitored by the SAP Benchmark Council (made up of representatives from SAP as well as hardware and technology partners). This process standardization assures customers of the benchmarking tests’ rigors and fairness. Acting as an auditing organization, SAP also checks the benchmarking results for completeness and correctness before officially certifying and publishing them on www.sap.com/benchmark.

The most popular and well-known of these benchmarks is the sales and distribution (SD) benchmark, which has been around since 1993 and is designed to simulate a core business process (selling, shipping, and delivery). The SD benchmark plays an important role in the SAP system sizing process.

SAP has a range of benchmarks that cover a variety of other topics as well (see Figure 1).

While two of the year’s four record-breaking benchmarks fall under the SD benchmark heading, the other two are from some of these lesser-known categories, demonstrating that these non-SD benchmarks are also quite active.

 

Type of SAP standard application benchmark

        

Includes

     

SAP power benchmarks

 

SAP system power benchmark

SAP server power benchmark

     

SAP NetWeaver Business Intelligence benchmarks

 

Business information warehouse (BW) benchmark*

Data mart (BI-D) benchmark

Mixed load (BI-MXL) benchmark

     

SAP ERP benchmarks

 

Assemble-to-order (ATO) benchmark

Cross application time sheet (CATS) benchmark

Financial accounting (FI) benchmark

Human resources — payroll (HR) benchmark

Materials management (MM) benchmark

Production planning (PP) benchmark

Sales and distribution (SD and SD-parallel) benchmark

     

SAP Supply Chain Management benchmarks

 

Warehouse management (WM) benchmark

     

SAP Advanced Planner and Optimizer benchmarks

 

Demand planning (DP) benchmark

Production planning and detailed scheduling (PP/DS) benchmark

Supply network planning (SNP) benchmark

     

SAP for Banking benchmarks

 

Transaction banking (TRBK) benchmark

     

SAP for Utilities benchmarks

 

Customer care and service (ISU/CCS) benchmark

     

Enterprise portal benchmarks

 

Employee self-service portal (EP-ESS) benchmark

People-centric CRM portal (EP-PCC) benchmark

     

SAP Customer Relationship Management benchmarks

 

E-selling benchmark

Interaction center benchmark

     

SAP Product Lifecycle Management benchmarks

 

Project system (PS) benchmark

     

SAP for Retail benchmarks

 

POS inbound benchmark

Store replenishment benchmark

     
   

*Since January 2008, the Business Information Warehouse benchmark has been replaced by the data mart (BI-D) and mixed load (BI-MXL) benchmarks.

Figure 1 Types of SAP standard application benchmarks

Record Breaker #1: ATO Benchmark

The first record-breaking benchmark of 2011 came in September with the assemble-to-order (ATO) benchmark.4 The ATO scenario is characterized by high-volume sales, short production times (ranging from hours to one day), and individual assembly for such products as PCs, pumps, and cars.

The ATO benchmark looks at the integration of process chains across SAP Business Suite components — including financials, logistics, HR, and cross-application components. The KPI of the ATO benchmark is the throughput across the complete process chain, measured in the amount of fully processed assembly orders per hour. During this benchmark, a record value of 206,360 fully processed assembly orders per hour was achieved (see Figure 2), roughly two times more throughput than the previous record from 2003.

Figure 2 ATO benchmark results

Record Breaker #2: The TRBK Benchmark

Later in September 2011, SAP saw another broken record from a transaction banking (TRBK) benchmark.5 The TRBK benchmark is divided into two business scenarios:

  • Day processing, which simulates the daily usage activities — by both users and integrated systems — that generally occur during daytime business hours. Examples of day processing include transactions from external payment transaction systems and real-time account statements.
  • Night processing, which simulates account balancing (a usual nighttime batch activity) by calculating interest and charges for each account across 20 value date days.

The benchmark was performed on a setup of five database servers and 21 application servers. For the day processing scenario, 56,518,000 postings to bank accounts per hour were achieved, and 22,382,000 balanced accounts per hour were achieved for the night processing scenario (see Figure 3). This is triple the results of the previous benchmarking record.

Figure 3 TRBK benchmark results

Record Breakers #3 and #4: SD Benchmarks

The SD benchmark is of high interest to SAP customers because its results are measured in SAPS,6 which can be used to gauge how well a system will meet sizing requirements. The results also allow customers to better judge what category machine they might need. With this popularity, it’s not surprising that in 2011 SAP saw two records posted in the SD benchmark category, one in the plain SD benchmark and one in the SD-parallel benchmark. Several categories within this benchmark type can also claim new records: The most prominent categories are two-tier/three-tier setups, number of processors, and operating system.

In September, a setup that was based on a two-tier Internet configuration and eight Intel processors running Windows was certified.7 The benchmark was able to simulate 25,500 users running at once, beating the previous record of 25,160 users set earlier in 2011 (see Figure 4).8

The other broken record occurred during an SD-parallel benchmark, which consists of the same processes, transactions, and user interaction steps as the SD benchmark. The difference between the benchmarks is in the technical data distribution. As part of a series of four SD-parallel benchmarks, one of SAP’s hardware partners achieved over 1,000,000 SAPS during a 180,000 SD-parallel user benchmark (see Figure 5).9 The series of benchmarks consisted of a 2, 4, 6, and 8-database server setup.10 Each server was equipped with eight Intel Xeon E-8870 processors and 512 gigabytes of RAM.

Figure 4 SD benchmark results
Figure 5 Results of the series of SD-parallel benchmarks

A Look Forward

The record-breaking benchmark results SAP saw last year have set a new standard that other benchmarkers will no doubt strive to outdo. We expect to see even more records broken in 2012.

These results also show that SAP software’s performance and scalability capabilities can match the latest hardware developments. To learn more about SAP benchmarking, or to participate, visit www.sap.com/benchmark.

Tobias Kutning (tobias.kutning@sap.com) joined SAP in 2001 as a database support consultant in SAP Service & Support with a focus on IBM DB2 LUW until 2007. In 2008, he joined SAP IT as a senior DBA. Since 2010, he has worked on the Performance & Scalability Team, responsible for SAP Standard Application Benchmark Product Management.

Bernd F. Lober (b.lober@sap.com) is Director of Product Management, Performance, and Scalability at SAP AG, responsible for benchmarks and sizing. He joined SAP in 1995. Previously, Mr. Lober worked at IBM EMEA and headed the Technical SAP IBM Competence Center in Walldorf. He holds advanced degrees in mathematics and economics from the University of Augsburg, Germany.

1 All data noted in this article is current as of November 2011. [back]

2 Certification number 2011008. [back]

3 For more information on the SAP server power benchmark result, see “How Much Energy Is Your Server Using? And How Much Could an Energy-Efficient One Save You?” by Tobias Kutning in the July-September 2011 issue of SAPinsider. [back]

4 Certification 2011033. [back]

5 Certification 2011035. [back]

6 SAPS stands for SAP Application Performance Standard. This measurement unit provides a performance yardstick; its results are determined by SAP’s de facto industry-standard benchmark, the SAP SD application benchmark. [back]

7 Certification 2011034. [back]

8 Certification 2011021. [back]

9 Certification 2011037. [back]

10 Certification 2011040, certification 2011039, certification 2011038, and certification 2011037, respectively. [back]

 

An email has been sent to:






More from SAPinsider



COMMENTS

Please log in to post a comment.

No comments have been submitted on this article. Be the first to comment!


SAPinsider
FAQ