When implementing a software solution, preparation will always be a cornerstone of success. You need to know the business expectations for the solution; you need an implementation roadmap to determine exactly how you will deploy the solution; and you need a plan to promote user adoption. Perhaps most importantly, you also need to make sure that you have a scalable hardware configuration that can support that software solution and all of the operations that you will use it for. How can you do this?
Let’s say that your company is preparing to implement an SAP Business Suite solution. To ensure that the solution will run optimally, the company wants to make sure that the hardware that supports the solution is properly sized. Using SAP’s Quick Sizer tool (available at https://websmp204.sap-ag.de/quicksizer), the company can determine the size and capabilities of the hardware it will need.
Using the output from Quick Sizer, your company can then view the results from the SAP standard application benchmarks so you can find the precise hardware that will fit your needs. Better still, because of the hard work of the SAP Benchmark Council, you can rest assured that these benchmarks are the result of rigorous, reliable tests, so you’ll have a good idea of the kind of performance you can expect from your hardware going forward.
For 20 years now, SAP benchmarks have been supporting customers in their bids to find the right hardware and software configurations to support their SAP landscapes (see sidebar). Throughout this time, the SAP Benchmark Council has worked to make the SAP standard application benchmarks among the most credible and influential application benchmarks in the IT industry. Let’s take a closer look at the history of the SAP Benchmark Council, how it has supported SAP customers in the past, and how it plans to continue to do so in the future.
The SAP Benchmark Council: A Focus on SAP Customer Needs
The first meeting of the group that would later become the SAP Benchmark Council took place on March 16, 1993, when SAP came together with technology and hardware companies to discuss how to ensure performance and scalability when running SAP R/3. The participants who joined the first meeting came from SAP partner companies including DEC, HP, IBM, Siemens-Nixdorf, and others.
Seeing the value in using benchmarks to demonstrate the scalability and capabilities of SAP’s software and partners’ hardware, the gathering was officially formalized and established as the SAP Benchmark Council in 1995. Participating partners in the council, which is staffed entirely by volunteers, include hardware vendors, database and operating system vendors, CPU vendors, vendors of virtualization software, and, most recently, cloud providers. Together with SAP, these partners act as the governing body of the entire SAP benchmarking program, striving to be completely transparent regarding benchmarks and their results.
The SAP Benchmark Council’s founding principle is that all members should have the same rights and responsibilities. Our main focus is meeting the needs of our customers. The council’s intention is to make it as easy as possible for SAP customers to find the hardware and technology they need to support their IT landscapes.
Part of living up to that principle means making certain that the activities around the SAP standard application benchmarks are as transparent and fair as possible, so no partner can push through a personal agenda. Rather, the underlying agenda of all participants is to generate a win-win-win situation for customers, who gain valuable information about partner solutions; for partners, who have an opportunity to prove the power of their solutions; and for SAP, which can demonstrate the scalability of its applications and provide sizing information to customers (see sidebar).
The SAP Benchmark Council meets once a month to discuss the development, definition, and certification process of the SAP standard application benchmark suite (consisting of more than 20 individual benchmarks). On behalf of the SAP Benchmark Council, SAP certifies the benchmarks submitted by its partners. However, all partners are responsible for monitoring the activities concerning these benchmarks and making sure that all participants adhere to the rules and the disclosure policy.
The full disclosure policy requires that every partner, as well as all customers, be able to obtain all benchmark data from certified benchmarks. The policy was created to enable a detailed review of the benchmark results to discourage any prohibited tuning and tempering of benchmarks.
All configurations certified through SAP standard application benchmarks are either available on the market at certification time or will become available within a six-month time frame. With the help of these benchmarks, customers gain valid, objective, and reliable hardware performance KPIs, which can be used for hardware sizing.
Keeping Up with a Changing IT Landscape
With the speed of technological advancement, the pace at which the council needs to react to changes to offer new or adapted benchmarks is one of the biggest challenges for the SAP Benchmark Council. However, the council’s agility is definitely one of its biggest advantages compared to other benchmark organizations, such as the Standard Performance Evaluation Corporation (SPEC) or Transaction Processing Performance Council (TPC). In addition, instead of creating artificial benchmark workloads, the SAP Benchmark Council can rely on SAP business applications, which provide an excellent foundation for benchmark scenarios.
When a new technology innovation appears, the SAP Benchmark Council sets up workgroups, staffed with experts from interested partners, who collaborate to find the right solutions to new technological challenges and to efficiently set up any new benchmarks. These workgroups investigate, discuss, and come up with solutions to benchmark challenges.
For example, one workgroup is currently determining how the execution rules of existing benchmarks can be adapted for cloud environments, as well as what these benchmarks may look like. The workgroup is testing qualities such as cloud elasticity — how quickly a service can adapt to changing customer needs — and variability between different benchmark results.
The workgroups are also responsible for figuring out how to integrate new requirements into the benchmarking process. For example, the addition of the SAP power benchmarks was the product of a workgroup collaboration.
A unique, permanent workgroup is the publication workgroup. One of the major benefits for technology partners that take part in the benchmark race is the opportunity to make their benchmark achievements public. And the customers also benefit because the results and publications are certified and standardized, so they are easy to understand and compare.
The publication workgroup was founded in 2001 to make sure that these benefits would stay in place. The workgroup defined a set of fair and competitive practices, along with a common terminology for the publication of information related to SAP standard application benchmarks. This framework is called the SAP standard application benchmark publication process, and is available online.
If a partner discovers that another partner is violating the publication guidelines, the partners will first attempt to resolve this violation by themselves. If the matter cannot be settled, however, it is brought to the workgroup for discussion. If the publication is found to be in breach of the guidelines, the workgroup will list the breach in the publication policy and violations section of the benchmarking website at www.sap.com/benchmark.
A Look Toward the Future
For 20 years now, SAP standard application benchmarks have helped SAP’s partners prove how well their solutions can perform, and have supported SAP’s customers in their efforts to effectively size and configure their productive SAP business solutions. And the SAP Benchmark Council has no plans to stop.
One recent topic on the agenda of the SAP Benchmark Council and its expert workgroups is establishing concurrent benchmarks. Concurrent benchmarking means running several independent SAP benchmarks (with different SAP system IDs) on a common infrastructure, so that the simultaneous operation of a large number of SAP systems on one physical server can be demonstrated, tested, and validated. The second hot topic is running SAP standard application benchmarks in cloud environments to determine the robustness and scalability of cloud offerings with regard to the operation of SAP applications. You can be sure to see the results of this work in the near future.
Even while we see continuous advancements in the hardware and technology industry, customers can rest assured that the SAP Benchmarking Council will be there to support them as they work to find the tools and platforms that will best fit their needs. For more details about SAP standard application benchmarks, visit www.sap.com/benchmark.
Tobias Kutning (firstname.lastname@example.org) joined SAP in 2001, worked as a database support consultant until 2007, and joined SAP IT in 2008. Since 2010, he has worked on the Performance & Scalability Team, responsible for SAP standard application benchmark product management.