With enterprise applications capturing wide swaths of information about customers and business activities, organizations are accumulating massive piles of big data that need to be managed. An important part of this growing volume of information is historical data. Archived data from business-complete processes — such as manufacturing, HR, and financials data from SAP ERP — has intrinsic value that can benefit current or future business decisions by helping to reveal trends, patterns, and correlations. Historical data also often needs to be retained for many years to comply with legal requirements, such as laws governing tax audits or product liability, or FDA regulations in the pharmaceutical industry.
Keeping historical data available in the main application database is a resource drain, however, so organizations generally move this data to a secondary storage location. While solutions such as SAP Information Lifecycle Management provide comprehensive functionality for data management tasks such as archiving, retention management, and system decommissioning, adding various storage locations to support these tasks can lead to increased — and costly — landscape complexity.
To help customers simplify their landscapes while addressing their archiving requirements, SAP offers a new database storage option within SAP Information Lifecycle Management that provides organizations with a less complicated, more efficient alternative for archiving SAP ERP data.1
Traditional Approaches to Storing Archived Data
In SAP environments, archived operational data, such as transactional data or master data from an SAP ERP system, is traditionally stored in a third-party storage system. This storage system needs to be certified for use with the SAP ArchiveLink interface to enable integration with the standard SAP archiving system, or, to work with SAP Information Lifecycle Management for data retention and system decommissioning, it needs to be a write once, read many (WORM)-like store that supports the storage of structured information using the WebDAV standard.
Organizations that are also using SAP Business Warehouse (SAP BW) have the option to move less frequently accessed (“cold” or “aged”) analytical data from their SAP BW systems to the near-line storage area of SAP IQ — a highly optimized data analysis server that enables SQL-based reporting and improved performance during data load and access (see the sidebar “What Is SAP IQ?”) — via a near-line storage interface. As part of SAP BW, SAP delivers a native ABAP-based implementation of the SAP BW near-line storage interface, but you can also leverage certified partner implementations, which are developed using a near-line software development kit (SDK) included in SAP BW. The near-line storage option is available for SAP BW systems regardless of whether they are based on SAP HANA or a traditional RDBMS database.
If implemented discretely, these traditional approaches to storing archived SAP ERP and SAP BW data create two silos of data (see Figure 1), which ends up adding to the complexity of your system landscape and increases costs.
A New Way to Store Archived Data with SAP IQ
Available with SAP NetWeaver 7.31 support package stack (SPS) 07, the SAP Information Lifecycle Management database storage option allows customers to consolidate their storage infrastructure on a single platform and substantially reduce storage-related costs. Instead of storing SAP ERP data in a traditional third-party data store via SAP ArchiveLink or in a WORM-like data store, you simply move the data to an SAP IQ database that can also store your SAP BW data (see Figure 2). This option not only allows you to take advantage of the archiving and reporting performance advantages of SAP IQ, it also enables you to leverage the full set of retention management capabilities of SAP Information Lifecycle Management, such as propagation of expiration dates, legal hold support, and automated rule-based data destruction, while storing the data in SAP IQ.
Detailed information on installing and configuring the SAP Information Lifecycle Management database storage option is available at SAP Service Marketplace.2 Before embarking on an implementation, however, it is important to understand the types of content that can be archived using this option, and the benefits provided by the storage capabilities of SAP IQ, so that you can get the most out of your investment. Let’s take a closer look.
What Types of Content Are Supported?
The SAP Information Lifecycle Management database storage option enables you to store two types of structured database content from SAP ERP in SAP IQ: archive files, which contain archived data, and archive indices, which provide access to the archive files. Many companies, such as retailers, generate large volumes of data in their SAP applications and need to archive items — iDocs, for example — on a daily basis. To be able to respond to inquiries — from shop personnel regarding certain materials or promotions, for instance — organizations maintain large archive indices of these archived items.
Note: The SAP Information Lifecycle Management database storage option is included with SAP NetWeaver 7.31 SPS 07 (archive index support only) and SAP NetWeaver 7.31 SPS 10 (archive file support). To use this option, you must have the SAP IQ server installed with a minimum required version of 15.4 ESD3 (18.104.22.16827). If you are running SAP IQ 16.0, you must be at least on version 16.0 SP1 PL1 (64-bit). Additionally, the code page of the server must be CESU-8. For the full details, refer to SAP Note 1737415.
Archive indices are archive information structures created with the SAP standard Archive Information System (transaction SARI) and stored in transparent database tables with the technical name ZARIX* (e.g., ZARIX1, ZARIX2). While archived data moves from the application database to the archive, the indices traditionally remain in the application database, and large indices in particular can still cause volume problems. Storing indices in columnar tables in SAP IQ allows the database to benefit from the high compression rates of SAP IQ, relieving it from unnecessary strain. To enable this functionality, you simply establish a connection to a secondary database (SAP IQ) in the Archive Information System via transaction DBCO (see Figure 3).3 The storage of archive indices in SAP IQ is supported as of SAP NetWeaver 7.31 SPS 07.
Archive files are Archive Development Kit files created by the data archiving process in your SAP ERP system. The archiving system writes the data to be archived to archive files according to the structure defined by the corresponding archiving objects, which must be enabled for SAP Information Lifecycle Management for use with its functionality. To store archive files from SAP ERP in SAP IQ, in transaction IRMPOL (SAP Information Lifecycle Management policy management), you simply specify that the SAP Information Lifecycle Management store point to SAP IQ. The files are stored as BLOBs in the read-only table space of the SAP IQ database. Archive files created in the SAP Information Lifecycle Management retention warehouse for system decommissioning can also be stored in SAP IQ. The storage of archive files in SAP IQ is supported as of SAP NetWeaver 7.40 SPS 05 and SAP NetWeaver 7.31 SPS 10.
In an upcoming innovation cycle, SAP intends to add support to the SAP Information Lifecycle Management database storage option for storing SAP ArchiveLink documents (which contain unstructured content, such as scanned invoices, that is attached to business objects in the SAP system) in SAP IQ. With this support, customers will no longer need to maintain an SAP ArchiveLink store for storing SAP ArchiveLink documents, enabling further optimization of the storage infrastructure.
A Fast and Efficient Deployment
To help customers quickly and efficiently adopt a cost-effective data aging and retention management strategy, SAP delivers the SAP IQ Near-Line Storage and Retention Management rapid-deployment solution.
This rapid-deployment solution, available since August 2014, helps organizations get up and running quickly to address typical business challenges caused by large data volumes, including:
- High maintenance effort for managing large data volumes in SAP BW and SAP ERP systems
- Poor performance of SAP BW and SAP ERP applications due to large data volumes
- IT complexity caused by heterogeneous system landscapes, including non-SAP near-line storage solutions
- Risks due to costly implementation efforts and unforeseeable implementation problems
The SAP IQ Near-Line Storage and Retention Management rapid-deployment solution includes best practices and services for a complete data aging strategy for SAP BW and retention management for SAP ERP applications. Figure 4 provides an overview of its delivery scope.4
Once you have fully set up your storage infrastructure using the rapid-deployment solution, you can move data from your online SAP BW and SAP ERP applications to secondary, secure, and cost-effective near-line storage based on SAP IQ software. This helps reduce the maintenance effort, increase the performance of your SAP BW and SAP ERP applications, and fulfill your retention obligations.
Paving the Way to a Holistic Big Data Strategy
By storing historical data from your SAP ERP or SAP BW applications on SAP IQ, you can:
- Minimize costs: You get all components, including storage, from one, trusted vendor. You also reduce costs by storing archive indices more efficiently, thanks to the high compression rate of SAP IQ compared to traditional databases. If you use the SAP BW near-line storage solution, you can further reduce the data footprint in SAP BW and optimize your investment in SAP IQ.
- Increase performance: With its columnar-style database, SAP IQ enables a fast archive index read without additional secondary database indices, resulting in high performance when accessing archived data. It also offers enhanced search capabilities using broader indices (since database space is no longer an issue), which saves time and enables faster I/O because the system accesses fewer layers (software, network, storage, and hardware, for example) compared to a standard solution using an external archive store.
- Reduce system complexity: By eliminating the need for third-party data stores and software, you can reduce the complexity of your IT landscape. This is particularly true if you also leverage SAP IQ for storing analytical data using the SAP BW near-line storage solution.
The SAP Information Lifecycle Management database storage option helps organizations simplify and streamline their archiving storage infrastructure, in turn paving the way to a holistic and efficient big data strategy. You can learn more about SAP’s end-to-end approach to information management at www.sap.com/pc/tech/enterprise-information-management/software/information-lifecycle/index.html.
1 While the database storage options support archiving data from all SAP Business Suite applications, SAP ERP currently has the most use cases, and is the main focus of this article. [back]
2 See the “Installation and Configuration Guide” for the SAP Information Lifecycle Management database storage option at http://service.sap.com/~011000358700001208472013E. [back]
3 For more information, refer to the SAP Help Portal under SAP ERP > SAP ERP Cross-Application Functions > Cross-Application Components > SAP Information Lifecycle Management > Additional Functions > Defining a Secondary Database (SAP IQ). [back]
4 More details on the SAP IQ Near-Line Storage and Retention Management rapid-deployment solution, including availability information, is available at SAP Service Marketplace (http://service.sap.com/rds-nls). [back]