GRC
HR
SCM
CRM
BI


Article

 

SAP's Exchange Infrastructure - The Remedy for Escalating Application Integration Costs

by Sven Leukert | SAPinsider

April 1, 2001

by Sven Leukert, SAP AG SAPinsider - 2002 (Volume 3), April (Issue 2)
 

The exchange infrastructure is SAP’s new XML-based application integration platform. Tightly integrated with SAP applications but open to non-SAP components, it provides a technical infrastructure and delivers business-process and integration knowledge — in the form of SAP’s predefined integration scenarios — to the customer. What’s more, the exchange infrastructure offers an integrated toolset to help you build your own integration scenarios by defining the appropriate messaging interfaces, mappings, and routing rules needed to connect application components or to create collaborative processes involving business partners.

In this overview, we will take a look at all the elements that make up the exchange infrastructure, and how they connect SAP components with each other, as well as with non-SAP components. Although the first release will not provide all the functionality discussed here, this introduction is designed to help as you plan for the short- and long-term future of your current system landscape. With a blueprint of the exchange infrastructure in mind, you can start to evaluate how to make a gradual, step-by-step transformation to a more open, flexible, and integrated system environment.

XML Data Exchange — Flexible and Open Communication

One of the key principles of the exchange infrastructure is that data is exchanged between application systems in the form of XML messages. XML provides a whole new set of standards and technical building blocks that — although they do not solve all the issues of semantic integration — provide a huge step forward in terms of interoperability.

XML is not only used for meta data, it is also the syntax for everything that is exchanged between heterogeneous systems. After all, when it comes to collaborating with business partners directly or via electronic marketplaces, we know that asynchronous and guaranteed messaging is the only option for critical business transactions. So why not make the external and internal integration look the same, and benefit from a homogeneous integration infrastructure?

With XML, you have a flexible basic format for transporting and transforming information. Message formats can be based on open XML standards for business documents, or can be custom designed. Using XML-based mapping, it is possible to transform messages to and from a common XML format or directly from inbound into outbound format.1

This approach is similar to ALE and IDoc communication in that the asynchronous XML messaging allows for the decoupling of applications. Since the exchange infrastructure is designed to use XML and open Internet standards to describe all the necessary meta data, this data can be kept in one central location, which shifts integration-relevant information away from the application. This makes for a much more flexible IT landscape, and one that can be more readily adjusted to evolving software or business conditions. Individual systems can be added, maintained, or taken out of the landscape, while the rest runs as usual.

Taking the First Step: Incorporating the Exchange Infrastructure Into Your System Landscape

Imagine a simple IT landscape. You’ve got:

  • An SAP R/3 4.0B system
  • An SAP R/3 4.6C system
  • A third-party application system
  • A legacy application

Even in this simple example, you would find various methods of data exchange:

  • Between the legacy and the third-party system, you find any number of custom-programmed file exchanges.
  • Between the third-party and SAP R/3 4.0B system, you find some BAPI calls.
  • Between the R/3 systems, you find ALE communication.

What can the exchange infrastructure do for this system landscape? Figure 1 illustrates the possibilities. For starters, with the exchange infrastructure you can take the existing, file-based data exchange between the legacy and the third-party systems and re-route it through the Integration Server, using a more modern XML-based interface to communicate with the third-party application component. This has several benefits:

  • Knowledge about the integration of these two systems is kept within the exchange infrastructure, making it more manageable and easier to change in the future.
  • The data exchange between these systems becomes monitorable — so you can tell which data files have been picked up and processed and when.
  • Later, the same data, once it is converted into an XML message, can be reused to communicate with other connected systems in this landscape. For example, via mapping, it can be made available to the R/3 system in the form of an IDoc.
Figure 1 A Simple Landscape Incorporating the Exchange Infrastructure

Many customers see this kind of scenario as a very important step on the way from their existing IT landscape to a more message-based, process-oriented environment with consistent data throughout.

Of course, the real value does not lie in the single interface integration described here, but in making it part of an overall integrated landscape, where other receivers can make use of the same data and entirely new, collaborative business processes become possible.

Let’s take a look now at the technology that makes the exchange infrastructure work.

The Path to the Exchange Infrastructure Is Simple and Painless

Introducing the exchange infrastructure into a system landscape, as we did in this example, is simple and painless. We didn’t even need to touch the BAPI and ALE communication between the R/3 systems.

Remember that we’re advising an evolutionary approach to adopting this technology. When the time is right, you introduce XML-enabled interfaces, utilizing the exchange infrastructure as a platform for communication between the connected application systems.

Understanding the Elements and Workings of the Exchange Infrastructure

The exchange infrastructure is made up of the following major components, shown in Figure 2:

  • The Integration Repository, which stores design-time integration information — such as interface descriptions or business scenario descriptions — delivered by SAP or entered by you.

  • The Integration Directory, which stores only the integration information that you’ve configured for your own IT landscape.

  • The Integration Server, the runtime environment for performing all tasks involved in controlling business processes and exchanging data between connected systems.
Figure 2 Overview of the Main Components of the Exchange Infrastructure

The Integration Repository and the Integration Directory: Synchronized, Central Integration Information

The Integration Repository and Integration Directory are used to store what we call shared collaboration knowledge: all the meta data that describes the integration of application components and business processes. Both the Repository and the Directory are built on Java technology and use XML standards for all object descriptions. They not only allow the integration of SAP components, they can also hold information about other systems’ interfaces and services, and can model business processes across heterogeneous landscapes. While the Repository and Directory are closely linked, they have very different roles.

The Integration Repository holds design-time information. As such, it serves a dual purpose: it is part of the development environment, and it also delivers a vast amount of accumulated design-time information that the customer can use as a resource in the configuration process.

The Integration Repository tools function as a starting point for the design of interfaces, XML message schemas, and business processes. SAP performs this kind of development and packages this integration knowledge in the Repository, so that it is available as a basis for configuring the actual integration landscape. It will also be possible for customers and partners to add to the Repository with their own applications developed on the SAP Web Application Server and using the exchange infrastructure.

If the Repository is the starting point for integration scenarios, the Integration Directory reflects the finished product — the integration that is actually in place in a particular landscape. The Integration Directory holds the configuration information for the customer’s integration landscape: which application components are actually installed in the IT landscape, which business processes are actually configured, which interfaces on which systems are actually called, and so on. When XML messages actually flow from component to component, all this data is made available to the runtime environment — the Integration Server — to control the execution of all relevant integration tasks.

Information in the Integration Directory is linked to and references the corresponding information in the Integration Repository. As a result, not only is configuration-time knowledge kept in synch with design-time knowledge, but the information is also stored centrally, rather than scattered across all the various participating systems and applications.

The Integration Server: The Communications Platform of the Exchange Infrastructure

The Integration Server is made up of a number of components. The main component is the Integration Engine, which handles processing, mapping, and routing services. A Business Process Engine will also be included in the Integration Server to control complex and collaborative business processes. Additional Integration Services may be included for particular integration purposes, such as special value-mapping or master data integration. Adapters enable the Integration Engine to exchange data with different application components.

In the future, as new SAP applications are developed based on the latest mySAP Technology — such as the recently announced Supplier Relationship Management (SRM) component — they will be able to communicate with the Integration Server without requiring additional adapters. Each new application will contain its own Integration Engine for handling the local messaging and integration tasks, and will be able to communicate with other applications, or directly with an Integration Server (see the sidebar below).

While all parts of the exchange infrastructure run on top of the SAP Web Application Server, participating applications and special adapters may run on any platform that supports basic Internet protocols, such as XML, HTTP, SOAP, and so on. Since all Integration Engines work off the same centrally stored and managed integration information, the term shared collaboration knowledge should now make perfect sense: knowledge about integration is shared over a distributed execution environment.

New SAP Components Will Be Designed for Direct Integration into Your SAP Landscape — No Adapters Needed

How would the exchange infrastructure integrate a newly developed mySAP.com component into your existing SAP landscape? Suppose you have a new mySAP.com component — such as the Supplier Relationship Management (SRM) solution — that needs to communicate with an existing SAP R/3 4.6C component (see Figure 3). How would that scenario be supported?

In this case, the SRM component needs to send an XML message asynchronously to an RFC interface in the SAP R/3 component. As you will see in many future SAP solutions, an Integration Engine is already incorporated into the SRM component.2 This Integration Engine takes the XML message, makes it persistent (to support guaranteed delivery), then forwards the message via HTTP to the Integration Server, which acts as the communications hub.

Here, the Integration Engine receives the message from the HTTP handler, makes the message persistent in a queue on the Integration Server, and then executes the logical routing and mapping steps by calling the appropriate services in the integration layer.3

Finally, after the Integration Server determines the physical address and protocol characteristics of the receiving R/3 system, the XML message is passed to an RFC adapter. This adapter then executes the asynchronous call with the RFC message data to the R/3 system via the tRFC protocol.

Figure 3 Integrating a New mySAP.com Component into Your Landscape

The Integration Engine: Processing and Delivery of the XML Message

The Integration Engine is part of the Integration Server, and helps to exchange all messages between the various components connected within the system landscape. It has two layers: Messaging & Transport for basic message handling, and Integration for routing and mapping.

Once an XML message is received by the Integration Engine, it is handled by the Messaging & Transport Layer. To this end, a pipeline in the Integration Engine executes integration services, such as logical routing or mapping, as well as more technical services, such as message persistence, queuing, and logging provided by the messaging layer itself. In this layer, the Integration Engine provides queuing functions for the physical decoupling of application components and goes beyond standard Internet protocols to guarantee the “exactly-once” delivery of messages. For example, messages are passed between Integration Engines as SOAP messages with specific header extension fields to provide reliable messaging.

Once the Integration Layer receives a message (with information on the sender and the message interface), the logical routing service determines the correct receivers and required interfaces by evaluating the corresponding routing rules, whether those rules are XPath expressions or Java code. The configuration of routing rules allows the Integration Engine to use the message content to determine the receivers for messages in a publish/ subscribe model. Routing rules help relieve the sending application from the need to know all the details about potential receivers and each of the message formats that the receivers will understand.

The mapping service then determines the required transformations, depending on the sender and receiver interfaces. After retrieving the required mapping from the Integration Directory, the mapping service can execute XSLT mappings or Java code (or a combination of the two in the right sequence) to the business content of the sender message. An XSLT processor and the Java execution environment are contained in the underlying SAP Web Application Server. Both XSLT and Java have the benefit of being a kind of universal language that can be “understood” and executed on multiple runtime environments. This is the basis for the evolution of the technical infrastructure itself.

Adapters Link Integration Engines to R/3 and Third-Party Systems

Adapters enable the Integration Engine to talk to different application components that are not able to speak to web service protocols natively. However, adapters address more than just technical connectivity — they also address semantic issues.

As illustrated back in Figure 2, third-party adapters link an Integration Engine to third-party systems like Baan, Oracle, and PeopleSoft, as well as to legacy systems. These adapters translate XML and HTTP-based messages into the (often very specific) protocols and formats of a given application system.

To integrate existing SAP components like R/3 Release 3.x and 4.x, the IDoc or RFC adapters establish connectivity between the exchange infrastructure and SAP systems. These adapters translate XML and HTTP-based documents into IDocs and RFCs. This ensures that companies can still seamlessly connect to their existing SAP infrastructure while leveraging the new exchange-based integration infrastructure. (For SAP systems based on the Web Application Server 6.20 and up, SAP supports direct systems connectivity without the need for additional adapters.)

A number of generic adapters can be used either to integrate application or technical systems by exchanging files or accessing databases, or to integrate other messaging systems such as IBM’s MQSeries via the JMS API. (Note that such an external queuing system is not needed for queuing messages within the Integration Server itself, but can complement the server as a communication partner for the exchange of messages.)

Built-In Monitoring, Tracing, and Error Handling

The Messaging & Transport Layer is built on top of the ABAP stack within the SAP Web Application Server. This provides SAP’s proven system infrastructure technology as a basis for features such as message persistence and queuing, and also allows for integration with the standard SAP monitoring infrastructure.

Depending on the configuration of the Integration Engine, XML messages can be logged at all points in the processing sequence, and can then be searched, displayed, and compared in a monitoring environment. The sending system can also request that the Integration Engine provide detailed tracing information as a message attachment. The Integration Server itself, along with important aspects such as queue status, can be monitored via CCMS.

The Integration Engine is designed to handle different error conditions in appropriate ways. For example, a temporary inability to reach a target system will be handled by a retry mechanism in the messaging layer. Certain error situations are detected automatically, such as the inability to deliver messages due to missing configuration information. The Integration Engine reacts by putting such messages in special error queues and notifying the administrator. For other kinds of errors (for example, application-level errors occurring as a result of an asynchronous messaging process), the Integration Engine logs fault messages returned from the application component and makes them available for follow-up processing.

A Day in the Life of an XML Message4

Let’s revisit our simple system landscape to get a better picture of how, with an XML message, our systems can be integrated. Remember that in this landscape, data from a legacy system was written to a file, routed through the exchange infrastructure, and sent to one of our SAP systems. In Figure 4, you can see a “blueprint” of an Integration Server containing only those adapters necessary for this particular scenario, as a file is transformed to an XML message, and then to an IDoc for use in an R/3 4.6C system.5

Figure 4 Blueprint of the Integration Server

For purposes of illustration, we will follow this XML message on its way through the exchange infrastructure, “room by room” so to speak.

Before we begin, though, let’s look at how an XML message is created in the first place. The legacy system, as part of its daily work, has just written data to a file on the file system. The data is plain, comma-delimited ASCII data. The file adapter in our Integration Server is configured to monitor that system, so it picks up the file, extracts the data from it, and an XML message is born.

If you think about the “anatomy” of our XML message (see Figure 5), its “head” contains the essential information for processing the message within the Integration Server: the message type, the sender (the legacy system), and so on. It also indicates that delivery to the target system should be asynchronous and guaranteed exactly once (quality of service). The body of our rough XML message looks very simple, actually — it is just a collection of tagged items, organized almost like a table of values.

Figure 5 A Rough Sketch of an XML Message

Now we’re ready to begin the Grand Tour. In the first floor of the Integration Server, we follow the newborn message on its way into the Integration Engine’s main room, the Runtime Engine. This is the central point from which all other rooms branch off. On its way to each part of the Integration Engine, the XML message will keep returning here.

Since the delivery of the message to the target system needs to be guaranteed by the Integration Server, we follow the message next to the “basement,” where processing queues and other integration runtime data are stored in the database. (The system keeps an image of the message here in case of processing errors or system failure.)

Now the message returns to the Runtime Engine room, and the actual processing begins. In the Logical Routing room, the exchange infrastructure determines where to send the message based on the message’s header information, and perhaps on its body as well (for content-based routing). To make these routing decisions, the Integration Engine quickly accesses the routing model data in the Integration Directory in the basement.6 The receiver information is added to the message’s header for later reference.

Once the Integration Engine has identified all the places the message needs to go — in this case, to just one SAP system — it then determines how to get it there. In the Technical Routing room, this information is retrieved from the Integration Directory: the receiving system is an SAP R/3 4.6C system, and the receiving interface is an IDoc interface. This is added to the header as well.7

Next stop is the Mapping room, where parts of the message’s body are taken off and reattached elsewhere (ouch!) as part of structure mapping (done with XSLT or Java). This extensive “cosmetic surgery” makes the XML message look like an IDoc; however, it still hasn’t gone through its full transformation.

Our message now returns to the Runtime Engine room one last time, and here it is pushed into the IDoc Adapter room. The IDoc Adapter takes the data, produces a regular IDoc, and sends it to the connected R/3 system via a regular tRFC.

The record of our XML message remains in the database, at least until it’s time for archiving or the data is extracted for analysis.

As we were taking this tour through the exchange infrastructure, you may have noticed that we didn’t enter the Business Process Engine and Additional Integration Services rooms. This floor is still under construction, but watch for new functions to be added on, such as value-mapping tables from a master data management solution, message correlation, and event management.

Business Process Engine Brings State Management to Messaging

Future additions to the Integration Server will include the Business Process Engine, which will introduce robust state management. Whereas the messaging layer works in a stateless manner, the Business Process Engine will be responsible for tracking multiple messages to provide the state management required for business processes. In other words, it will manage the correlation between messages, and “remember” the status of the exchanged messages — what is done, and what is left to do — based on the messages’ sequence and content. (Currently WebFlow technology handles these processes, but the Business Process Engine will expand these functions. For more information, see the article “Bursting the Boundaries of Your Current Business Processes” by Alan Rickayzen in this issue of SAP Insider.)

In this role, the Business Process Engine can use the messaging layer to trigger the execution of process steps on other component systems, suspend the process, and resume execution when response messages are received.

Future System Landscapes with the Exchange Infrastructure

Now that we’ve laid out some of the possibilities for your current landscape, let’s fast forward to what a more complex system landscape may look like in the future, making full use of the integration capabilities of the exchange infrastructure.

Figure 6 illustrates three sub-landscapes: one for intra-enterprise (internal) process integration (which we will call Production, for our purposes), one for running collaborative business scenarios with business partners in a private exchange setting, and one for testing purposes. You’ll notice the Integration Repository is now shared by all three landscapes as a common development and meta data repository. Multiple Integration Servers and Integration Directories are set up to separate the individual landscapes from each other for security, load balancing, or software logistical reasons.

Figure 6 Looking Ahead to the Next Steps for the System Landscape

Existing SAP and third-party components can be connected to a separately deployed Integration Server, which, for these components, acts as a communications hub. New components deployed on the SAP Web Application Server can use their built-in Integration Engines to communicate directly with each other, or through an Integration Server with other components — all depending on the configuration of the individual Integration Engines. It is important to point out again that while a distributed execution environment is possible, all Integration Engines use the same centrally stored collaboration knowledge.

Conclusion

Are you thinking about what your company’s IT landscape will look like in the next year, in five years, and into the future? Are you planning to set up new, collaborative business processes, or integrate systems previously not connected? Do you foresee the consolidation of functionality, now spread over numerous legacy systems, by introducing new application components?

Then consider how your organization might benefit from the exchange infrastructure, used in a step-by-step process to arrive at a homogeneous infrastructure for integrating a heterogeneous world of application components and business processes.


1 Why would you choose anything but direct inbound-outbound transformation? Not only is a common XML format a more standards-based approach, it can actually reduce the total number of necessary mappings. Suppose you have 2 applications sending sales order documents, and 3 applications receiving sales order documents, each one with a different sales order interface. If you individually map each sending interface to each receiving interface, you get a total of 2*3=6 mappings. On the other hand, if you introduce a “common” XML sales order format in the middle, and map each sending and receiving interface to that, you only get 5 total mappings. Extrapolate this to 10 sending and receiving applications each, and you can imagine the benefits!

2 SRM contains a built-in “pipeline” that takes the XML message from the application proxy, which was generated from the interface definition in the Integration Repository during SRM application development.

3 Depending on the configuration of the pipeline, additional logging steps may be included while processing the message.

4 Although you are probably aware that XML processing does produce a certain overhead, I want to reassure you that it will not take a whole day to process a single message!

5 For simplicity, the Integration Repository and Directory in this example have been included in the Integration Server, although a distributed installation will be possible as well.

6 We are not really accessing the Integration Directory directly, but instead the routing model data, which has been cached in the Integration Engine’s runtime data store.

7 However, had the message been destined for three receivers, there would have been copies made in the Integration Server, and each one would be processing separately.


Sven Leukert joined SAP in 1998, after studying mathematics in Berlin and receiving a Ph.D. in mathematics from the University of North Carolina, Chapel Hill. He worked in product management for CRM Middleware until 2000, and has been involved in the development of the exchange infrastructure from its beginnings. He is currently a product manager for the exchange infrastructure, and is responsible for selection of pilot customers.

An email has been sent to:






More from SAPinsider



COMMENTS

Please log in to post a comment.

No comments have been submitted on this article. Be the first to comment!


SAPinsider
FAQ