Companies grow by acquisition, meaning more systems added to the legacy list. New distribution channels emerge. Electronic trading with business partners is needed to improve efficiencies. ‘Front office’ applications are needed to give added functionality, but they must be integrated with back office systems. Underwriters want to interact with trading hubs and receive their bound policies automatically. The business does not want its processes to be constrained by the location of the data. Third parties want to consume and even add to enterprise data. The Market and trading partners are moving to adoption of ACORD messages. The list goes on and on. The question for the CIO is how do I integrate systems? How do I move data around the enterprise? How do I do all this in the most flexible and timely way? The promises of SOA and ESB driven architectures would suggest that this is the answer; but is it? According to Information Week there is a dark side to SOA: 24% said projects fell short of expectation; 55% said more complexity introduced; 41% said projects cost more, but failed to generate expected returns; only 7% said expectations were exceeded. Over two blog pieces we firstly try to define some terms and concepts and secondly to consider whether SOA/ESB maybe the answer to a CIOs prayers.

According to Microsoft there are some myths about SOA that need to be exploded: SOA is a technology; it isn’t, it is a design philosophy independent of any product, technology or industry trend. SOAs require Web Services; SOAs may utilise Web Services, but utilisation of Web Services does not necessarily constitute a SOA. SOA is new and revolutionary; in fact, EDI, CORBA and DCOM were conceptual examples of SO. A SOA Reference Architecture reduces implementation risk; in fact SOAs are like snowflakes, no two are alike. SOA requires a complete process and technology overhaul; not true, SOA should be incremental and built on your current investments. We must build an SOA; remember, SOA is the means not the end!

An Enterprise Service Bus (ESB) means different things to different people: “A Web-services-capable infrastructure that supports intelligently directed communication and mediated relationships among loosely coupled and decoupled biz components.” Say Gartner. “To put it bluntly: If you have WebSphere MQ and other WebSphere brokers and integration servers, you have an ESB.” say IBM. “A standards-based integration backbone, combining messaging, Web services, transformation, and intelligent routing.” say Sonic Software. “The ESB label simply implies that a product is some type of integration middleware product that supports both MOM and Web services protocols.” say The Burton Group. “Never drink it at lunchtime” said an old colleague.

We need to agree on what an ESB is and does. Microsoft suggests: Message broker; message transformation; message validation; message routing; exception management and message oriented middleware. The latter should provide adaptation and service orchestration. ESB is one important building block of a service-oriented infrastructure. The remaining aspects are: Service registry/repository; service management and security. In the Microsoft world this of course equates to BizTalk Server, SQL Server and .NET Framework. There are of course alternatives. OK, lets go with that definition.

So, essentially, what happens in an ESB? Data (messages) come in via an ‘On Ramp’ with a meta data ‘envelope’ that identifies what the message data is so the ESB knows what to do with it. It is also possible to utilise elements of the message content as the controlling factor if meta data is not available. Data goes out of the ESB via an ‘Off Ramp’. On and Off Ramps can be custom built or utilise products such as MQ Series. The ESB would then dynamically invoke the relevant services to transform and process the message according to ‘Itineraries’ (Itineraries being a series of steps or processes that need to be performed on the message data) hence the term ‘Service Orchestration’. The ESB could also dynamically route data to the relevant ‘Off Ramp’. Another important aspect of an ESB is to implement additional business processes by choreographing multiple services to create modular business processes, or it can aggregate services to create ‘uber’ services. The ESB will provide Exception Management that undertakes message specific exception handling. Custom applications may issue exception messages, the ESB will subscribe to them. Exception handling is thus unified into one place.

That is all very conceptual. Let’s consider a practical example. Say you want to join RI3K and you want to receive electronically all the risks you subscribe to in order to automatically update a back office system. You use Web Connectivity as the gateway to receive the policy messages from RI3K in ACORD format. Web Connectivity will ‘land’ the message into its proprietary database. From there it is sent via MQ Series into your ESB (the ‘On Ramp’); the ESB is based on BizTalk Server. The ESB recognises this as a policy record from RI3K, performs any necessary validation and transformation. The ESB knows that for this type of message it needs to invoke a ‘create policy’ web service in order to get the policy into the back office system. If there are any errors as a result of trying to invoke the ‘create policy’ web service then these are passed to the Exception Handler. If no errors you have a policy in your back office system.

Another one: you receive an electronic file of submissions from a broker so that you can clear each one and authorise the broker to proceed to quote. They are received in XML format. The ‘On Ramp’ processes them into the ESB. The ESB realises what type of message this is and invokes a web service to perform the clearance function for each submission (checking that we do not already have the submission or that we are not already too exposed to that insured). The result of the clearance process on each submission is reflected in an outbound message. The outbound message is routed to the ‘Off Ramp’ from which it can be transmitted back to the broker.

In the next part we will move on and try to answer the original question - Service Oriented Architectures (SOA) and Enterprise Service Buses (ESB) – ‘Emperors new clothes’ or integration best practice?