I am currently finding myself having discussions about the best way to provide globally operating Enterprises with a scalable Integration Architecture.

There is the Event Driven Architecture (EDA) approach, which has been around and proven for quite some time now. I believe it was one of the reasons Message Oriented Middleware was born and why IBM came up with MQ Series in the early 90s.

On the other hand there is the Service Oriented Architecture (SOA) paradigm that is nowadays lived and breathed by many vendors which have ‘SOA Ready’ software stacks to sell.

In order to not get opinionated for the wrong reasons, I had a look at the basic principles and underlying technologies.

And I was quite surprised by the result…

Consistency Model: BASE VS ACID

Verdict: EDA.
According to the CAP theorem, out of network partitioning, availability and consistency you can only achieve two at the same time. That is why Amazons IaaS offering applies BASE (Basically Available, Soft state, Eventually consistent) across their globally distributed infrastructure.
A global enterprise faces similar challenges. Where systems are distributed across countries and continents, BASE is the way to go in order to integrate a global enterprise.

ACID (Atomicity, Consistency, Isolation, Durability) is important for applications and systems which have immediate consistency as a paramount objective.

Scalability: High VS Low

Verdict: EDA.
SOA is emphasising on service calls, which creates dependencies on all service provisioning systems to be up and running and requires distributed transactions (e.g. WS-AT) or additional compensation flows. It is hard to predict how many service calls a service provider needs to cater for during peak load.

EDA uses local transactions to put event messages on a queue. The routing of messages is done via managed subscriptions, which allows to examine the exact number of subscribers.

Communication: Asynchronous & Message Oriented VS Synchronous Request-Response

Verdict: EDA.
Albeit it is true, that SOA can be operated over messaging infrastructure (e.g. SOAP/MQ or SOAP/JMS) the real problem lies in the request to do something and the associated wait for a response, which means systems depend on other systems to complete a task and respond (see also scalability).

Another drawback on SOA is that it is impacted twice by network latency, because there is always a service request plus it’s response whereas EDA doesn’t need a response if there is none required.

Messaging Infrastructure Setup: Complex VS Simple

Verdict: SOA.
On a pure Messaging Infrastructure level SOA/HTTP wins over EDA/MOM, because of technology like DNS, which help to automatically resolve service end points. To be completely fair though we need to note that we are really comparing two  pre-existing infrastructure assets (HTTP/DNS) with Message Queues, which is more a foundational layer of both paradigms. However if you go down either of those two routes you need to be aware of what you can re-use.

Endpoint Layer: Infrastructure VS Application

Verdict: EDA.
In terms of (OSI type) layering, a Service endpoint requires more layers then a message endpoint.

A Service usually requires network, DNS, platform (Java/.NET), App Server, Web Server.

A message endpoint requires network and a queue infrastructure.

Ownership: System Owner VS Multiple

Verdict: EDA.
Because SOA believes in re-use through orchestration, an orchestrated service can involve multiple services provided by multiple system owners.
This can create conflict in terms of ownership and responsibility to adhere to a specific Service Level Agreement (e.g. Platinum Services).

In EDA the system owner of an event message emitting system is clearly defined as the owner of that message. Any type of changes to that message need to be agreed on with this owner.

State: Stateless VS Stateful & Stateless

Verdict: EDA.
Let’s be clear: In Enterprise Integration, you do not want state. Application architecture might need state, but even there, a good application architect will try to avoid it because it is expensive and does not scale or failover easily. That is one of the major problems with service orchestration as soon as it requires compensation flows or distributed transactions. And that is why a Service Orchestration Engine might be of great use as part of an application or system, but not in Enterprise Integration.

In Case of Error: Local Transaction Rollback VS Distributed Transaction Rollback or Compensation Flows

Verdict: EDA
EDA relies mainly on local transactions for message delivery and pickup. Even if there are transactional clients involved the local system context remains, because – unlike SOA – there are no calls to distributed services and systems necessary.

SOA compensation flows add complexity through ‘Undo’-services in original and canonical form.

Decoupling: Canonical Message Form & Hub VS Canonical Service Endpoint & Service Bus

Verdict: Draw

Interface Contract: Message VS Service

Verdict: Draw

Provider Uptime Dependency: No VS Yes

Verdict: EDA
Another major drawback of SOA as an Enterprise Integration approach is it’s dependency on all the service provisioning systems. The service providers need to be up and running and must have enough spare capacity to service the request.

EDA on the other hand notifies other interested systems of an event that has happened. Those systems are free to pick up the event message when they are ready and more importantly it does not prevent any other system from completing it’s task.

Consumer Uptime Dependency: No VS Yes

Verdict: Draw/EDA.

If the service response is delivered asynchronously then SOA and EDA behave in the same way.

In case you are using synchronous service invocation (e.g. over HTTP) the EDA response message can sit in a queue until the receiving system is ready to pick it up, whereas the SOA response might never be received.

Transaction Handling: Local VS Distributed

Verdict: EDA.
See “In Case of Error” above.

Message/Service Peak Load Management: Subscription Based/Predictable VS Random/Unpredictable

Verdict: EDA.
If a system X does something (e.g. Account create) 500 times per minute then we know that we will receive 500 notifications during that minute. The Integration Hub knows how many receivers this event notification has because systems subscribe to it.

If system X now calls a service 500 times then the provider of this very service might or might not have the capacity to deal with it because there is no reliable way of predicting how many service calls that provider will get during the minute from other systems.

Network Traffic & Latency: One Way Message Delivery VS Request/Response

Verdict: EDA.
Event Driven Architecture, in most cases, delivers a message to the underlying messaging infrastructure and is done (fire & forget).

Service Oriented Architecture  works in a Request/Response paradigm and is hence affected twice by network latency.

De-Comissioning of Event/Service Provider: Simple VS Potentially Complex

Verdict: EDA.
Because there is no clear understanding of who is calling particular services, turning off services might have unwanted effects across the enterprise.

In case an Event emitting system has been turned off the event simply will no longer be received by other systems. Alternatively the same event can be emitted from another system if the business deems that event necessary.

Summary

It’s a quite clear Pro-Event Driven Architecture result, which is a bit of a surprise given the fact that there is still a lot of hype and products (Which one was first – hype or product?) around SOA.

But it does not mean that SOA is inherently wrong or faulty. The context for this comparison is very clearly set to ‘Highly Scalable Enterprise Architectures’ for globally operating Enterprises. There might still be a valid reason to go SOA if the context provides a fit.

Cheers,
Andreas Spanner

Related Links:

• http://soamastery.blogspot.com/2009/06/soa-process-driven-message-driven-or.html
• SOA: A Compelling but Specious Argument
• MSDN: Using Events in highly Distributed Architectures

Gartner Analyst Alistair Newton raised data ownership as one of the new main concerns for CIOs in the Banking industry. His view ecompasses handing value-added data back to the Customer for re-use on other related services. Banks need to be managing data internally in a way that the customer can access and update the part that he/she knows best about, like Name and Address for example.
This imposes demands on a banks ‘Single View of Customer’. More importantly it demands (the right!) MDM, Data Quality and Integration Architecture.
In this article we will explore different types of Integration Architectures and come up with a best fit for purpose.

A New Paradigm

“Customer Own Their Data” is quite remarkable considering the MDM and Data Quality discussions in recent years. Not everyone was convinced a MDM project is necessary at all. A lot of focus was put on Functional Integration (i.e. Services/SOA), less so on true Data Integration (MDM/DQ).

For us Customers to own our data makes sense. If my name is ‘Andreas’ but I want to be called ‘Andi’ – from a Customers perspective the bank should comply with that wish. If you then look at the systems landscape inside of large enterprises only few seem to be ready to satisfy a simple wish like this. And the reasons for this are plentyful:

• Failing or incomplete SOA projects
• New data aggregation applications (e.g. web portals) are built, instead of integrating the data backends
• IT projects overrun in time and budget
• Business and IT run as disconnected entities

So?

Firstly, all the systems which require Customer data need to use the same data records of the entities, which are owned and maintained by the customer.

Why? Because, if I change my name now from ‘Andreas’ to ‘Andi’ through the Online banking interface, I’d like to see that change on my next account statement, too.

In order to do so, we need either

1. Shared Data or
2. Data Update Notifications

It is important to note that a central MDM Master is now becoming the MDM slave for the Customer owned entities (like Name and Address for example). Those MDM entities are now ‘mastered’ by the Internet Banking system.

‘Shared Data’ means there shouldn’t any Customer data, except a global identifier, be lying around anywhere else. An MDM master can be such a data repository, but making all the legacy systems using the MDM master is a whole different story.

‘Data Update Notification’ means I am receiving a notification that Customer data has been updated.
Best Practice is to send the whole updated record around instead of just the identifier. This is to save the unnecessary overhead of subsequent getCustomerData requests from all insterested systems.

This is what it means for systems to use the same Customer data. And it is where an Enterprise Architect should start to think about scalability.

Scalability

In a purist SOA/ESB way we would create a single service to set&get the latest and greatest customer data. This service would ideally be unique across the whole enterprise. The peek&idle load of those service calls from all enterprise wide service consumers would be hard to predict and you could end up with the data repository access as the bottle neck. Sure caching is an option, but then you need to think about cache location, cache-size, examine cache hit and miss counts and cache-invalidation strategies. And I’d argue then that this is not part of the core problems you should spend a large portion of your thinking on, when solving your Enterprise Integration challenges.

Remember, aim is to make the Account Statement System print ‘Andi’ instead of ‘Andreas’. If the system, which the Account Statement System uses to get the latest Customer data, is down during my update, the service call and hence my name update will never make it onto my statement. Best case senario is that I will get an error message like “Dear Valued Customer, Due to an Internal Error the requested operation cannot be completed. Please try again later”. It should perhaps read “Due to the wrong Enterprise Integration Architecture the request cannot be completed.”

Solution Architecture

How about sending a ‘Customer Details Updated’ notification to all systems that hold/store my customer data? There will be only one notification per Customer update, no unmanageable sub-sequent service call load, caching has become unnecessary and if a system is down the message will be delivered through a queue whenever the system comes back online

The thinking behind these two Solution Architecture approaches  are quite different. The SOA approach is based on the ACID paradigm whereas the EDA (Event Driven Architecture) approach is based on BASE. In case you want a refresher, Werner Vogel (Amazon CTO) has written a great article on that topic.
The ACID approach is perfectly valid in a non-distributed Application architecture, on an Enterprise Integration Architecture level things change.

Ready to hand the data- ownership over to the customer?

It seems as if the new way to go is to hand data-ownership over to customers. Customers should be happy and CIOs have to worry less about the Match&Merge step withing their MDM and Data Quality projects, at least for the Customer owned entities. However, as shown above, it is still paramount for CIOs to get the Enterprise Integration right first – and that is nothing new.

What’s your view?

Andreas Spanner