Changing the delivery of IT

Tony Bishop

Subscribe to Tony Bishop: eMailAlertsEmail Alerts
Get Tony Bishop: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Related Topics: SOA Best Practices Digest, SOA & WOA Magazine

Cloud Computing: Blog Post

Scaling SOA with Complex Systems Engineering

Achieving business agility when scaling SOA implementations to the enterprise level requires a new approach to engineering

Ever since ZapThink published our Business Agility as an Emergent Property of SOA ZapFlash, we've been explaining in our Licensed ZapThink Architect course how SOA implementations must be complex systems in order to deliver on emergent properties like business agility. Yet, even though we've expanded our treatment of Complex Systems Engineering (CSE) in the latest version of the course, the reaction of most of our students is typically one of perplexity.

Not that we're really surprised, however. Breaking away from the Traditional Systems Engineering (TSE) way of thinking is a huge leap for most technologists, as it shakes to the foundation how they think about architecture, not just SOA in particular, but even more fundamentally, the role IT plays in the enterprise.

Complex Systems: Order from Chaos in Nature
Complex systems theory is especially fascinating because it describes how many natural phenomena occur. Whenever there is an emergent property in nature -- that is, a property of a system as a whole that the elements of the system do not exhibit -- then that system is a complex system. Everything from the human mind to the motion of galaxies are emergent properties of their respective systems. Fair enough, but those are all natural complex systems, and we're charged with implenting an artificial, human-made complex system. How we take the lessons from nature and apply them in the IT shop is a question that engenders the perplexity we see on our students' faces.

There is a fundamental flaw in this distinction, however. Making such a distinction between natural and artificial systems is basically a TSE way of thinking because it separates people from their tools. In a traditional IT system, people are the "users," but not inherently part of the system. In many complex systems, however, people aren't just part of the system, they are the system.

In fact, any large group of people behaves as a complex system. For example, take a stadium full of people doing the wave. Each individual in the crowd decides whether or not to participate based upon the behavior of other people, but the wave itself has "a mind of its own" -- in other words, the wave behavior is an emergent property of the crowd. Another example would be a traffic jam. An accident in opposing traffic will slow down your side of the freeway every time, even though each individual knows that slowing down to look will cause a jam. You and hundreds of people like you can decide not to slow down to look in order to avoid creating a jam, but the jam forms nevertheless.

In the wave example, no technology of any kind takes a role, while in the traffic example, vehicles affect the behavior of the system to a certain extent. In fact, changing the technology can have a dramatic impact on the behavior of the system: if the traffic consisted of trains instead of automobiles, your train might not slow down at all for a problem on a neighboring track. But regardless of whether it's made up of trains or automobiles, the system includes individual people making individual decisions based upon their personal point of view within the system, and emergent properties result, just as they do in a natural system with no people involved at all.

The Enterprise as a Complex System
Any human organization is, in fact, a complex system, including those unwieldy beasts we refer to as enterprises. Enterprises all have policies and managers and lines of control, but the overall behavior of the enterprise emerges from the individual behaviors of the participants in it. Furthermore, the emergent behaviors of corporations and governments may depend entirely on the people who belong to such enterprises, independent of technology. But when we do include technology in our enterprises, we can dramatically affect the emergent behavior of those systems, just as switching from cars to trains changes how traffic behaves.

So, what do you get when you take traffic and subtract the people? A parking lot! Without the people, what was a complex system is now little more than a collection of individual, traditional systems, namely the cars themselves. Each auto is a traditional system in the sense that the properties it exhibits are the properties its manufacturer designed into it. The best you can expect with TSE, after all, is to deliver a system that does what it's supposed to do.

Too often in the enterprise, people confuse complex systems with collections of traditional systems, which is just as big a mistake as confusing a parking lot full of empty cars with a traffic jam. In fact, architects are often the first to make this mistake. Of course, it's certainly true that some architects are too focused on the technology, leaving people out of the equation altogether, but even for those architects who include people in the architecture, they often do so from a TSE perspective rather than a CSE approach. But no matter how hard you try, designing better steering wheels and leather seats and the like won't prevent traffic jams!

Complex Systems Thinking and SOA
In traditional systems thinking, then, we have systems and users of those systems, where the users have requirements for the systems. If the systems meet those requirements then everybody's happy. In complex systems thinking, we have systems made up of technology and people, where the people make decisions and perform actions based upon their own individual circumstances. They interact with the technology in their environments as appropriate, and the technology responds to those interactions based upon the requirements for the complex system as a whole. In many cases, the technology provides a feedback loop that helps the people achieve their individual requirements, just as brake lights in a traffic jam help reduce the chance of collisions.

Such complex systems thinking has been a common theme in many of ZapThink's articles for several years now. Here are some examples:

  • In Best Effort SOA and the SOA Quality Star, we discuss how the business agility requirement complicates the SOA quality challenge. Because agility is an emergent property, we have to establish continuous quality policies that ensure that the delivered system is sufficiently agile. As a result, there's always a tradeoff between agility and quality we call "Best Effort SOA."
  • In The Buckaroo Banzai Effect: Location Independence, Service-Oriented Architecture, and the Cloud, we explore the "Next Big Thing" as SOA, Cloud Computing, Web 2.0, and mobile presence converge. Our conclusion? "The Next Big Thing isn't a cloud in the sense of abstracted data centers full of technology; it's a cloud of people, communicating, creating, and conducting business, where the technology is hidden in the mist."
  • In Resilience: The Missing Word in the SOA Conversation, we discuss how SOA implementations must be resilient, that is, they must have self-righting tendencies that help them recover from adverse forces in their environment. Resilience is a property of the component systems in a SOA implementation that allows the overall system to exhibit the emergent property of business agility.
  • Finally, in the more recent The Christmas Day Bomber, Moore's Law, and Enterprise IT, we introduce the concept of a "metapolicy feedback loop" that explicitly describes the relationship between humans tackling governance in the enterprise and the governance technology they leverage for the task. Only by taking a complex systems approach to the problem of governance do organizations have any chance of dealing with the explosion in the quantity and complexity of information in the enterprise over time.

The common elements to all of these arguments are the feedback loops between people and technology at the component level that enables the overall system to continue to meet requirements as those requirements change -- the essence of business agility.

The ZapThink Take
If you still find yourself perplexed by this whole complex systems story, it might help to point out that complex systems aren't necessarily complicated. In fact, in a fundamental way they are really quite simple. Traffic jams may be difficult to understand, but individuals driving cars are not. Best practices like Metadata-driven governance, the Business Service abstraction, and infrastructure and implementation variability, to name a few, are well within reach of today's SOA initiatives. And the great thing about complex systems is that if you take care of the nuts and bolts, the big picture ends up taking care of itself.

For organizations who don't take a complex systems approach to SOA, however, the risks are enormous. As traditional systems scale, they become less agile. Ask any architect who's attempted to hardwire several disparate pieces of middleware together in a large enterprise -- yes, maybe you can get such a rat's nest to work, but it will be expensive and inflexible. If you want to scale your SOA implementation so that it continues to deliver business agility even on the enterprise scale, then the complex systems approach is absolutely essential.

More Stories By Jason Bloomberg

Jason Bloomberg is the leading expert on architecting agility for the enterprise. As president of Intellyx, Mr. Bloomberg brings his years of thought leadership in the areas of Cloud Computing, Enterprise Architecture, and Service-Oriented Architecture to a global clientele of business executives, architects, software vendors, and Cloud service providers looking to achieve technology-enabled business agility across their organizations and for their customers. His latest book, The Agile Architecture Revolution (John Wiley & Sons, 2013), sets the stage for Mr. Bloomberg’s groundbreaking Agile Architecture vision.

Mr. Bloomberg is perhaps best known for his twelve years at ZapThink, where he created and delivered the Licensed ZapThink Architect (LZA) SOA course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, the leading SOA advisory and analysis firm, which was acquired by Dovel Technologies in 2011. He now runs the successor to the LZA program, the Bloomberg Agile Architecture Course, around the world.

Mr. Bloomberg is a frequent conference speaker and prolific writer. He has published over 500 articles, spoken at over 300 conferences, Webinars, and other events, and has been quoted in the press over 1,400 times as the leading expert on agile approaches to architecture in the enterprise.

Mr. Bloomberg’s previous book, Service Orient or Be Doomed! How Service Orientation Will Change Your Business (John Wiley & Sons, 2006, coauthored with Ron Schmelzer), is recognized as the leading business book on Service Orientation. He also co-authored the books XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996).

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting).

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.