You know it and I know it: Most large organizations suffer from silos, with duplicate software projects in active development. The reasons for this vary — a political power-play, an accident, a belief that it's easier to reinvent the wheel than drive someone else’s car.

But the results, as every enterprise architect knows, are the same: Duplication is costly, and ultimately it eats into the organization’s ability to function as an agile enterprise — that is, one that can freely and spontaneously exchange information as a whole, and where new initiatives are taken on by many people working collectively, instead of only a few working in isolation.

How to wrestle this problem to the ground? I suggest thinking in terms of the federated architecture pattern, an approach that focuses on software, processes and developer behavior. This approach leads to a more agile organization, as it fosters interoperability between lines of business and information sharing. John Zachman, creator of the Zachman Framework for Enterprise Architecture, has positioned this approach as a practical way to employ architectural concepts in complex, multi-unit enterprises. And he has noted that its success requires someone (YOU) figuring out and actually creating and managing what is to be federated — i.e., that which is to be centralized, optimized, integrated and reused by the overall enterprise. (You may want to read Smart Architect’s interview with John Zachman on recent changes to his framework here.)

Building on Zachman’s definition, federated architecture is more than a fancy term to describe an enterprise-wide approach to software. It's a model to share concerns across the enterprise, and to identify common business problems, infrastructure requirements, development efforts and data needs. It requires, in effect, that you build a system that encompasses the entire organization, so that the business is able to adapt to changes in the market more quickly and efficiently, and in a much more cost-effective manner.

As Fig. 1 shows, a federated architecture solution provides the platform to align disparate systems, data sets, other internal and external cloud-based systems, legacy systems, and infrastructure, all via an organization-wide taxonomy. From the purely organizational perspective, this is a difficult task: You need to align the business, the data owners and all stakeholders, the corporate IT strategy, and all of development. And if that weren't difficult enough, there are technical hurdles beyond the norm.

Indirectly related to this is a means to perform communication and integration across the components: Complex event processing (CEP). CEP involves designing your systems to respond to events that occur across all layers of the organization's IT systems. CEP, which in a federated architecture goes way beyond plain-vanilla middleware, becomes a driving force of systems integration and higher operational intelligence, and as such requires corporate wide buy-in and cooperation. Successfully accomplished, the result is a unified architecture, and a common development approach, for all of the systems in the enterprise — no more silos.

Integrating this into the cloud also ties into everything we’ve discussed here over the past year: further reducing costs, increasing efficiency, and raising overall system quality and uptime. (For more on those issues, see, for starters, Time to Revisit Your Cloud Strategy, Architecting the Cloud: Which Workloads Should Move First, or A Technical Perspective On Which Workloads Should Move to the Cloud.)  It also leads to that elusive agile enterprise, where individual components within the architecture — and the people involved — are part of a larger system, one where changes and improvements affect the organization as a whole, with benefits shared. 

Additionally, the federated approach helps to achieve operational independence: Developers for one part of the system can help (or even take over) when emergencies arise in other parts. Examples include software failures, architectural deficiencies and changing requirements that demand additional expertise. Being able to quickly find others within the organization to plug in and help in these situations is a big win. This is a key value that’s lost in the “silo approach.”

For the past year, I’ve been involved in a project where a set of disparate financial systems are being integrated into a single federated portal. As the system brings together elements of trade execution and supporting analytics, the problems have been wide and varied, from normalizing technology selections, to ensuring data-access rights, to meeting latency requirements.

In an upcoming blog, we’ll explore an architectural solution for a federated system based on this real-world experience. Your own federated systems will vary, and of course they don’t need to be a Web portal solution specifically for these lessons to apply. Regardless of what you’re building or integrating, you’ll face the struggles of coordinating numerous stakeholders across the organization, getting them to agree on an architectural direction, aligning their schedules and deployment plans to one another, and truly implementing the systems according to the agreed-upon plan.

In the meantime, let us know if you have an example of how you have implemented a federated architecture to drive agility – and how you did it. 

Let’s face it: The IT Infrastructure Library (ITIL) is the 800-pound gorilla in the world of frameworks. In so many respects, ITIL has had a tremendously positive impact on the ability of companies to deliver IT service support capabilities. Yet, over the course of its 20-year history — particularly with version 3 — the ITIL framework has had applied to it some extended interpretations of just what it is and does. Some have concluded, for example, that it offers best practices for portfolio or service management, when in fact ITIL doesn’t even specifically define good processes for IT infrastructure and operations disciplines such as change management. Rather, it offers guidance, not “must-do” mandates.

These expanded notions of ITIL’s domain, though benign in intentions, have sometimes created tensions among various communities in the enterprise. Application developers didn’t necessarily appreciate ITIL V2 delving into application management, seeing it to be outside the role of the IT infrastructure and operations staff – the target audience for ITIL guidance from the outset. And enterprise architects may have felt their own Open Group architecture framework (TOGAF) territory was invaded by the V3 Service Strategy book that, despite its excellence, discussed activities outside the traditional scope of IT infrastructure staff.

"ITIL was designed for and is almost exclusively consumed by infrastructure management and operations, and V3 caused some problems,” explains ITIL expert Brian Johnson, who was part of the U.K. government team that originally created the ITIL approach. Johnson has authored many ITIL books and now is Principal Services Architect at CA Technologies. An example, in his opinion, is that the Service Strategy book shouldn’t be in V3 of ITIL. “It talks about the whole issue of designing services from a marketplace perspective. It is a genuinely excellent book that addresses subject matter outside the remit of most in-house IT service providers (and ITIL consumers) and would have been better placed in one of the OGC Programme Management guides.“

The release of ITIL V3.1 in July is aimed at addressing some of the more contentious issues that have arisen, and it takes steps such as placing into better context where ITIL does serve as a best-practice framework and where it does not. In other words, for instance, it acknowledges that it is at the discretion of the organization how portfolio-, project-, or security-management fits with the ITIL framework, if at all. While it is entirely reasonable for ITIL-adherents to have ideas on how to work with other good practices and practitioners of other equally important disciplines, including enterprise architecture, ITIL is not best practice for these disciplines, Johnson says.

That comes as welcome news for many, (enterprise architects not least among them!). Perhaps a lessening of friction among parties may lead to better relationships among enterprise architects, developers and IT infrastructure professionals — relationships that could be quite healthy for enterprise efficiency and productivity, too.

So Happy Together

Enterprise architects, application developers and IT infrastructure staff, for example, may want to put their heads together and revisit ITIL V1 and its Software Lifecycle Support book, as well as the V1 Business Perspective series. (In particular, Johnson recommends the book ‘Understanding and Improving’).

These books, Johnson says, do not claim to offer best practices for development, programming, database design or any of those things. But they do provide guidance for IT operations staff working with those involved in service creation, which of course includes the development organization and the architects charged with setting enterprise standards for removing complexity from application portfolios and enabling business responsiveness. “The books just explained where the different pieces came together,” Johnson says of the various elements that play into the software lifecycle. “Strictly speaking, that’s an Enterprise Architecture view.” And wouldn’t it be a good idea, he suggests, for enterprise architects and IT service professionals to come together at some point in the creation of new services and applications, so that they will effectively run on the infrastructure?

“It’s important to communicate how you can help one another,” Johnson says. Business services as well as IT services are bound to create storage, availability and recovery issues — all IT responsibilities and ITIL disciplines. So, when new programs of work are undertaken, it makes sense for enterprise architects to figure out what all the interfaces are, and make sure all the right people, IT operations included, are consulted at the right time throughout the design process. 

At the same time — given the weight ITIL has long carried in many organizations — enterprise architects, among others, may expect more from ITIL than what it in reality delivers when it comes to creating services. In which case, those expectations should be tempered: V3 didn’t provide a method to go from service strategy to service design.

“Right now, there is a disconnect between what an architect is probably expecting and what is delivered,” Johnson says. ”You don’t ‘design’ services using a catalog, for example. A catalog is where you put things. A catalog might offer reuse capability, it can store blueprints of designs, but it just cannot create new service designs.”

That said, the definition of design is often a moveable feast, Johnson notes, taking us right back to the importance of inter-discipline communications. “An operations manager might consider a new service to be, say, provisioning a server--but from a business perspective that would be an invisible component of a service that the business requires to process insurance claims in a different way. So the design will include inputs from many domain experts—including ITIL experts,” he says.

While ITIL does have a view on enterprise architecture, it is no more or less than a domain opinion on how best to interact with another domain’s best practices . Used incorrectly or as a blunt instrument, Johnson says, ITIL can in fact create more problems than it solves. But, as he also emphasizes, value will result when different parts of the organization come together to review earlier and current ITIL guidelines and come to their own interpretations as a group of how best to utilize them for the benefit of the particular enterprise. “It all comes down to attitude, behavior and culture,” he says, if enterprise silos are to be successfully bridged. “Stop talking and listen to each other.”

ITIL® is a Registered Trade Mark, and a Registered Community Trade Mark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.

About the Expert:

Brian Johnson, CA Technology Services

Brian was part of the U.K. Government team that created the ITIL approach. Working for the U.K.'s Central Computer and Telecom Agency (now part of the Office of Government Commerce), Brian authored many of the ITIL books and designed both the ITIL business perspective series and version two of ITIL. He founded the ITIL user group (itSMF--and is an Honorary Life Vice President). His version one book, Software Lifecycle Support, written with software testing guru Richard Warden covered the roles of IT operations and development in designing IT services, one of the central tenets of the version three ITIL books.

Working with CA Technologies, he helped a major client achieve ISO accreditation for their IT operation. Brian has authored, or co-authored, more than 20 titles on ITIL, project management and related subjects. Putting theory into practice, he led both the first successful government implementation of ITIL best practices at National Savings, as well as one of the first private-sector examples at General Accident.

A few weeks ago, Editor and Community Manager Paula Klein wrote here that you could expect to see some new interest groups surface on Smart Enterprise Exchange this year. Making good on that promise, this week I’m happy to be part of the debut of new, exclusive resources for the enterprise architect community. Smart Architect is a group we have created to enumerate, explore and engage with you on the many issues enterprise architects face as they strive to bring greater agility to the business.

Those issues can be pretty monumental, I know. Enterprises have accumulated innumerable legacy business processes and stand-alone services in silo applications that often are linked only by point-to-point integrations. On top of that, the frameworks, reference models and other forms of documentation that have been the traditional outcomes of enterprise architecture (EA) were likely to be treated tactically instead of strategically.

The result of such past portfolio practices? IT organizations’ budgets now stagger under the weight of operational expenses that result from maintaining stove-piped systems, even as IT leaders face new pressures to embrace modern design paradigms — like the cloud — that promise to accelerate business opportunities.

The good news, I think, is that there’s never been a better time for enterprise architects to demonstrate that their discipline can — and should — have strong application to a dynamic business strategy. Today, EA skills are firmly focused on building organic plans to deliver adaptable business capabilities for the organization. This work should prepare the enterprise to reduce IT complexity so that the business can function with efficiency and agility. Those two features are critical today when business-adoption cycles are fast-paced; when the cost-lowering proposition of reusable business services is increasingly critical; and when a consistent and logical view of exploding enterprise data sets matter — as it undeniably does — to help inform corporate next-steps.

All of this sets the stage for your new Smart Enterprise Exchange group. In the coming months, you’ll find informative articles featuring real-world insight from your peers about how they’re forming their EA strategies and preparing for new challenges. You’ll find slide-casts to help you lead your EA modernization efforts. We’ll also feature blogs by expert architects on timely topics, from making way for cloud services to maximizing service-oriented architecture (SOA) returns.

Above all, we hope you’ll join this group, offer comments and actively participate in its content. I’m looking forward to making your voice a part of our vision. Please feel free to contact me with your ideas, comment on my blog, or maybe even start a discussion thread of your own. Thanks!