To the cloud we go. At Nemertes Research, we just completed our 2011 enterprise IT benchmark study, and we find very high interest in private clouds. In fact, 35 percent of the 240 organizations participating in the benchmark will have a private cloud within the next two years.

Nevertheless, as businesses embark on the road to virtualization and cloud environments, an unexpected speed bump is appearing: the issue of self-provisioning.

When analysts talk about private clouds, we assume everyone is on the same page: A cloud is a metered, multitenant, accessible, elastic and self-provisioned service offering. Right? Well, maybe. Most enterprise IT professionals agree with these characteristics, though many say they have a private cloud even without some of these fundamentals. The factor that they most resist is automated self-provisioning.

Despite the fact that most respondents with private clouds indicate they will eventually offer self-provisioning, some pushback remains. Why? The biggest reason expressed by nearly half (48.5%) of organizations is sprawl — or the self-provisioning of virtual machines and storage that never spin down, even long after they are no longer in use.

Sprawling Concerns

The positive side of sprawl is that it demonstrates the ease with which system operators can allocate servers and storage in seconds with just a few mouse clicks. It’s also the bane of many a virtual infrastructure because it eats storage and CPU resources and ties up network resources; basically dead weight. “Our challenge is managing waste and service sprawl. It's tough enough when you have a highly skilled team managing it today,” says the CIO of a small publishing organization. But he fears that moving to a self-service portal environment "will make sprawl management really tough. Blowing through 200 terabytes in the first week would be a problem.”

What it comes down to for many operations staff is balancing full automation with human involvement to better manage sprawl and retain good processes. It occurs to me that the arguments against cloud provisioning could be just another example — like social media and consumer technologies — of IT having trouble giving up control of business services. Perhaps there are some similarities, but I believe that cloud self-provisioning concerns can be easily addressed with better collaboration and governance.

System operators who want to remain in the workflow aren’t only protecting their egos; the reality is that IT is ultimately responsible for the infrastructure, which IT’s customers — both development teams and business units — are provisioning. Some IT professionals are very concerned about unnecessary self-provisioned systems that may boost costs and waste resources (more justification for charge-back, however). For others, it’s simpler: “dealing with insufficient competence on staff outside of IT,” says the CTO of a state government organization . And, for still others it is the legitimate concern of losing touch with the customer.

Governance, Processes Needed

What’s really critical, therefore, are proper policies and procedures for managing the costs associated with self-provisioning. In organizations where business units have development teams, there’s already a process for requesting compute and associated storage resources. There’s no reason that once self-provisioning is implemented, governance should go missing. “We want to do it more generally, but we have to have rational constraints, some kind of background business processes,” says the CTO of a large higher education institution about self-provisioning options.

Ultimately, it’s important to separate the technology of self-provisioning from the process of self-provisioning. Self-service portals, tools and service catalogs are available today. Needed now are processes and procedures that deal with self-provisioning to make it effective, efficient and flexible enough for both IT and the business units to meet customer needs. That’s when the speed bump will no longer prevent a smooth ride to the cloud.

Share you experiences with me and your peers. Add a comment.

Thanks to fast moving technology companies like Apple and Google the IT world is now obsessed with innovation. Not to say innovation has been unimportant in the past, but it's now a close second in the techie buzz word dictionary only to "cloud." However, it's hard to say "innovate" without giving employees at least the right environment to do so. After doing some reading on the topic I thought I would share my thoughts on ways to get your IT organization to at least start sharing ideas that could form a good foundation for innovation.

First I highly recommend watching Steven Johnson's amazing TED talk on innovation titled "Where good ideas come from." He tells a humorous and thought provoking story on how ideas can connect, mature and eventually flourish. If you find the talk interesting then go out and buy his book of the same name (or you can watch this 4 minute video overview of the book). The book covers in much greater detail what types of environments help foster innovation.

Openness serves ideas better than walls - This is often a tough one in traditional IT departments. There is a strong belief that the thoughts and ideas in your head (tacit knowledge) are your intellectual property and by keeping it to yourself you stand to gain more by sharing it. This is the classic job protectionist line of thinking. But when it comes to innovation it just doesn't have the same rate of success as sharing ideas and being open. A team that openly shares their ideas is a team that will be more successful. Ultimately ideas that work are often ones that connected with other ideas.

Ideas are hungry for other ideas - As said above many individuals' ideas aren't fully formed and they need to connect and be discussed to become a reality. Johnson likes to use the coffee house as an environment that generates informal discussion. It's a comfortable environment where people can talk freely and allow their thoughts to connect and form new ideas. Try to create an environment during team meetings that promotes free form discussion where you don't have the formal organizational structures (i.e. judgment and hierarchy). This could be literally going to a coffee house, a team lunch outside the office or a walk around the office park.

Follow the improv rule of "yes and" - If you know anything about improvisational comedy then you are well aware of the most important rule "yes and". Meaning you don't disagree with your fellow actor but rather agree and add onto that thought and push the scene/plot further. This is a great rule to follow in a team meeting by taking a suggestion or idea and pushing it further. It also promotes openness.

Put all your junk on the table - In Johnson's book he talks about the movie Apollo 13 and how the team on the ground that is trying to save the astronauts puts all their spare parts on a table and are told "figure it out." Put all your components, systems and processes on the conference room table (figuratively that is) and think about where they either work well together today or are disconnected. What are the biggest problems for the business or IT right now? Can these parts be put together in a different way to solve it? Do we need all these parts?

Are there leftovers? Can we make a soup? - My grandmother could always make a great soup out of last night's leftovers and available items in the kitchen. Like soup, some of the most amazing innovations have come from what many see as junk or waste from an existing product or process. From an IT perspective what waste or noise do your applications or processes generate that could be used in other processes or applications? Is there gold stored in those log files or data exports that we could use if we mine it?

Innovation in IT is often about automation - The low hanging fruit for innovation in an IT organization is usually automation. It could be automating a manual process, or even getting rid of an expensive or unnecessary process. Don't feel that innovation can only be achieved by buying a new piece of technology. Instead look at ways to get rid of technology that isn't adding value or could be slowing your team down.

Since the completion of my book, Next Generation Datacenters in Financial Services, (Elsevier Science, 2009), enterprise IT has continued on the journey of dealing with an explosion of data and devices as well as economic constraints.

Many concepts covered in the book resonate even more today than when it was written. In particular, four of the concepts continue to be applied or adopted in most firms’ data center strategies today. These concepts and their takeaways are:

1) Concept: Fit for Purpose — “As Needed — When Needed”

Takeaway: Dynamic allocation of IT supply to meet real-time demand should be included in service contract requirements. These should encompass guaranteed execution response, throughput volume, work characteristics, and time constraints or cost/margin rules.

Bottom Line: Efficient and effective execution of work on provisioned infrastructure should be tailored to the specific processing requirements of the business.

2) Concept: Real-Time Management — “Sense & Respond” 

Takeaway: Data center infrastructure operations that incorporate real-time transaction workload management synchronize work characteristics (demand) with resource capabilities (supply) to ensure that IT fulfills the service contract requirements of the business.

3) Concept: Alternative Sourcing Models (Cloud) — “Manage IT as a Supply Chain”

Takeaway: IT is the new supply chain of the business. With this new model comes the need for management-science discipline that is married to an IT-capability discipline. Together, these will enable fulfillment strategies that best match the needs of the business in terms of strategy, performance, regulatory, costs, financing models and security requirements. As firms continue to exploit Internet-connected delivery of IT services (e.g., cloud computing in all forms — software/platform/infrastructure via private, public, trusted or hybrid fulfillment models) inside or outside of the firm, quality of fulfillment can be maximized to meet the needs of the business.

4) Concept: Integrated and Provisioned Footprints — “One Size Does Not Fit All”

Takeaway: Business workloads vary across different types of application patterns. For example: electronic commerce vs. financial reporting vs. data mining vs. batch processing. Technology has matured to the point of enabling firms to set up and tear down infrastructure with automated provisioning where infrastructure components are combined in a tailored manner to ensure that the right types of resources (network, compute, I/O, disk, operating system, application container, security services, etc. …) are provided at the right time based on workload requirements and business policy. It is important to note that since the book was completed, the industry has seen this concept take root in what Gartner Group calls a “computing fabric.”

Many firms are building out next-generation data center infrastructure capabilities with production success. Two leading industry examples where CIOs/CTOs have employed strategies similar to those outlined in the book are Rooms To Go and NYSE Euronext:

• Rooms To Go

Rooms To Go is America’s leading independent furniture company with 150 showrooms, nine distribution centers and the largest furniture inventory. The customer experience and supply chain at Rooms To Go are highly dependent on IT. The firm employed a strategy to manage its infrastructure as an integrated portfolio that can be reprovisioned based on business purpose. Company executives were quoted as saying, the strategy “improved performance, asset utilization, capacity planning and helped the company minimize business risk and respond more quickly to growth.” (Source: CA Technologies website )

• NYSE Euronext

NYSE Euronext and its CIO Steve Rubinow are focused on doing things faster, including communication, development and, of course, competing in the low-latency trading arena. To do this, they have brought two big, new data centers online and phased out operations at 12 old ones. The move represents a half-billion dollar investment in the midst of a severe financial downturn. It positions the NYSE to become a global trading exchange. The core of the infrastructure design is in an integrated footprint infrastructure provisioned and tailored for low latency, highly computational and high-throughput workloads. (Source: Wall Street & Technology 2010)

Smart Enterprise Exchange members can download the first two chapters of the book here.

Attending Interop in New York reminded me that IT is many things to many people. To the vast majority of attendees, it’s about getting help to daily problems like how  to provide security and data access in the ever-changing  worlds of cloud and virtualization.

For them, networking guru, Jim Meltzer, a founder of Ashton Meltzer & Associates, offered a wealth of information about integrating and managing end-user services over LANs, WANs and virtual networks — and how to optimize those networks.

Elsewhere, Paul DeBeasi of Gartner, along with Craig Mathais of Farpoint  Group, listed the top 10 key issues in wireless and mobile — think proliferation and standardization — and how to manage the mess.

But I found the perspective of Bitcurrent’s Alistair Croll, the most refreshing — and arguably, the most important for enterprise CIOs. With the title, The Democratization of IT, Croll’s presentation on October 20 took the 50,000-foot view, outlining nothing less than the reasons that IT as we know it will continue to become unrecognizable to old-generation IT execs.

It’s not that we haven’t heard this type of prediction before, but Croll’s view of the consumerization of IT offered some new insights. Specifically:

• IT no longer has a monopoly on providing technology — adoption and application development are emerging from the grassroots of the organization.

• IT is no longer a proprietary operation; managed services are competing with your  IT department to offer services to your business; and — sadly, for some IT execs — your CEO would rather use those providers because they are in the IT business while your core competencies lie elsewhere.

• Line-of-business users can build their own aps cheaper and faster than IT can; get over it and find a way to make it work. “The back door is wide open,” according to Croll.

• There is more information, and more sharing of information, now than ever before. We are in a generation of instant knowledge and public disclosure.

• Users  are becoming better at IT than you are; especially if you are tied to traditional ways.

In a nutshell, Croll believes that with social media and ubiquitous computing, the “genie is out of the bottle." So what does he suggest that IT do to demonstrate its value? "Stop looking for a cork; start deciding what to wish for."

More precisely, he says, IT could be looking to find the next killer app for salespeople where they can track their leads via feeds, calendars and social media; then offer it as an app. His general advice Is to focus less on how to regulate, govern and secure. That “misses the point.”

Instead, fund different models and follow the consumer’s lead, he says. Companies that recognize that a slow-moving IT department won’t drive the business will thrive; others won’t.

Sober advice — or nice conference soundbites without real teeth?

Let me know what you think...

By Robin Bloor

It’s difficult for a CIO today not to be considering a cloud-related strategy. Over the past three years, these hosted services have acquired marketing sparkle and every IT vendor worth its socks has developed offerings.

For large enterprises, two things have become clear: First, Google, Yahoo and other Web-based businesses with very large data centers have demonstrated economies of scale by pointing thousands of servers at a single application. Second, and more important, Amazon has established a thriving and fast-growing business by providing storage and virtual machines on an hourly rental basis.

These two developments have proven that cloud computing has legs, although it is still evolving today. The critical question for the CIO, however, is what to say when the CEO asks, “What are we doing about cloud computing?”

My answer is that the primary motivation for moving systems into the cloud is to reduce costs: staff costs, establishment costs, energy costs and hardware costs. The cloud will not significantly improve the key business processes of your organization, though it will help you develop applications more quickly.

Currently, the major infrastructure-as-a-service (IaaS) options deliver only Intel-based resources running Windows or Linux. And sadly, only about half of all data center applications run on these operating systems. Even with the best intentions, you won’t be moving the other 50 percent to the cloud anytime soon. And some of the Windows or Linux applications aren’t really candidates for IaaS in any case. Anyone using Microsoft Exchange in-house, for example, might move to a hosted service. Call it cloud computing if you like, but in truth that’s business as usual.

Hosted e-mail systems existed long before cloud computing, as did hosted Web sites and other software-as-a-service alternatives to data center applications.

Low-Hanging Fruit
My advice to large organizations, then, is to begin with the low-hanging fruit. But, as for other applications, prove that you can run them in a private cloud first before you turn to an Internet-hosted cloud model. Management will be the key: resource management, application management, performance management, service management and recovery management. A good rule of thumb: If you can’t make it work in your own data center, it’s not going to work in the cloud.

Consider software development in all its aspects. Many companies already supplement this activity with cloud resources because development software is portable and there are obvious benefits. For example, developers no longer have to negotiate with data center staff to get extra resources. It can take days or weeks for a data center to make an extra server available, but in the cloud, you can get one in minutes, pay for it by the hour, and scrap it when it’s no longer needed.

When you look at operational systems, however, you run into more complexity. It’s true that some stand-alone systems can evaporate into the cloud with few consequences, but only if your cloud service provides a management interface and is secure. Most applications are not stand-alone; they have dependencies, and dependencies are not cloud-friendly because they don’t port very well.

The Role of Virtualization
Virtualization, therefore, is the litmus test and should precede any broad adoption of cloud computing. You may also call this server consolidation, or server virtualization. Some view it as creating a private, or internal, cloud.

Yet, virtualization has its own complexities. Anecdotal feedback from large sites that have pursued virtualization projects suggests that diminishing returns nearly always set in once software development and stand-alone systems have been virtualized. Eventually, a point is met where the payoff falls below zero, as management costs escalate and managing the set of virtualized resources becomes increasingly difficult.

All of this must be considered before implementing cloud computing.

[Robin Bloor is President and Chief Analyst, The Bloor Group, and founder, Bloor Research. He is also co-author of the books, Service Oriented Architecture for Dummies, Service Management for Dummies, and Cloud Computing for Dummies. Read his blog here.]

Robin is also a member of Smart Enterprise Exchange and will reply to your comments here.