The prefix “Big” (or sometimes “King”) has been used for a commodity that changed the nature of society and commerce, for example Big Oil, Big Coal and King Cotton. Well, there is a new Big commodity on the block, and it is promising to reshape commerce, businesses and even society. This is Big Data.

Big Data refers to the confluence of multiple rivers of data, including CRM; network analysis; social media; tracking research; behavioural targeting (for example, advertising company WPP Group’s plan to profile 600 million people in its Xaxis database); public data; and the harvesting of information from social activities such as geo-location platforms like foursquare and social shopping sites such as Groupon and Gilt Groupe.

For retail businesses, the initial deliverable from Big Data — and its relative, business intelligence — is granular knowledge about who is doing what and using that information to improve marketing offers and customer service. Often  quoted examples include: U.K. retailer Tesco and its customer loyalty card operation; Amazon.com, and its book recommendations; and Bharti Airtel, the Indian telco that has used network analysis on its more than 3 billion calls a day to reduce churn.

However, the history of Big Data so far suggests that there will be more losers than winners as a result of the new technologies — although the winners will do very well. For instance, researchers at the Butler Group and The Economist have estimated that up to 70 percent of CRM systems produced negative return on investment (ROI). Although Tesco and a few others have leveraged their loyalty card programs to great commercial success, most programs don’t perform as well. And some tech companies that thought they would make their fortune with business intelligence (BI) and Web analytics are being undercut by the launch of the free Google Analytics.

CIO Challenges

One challenge for CIOs and their departments will be in deciding who owns the data. The cost/ benefits of using standardised solutions is likely to massively outweigh the benefits of home-grown solutions, but there are trade-offs. If solutions are produced internally, CIOs may well find that their corporate position is further entrenched. If the solution is packaged by a company like SAP, on the other hand, then the internal owner could be the finance or marketing department.

To paraphrase scientist Niels Bohr: Forecasting is very hard, especially about the future. But five predictions that seem fairly safe include:

1.     The amount of data available will continue to grow exponentially. We are entering the Petabyte Age — where kilobytes were once stored on floppy disks, megabytes on hard disks and terabytes on disk arrays, petabytes are stored in the cloud.

2.     The amount of data that decision makers will look at will actually continue to decline because there is too much and it is not useful or accessible. Consider the exceptions: Dow Jones Index, a single number to explain the stock exchange; a credit rating agency’s simple AAA to D rating; or the net promoter score (NPS), the “one” customer satisfaction number that its developer, Fred Reichheld, believes companies should be looking at. The point is: Simple is better and less is more.

3.     Similarly, most users of information will not know about or understand the processes that are used to aggregate disparate strands of Big Data, nor will they know about or understand how Big Data is turned into simple numbers.

4.     The winners in the Big Data battle will be those whose systems become standard, and history suggests that the best systems will not necessarily be the winners.

5.     Some of the systems and data are going to be free, just as Linux, Apache and Google’s Insights, Analytics and Earth are in the public domain. These offerings make it harder to predict which aspects of the Big Data system will be profitable.

For enterprise IT departments, two key skills that will be in high demand as a result of Big Data will be the ability to add an understanding of “Why?” to the increasingly accurate picture of “What?” and the ability to simplify the complexity of Big Data. For example, businesses will need to buy or develop systems that direct them to offer this person this product at this time at this price or deal. The need to add the “Why?” element is likely to be seen as an opportunity for a wide range of providers — from anthropologists to market researchers, behavioural economists to semioticians, and social scientists to trend hunters.

Where can Big Data go next? Consider the comment by Google’s Eric Schmidt in 2007 when he forecast that Google would be able to answer such questions as, “What shall I do tomorrow?” and “What job shall I take?” Interesting possibilities indeed.

******

Ray Poynter is author of The Handbook of Online and Social Media Research and Executive Vice President with Vision Critical.