The "Big Data" RevolutionDuring the Industrial Revolution, a key challenge was mining and transporting enough of the abundant coal and iron ore deposits to feed the growing demand. Today, in our information-based economy, data is the new vital raw material and there is certainly no shortage of that raw material.
Every day, every second, huge volumes of data are being created as a by-product of business and individual activity. Companies are tapping seemingly endless streams of data about their customers, suppliers, and operations, while sensors ? which increasingly are embedded in devices such as mobile phones, smart energy meters, automobiles, and industrial machines ? are also recording and communicating data. Contributing further to this deluge of data is the use of social media sites, smart phones, and other consumer devices (including PCs and laptops), by billions of people around the world.
The result of all this activity is known as "big data." A recently released report from the McKinsey Global Institute titled "Big Data: The Next Frontier for Innovation, Competition, and Productivity,"1 defines big data as "data-sets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze."
McKinsey estimates that, in 2010 alone, "enterprises worldwide stored 7 million terabytes of new data on disk drives" and "consumers stored more than 6 million terabytes on devices including PCs and notebooks." U.S. and EU organizations had a combined online storage capacity at the end of 2010 of roughly 27 million terabytes The volume of new data is expected to continue growing by 40 percent every year, while IT budgets are expected to grow at a 5 percent annual rate.
Over 20 years ago, the Trends editors saw this opportunity on the distant horizon. Working with clients in several industries, they identified analytic techniques designed to harness this big data to dramatically improve decision making. At the time, however, the hardware and software technology seriously limited the scope and timeliness of these analyses ? and consequently the impact they could have in the marketplace.
Since then, exponential growth in the price-performance of computing, digital storage, and network bandwidth have converged to remove technological barriers and bring down the cost of collecting and analyzing these huge data-sets. Therefore, much of the large-scale data thats long been collectable can finally be stored, aggregated, and filtered cost-effectively. Then, most importantly, companies can economically carry out deep analyses resulting in new and profound levels of insight.
In short, were at a crucial inflection point ? a defining moment when the capabilities of big data are finally catching up with the opportunities, which will enable big data to move from the periphery to the center of commerce in most industries. According to the McKinsey report, "there is strong evidence that big data can play a significant economic role to the benefit not only of private commerce, but also of national economies and their citizens."
For businesses, these immense data-sets will do more than simply provide incremental economic benefit. Their impact will be much more dramatic. The way they are leveraged will determine the economic winners and losers in one market after another. In fact, according to a recent article in The Economist,2 "Companies that can harness big data will trample data incompetents. Data equity, to coin a phrase, will become as important as brand equity." Not surprisingly, as the McKinsey study highlights, leading-edge businesses are aggressively embracing the use of big data.
As those businesses are demonstrating, there are five broad ways in which big data can be used to create value. These five ways are:
? First, by creating transparency. That simply means making relevant data available for analysis by departments and business functions across the enterprise. For example, giving transparent access of R&D, engineering, and manufacturing databases to engineering would enable higher quality, lower costs, and shorter time-to-market. ? Second, by enabling meaningful experiments. Today, we can collect enormous volumes of transactional data in low-cost digital form. Leading companies are using data collection to facilitate "controlled experiments" that make better management decisions possible. Its then possible to change one policy or other parameter in a given store, assembly line, or marketing campaign and to track statistically meaningful results of that one change. ? Third, by customizing actions to more effectively address a population segment. Big data allows ever-narrower segmentation of customers with much more precisely tailored products or services. This is already a common technique employed in marketing and risk management. However, it could prove revolutionary in one-size-fits-all public sector organizations. ? Fourth, by enabling better and timelier decisions. Sophisticated analytics can substantially improve or even replace human decision-making. Systems have already proven superior to humans in the fast-moving, complex world of foreign exchange trading; and thats just the beginning. Emerging analytic systems can analyze enormous data-sets in ways that vastly improve on the traditional approach, in which an individual looks at averages or small sample sets on a spreadsheet. Real-time analysis of entire data-sets collected from customers, employees, or even sensors embedded in products will enable faster and better decisions in fields ranging from inventory planning to medical diagnosis. ? Fifth, by enabling computer-assisted innovation. Big data can be used to improve the development of the next generation of business models, products, and services. For instance, sensors installed inside the current generation of products will increasingly give designers insights for improving the next generation of products.
This revolution in data is predictably disrupting established industries and business models. Here are just four examples:1. IT firms are making inroads into health-care markets. Google Health and Microsoft HealthVault, for example, are now giving their customers the ability to monitor their own health and to keep track of their treatments.
2. Manufacturers are moving into the space of service companies. With the placement of sensors in their products, manufacturers can monitor those products and alert customers to needed repairs before products fail. BMW is one company that has become proactive in using vehicle data to generate service business. It uses sensor data from each car to inform customers when their automobiles need servicing.
3. Insurance firms are gaining the ability to move beyond the long-established practice of simply classifying drivers based on their age and gender. With the ability to monitor individual driving styles, rates can be based on a persons individual competence, or recklessness, rather than on a generalized grouping.
4. Government is changing as well. The accessibility of big data is improving the ability of tax authorities to spot "spongers." This means, for example, people who claim unemployment pay as well as benefits for work-related injuries are more likely to be caught.
Its not just businesses and governments that are benefiting from the "big data revolution." Although some fear the downside from a loss of privacy for individuals, its clear that their lives will also be enhanced by access to more useable information. Consider just three of the countless examples:
1. Shoppers using smart phones scan bar-codes on items in one store and immediately discover where there is a better price.
2. McKinsey argues that when retailers exploit identifiable "big data marketing and merchandising levers," they will not only improve productivity, but they will also improve the shopping experience for consumers. For example, consumers who opt-in to marketing programs that use big data to better target offers will find products that better match their needs and spend less time looking for those products at the right price. This should increase the real value-added of the retail sector.
3. In healthcare, analyzing patient clinical and behavior data has enabled preventive care programs targeting the most appropriate groups of individuals. Castlight Health is a company that analyzes big data to make information on healthcare pricing available to patients in large health plans.
Whats most obvious is that big data will have a major impact, not just on traditionally data-oriented functions, but in virtually every sector and component of the economy.
Given the rise of "big data," we offer the following five forecasts for your consideration:
First, the success of individual firms will become firmly rooted in their use of big data.
All companies that wish to stay competitive will need to take the big data opportunity seriously.3 Both established companies and new entrants will drive innovation and capture value through deep analysis of information. The McKinsey study found that this is already beginning to happen in every sector that was examined.
Second, the use of big data will generate new waves of productivity growth and consumer surplus.
The increase in actionable information will improve efficiency and effectiveness. This represents a move past the benefits of the first stage of the Internet, where computing and low-cost communications were combined to facilitate commercial transactions. In the next stage, Internet-scale data-sets will be leveraged to establish whole new businesses, as well as to predict consumer behavior and market shifts that are crucial for established businesses. In the process, companies will discover ways to do more with less, producing higher-quality products and services. This productivity growth will lead to savings that will accrue to both firms and their customers. For example:
? According to McKinsey, retailers that fully leverage big data could potentially increase their operating margins by more than 60 percent. ? Similarly, the U.S. healthcare system could easily create $300 billion a year in additional value, mostly by improving treatment decisions through more complete information. ? Consumers could realize savings of more than $600 billion a year worldwide just from services enabled by "personal location data."
Third, big data will impact all industries, but some will reap greater benefits.
Understandably, the opportunities and challenges differ across sectors. The ones that are poised to make the largest and most immediate gains from the use of big data include finance, insurance, and government, as well as the computer and electronic products sector and the information processing sector.
Fourth, the biggest constraint on the potential of big data will be a shortage of necessary talent.
By 2018, the United States alone will face a shortfall of nearly 1.5 million managers and analysts with the know-how needed to exploit analyses of big data to make effective decisions. Perhaps even more worrisome, were likely to face a shortage of between 140,000 and 190,000 people with the deep analytical skills required to transform big data into actionable information. For many companies, this shortage will be a significant hurdle to realizing the full potential of big data.
Fifth, maximizing the potential of big data will force businesses and society to address fundamental assumptions and priorities.
Even as companies exploit the opportunities, they will have to address the broader societal implications that could limit the benefits of this new resource. For example, as the McKinsey report points out, "Many citizens around the world regard this collection of information with deep suspicion, seeing the data flood as nothing more than an intrusion of their privacy." Policies related to privacy, security, intellectual property, and even liability will need to be addressed in a big data world.
References
1. McKinsey Global Institute, June 2011, “Big Data: The Next Frontier for Innovation, Competition, and Productivity,” by James Manyika, Michael Chui, Brad Brown, Jacques Bughin, Richard Dobbs, Charles Roxburgh, and Angela Hung Byers. ⓒ Copyright 2011 by McKinsey & Company. All rights reserved. http://www.mckinsey.com/mgi/publications/big_data 2. The Economist, May 28, 2011 “Building with Big Data.” ⓒ Copyright 2011 by The Economist Newspaper Limited. All rights reserved. http://www.economist.com/node/18741392 3. The New York Times, May 13, 2011, “New Ways to Exploit Raw Data May Bring Surge of Innovation, a Study Says,” by Steve Lohr. ⓒ Copyright 2011 by The New York Times Company. All rights reserved. http://www.nytimes.com/2011/05/13/technology/13data.html
