Despite its promise to revolutionize society, the deluge of data gushing from credit card transactions, social media posts, smartphones and other sources has largely gone unused because those gathering it often haven't a clue how to take advantage of it.
If analyzed properly, experts say, this information tsunami could greatly enhance our knowledge, allowing for more accurate predictions about everything from our susceptibility to disease and natural disasters to our electricity needs and consumer preferences.
Yet recent studies have found that much of the compiled material is ignored, mismanaged or lost -- particularly by businesses, which gather most of it. In a report this month by Oracle(ORCL), 93 percent of 333 executives surveyed estimated that failing to take advantage of all their information cost them an amount equal to 14 percent of their annual sales, on average. For a $100 million company, that's $14 million every year.
"It's really a staggering number," said Rod Johnson, vice president of industries strategies at Redwood City-based Oracle, whose products help businesses manage the flood. And the problems for companies will only grow worse, he added, because "the data volumes are increasing at dramatic rates."
Digital data produced
Generated largely by the explosion of computerized gadgets, the data includes records of supermarket sales, insurance claims, flu outbreaks, demographic changes, X-rays, traffic statistics, surveillance camera images and oceans of other details, much of it on video.
In November alone, "30 million pieces of content were added to Facebook," according to an analysis by research company Gartner, which also noted that "the average teenager sends 4,762 text messages per month."
With so much information today, some experts say, forecasts of future trends can be made with greater confidence, and important patterns or tendencies can be spotted more easily. In the past, companies and researchers depended on limited and sometimes unreliable data samples.
Many firms can benefit from the torrent of data, said Rafiq Mohammadi, an executive with Autonomy, a British company Hewlett-Packard(HPQ) bought last year that helps customers understand and act on the information swirling around them.
With Autonomy's assistance, he said, a U.S. drugmaker he declined to name regularly checks dozens of Chinese publications for problems that could hinder its supplies from that country. Another monitors tweets so it can design marketing campaigns. For a third, involved in lawsuits over the 2010 oil spill in the Gulf of Mexico, he said, "we are managing huge amounts of information for them as they are preparing for this litigation."
Using data fully can enable retailers to boost their operating margins -- a measure of revenue minus certain production costs -- "by more than 60 percent," according to a study last year by the McKinsey Global Institute, which monitors the economy.
Given such assessments, a lot of companies hope to cash in on the information deluge, said Mike Franklin, who runs a UC-Berkeley laboratory that has received nearly $25 million from federal and private sources to develop new methods for analyzing massive amounts of information. But he added that many of them "have a nagging feeling they are not getting all the value of their data that they should be able to extract."
That's partly because at many firms, "management has not specified how it will be used: who will make what decisions or provide what services with what data," concluded a study in June by the Massachusetts Institute of Technology and other universities. In addition, experts say, companies often lack the technology and skilled personnel to make sense of their data or it simply disappears like so many misplaced socks.
"In the last year, 69 percent of businesses experienced some form of information loss for a variety of reasons, such as human error, hardware failure, security breach or lost and stolen devices," concluded a recent survey of 4,506 business information technology specialists by Symantec of Mountain View.
Such incidents raise widespread fears.
"There's not enough discussion around privacy and the control consumers should have over their own data," said Lillie Coney, associate director of the Electronic Privacy Information Center in Washington, D.C. Among other concerns, she said, some information collected about individuals "is just wrong."
Similar worries arose from a study this month by the Pew Research Center. Of 1,021 Internet experts it surveyed, 39 percent agreed with the contention that the data deluge "will cause more problems than it solves," in part because it "will be misused by powerful people and institutions with selfish agendas." But 53 percent concluded "the rise of big data is a huge positive for society."