The Big Deal On Big Data

March 23, 2012 – In tech circles, the new paradigm of handling massive volumes of information has dominated the discussion about the future of enterprise for a while now. Two years ago, a cover story in The Economist offered tips on how to prepare for the “new world” of Big Data. As recently as February 12, the New York Times reported that data measurement could be the 21st century equivalent of the invention of the microscope.

So, what’s the big deal about Big Data?

Until recently, Big Data has been the darling of the largest companies – Facebook mines 845 million users’ constant stream of comments, links and photo postings to understand user motivation and optimize services; Google analyzes billions of search results to tailor high-value advertising programs. Even outside of the tech world, American Express and Visa have spent billions creating data banks capable of managing immense volumes. Storage and analytics are not an issue for these well-capitalized giants, but most organizations have lacked the technologies and specialized skills needed to harness complex data and extract useful insights.

But as the cloud resolves issues of data storage and management, and mobile and social technologies allow for easy and immediate information sharing, Big Data now can be crunched and shared throughout an organization, leading to better decision-making and outcomes for all sorts of enterprises.

And that’s the big deal about Big Data. We are in an era in which Big Data is democratizing decision-making by delivering actionable insights from any data; in the process, it’s transforming organizations and end users to be predictive and proactive.

Take the example of companies like Zynga, which has defined the category of social games, or OMGPOP, which has the fastest-growing mobile game of all time, and which Zynga is acquiring. Millions of simultaneous users play Farmville or Draw Something (which scaled up to 36 million users in 6 weeks), which requires a simple, fast, elastic database (called a NoSQL database, a new category of product designed for the age of big data), that can process and deliver massive amounts of user and other types of data in real time. What does this mean? “Snappy” response times mean better overall user experience and a higher likelihood that users will choose to spend more time with your game or application. Maybe even “game over” for the competition.

Despite the dismally high unemployment rate in the country, the average open job only receives 4 resumes. This is because even with the proliferation of myriad recruiting software packages and services, it can still take months for a small or mid-size company to fill a job opening. By capturing and analyzing data across hundreds of job boards and social networks, and offering that as a free, SaaS-based service, an innovative vendor can bring together employers and the best candidates for the open position. If the Chicago Meat Authority needs a new specialist butcher, rather than posting to a number of different job boards and then sorting through the e-mails themselves, they can post through SmartRecruiters, which will instantly post the job on the best boards for that type of position, sort the responses and bring back the top choices based on experience, geography and salary expectations.

The days of waiting for the repairman to arrive between noon and 3 p.m., only to have him reschedule the appointment because he doesn’t have the right part, are over. With the rise of tablets and end-to-end field service applications that include entitlements, parts management, social collaboration and mobile, companies can manage the geographic locations of their workers in real-time, as well as providing remote access to the company’s database to find the quickest, easiest solution to each customer’s problem. Using a mobile device like an Apple iPad, a kitchen appliance repairman can anticipate and manage his daily work assignments. Before each appointment, he can access his company’s database to find the most common source of a problem on any particular appliance, eliminating the guesswork and surprises that can slow down the repair process.

Big data insight can show sales managers which deals – among the hundreds they are working on in any given month – are most likely to close, where they should focus their efforts, and in the end, helps them increase win rates. It’s a step away from the intuition that has driven these decisions and major step forward in terms of planning, all the way down to the supply chain.

Down the Road

A few years from now we see new areas emerging, like Data as a Service, where companies of all sizes are able to access all sorts of data streams—some directly relevant, some unrelated—for computer engines to combine and process in search of new insights. Imagine nonstop rivers of real-time data and the systems that continually organize and manipulate it to make it useful for companies and consumers alike.

We’re looking at real predictive data mining – insights that come when the user has no clue what question to ask. And with the field-leveling advances entrepreneurs are making on the shoulders of Big Data today, those predictive insights will empower Silicon Valley, Wall Street and Main Street alike.

This post originally appeared in Forbes online.