Big Data – how to address resistance to adoption

The term “Big Data” results in 770 millions search results in less than a second on Google.

In that sense, it is a way bigger than the most searched celebrity Kim Kardashian with 190 millions search results. Most of these search links are populated by education material, vendor’s product details, seminars and why one needs Big Data. Which page actually starts giving links about practical use of Big Data in an enterprise is still a mystery.

“Big data ? I don’t think I need it..”
Mid size companies are yet allergic (if I may say so) to a term “Big Data”. They are worried about cost, and stories they have heard about side effects of failed implementations. In fact, the word “Big” in Big Data acts as a repellant for many companies. Why don’t we simply say “adoption of unstructured data” rather than Big Data ?

The status of Reporting and BI in mid size companies is not very encouraging. Reporting and BI is like reading a newspaper. We don’t read the entire newspaper. We go to our preferred section… browse whatever we can and keep the newspaper aside. However, by doing this, we may be missing on some powerful editorial or a small news byte useful to us. Its interesting to observe the shift from traditional newspaper to electronic news aggregators. The compelling features propelling this shift are : easy search, interactive drill-down, and the ability to archive / share. This gives a reader the much needed power to consume news the way he desires.

Unstructured data – the key differentiator
Traditionally, Reporting and BI don’t need unstructured data. They are designed to answer known questions. They are instructed to give us predictable news in a fixed format. Analytics is supposed to act like an electronic news aggregator. However, we need to provide analytics platform all the necessary power of electronic news : aggregation, search, archival and easy adoption. If we fail to do this, its going to be another BI – Predictive answers to routine questions.

Many companies take the “wait & watch” approach for Big Data and analytics. But wait for “how long” and watch “who” is not defined. Another reason of shying away from adopting unstructured data is our top-down approach while thinking about analytics. We first try to justify why we need Analytics when we have in-house BI. In the absence of practical approach and hands-on experience in collection of unstructured data, we will never be able to justify investment in analytics over existing BI investment. Unstructured data has the power to provide a substantial differentiation in the value proposition of analytics over BI.

The right approach – keep it small and simple
The right approach will be “bottom-up”. First create a low cost data aggregation platform, then provide search and query facility powered by visual display and then finally integrate with your existing BI platform. You don’t even need great investment in this due to advanced open-source technologies such as Hadoop, Hive etc. Ideally you shouldn’t even discuss this project in top level board-room meetings. First adopt unstructured data, create useful insights and then present a business use case to the senior management. Remember how Linux and open source entered in enterprises in early 2000 ? With small but successful footprint in the beginning, Linux made major inroads into enterprise platform over the period of time. Similarly, when you start implementing smaller projects of unstructured data aggregation, you will be actually creating an unstructured data consumption culture which is the stepping stone to Analytics.

Once you are comfortable with consumption of unstructured data and BI, you can start building a strong case for investment in analytics. By integrating unstructured data with existing BI, you have already started the process of finding unknown answers to known and unknown questions i.e. your first step toward Analytics.

Your journey towards Analytics should be well planned and in a phased manner. I would suggest the following four stages :

A] Shift from Reporting to BI
If you are in Reporting phase, build a strong case of BI. Reporting is “As is”, where as BI is a “snapshot” from an ocean of data with the ability to “drill down” if required

B] Adoption of unstructured data
Build a culture of adoption of unstructured data. Build in-house POCs and use cases for various functions like legal, Accounting / Finance, Inventory, HR and CRM. Show how you can enrich their existing data.

C] Integration of unstructured data with BI
Integrate unstructured data with existing reporting and BI

D] Adoption of Descriptive analytics before predictive analytics
Build strong case for analytics with focus on “why somethings happened” (descriptive analytics) before jumping into “predicting what will happen” (predictive analysis)

Would like to know more from you on practical implementations. Drop in your comments and suggestions.