Big Data isn’t just another sham or fad that has emerged recently but rather if employed appropriately, it is a crucial component of the digital transformation journey of every enterprise. It plays a vital role in various innovative exercises that these enterprise undertake to obtain an edge – from getting insights into customer behavior to making important decisions around product features or charting out an effective go-to-market strategy.
Most CIOs when asked about their Big Data strategy will have convincing answers around its significance and the tools they have invested in. Occasionally, you may hear some unsatisfactory voices as well. However, in my opinion there is yet a large gap between the actual promise and potential Big Data can deliver versus the value being currently derived.  On one hand, key commercial companies like Hortonworks and Cloudera behind the success of Hadoop, a centerpiece of the Big Data platform have shown steady growth in their revenue and customer acquisitions. But on the other hand, if you look closely, they have had to spend a massive amount of money in sales and marketing – essentially in convincing customers. Why is that happening?
I personally believe that the Big Data initiative is not limited to just adopting the Hadoop platform. In my view, the key driver for a CIO should be to find value out of the remarkable data the enterprise has access to. To explain my idea of finding value in Big Data, I’d like to draw an analogy with an exercise that involves constructing railroads around hilly and mountain terrains. In the earlier days, if an enterprise was to construct this railroad, they would go about laying down the rail tracks post leveling the uneven ground by discarding the unwanted soil and stones. They would thus incur high costs and energy expenditure in laying the tracks that they wouldn’t find the time, energy or inclination to investigate if the things they discarded could be of any value to them. If they encountered mountains in their path, they would simply choose an alternate path rather than choosing to employ innovative methods like digging through the mountain and constructing a tunnel, because the latter would be a very unwieldy and expensive affair. There could have been a possibility that the tunnel could have resulted in a shorter distance and route but it wouldn’t be investigated into for lack of means and funds.
In context, the rail tracks are essentially the expensive database technologies that most enterprise invest in today, the discarded soil and stones are data logs and archives while the mountains are other sources of data. If one could obtain an approval for additional budget going through the long management approval process, a data warehouse project could be implemented – which is the equivalent of constructing a tunnel and trying to investigate what is behind the mountain. (It is a different matter that most DWH projects spent more time and money in technologies rather than in investigating the value they could derive out of data).
Fast forward to the new era. You now have the tools and the means of combing through multiple ‘mountains’. These tools, being open source are inexpensive. There are enterprises now, talking enthusiastically about the goldmine they have discovered while combing through mountains using these tools. Some are also speaking about the massive value they found after processing the dirt that they were earlier ignoring or discarding – in archives, logs and unstructured data.
Now you have options available either to comb a mountain, dig through the mountain and reach the other end, and even climb up the mountain in real quick time without spending a fortune. Everyone is excited about these prospects and assertive of the value it has to offer. However the real problem starts here. Everyone wants to discover the goldmine and a vast majority also know the tools that should be and can be made use of. But very few know how to go about using these tools. To effectively use these tools one needs a nimble and innovative approach. What has worked for others would not necessarily work for you. In the earlier days, you bought Oracle or SQL Server because everyone else bought it and it served the purpose. There was one standard approach to utilize those technologies. However, now you have the flexibility to adopt different methods and possibly have different needs as well. Your terrain is different and so you need to have a greater measure of agility and innovativeness in your approach. Now you can start small, experiment, learn from it, find some value, improvise and move ahead to find some more incremental value. You don’t need to wait till the end of the journey to derive value. You could find value incrementally all along the big data journey.
The mantra of this journey is to start small, be agile, be ready to change tracks and be ready to connect the dots. Remember that, in this journey there is definitely more value, you just have to take one step at a time. You don’t have to get bogged down or overwhelmed by the ‘Bigness’ of Big Data. Your project like yesteryears, don’t have to have start with big budgets and long planning and designing cycle. You have many tools and technologies available that allow you to start small and grow big. The value of Big Data can be found in these small steps. You don’t have to worry about how to ride that ‘big elephant’, instead you can simply start with these tools and get comfortable to ride the elephant eventually.