Building big data capabilities without getting lost in it
I've been staggered by the explosion of the data storage volume we require as our multi-media content expands exponentially each day. And there's a question I've often mused on... “Where does all the electronic data we’re creating across the world actually get stored - our digital photos, digital books, newspapers, magazines, chat room content, video links, electronic records etc.?” Physically, I mean? Are there electronic filing cabinets in space? Even with cloud computing, will we run out of e – storage space eventually?
So it blew my mind to read in a recent IBM report, and they should know, that every day we are now creating 2.5 quintillion bytes of data (18 noughts for the uninitiated) and even more mind blowing - that 90% of the data in the world has been created in the last two years alone ... no wonder it’s now being labelled 'big data'! For the likes of IBM, Amazon and Google whose business models have data at their core, this is a tough but familiar business challenge; but it’s heavy going for the rest of us and when it comes to accessing, synthesising and generating actionable insights from such 'big data' it’s fair to say that many non-digital companies are still at absolute first base.
How timely then to read the most recent issue of Harvard Business Review (October, Getting Control of Big Data) which I found extremely thought provoking. This should be compulsory reading in boardrooms and definitely helped me grasp some of the fundamentals as well as the organisational capability issues involved in harnessing the power of big data.
The core problem is that 'big data', true to its name, is overwhelming for people to process. Even if it is heavily distilled by computers, at some point it needs to be made manageable and translated into actionable insights without becoming a stream of numbers with all the facts and none of the finesse. The quantities of data thrown up by the vast range of live global sources, everything from social media interactions, GPS signals and website interactions is literally mind-boggling. Research has also shown (the Harvard Business Review informs us) that companies who are delivering on this data challenge really do perform better - well at least on key financial performance indicators.
It’s obvious that businesses need to do something to tackle this. But what? And how? As always a key challenge is that of talent. Who should you be employing to crack the data challenge? What skills do they need? Where can you find them? Do they even exist? And the last question is by no means a rhetorical question. Since much of what contributes to the noise of big data didn’t even exist a decade ago, we’re all facing this massive digital data revolution at the same time. And like previous revolutions it will inevitably bring winners and losers. But fortune favours the brave!
The sort of ‘data scientist’ that can help steer an organisation’s approach to dealing with big data is as likely to have a background in scientific research (anything from genetics to ecology and systems biology – in other words people who are used to making sense of unstructured masses of data) as computer science, and preferably a mix. Oh and yes, they must also be able to write code. Add to that an understanding of the organisation and how it operates from a strategic and customer perspective and an ability to converse at C-suite level and you could be excused for thinking it’s like the search for the alchemist’s stone! Not surprisingly the race to grab the few people with the right mix of skills is like nothing seen before. Only belatedly are leading Universities appreciating there is a growing market for this skill set and introducing courses that will slowly develop the right candidates. So what to do in the meantime?
The Harvard Business Review is predicting a move from gut or instinct based decision-making cultures to those where data steps up to play a much more significant role. Far sighted CEOs are tackling the challenge head on. They are rethinking how insights from big data can be generated and applied – how they can inform the company’s decision making processes. They are thinking about the new leadership challenges involved and how the culture needs to change so that big data can inform the big decisions without losing the connection with real customers’ wants, needs and motivations.
And, as with all such changes, while data is a core business asset with enormous value, it doesn’t create competitive edge on its own and should never be viewed as an end in its own right. It’s the organisation’s ability to determine the patterns that others with access to the same data have missed and to create powerful insights on which they act that ultimately creates competitive edge. So I believe that we should not get lost in big data. There is still a massive role for big creative leaps - the one’s that come from analysing, combining and connecting data in new and inspiring ways. And, ultimately there is still a big role for insights gained by connecting with consumers as people not simply as data points.