My New Year’s resolution is to be more consistent with this blog, so please help me as any comments and feedback is motivational!
At this time of the year there are more predictions and forecasts than you poke a stick at, so I’m not going to add to this, but rather delve deeper into one prediction that everyone seems to agree with Big Data. If you have read some of my previous posts you would remember that I believe Big Data is the tools and techniques that help us understand what the data is trying to tell us, and how we use that knowledge.
During the last couple of years I’ve been reading as much as I could on the subject and in this process have become a member of a number of Big Data and Analytics groups. And although I’ve had this kind of interest in a number of different subjects I have never known a group of professionals who are more willing to share their IP. I’m absolutely amazed at how much of the basic work is being developed in the open community.
This willingness to share at almost any level is fuelling this field at an amazing rate. Over the last two years the discussions have mapped progress in this field. Starting with, “What is Big Data?” and “Is there any real practical use?” Through the stages of how to get started and what the team should look like, progressing to about a year ago, where it grew into what were the best tools and flowed naturally on to what algorithms where suited to which jobs. Now the discussions have become more sophisticated, for example how to use a combination of algorithms which enhance weak signals to provide the best predictions.
(This reminds me my journey learning Photoshop, you learn the basics-opening closing, saving, cropping, white balance, etc. Then you discover the tools that allow you to do things like correcting exposures with levels and curves as well as filters to enhance your image and finally you find people who have worked out how to combine functions to fix difficult problems or do magic to your image; for example the use of the high pass filter to enhance sharpness, or the combination of layers in certain ways to ‘pop’ that sunset!)
So why am I so confident that this is the prediction that everyone might have got right… well it seems to me that we’ve moved into Gartner’s – “Slope of Enlightenment” in Big Data! So much so that I’ve rolled up my sleeves and hit the keyboard over the last few weeks to turn some of my theoretical knowledge into practical experience… which I’ll share here in the coming weeks.