Don’t Shout I’m Back – My Big Data, Social and Mobile Holiday

After committing to this blog and posting just two posts, I went silent. We’ll I’ve been on holiday, two weeks of beach, reading and the Australian Open… all ready for the year to come!
Do you ever stop and think about how different life is now compared to the ‘way it used to be’. Let’s just compare my two weeks of to just 15 years ago. How the nature of data, information and access is changing the way we live!
To start we decided rather late in the piece to go away, being peak season we were expecting availability to be scarce and prices to be extortionist. 15 years ago, (then), my wife would have taken to the phones and after several hours we would have been in a position to make a decision, in this case within 15 minutes we reviewed the options and had a booking with the deposit paid within 20! So welcome to the age of the Internet… but that is not that interesting. What was interesting, (and unfortunate for the vendor), is that we could see this particular set of units did not have a high occupancy and negotiated a much ‘fairer’ price. Sometimes we ignore the price transparency that we enjoy today!
I read a great deal but it’s all ‘work’ related, so I’m lost when it comes to what’s worth reading for entertainment. For me sitting on a beach is about being fully covered and protected while reading. (For people who don’t know me, I’m the kind of person who burns, I don’t tan.) In the past this would mean getting recommendations from friends and an extended trip to the book store/ newsagent to find stuff to read… now it was a quick ‘trip’ to Amazon books. After selecting one book… the recommendation engine suggested a few others, which also got loaded onto the Kindle. Once again nothing new here, but sitting on the beach I was expecting to be the only nerd, however as I scanned the beach at least 80% of the people reading were using a device, rather than filling pages with sand!
In the past I’ve managed to watch the major games of the Australian Open, but this year I got to watch the championship build through the two weeks, a totally different experience. It seems to me like the first week are the warm-up matches for the top players. However, as they move into the second week there is a perceivable shift in play. Now, for better or worse it seems like tennis is moving to look more like baseball, heavy heavy statistics! You will notice on TV every few games the commentators are throwing some fact and stat at you. However if you went multi-screen and also participated in the online experience you could get overwhelmed with the information available.
I’ve discussed the ‘slamtracker’ that Tennis Australia has on their website that tracks the way the opponents are playing and how they are achieving their ‘key play objectives’. The idea is that each player has a certain style of play and as such to beat them you have to adapt your style to overcome their strengths and exploit their weaker side. (For example Nadal served almost exclusively to Federer’s backhand!) Now they have added a social monitoring as well, to me more for interest than predicting the outcome of the game. So while the game is being played you can not only see all the statistics in-front of you, you can see how well they are sticking to a winning game strategy and see what people are thinking. Like the massive drop in attitude towards Nadal as he left the court during the final, and expressed with the boos from the crowd when he got back!
All in all…Tennis Australia today is where Social, Mobile and Big Data come together in one place!
So sitting on the beach, reading my kindle with an eye on the tennis… did mean a beach holiday was very different to what I recall just 15 years ago!


The Storage World Accelerates – Hardware or Software


I just wanted to say I told you so! (Thanks for that, I do feel better now). But don’t go away, I have a way for you to keep one of your NY resolutions!

Last year I said that “Storage was not Snorage” here and was I right?

Well at the end of the year I found this article, “The 10 Coolest Storage Startups of 2013” (here), isn’t that great? Firstly who would have dreamt of a headline about storage a couple of years back, and secondly they had to choose the 10 coolest ones!!

Silicon Valley has turned back to enterprise IT! For a number of years the buzz was all about the ‘cloud companies’ and everyone wanted to be the next Facebook or Google, but now the money seems to be squarely behind the enterprise now. The venture capitalists have placed their bets and they don’t like to get it wrong!

In the storage space it’s proven to be more interesting than I had predicted! The current ‘battle ground’ is between the traditional way of building storage systems and the use of standardised components with intelligent software. The fundamental attribute of storage is resiliency, after all the purpose of persistent storage is to persist! While the traditional approach of providing ‘no single point of failure’ was through engineering a systems with at least two of everything, the new ‘software resilient’ architecture uses smart software on many standardised systems.

I’m always amazed that we technical people who should be rational and logical, tend to lose perspective in these times of religious fervour! The issue is that the traditional architectures provide far more than just resilience, they are engineered for a specific purpose and thus deliver against a set of service levels. These systems resolve the sharing issues like QOS etc. On the other side of the camp the software developers recognised it has to scale-out from the get-go, needs to be reliable across “not-that-reliable” hardware!

So what is the right way ahead, well I have to say it looks like BOTH ways have their pro’s and con’s and as always in IT, it depends!! So what are we seeing? Firstly traditional workloads tend to gravitate to traditional architectures, and with advances in FLASH and FLASH arrays there are new opportunities and new ways to solve old problems. For example instead of having multiple physical copies of your database, (to separate the loads), today using XtremIO you can run them all on the same array, using the snap copies, and have performance and reliable low latency…to boot!

Then at the other end of the spectrum you have Big Data workloads, where keeping the data close to the CPU pays its dividends in massive processing jobs! So a system like ScaleIO has some interesting applications.

However, in life the ultimate performance may not always be the best solution, when you have to consider all the other aspects of running a production system. Hence systems like Isilon have become very popular in the Big Data space.. a scale-out architecture that implements the Hadoop file system. Here for a small drop in ultimate performance you gain an enterprise class storage system that does not take a team of engineers to maintain etc.

In the end I think the real battleground is how to seamlessly use the right tool for the right job. The challenge is the glue that makes these tools look like one system, with little to no duplication, no increase in management and the automation of the optimal placement of data, at any point of time. Now that’s interesting!

So when your storage vendor arrives with their hammer and tries to convince you that everything is actually a nail. Don’t get religious about it, using the right tool for the right job is always the best option, not only is the TCO always less and performance better, but you get to spend nights and weekends with the family. (And wasn’t that one of your New Year’s resolutions!)

The Year of Big Data – 2014

My New Year’s resolution is to be more consistent with this blog, so please help me as any comments and feedback is motivational!

At this time of the year there are more predictions and forecasts than you poke a stick at, so I’m not going to add to this, but rather delve deeper into one prediction that everyone seems to agree with Big Data. If you have read some of my previous posts you would remember that I believe Big Data is the tools and techniques that help us understand what the data is trying to tell us, and how we use that knowledge.

During the last couple of years I’ve been reading as much as I could on the subject and in this process have become a member of a number of Big Data and Analytics groups. And although I’ve had this kind of interest in a number of different subjects I have never known a group of professionals who are more willing to share their IP. I’m absolutely amazed at how much of the basic work is being developed in the open community.

This willingness to share at almost any level is fuelling this field at an amazing rate. Over the last two years the discussions have mapped progress in this field. Starting with, “What is Big Data?” and “Is there any real practical use?” Through the stages of how to get started and what the team should look like, progressing to about a year ago, where it grew into what were the best tools and flowed naturally on to what algorithms where suited to which jobs.  Now the discussions have become more sophisticated, for example how to use a combination of algorithms which enhance weak signals to provide the best predictions.

(This reminds me my journey learning Photoshop, you learn the basics-opening closing, saving, cropping, white balance, etc. Then you discover the tools that allow you to do things like correcting exposures with levels and curves as well as filters to enhance your image and finally you find people who have worked out how to combine functions to fix difficult problems or do magic to your image; for example the use of the high pass filter to enhance sharpness, or the combination of layers in certain ways to ‘pop’ that sunset!)

So why am I so confident that this is the prediction that everyone might have got right… well it seems to me that we’ve moved into Gartner’s – “Slope of Enlightenment” in Big Data! So much so that I’ve rolled up my sleeves and hit the keyboard over the last few weeks to turn some of my theoretical knowledge into practical experience… which I’ll share here in the coming weeks.

Making Data tell a Story

Great, simple and compelling (just over 3mins) and if you like this, have a look at Rosling’s TED talks.

Data Science 101

I don’t think anybody does it better than Hans Rosling. In the following video he helps to explain population growth, child mortality, and fossil fuel usage based upon wealth. I love how he uses toy blocks and chips to help visualize his point.

See the original post from the Guardian, Hans Rosling: the man who’s making data cool

View original post

The Future of I.T. – This is IT!

In the past four blog posts I’ve been discussing the individual trends of Social, Mobile and Big Data. To save you time if you haven’t read them I argued that the underlying value of each of these trends are:
–          Social’s value is tapping into the community or crowd
–          Mobile’s value is in the fact that everyone and everything is going to be connected.
–          Big Data’s value is in understanding what the data has to tell us.

Now I gave examples how each of these trends have been used by companies and people to change the way that they do things. These niche applications are very specific to an organisation or person, however what everyone wants to know are the benefits that are applicable to most organisations.

What I’ve found is that the benefits multiply when you look for applications at the intersections of these trends.


Social meets Mobile

When the crowd is connected it revolutionises engagement, giving rise to new support models, new way to achieve customer intimacy, new merchandizing methods and new ways industries.

Many organisations are facilitating the service of their products by the crowd. Look at any technology company and you will find a ‘discussion forum’ and or a connection between their products and the crowd. Crowd sources customer service is becoming the norm for the IT industry and its improving customer satisfaction. Not only does the crowd not involve a call centre and a level one script but it also offers real-world advise on best practices!

I took up cycling about two years ago, and to keep motivated I used an app called Strava. At the time I spoke about how this application that tracks people riding and allows them to virtually ride with others, as well as compete against them gave a new dimension to the cold mornings out. Since then there have been a flurry of other initiatives where communities have connected to each other to change the nature of interactions. Everyone knows about garage sales but today the largest garage sale of all eBay has gone way beyond the local neighbourhood as well as past just second had stuff. Craft markets are being replaced by and even borrowing the neighbour’s car is now being put online with or where you can rent your neighbours car in  peer-to-peer rental system.

Big Data meets a Connected World

The ability to augment the human with data can dramatically alter any interaction and any decision. Also understanding what ‘things’ are telling us could improve reliability, performance and efficiency.

Although Google Glass is getting a reputation to be the “nerd’iest” thing on the planet, think about the concept of providing information to you as you do about your normal daily activities. For example you look at an item in a shop and can instantly get the details of the item, the reviews of others and comparative pricing, won’t that change your shopping experience?

Is there information in your organisation that would help someone make a better decision, improve their service to a customer, I’m sure the answer is yes. The issue is getting the relevant information to the relevant person at the relevant time! So we need to start thinking if you would like a side of data with that.

Secondly, in this complex technological world how much better could our systems perform if we understood exactly what was going on. For example: EMC runs a relatively large e-mail set-up to support 60 000 odd users. This single, complex system does have a lot of moving parts and as such does suffer issues that affect users. EMC undertook a project to capture the logs and information from the e-mail system and the infrastructure it runs on and then correlate this to ‘failures’ as reported to the helpdesk. Now EMC can predict when a situation is forming which is likely to affect users, and these can be averted by taking action.

The information is out there, we just have to understand what it is saying.

Crowd meets Big Data

Understanding that the crowd is saying and where they are going is key to anyone producing or servicing people!

A fun example is, a dating site that took a ‘big data’ route to differentiating itself. I believe they have about 500 questions that they ask of people to enable them to match couples up. According to their entry on Wikipeadia, (here), their couples have a better ‘quality score’ than the average and twice as likely to report their marriages as ‘extremely happy’. (Their stats.)

Well do I need to say more if there is some evidence that it works in a subject as complex as relationships and love?

Understanding the social trends leads to a predictive enterprise, revolutionises the development of new products and services as well as gives a new dimension to branding and marketing.

To summarise, if you are looking to drive meaningful change to your organisation all you have to do is leverage the Internet of Everything, tap into the power of the crowd and understand what the data is telling you!

The Future of IT 4: Big Data

big data


Speak to the average IT person about Big Data today and you will get one of three answers:-
–          We don’t have it, that’s just for the large on-line guys!
–          Speak to the BI team, they are responsible for data analytics.
–          We don’t know how to build the business case.

Ask for a definition of “Big Data” and that’s where the fun starts. Some will talk about the classic 3 V’s – velocity, variety and volume, but seldom do you hear anything that gives you a clear idea of neither the value nor the methodologies involved.

I think that this name “Big Data” is a misnomer!  Firstly I’m not convinced that it’s about “Big” I have spoken to people who have gained immense value from understanding their small data. Secondly, I agree with Harper Reed who says it’s not about the Data .. it’s about the Answers!  (Harper was the CTO for the Obama campaign and ran an immense Big Answers project, have a look here for more.)

To me, this thing called Big Data is about understanding what the data has to tell you!

How? Simply by bringing relevant data together to create context, then present that data in a human digestible form.

Bringing data together is where the magic starts, we bring all kinds of data together and the technologies allow us to link, map and/or match this disparate data together. This is very different to the traditional technologies. When people ask why we can’t use the existing technologies I say there are two problems, Relational Databases and Relational Databases.

The first problem with relational technologies is that they are relational! That means you need to analyse the data and structure it before you begin. The result of this is that you have pre-determined the scope and results that you will ever get out of the system. This is not valid in a situation where you don’t know what you will be looking for in six months time.

The second problem with relational technologies is the architecture was built for transaction processing workload. The “Big Data” workload does not fit a TPC-x model at all.  In the end if you attempt to do this work and scale it, it will cost a fortune!

I know that might be too simple for some, but in my simple definition “Big Data” turning data into something that people can use.  To expand on this idea we have to acknowledge that the human mind is better at processing certain types of stimuli, and is limited in the scope of what can be processed.  In 1956 a study was done that showed that the human mind can respond to an average of five, plus or minus 2 stimuli at any one time.  You know that you can stare at a screen full of numbers all day and not be able to detect the trend or an incorrect entry; however you can spot a dead pixel on a HD TV in a moment!  Even relationships have a finite number, known as Dunbar’s number which is 150, above that number and you cannot retain a stable social relationship.

That is why the techniques in ‘Big Data’ allow us to model and visualise the data… to create info-graphics to allow us to grasp the meaning and to build on the knowledge we gain.


We are transitioning to a world which is real-time analytics enabled! (I will talk about this in the future when I bring this altogether.) Unlike today, where everyone has compartmentalised traditional and “Big Data”, much like on-line and batch processing, into two different computing disciplines. The future has to be one where these two worlds work hand in hand or perhaps future architectures incorporate both as a fundamental design.  This is how the large internet companies operate today; imagine if you purchased a book from Amazon and a week later they sent you a recommendation for another book, would you open the e-mail?

So let’s just summarise this down to ‘Big Data’ = Understanding.

Next: Bringing it All together.

The Future of IT 3: Mobility = Connected

Mention “Mobility” to any IT person these days and they go a bit ashen, and the conversation invariably turns to MDM, (mobile device management.)  When you chat to these people you realise that MDM may stand for “Massive Device Mess). There are always areas where the early adopters see the advantage that a new technology provides.  Hospitals have seen how they can un-tether their staff from the office terminal and make them more productive and effective by entering data by the bed side. Restaurants similarly use devices to take orders that are immediately sent to the kitchen. Even airlines have replace pilot’s map cases with tablets, hopefully without the in-air entertainment system installed.

How else have things changed? Let me ask you, do you check your e-mail while in bed? Perhaps you check your e-mail before you check your partner?  You join 71% of the respondents to a recent survey, who said they check e-mail first thing in the morning, while another 17% are on Facebook. Then before we go to sleep 47% of us do a final check of e-mail while 27% make sure we know what our friends are up-to on Facebook.  However in-between… we check our devices while commuting, at work, and even while having a cold beverage after work, we glance at our devices.

The header picture I took on a holiday in Fiji. Being school holidays in Australia the hotel pool was crowded and as I walked I noticed that of the approximately 40 people, only two had paper books, the rest on some device. In-fact there were only two places in the hotel where Wi-Fi was available, a café and the pool!

So the core value of mobility is the fact that we are always connected!

The Future
Let’s expand this idea because it’s not just “us” who are connected, but we are in the process of connecting everything!  At Cisco Live earlier this year the keynote speaker, Carlos Dominguez , demonstrated the Philips Hue, a Wi-Fi light bulb. Taking a picture of the Cisco Live Logo the bulbs changed colour to emulate the sign’s hue!  Now I have to say that when a light bulb is connected we are really getting to a place where everything is connected and the world is rapidly moving to the Internet of Everything.

Just consider the potential of being able to communicate with everyone, while measuring and controlling everything!  As I ‘dad joke’ with my children, “light switches also have an off position, you should try it!” Now consider no light switches, just rooms that light up when you walk towards them, as the GPS in your pocket is communicating your movements to your home control system which is predicting where you are going.

Smart cities are being designed and everyday objects being instrumented, in-order to make city life more efficient and safer. Kevin Bloch, CISCO CTO talks about the ‘electronic’ lamp post. A street lamp post which has multiple devices built into it, from measuring weather conditions to safety cameras and yes a simple light sensors to turn the bulb on and off.

So Mobile = Connectivity

Next: Big Data.