What’s the value in Big Data? The Push Shopping example (Big Data, Part 3)

By Clive Gold – Marketing CTO, EMC Corporation, Australia and New Zealand

Talking about a Telco and its data might not resonate with you so let me expand my example, before we discuss how to solve the problem.  Go back to Part 1, here, of this series, and consider how a retailer could turn this near shopaphobic into a future shopaholic? As discussed online is changing the shopping experience, more than just as a substitute for physical.

Building on the online shopping experience, I described a substitute for physical shopping, now think beyond that.  We know about online auctions, but I also use Catchoftheday and Scoopon (yes I have a frustrated wife that thinks we get boxes of useless gadgets, and yes… I’m in the process of re-doing my open water divers licence – it’s been years).  These are interesting as it’s an example of push selling, not new but much faster and group purchasing.  Now think beyond that, the offer I get from these systems is not tailored to me, so I ignore almost all of them! However if they were in line with what I was interested in today, well that’s another story.

Firstly a health warning, don’t panic! As you live you leave a digital trail. Ignore for now the privacy and security issues. What if all this data was in one place, (Isilon-read Part 2), here, then a system could analyse this and tell me stuff I didn’t already know.  Here is an interesting example: I purchase an iPad and I comment on Facebook that I hate the soft-keyboard. Imagine if you sold iPad cases and you had these two bits of information and you could see I purchase a lot of electronic gadgets; you send me an offer of an iPad case with a Bluetooth keyboard built in… and you have me.

The problem is we are talking about two big problems, lots of data and looking for connections I don’t know I’m looking for. Traditional databases just can’t hack this, not in size and not in structure, so enter Greenplum. Designed to do this work it not only scales, (performance) but does not require you to pre-define the structures, which then limits you to finding things that you know you are looking for.

The example I like to use is a case study where a major Telco in the USA, was analysing their mobile call data records. They found that when a person cancelled their service, within a short period of time, up to six of the top 10 people they used to call would also cancel (not surprising, they are the people they talk to).

So, finding the un-known, un-known’s is an interesting topic for another day. For now, being able to store and analyse BIG DATA has so much potential.

Perhaps now having this digital trail can enhance our lives!

What’s the problem with storing BIG DATA? (Big Data, Part 2)

In part one, here, I teased that the telco could not get the data they needed and they could not analyse it.

Let’s drill down into the first issue, if the telco really wants to ‘know’ me as a customer they would need to pull all my call data records together, (perhaps the call data records of the people I called) and internet usage (URLs visited)! Now doing this for me and then the rest of their customers individually, would not be practical. They would need this data for all their customers. That is a lot of data, 100’s of Terabytes or maybe even Petabyte(s) of data, and keeping this in one place is hard.

Data storage is not like a big box you just throw stuff into; you need a structure to make it work. The structure is created through software that creates RAID groups, file systems and volume managers, which result in limitations at scale. The size is limited by software that is created with a fixed data word size. You do the math and the word size limits the amount of space that can be addressed. So in a 32 bit world the biggest single box that can be created is in the TB range, not big enough for our Telco.

One answer is to just re-compile onto a 64bit system, after all Windows 7 has a 64 bit version! Yes you can do this and publish a new competitive spec sheet, but it won’t perform at scale! The layers of software, (described above) cause an architectural bottleneck, (queues and messaging busses). Also there are structures that you need to keep in memory which creates a cost problem.

Perhaps you throw masses of hardware at it and cluster a number of systems together, using a coordinator or manager system to make it run. Let’s not go into the messaging and management issues with this approach!

So what is needed is essentially a petabyte scale USB memory stick! That is what Isilon delivers – the company EMC just bought. Technically Isilon is elegant in its simplicity, in that it does the job of the RAID, Filesystem and Volume manager in one piece of software which was designed from scratch for scale! This system that is branded OneFS, is the magic. It is fast, as its one layer, it’s scalable as it is duplicated in each node, giving a peer network of nodes which scale linearly.

Now with this giant USB drive, the telco can put all this data in one place and if it could analyse it in almost real-time they could have an incredible competitive weapon…

To be continued in Part 3…!

Consider ‘Push’ shopping! (Big Data, Part 1)

By Clive Gold – Marketing CTO, EMC Corporation, Australia and New Zealand

I hate physical shopping, yet I kind of like shopping online (I’ll even confess to enjoy it, if the wife is not around). Why? There are a couple of reasons – transparency and control.

Transparency because within a few minutes I can compare prices and features from all over the globe and become an instant expert. Perhaps not expert but at least well informed, compared to the last time I asked a shop assistant a question. They simply read the box, which I had done, and then tell me they don’t know.

Everyone loves giving their opinion, (your reading this!). So having decided on my most likely purchase, I look for reviews, problems, issues, etc. Now I get to hear hundreds of user’s opinions about the product that either reinforce my decision or change it for me. The other day a media-player with all the features and at a reasonable price looked good. However, on the chat forums it had more complaints than a plane load of Poms…  so I went to my second choice!

In the end it’s probably that I’m totally in control, I decide on my terms and in my time. There are no extraneous other factors affecting my decision. A perfect situation… well there is no such thing.

The downside of shopping online is you instantly become a target for the retailer, and essentially they begin spamming you. The problem is I don’t know if I should take offence or not… I’m interested in their stuff, I bought some of it, but I’m not interested in all their stuff!

Last piece of this puzzle is I don’t like getting ripped off! For example last week I re-jigged my home telecommunications setup. Combining two teenagers with a 12GB broadband limit was causing a great deal of pain. So I compared my $65/month set up with what was available in the market. Oops, for about $10 more a month we could get 10x the data, and by the way the landline with calls included that will save us about $100 per month. Very difficult decision to make!

So why would this major Telco provider not keep me as a customer and suggest a new plan? The reason is not profiteering as it will now cost them far more to win me back, but data! Today they are literally clueless, they don’t have all the data they need in one place, and they can’t analyse it! But that is changing…

To be continued in part 2…

So what is ‘Big Data’?

By EMC Marketing CTO – Clive Gold

I know most of you are familiar with Cloud, so with the theme for EMC Inform being “Where Big Data meets Cloud”, let me expand on the rapidly growing, ‘Big Data’ subject!  (No apologies for the pun/dad joke)

I recently read a whitepaper from the Kimball Group entitled
The Evolving Role of the Enterprise Data Warehouse in the Era of Big Data
Analytics. I found it fascinating as well as an insightful analysis of how Big Data is an issue that needs addressing and the key shifts that have resulted in this new opportunity.

Interestingly it also shows the expanded use cases for businesses looking to capitalise on their Big Data. To me most importantly, it reveals that not a single organisation will be untouched by its effect, because so often these discussions centre around ‘genomic research’ and ‘geophysical analysis’.

It said “the use cases come in all shapes and sizes and formats, and require many specialised approaches to analyze. Up until very recently all these use cases existed as separate endeavors, often involving special purpose built systems. But the industry awareness of the “big data analytics challenge” is motivating everyone to look for the architectural similarities and differences across all these use cases. Any given enterprise is increasingly likely to encounter one or more of these use cases. That realization is driving the interest in system architectures that addresses the big data analytics problem in a general way.”

If you want to read the entire paper – I highly recommend it! You can download it here.

So what has this to do with EMC? Well the paper also talks to the sheer magnitude of new opportunities in this space and makes it clear that systems to support big data analytics have to look very different than the classic relational database systems from the 1980s and 1990s.  Not only this but the task of storing massive amounts of data in also a challenge for traditional scale up technologies. EMC has been investing in solving the issues and providing these new generation capabilities. Developments such as ‘cloud optimised storage’ ATMOS, and acquisition like Isilon and Greenplum, results in EMC offering in impressive and interesting range of technology to help you realise your ‘Big Data’ potential.

Hopefully I have sparked your interest, if so read on about what Chuck Hollis’ as been saying about the subject, just a few quick links:-
–   Big Data: http://emc.im/i0aV49
–  Are you ready for big data? http://bit.ly/hVH1B1
–  More Isilon Big Data Magic” http://bit.ly/hVW35B