I’m not sure people understand big data and mining the information held within. I’ll summarise a conversation I had today.
Me: Can I have that data-set please?
Other Person (OP): Its got over 20 million records in it, what do you want to know?
Me: I am thinking about x, and think your data set may answer some questions.
OP: What exactly are you looking for?
Me: I don’t really know, until I’ve seen the data and what information it holds.
OP: How do you know my data-set has the information you need?
Me: I don’t. but its the best chance I’ve got.
and so on and so forth…
I think sometimes big data mining is a bit like mineral mining. You can take samples and investigate indicative factors, but until you take hold of your pick-axe, you’ll never know exactly what is down there. Hopefully I’ll get access to the output and see what can be discovered from it. I am already thinking about visualisation techniques to find the shiny nuggets of data held within.