Home Sponsored Announcements Collecting data with a purpose to avoid the 'Big Data Trap'

In the age of Big Data, the rule seems to be “Collect everything now, figure out what to do with it later”. Unfortunately, this is often a costly mistake. Cloud storage is getting cheaper by the day – so you certainly can do it – but the real question is why should you do it?

By kyeling, December 18, 2017

Without a strategy for data, the methodology of getting a lot of storage space, gathering all the possible data and then setting up a team to analyse it can easily turn you into a data hoarder. Although it feels great to brag about how many Tb of data we have stored, we tend to forget that having more Tb of data will limit how fast and flexible we can be when the time comes to analyse it: quality over quantity still applies to data analysis.

Firstly, we need to go back to why we collect data. It is the first step in a process that also involves transforming that data into a format suitable for analysis followed by a methodical examination of relationships, patterns and trends to find answers to questions that are usually not immediately obvious. Collecting data is after all just a means to an end.

Also, deciding what data to track can be hard as it’s not always obvious how certain variables can be used in the future or even what the business goals will be down the line. The fact is, we are constantly expanding our analysis and enriching our datasets, so we may easily overlook the relevance of new data as it becomes available. In my experience as a consultant and CDO I have found myself in situations where I was invited to join or execute a “dream project” involving “Big Data” just to find out that the data collected was often duplicated, redundant or simply not fit for the purpose. After spending years and large sums collecting and storing this data, they realized that doing so without first having a clear use case was an expensive mistake.

Other usual mistakes that I find include collecting data that:

  •     Cannot be linked to the existing datasets by any universal identifier/key or used to enrich existing datasets;
  •     Is not needed to solve any underlying business problem or cannot be associated to any of the goals or strategic priorities;
  •     Has a large cost to collect, store and analyse but has no clear value proposition;
  •     Is collected erratically or generates datasets that are unbalanced in ways that cannot be tracked and corrected, generating biased analysis and leading to wrong conclusions
  •     Has no associated plan on when and how to be used, what are the desired results to be achieved and how they can be measured.

The best way to prevent falling into the “Big Data trap” is to start by defining a strategy for data, setting clear business goals and assessing the value to the business, then working backwards to find out what kind of analysis and datasets we need to best achieve these goals, pointing us towards what we need to collect and analyse both now and into the future.

After all, if the reason we need information it to make better decisions, understanding which ones are the most important and deliver the best return to our businesses will guarantee the relevance of our projects, contributions towards the overall success and ultimately improve the satisfaction of both our teams and stakeholders.


With 50+ Speakers, 300+ senior data and analytics executives, over 3 exciting days you will indulge in all things data and analytics before leaving with strategic takeaways that will catapult you ahead on your journey

· CDAO Sydney is designed to bring together senior executives in data and analytics from progressive organisations
· Improve operations and services
· Future proof your organisation in this rapidly changing technological landscape
· CDAO Sydney 2-4 April 2019
· Don’t miss out! Register Today!
· Want to find out more? Download the Agenda



Australia is a cyber espionage hot spot.

As we automate, script and move to the cloud, more and more businesses are reliant on infrastructure that has the high potential to be exposed to risk.

It only takes one awry email to expose an accounts’ payable process, and for cyber attackers to cost a business thousands of dollars.

In the free white paper ‘6 Steps to Improve your Business Cyber Security’ you’ll learn some simple steps you should be taking to prevent devastating and malicious cyber attacks from destroying your business.

Cyber security can no longer be ignored, in this white paper you’ll learn:

· How does business security get breached?
· What can it cost to get it wrong?
· 6 actionable tips



Popular News




Sponsored News