Big Data, Small Steps

Posted: March 21, 2013 by Matthew Hanwell in Big Data, Human Resources, Technology

Having worked with and in HR and HR systems and technology for over 20 years one of the things that has fascinated me is the data that is captured in the course of an employment relationship. And how over time that data forms an intimate historical record of each and every employee, and collectively a comprehensive overview of the life and health of our organizations. There is data directly captured about employees, but also organizational data, structures, relationships, and then system data that are recorded as transactions happen, again providing another rich source of data. This is just the data in traditional HR systems and databases. If we then consider all the data related to people and their work activities within an organization, within our corporate firewalls, the amount of data explodes, from emails to discussion forums, from voice to video, from every electronic interaction each of us have in our work. This can be structured or un-structured data. However, no matter how big this is, it remains minute in comparison to the amount of data that is being created each and every day on the web and by everything that is connected to it, be it human or not – That is BIG data.

I have just attended HRN Europe’s spring event in London (HRTechEurope) where I gave a keynote presentation, sharing my thoughts on Big data, HR reporting and analytics, titled ‘Is HR sitting on a big data gold mine’. My belief is that no matter where we are in HR, we are sitting on a data goldmine, the data is there, the tools are available to extract it, and process it, the challenge is do we know what to do with it, and its potential value to businesses and business decision making? One of my intentions was to challenge participants to think about the extent to which they are using the data sitting within even their current HR systems and databases.

I’m sure there is a huge amount of information published on the topic of Big Data, it is one of those terms that is thrown around and often used without a precise definition, except that we are talking about something large, very large. I think we can all understand that the amount of data is exploding, we are generating, and storing more electronic data than ever, and according to some estimates this is doubling every year! The growth is exponential, with no sigh that it will slow down.

Some of the numbers are staggering, close to 300 Billion email messages sent per day, 1 Billion Facebook users, 500 million Twitter accounts, 48 hours of video uploaded to Youtube every minute, 4.5 Million photos uploaded to Flickr. Wow – so the volume, the numbers are huge, but the speed is equally staggering, being in near real time. And then the Variety, no longer limited to structured data, conveniently stored in fields, according to pre-defined data models conveniently within databases, now we are talking about messages, updates, likes, relationships, location, sound, voice, text, video…..

The promise of big data, (and for that matter any reasonable quantity of data), is that we can make better decisions. We can focus our scarce resources better, improving business outcomes and we can do so in areas that have been dominated by gut feeling and intuition. HR must fall into this category!

I think HR has historically struggled even with relatively small quantities of moderately stable data! We have struggled with definitions, with consolidation, with reporting, with interpretation, even presentation and storytelling. The classic challenge being to produce an accurate headcount report (particularly in global/complex/changing environments) – only to be challenged and results compared with a different number from Finance, and then being expected to explain the differences. Today business leaders, of course still want to know how many people work for them, but increasingly they are looking to have data and information to support them making more strategic decisions.

I believe the tools and technology are now available to automate the extraction of most structured HR data, replacing the once almost manual and typically labor intensive work. The tools are available to process the raw data and deliver us with rich vivid multi-color, multiple dimensional views and 360 degree perspectives. This change in capabilities is like moving from X-Ray machines to MRI scanners, enabling far greater understanding and insight to be potentially gained. However with this enhanced capability comes increased expectation and requires upgrading our skills to be able interpret new forms of information.

If we are to improve decision making, we need to be able to convey the information to decision makers in ways that they can understand. In particular as we include analysis of more unstructured data, our ability to present, to visualize becomes critical to understanding. A picture no longer paints a 1000 words, it paints terabytes! As we present information, we need also to be careful to identify and visualize the outliers; Hotspots and cold spots; it can be dangerous to simply aggregate data related to people, assuming a homogenous data set. Average overall attrition rates fall into this danger zone.

In my experience our reporting and analytics efforts typically address historical data, data that represents a past state. Like driving using only using the rear view mirror. Business generally want to know where they are going, when to change direction, when to accelerate, when to slow down, what is ahead, and this includes their understanding key attributes and capabilities of their workforce. Enter the world of predictive analytics, the ability using available data, together with sophisticated algorithms to forecast the future. In the same way that some Police forces are able to predict crime in certain areas (using software called PredPol), and can therefore police those areas more vigorously, HR should be able to predict key workforce attributes (engagement, performance, development, retention/attrition or whatever else), and proactively target their interventions.

So Big Data is huge, vast, fast and amazingly assorted, and only getting bigger. Before going after the big data prize, I suggest that most HR departments would benefit from taking a couple of small steps; look at what they are currently doing, what data is already captured, review their current HR Reporting and Analytic capabilities, not just technically, but assessing the extent to which they can answer key business questions, and that these answers are used to drive business decisions and action. Once you have extracted the value from your existing data sources it may be time to consider expanding into the big data universe.


  1. Matthew Hanwell says:

    Talking of Big Data: McKinsey & Co in their quarterly, ask what is your Big Data plan…

  2. Matthew Hanwell says:

    The slides from my keynote at HR Tech Europe – Spring Warm-up – London:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s