Research shows that companies in the essential industries are improving their data management in anticipation of the exponential growth of big data in the coming years. Andy Hamlyn, group director AMT-SYBEX, takes a closer look at the challenges companies face and how prepared they currently are
With an increasingly digital universe, the amount of digital information in the world is only going to increase.
The concept of handling large volumes of data is obviously not new to the energy industry. Delivering energy requires the coordination and control of large volumes of data that relate to geographically dispersed assets performing core business functions, in a safety critical environment. Recently, energy companies have seen their already vast data sets steadily increase with the introduction of digital technologies such as smart metering and mobile field devices.
Growth explosion
However, as we enter a new era of big data, the rate of data growth will be explosive. Whether this is user generated through digital technologies like high resolution videos/photos, or machine generated from sensor networks, customer behaviour tracking and countless other sources, there is no doubt that both the amount of data is increasing and the data objects themselves are getting bigger. Companies that are able to respond quickly and proactively to incoming data will gain a competitive advantage by being able to improve business efficiency, reduce risk and support strategic decision making more effectively.
Energy companies in particular can leverage their deep heritage in managing big data, the new technologies emerging to handle big data, as well as worldwide interest and investment in big data, to capitalise on the vast opportunities.
With the volume of data that energy companies collect set to increase year-on-year, AMT-SYBEX conducted research to find out what the scale of the challenge would be and how big data is currently perceived within energy companies.
Research results
The research showed that in the last five years, companies have seen a growth in data of between 26-75%. Respondents unanimously agreed that the future growth of data will be ‘exponential’ due to the advent of smart meters, the Smart Grid, mobile reporting and the increase in digital imaging. Respondents suggested this could be anything from 100-1,000% growth over the next five years.
As a result, big data is increasingly being seen by senior management levels as a strategic issue, not just limited to the remit of the IT department. The research from AMT-SBEX shows that the majority of companies agree that the management of big data is an important or very important strategic issue within their organisation.
With data quantities set to increase over the next five years, one of the biggest challenges utilities feel they face is joining-up the data they have from across the organisation – from different technology sources and between their own and other organisations they work with.
Reconciling data from disparate sources presents numerous difficulties – from ensuring accuracy of capture, taking account of differences in timing and measurement (e.g. measuring gas volumes at different times), and allocating a person responsible for collating and reconciling that data. The underlying problem is that often organisations lack consistency in how they report and represent data across different departments. This suggests that traditional data management tools may not be sufficiently robust and there will be a greater need for integration between systems.
With energy businesses facing high velocity data coming from a variety of sources in various formats, linking this data together and deriving meaning from the information is a key step to achieving benefits. The more connected it is, the more related it gets, the more value can be derived.
Summary
It is clear that companies need to focus on making their data more easily transferable and accessible from across the organisation. AMT-SYBEX can help companies to deploy more integrated systems for capturing and processing data, and make this information more easily shareable.
The research showed that the majority of companies are realising the business benefit they can gain from the data they have and are already implementing initiatives to improve their data management to prepare for the universally anticipated increase in big data.
All the companies surveyed either had specific initiatives underway to improve data management within their organisation or had plans to do so in the pipeline. This contrasts to the findings of the AMT-SYBEX 2006 report, which found that less than a quarter of respondents were implementing a top-down initiative for data quality improvement. The report concluded that for most organisations surveyed, there was a significant gap between words and deeds with respect to managing data quality. In light of the findings, it is fair to say that, on the cusp of the next wave of big data, companies are now more conscious of the need to improve their data management processes, with most of the respondents saying that they thought their company had made improvements in how it manages data over the last twelve months.
Companies have made progress to get where they are today, but the forthcoming demands of the smart and digital world will be a game changer for the essential industries.
