In a world where big data and open data initiatives are current hot topics and relevant solutions are being vigorously pursued across many sectors, it seems scarcely credible that data is a significant issue for energy demand research. Yet “the lack of access to good quality, high resolution energy data of the statistical quality that most other disciplines would consider a pre-requisite for the pursuit of good science and robust conclusions” is acknowledged by the RCUK Centre for Energy Epidemiology (CEE) as a key issue facing the end-use energy demand sector.
I entered the world of academia after a career in data warehousing (DW) and business intelligence (BI) so my initial instinct was that these type of approaches might provide an effective solution. Inevitably it turned out to be more complicated than that. DW/BI solutions work well when there are regular data flows and well-defined, predictable analytical requirements but I quickly discovered this was the precise opposite of the data landscape in the energy research sector.
The data requirements of individual researchers or projects are incredibly diverse and specific to the context of each project. Data issues, however, appear to be broadly similar across the board. Data doesn’t exist or is difficult or impossible to access. Where data can be accessed, there are often problems with the quality of the data or accompanying documentation as well as frequent difficulties in linking relevant datasets.
There appear to be many barriers to effective data sharing – individuals and data subjects have data privacy concerns (which are comprised of genuine issues as well as artificially inflated fears stoked by an omnipresent media ever searching for the next story); government departments fear being on the receiving end of one of these media stories should any of its (shared) data be mislaid; while commercial organizations and academic institutions generally want to utilise data first and foremost for their own benefit. My, somewhat cynical, observation is that everyone is keen on data sharing until it comes to their own data.
It’s not all doom and gloom though. The government’s Open Data and Big Data initiatives combined with the Administrative Data Service (ADS) promise to make useful datasets accessible to the research community. Academic institutions are focusing on research data management while many research councils are mandating data management plans as part of funding proposals that include long-term archival and sharing of data.
This is a short, and probably sketchy, summary of many discussions I’ve had over the last 12 months. To gain a better understanding of data issues and requirements, it would be extremely valuable to get input from other practitioners in the energy demand sector (and other related sectors). Please take a few moments to complete a short web survey about data and your work.
