All posts by Andy Dudfield

Off to Scotland we go

It will come as no surprise to readers of this blog to hear that the Office for National Statistics is heavily involved with working in and around large amounts of data, which is why we are proud to once again sponsor Open Data Camp.

As we get ready for the trip to Aberdeen, ONS Chief Publishing Officer Andrew Dudfield, explains how ONS Digital are working to make more data more open.

The organisation is, in part, formed around a large data production process. We initially collect information. This has been traditionally collected via surveys filled in by individuals and business and is over time being augmented via a wider range of other data sources that help us better understand the world we are measuring.

Once we have the data, we need to store and process it. Again, this is an area of immense change. Not so long back, this would be in processing paper forms so that we could apply statistical methods to the results. We are now utilising a wide range of large scale data storage solutions to ensure the widest range of well organised, well labelled, consistent data is available to our teams of experts to work with and offering them a wider range of technical tools than ever before to unlock the insight within this. That insight is, in part, what is then published as statistics.

The publishing of these statistics is also transforming. We are trying to move from a world of inconsistent excel documents published on the pages of our website, to forming an openly available layer of data that anyone can easily access and use to answer a wide range of questions.

Within that, the word ‘open’ is important. We want to publish data that is open for as many people as possible to use. The use cases for this are near endless, but could include automated fact checking of statements, using the data to augment the responses given via voice controlled interfaces, producing automatically updating charts containing the latest data we publish and so on and so on.

We have started this journey by producing some Beta functionality on our website and via a new API. These are the thin edge of a much wider project and one we want to evolve in line with the needs of the users of this information. This, in part, is why I and a handful of my colleagues are heading to Aberdeen in November to attend the 6th Open Data Camp. This unconference has been touring the UK for the last few years, drawing together a group of likeminded folks who are interested in the importance and utility of the kind of open information the ONS is producing. We have sponsored the conference on several occasions and are pleased to do so again as we believe those attending and the communities of practice they represent are a vital audience in helping us deliver the best digital and data services possible.

Over the conference weekend we will be looking to learn from and with a range of professionals from a wide variety of backgrounds and will be happy to discuss what we are working on right now and share some thoughts on where we may go in the future. As we continue the transformation I described above, we will be continuing to hire digital and data staff across our three office locations (London, Newport and Titchfield) and I would be very happy to chat to people about some of the amazing roles we are hiring for as well

Looking forward to it.

Open Data Camp – 10 out of 10. Would attend again

This post was originally published on the ONS Digital blog

In October this year a group of like-minded folks will be meeting at Queen’s University in Belfast to chat about open data. They will be doing so under the banner of Open Data Camp, an unconference for those interested in making information from a wide range of sources “open”.

For those unfamiliar with the concept of an unconference, it is a format based more around peer-to-peer learning, creativity and collaboration.

In the context of this unconference, open can mean many things. In the depths of the technicalities of machine readable serialisation, the legality of data reuse, or how to convince your boss that making information available in only PDFs tends to be, shall we say, sub-optimal.

The reason I am writing this post here on @onsdigital is because I am pleased to say we are sponsoring Open Data Camp. We get asked to be involved in a lot of different conferences, but are not able to say yes to many, so I wanted to write a little about why this one is important to us and what we hope to gain from the weekend.

Primarily, we do this because we care about this community hugely and want to help ensure events like this can be financially viable. We also do it because ONS data being open is important if we are to ensure the greatest possible social and economic benefits for the public.

I’ve written about some of this over in my own blog, but don’t take my word for it, listen to Sir Tim during his Ted talk .

As part of this, Tim suggests:

“What you find if you deal with people in government departments is that they hug their database, hold it really close, so that they can build a beautiful website to present it.

I would like to suggest: sure, make a beautiful website, but first, give us – all of us – the unadulterated data. We have to ask for raw data now.”

I still regard this as pretty much the most important statement anyone has made about what the digital relationship between the citizen and the state is and what it could be.

Here at ONS we have been thinking about this a lot, have done some things that hopefully start us in the right direction for opening things up and know we have an awful lot more ground to cover.

We started with defining (with the help of Leigh Dodds) some open data publishing principles.

We are using these to inform the work we are currently doing on a project we are referring to as “Customise My Data”. This is a project to make some fairly fundamental changes to the way we publish data. The goals are to ensure that we move away from being an organisation that publishes excel and into one that publishes a consistent backbone of data, that allows users to breakdown our data into smaller parts and enables machine readable access to statistics (not spreadsheets)

I am hoping to pitch a session at open data camp around how we can make the data we publish through this system as open as possible and as useful for our users as we can make it.

A few final tickets for the event are going to be made available soon (it is free to attend) and I look forward to having some interesting conversations with those of you I get to meet over the weekend.

If you are unable to attend, but are interested in offering feedback on the work we are doing around open data at ONS, please get in touch with me directly, or register your interest to take part in the user research we are undertaking. We have upcoming sessions in London, Sheffield and Liverpool.

Photo available under Open Government Licence v3.0