Deathlines - a small peek into the BBC's archive taxonomy

Well, first post in a long time. I have a new job now and everything. But that's for another post, now I want to talk about a great internal hack day we had at BBC R&D's offices at Kingswood Warren, based around cool things we could do with the BBC's archive.

I wasn't even really planning on doing a hack, I just wanted people to see the cool data I had available from my work on the DMI data migration project, such as our Lonclass categorisation taxonomy (or thesaurus if you will) and our P4A production reporting data which shows incidental music etc.

But Simon Delafond (from BBC Online Media Group) and I were chatting and Simon mentioned that he would love to take information about important events in history and put them on a timeline. As an example Simon mentioned the death of Queen Victoria. That sparked an idea, and I showed Simon my copy of the Lonclass database, including -- sure enough -- an entry in the taxonomy about the death of Queen Victoria:



I didn't have the references to programmes to hand, but even the term from the catalogue on its own was useful, especially as we realised that the way the Lonclass data was constructed, we could extract all "death events" from the database. Note that "612.673" at the start of the subject, that's the Lonclass term for death! I could go into more detail on how those terms work, and if Dan Brickley has his way I will do exactly that some time, but for now suffice to say that searching this file for ">612.673" was enough to find all "death events" in the taxonomy.

But we had no way of mapping those events to an actual date that we could put on a timeline. So, like the good proto-semweb geeks that we are, we thought dbpedia would have that info. Tim Dobson, sitting across the table being very helpful with servers and stuff, suggested that we use YQL and a short while later we had a script that took a name (Screaming Lord Sutch was out favourite for testing), performed a Yahoo search limited to wikipedia, took the link from the first result, turned it into a dbpedia resource, found the JSON version of the dbpedia page, and parsed out the "deathdate" from the JSON file.

From there, all we had to do was make an XML file with the results, and feed it to a nifty timeline flash app that Simon had commissioned when he was producer of the Memoryshare project on bbc.co.uk.

The results are in the screenshot above, and why not, I'll put them on my server for posterity.

So have a click around, and be sure to use the funky navigation tools on the left and right of the flash app. Remember I didn't write the app, I just provided the data! And I know the links don't work, but the data is all there in our huge Infax database, so one day we should be able to link to archive footage in this way.

I hope this can give a sense of what can be achieved with the amazing data we have on offer, a few pre-prepared tools, a few different minds being brought together, and a sense of mischief.

Thanks of course to Ant Miller for organising the day, John Z from the R&D Archive research team and Tony Ageh for sponsoring various aspects of the day, and all the hackers who turned up!

Events ML

Something I hadn't noticed before... IPTC EventsML Working Group.

Right now "The project scope is limited to the definition of an information interchange standard for newsworthy event information", but I'm sure they will expand the scope eventually to handle other types of events... I haven't looked into it yet but I hope they're thinking about integration with iCalendar...

Publishing our metadata in HTML

In the BBC English Regions web CMS I've been involved with building, we have six controlled vocabularies for descriptive metadata (location, name, subject, audience, BBC brand, time period), and we manage about 85,000 terms in those CVs. We sourced some of the CVs from external suppliers and modified them, some we built from scratch. We also have a team that maintains the existing terms and allows our journalists to suggest terms and have them vetted and finished off by the metadata specialists.

A few of us have some ideas around publishing out this metadata, we just have to run it past search to check what format we should use. I’m hoping we can just do

<meta name=”location” value=”BBC/C/Devon,BBC/C/London”>

etc. That would be very useful, in the sense that people could pick it up and do amazing things with it... maybe we can do that stuff ourselves, but even if we don’t, the data is there ready for someone else to play with.

As well as the text-based names, we have the lat/long data for those locations in an XML file somewhere, I might be able to get that extracted and put into the HTML as well. We aren't allowed to publish the complete location CV due to licensing restrictions, but we can extract pieces of it for certain purposes.

We haven’t really decided on the format yet, it might be worth making it RDF or something from the start -- although I’m not sure how useful that would be unless our content is all well-formed XML, and while we’re pretty close, I don't think we've sorted out all the issues of & in URLs etc.

Thoughts on a BBC API

After seeing some cool newsy metadataey things recently like aaronland's New York Times widgets and Stef's BBC News wikipedia thing, and talking to Tom L about some of his grand ideas, I've been thinking more about where the BBC should be in the semantic web, the lazyweb, etc etc... and in the end I figure we at the BBC should be able to build the content equivalent of Google API or Amazon API.

Basically, Google provides a bunch of web services (via SOAP and REST) and exposes their core functionality to developers around the world. They provide the engine, developers can build their own interface, as long as they follow a few rules. Amazon does the same thing, although of course they're not being nearly as nice as Google because they're really just opening up their shopping cart to others, they still get the money from any sales.

So if Google can provide the definitive search API that everyone can use, and Amazon can provide a book buying API, what could the BBC offer? The list is endless...

• GIS services: postocde<->location name<->lat/long mapping


• TV/radio listings services: what's on today, what's on this week, in a particular genre, on a paricular network, what's on right now


• Search services: BBC search via API


• metadata services: classification engine??, list of terms, integrate with BBC Search to show stories/pages that match metadata terms


• information services: weather, traffic info, financial data...

A simple application of this would be something that takes a person's postcode, uses the postcoder API to translate it to a lat/long, then uses the BBC search to find all content marked with locations within 10 miles, and presents a custom page for that user. Of course that's just the beginning!