Deathlines - a small peek into the BBC's archive taxonomy

Well, first post in a long time. I have a new job now and everything. But that's for another post, now I want to talk about a great internal hack day we had at BBC R&D's offices at Kingswood Warren, based around cool things we could do with the BBC's archive.

I wasn't even really planning on doing a hack, I just wanted people to see the cool data I had available from my work on the DMI data migration project, such as our Lonclass categorisation taxonomy (or thesaurus if you will) and our P4A production reporting data which shows incidental music etc.

But Simon Delafond (from BBC Online Media Group) and I were chatting and Simon mentioned that he would love to take information about important events in history and put them on a timeline. As an example Simon mentioned the death of Queen Victoria. That sparked an idea, and I showed Simon my copy of the Lonclass database, including -- sure enough -- an entry in the taxonomy about the death of Queen Victoria:



I didn't have the references to programmes to hand, but even the term from the catalogue on its own was useful, especially as we realised that the way the Lonclass data was constructed, we could extract all "death events" from the database. Note that "612.673" at the start of the subject, that's the Lonclass term for death! I could go into more detail on how those terms work, and if Dan Brickley has his way I will do exactly that some time, but for now suffice to say that searching this file for ">612.673" was enough to find all "death events" in the taxonomy.

But we had no way of mapping those events to an actual date that we could put on a timeline. So, like the good proto-semweb geeks that we are, we thought dbpedia would have that info. Tim Dobson, sitting across the table being very helpful with servers and stuff, suggested that we use YQL and a short while later we had a script that took a name (Screaming Lord Sutch was out favourite for testing), performed a Yahoo search limited to wikipedia, took the link from the first result, turned it into a dbpedia resource, found the JSON version of the dbpedia page, and parsed out the "deathdate" from the JSON file.

From there, all we had to do was make an XML file with the results, and feed it to a nifty timeline flash app that Simon had commissioned when he was producer of the Memoryshare project on bbc.co.uk.

The results are in the screenshot above, and why not, I'll put them on my server for posterity.

So have a click around, and be sure to use the funky navigation tools on the left and right of the flash app. Remember I didn't write the app, I just provided the data! And I know the links don't work, but the data is all there in our huge Infax database, so one day we should be able to link to archive footage in this way.

I hope this can give a sense of what can be achieved with the amazing data we have on offer, a few pre-prepared tools, a few different minds being brought together, and a sense of mischief.

Thanks of course to Ant Miller for organising the day, John Z from the R&D Archive research team and Tony Ageh for sponsoring various aspects of the day, and all the hackers who turned up!

Innovating our way around the country

I'm lucky enough to be involved with the BBC Innovation Labs, run by Matt Locke and the BBC Creative R&D team (er I think they're still called that?)

The Labs are basically a way to involve the new media "indie" community in developing new ideas that meet various strategic goals of the BBC (eg how to reach an under-served audience, or how to exploit a particular type of content we own, etc), giving them a bit of cash to sit with us for five days and work through an idea, to which we get first rights but non-exclusive access.

Along with Mark, Ian and Matt, I've been asked to come along to the intro sessions to talk about the BBC's web infrastructure. The idea is that agencies will have an idea of how their pitches will run on our architecture and what they will have to integrate with. Going by some of the demos we've seen recently, a lot of agencies don't seem to think they would have to integrate with anything, so getting the word out has to be a good thing.

It also helps that we can talk about what will be coming in the next year or so as the "BBC 2.0" project kicks off... we will have a nice scalable service-oriented architecture for people to build upon. Well, that's the plan anyway.

It raises a few interesting points:

• Where are the techy agencies?
  It seems that most of the people pitching to us have been "new meedja" agencies in the sense that they are used to building corporate web sites for corporates, with minimal technical input. But if we want some solid recommendation software either built or integrated, or a message queuing system integrated and supported, where do we go? We've got the money to spend... (well, we will have soon, if all goes to plan..!)


• How to make it worth an agency's while to build for our architecture?
  We want people to build tools that will integrate with our single sign-on, ratings systems, file stores, social networking / friends systems, etc etc... but if they build exclusively for our needs, they won't be able to re-sell their products to others. So either (a) that's fine but we have to pay more for the work, or (b) we have to come up with a compelling offer to these guys. One way might be to create open source implementations of each of our components, and make them all freely available to be integrated into other products (under, say, the Apache license). But as at least part-funders of these products, wouldn't licence fee payers expect to see a share in any future proceeds of products built on our shared infrastructure?
  
  Another way would be to exclusively use shared standards and make sure an open source product already exists for every component we use. That might be fine for SSO (using, say, JAAS) but I can't see it working for some of our more obscure components.

Lots of food for thought... which is after all what the innovation labs days are all about.

Publishing our metadata in HTML

In the BBC English Regions web CMS I've been involved with building, we have six controlled vocabularies for descriptive metadata (location, name, subject, audience, BBC brand, time period), and we manage about 85,000 terms in those CVs. We sourced some of the CVs from external suppliers and modified them, some we built from scratch. We also have a team that maintains the existing terms and allows our journalists to suggest terms and have them vetted and finished off by the metadata specialists.

A few of us have some ideas around publishing out this metadata, we just have to run it past search to check what format we should use. I’m hoping we can just do

<meta name=”location” value=”BBC/C/Devon,BBC/C/London”>

etc. That would be very useful, in the sense that people could pick it up and do amazing things with it... maybe we can do that stuff ourselves, but even if we don’t, the data is there ready for someone else to play with.

As well as the text-based names, we have the lat/long data for those locations in an XML file somewhere, I might be able to get that extracted and put into the HTML as well. We aren't allowed to publish the complete location CV due to licensing restrictions, but we can extract pieces of it for certain purposes.

We haven’t really decided on the format yet, it might be worth making it RDF or something from the start -- although I’m not sure how useful that would be unless our content is all well-formed XML, and while we’re pretty close, I don't think we've sorted out all the issues of & in URLs etc.

Thoughts on a BBC API

After seeing some cool newsy metadataey things recently like aaronland's New York Times widgets and Stef's BBC News wikipedia thing, and talking to Tom L about some of his grand ideas, I've been thinking more about where the BBC should be in the semantic web, the lazyweb, etc etc... and in the end I figure we at the BBC should be able to build the content equivalent of Google API or Amazon API.

Basically, Google provides a bunch of web services (via SOAP and REST) and exposes their core functionality to developers around the world. They provide the engine, developers can build their own interface, as long as they follow a few rules. Amazon does the same thing, although of course they're not being nearly as nice as Google because they're really just opening up their shopping cart to others, they still get the money from any sales.

So if Google can provide the definitive search API that everyone can use, and Amazon can provide a book buying API, what could the BBC offer? The list is endless...

• GIS services: postocde<->location name<->lat/long mapping


• TV/radio listings services: what's on today, what's on this week, in a particular genre, on a paricular network, what's on right now


• Search services: BBC search via API


• metadata services: classification engine??, list of terms, integrate with BBC Search to show stories/pages that match metadata terms


• information services: weather, traffic info, financial data...

A simple application of this would be something that takes a person's postcode, uses the postcoder API to translate it to a lat/long, then uses the BBC search to find all content marked with locations within 10 miles, and presents a custom page for that user. Of course that's just the beginning!

Graf review on content management

The Graf review on the BBC's online offering was finally released today, after 18 months of development.

So what does it have to say about content management at the BBC? Some interesting points...

(Just notes-to-self for now, I'll come back and write about individual points later)

Page 20 of the main review: "The BBC has also employed a number of technical solutions to enable them to deliver news and information content quickly and effectively, over a number of platforms. The development of the News and Sport Content Production System (CPS) represents a significant investment in such technical solutions, with total costs amounting to £3.8m in 2003"

A constant theme is value for money, which resonate's well with Mark Thompson's recent prouncements to make all BBC services pass a "public value test". The term "cost per unique user", "CPUU" is referred throughout, obviously meaning the cost of content production divided by number of unique users. It will be interesting to see this statistic for a lot of our content. According to page 21, CPUU for weather is 1.8 pence and Nations and Regions (excluding Northern Ireland) is 20.5 pence. It will be interesting to see if this figure changes after our CMS has finished rolling out to the English Regions sites...

Page 21 again, "the BBC is currently developing the technology to track traffic exclusively to external sites." Well I thought the /go/ system would do that, but if you say so... anyway in the CMS, adapting our external weblinks to use a system like this (or to move from one such system to another) will take about 20 seconds of XSLT coding.

And more on page 21, "links from sites or pages are not... measured across the site." Another easy peasy thing for us to do with the CMS...

Page 22, "The total costs of the launch of the
search engine (worldwide and BBC site) were £414,268, and the current running costs of the
BBC Online search tool are £476,000."

Documentum specifically gets a mention on page 24!

Page 27, "Applications developed by the BBC, such as DNA have also enabled user-generated content to
be more stimulating for the user and more efficiently managed. The current growth in web log usage also allows users to contribute richer content (e.g. to news stories) in the form of text, pictures, and audio and video clips."

Page 32: "BBC Online’s ambition to syndicate online content to other providers (for example, free and non-exclusive arrangements for commercial websites such as www.streetmap.co.uk to carry BBC news headlines), and to other devices (for example, free and non-exclusive arrangements with mobile providers to ensure position of BBC Online on WAP versions), is again, a means to drive towards 100% reach. Reach is a key means to ensure that increasing numbers of licence fee payers can derive some value from the BBC’s online services. This strategic goal does, however, risk the BBC being perceived by commercial operators as an aggressive, and unfairly advantaged competitive force. Submitters to the review also argued that the BBC’s current
inconsistent approach to linking, the prominence of BBC Online results in its search engine, and the low level of joint venture or externally commissioned projects have compounded this
impression."

Page 35 has a graph of BBC Online Expenditure, charted over the years... the division with the most expenditure is of course News with 15.5m in 2003/04, second is Nations & Regions with 11.6m and third is Factual and Learning with 10.1m.

That'll do for now, more on the technology assessment appendix tomorrow :-)