AP Digitisation project announced

It has been a while since my last blog post, so I thought I should try to get back into letting the world know what I've been up to.

From May 2011 to November 2013, through my company Clueful Media, I was consulting at the video arm of the Associated Press. Most people aren't aware but while all the text and images work of the AP comes out of New York City and New Jersey, the video side is based in Camden, North London.

In 2011, AP kicked off a huge programme of work called Video Transformation, recognising that video is one of the few remaining areas of growth in the news industry, and ensuring that AP can retain its leadership position in news sourcing and distribution.

It has been a huge success, and I was involved in two parts of it: the AP Video Hub and the Digitisation project.

Another small part of my work there was managing the transcoding of the British Movietone archive after AP acquired the distribution rights. In 2015 the entire Movietone archive was uploaded to YouTube (in low resolution) and the high res versions are available from http://www.aparchive.com/.

Deathlines - a small peek into the BBC's archive taxonomy

Well, first post in a long time. I have a new job now and everything. But that's for another post, now I want to talk about a great internal hack day we had at BBC R&D's offices at Kingswood Warren, based around cool things we could do with the BBC's archive.

I wasn't even really planning on doing a hack, I just wanted people to see the cool data I had available from my work on the DMI data migration project, such as our Lonclass categorisation taxonomy (or thesaurus if you will) and our P4A production reporting data which shows incidental music etc.

But Simon Delafond (from BBC Online Media Group) and I were chatting and Simon mentioned that he would love to take information about important events in history and put them on a timeline. As an example Simon mentioned the death of Queen Victoria. That sparked an idea, and I showed Simon my copy of the Lonclass database, including -- sure enough -- an entry in the taxonomy about the death of Queen Victoria:



I didn't have the references to programmes to hand, but even the term from the catalogue on its own was useful, especially as we realised that the way the Lonclass data was constructed, we could extract all "death events" from the database. Note that "612.673" at the start of the subject, that's the Lonclass term for death! I could go into more detail on how those terms work, and if Dan Brickley has his way I will do exactly that some time, but for now suffice to say that searching this file for ">612.673" was enough to find all "death events" in the taxonomy.

But we had no way of mapping those events to an actual date that we could put on a timeline. So, like the good proto-semweb geeks that we are, we thought dbpedia would have that info. Tim Dobson, sitting across the table being very helpful with servers and stuff, suggested that we use YQL and a short while later we had a script that took a name (Screaming Lord Sutch was out favourite for testing), performed a Yahoo search limited to wikipedia, took the link from the first result, turned it into a dbpedia resource, found the JSON version of the dbpedia page, and parsed out the "deathdate" from the JSON file.

From there, all we had to do was make an XML file with the results, and feed it to a nifty timeline flash app that Simon had commissioned when he was producer of the Memoryshare project on bbc.co.uk.

The results are in the screenshot above, and why not, I'll put them on my server for posterity.

So have a click around, and be sure to use the funky navigation tools on the left and right of the flash app. Remember I didn't write the app, I just provided the data! And I know the links don't work, but the data is all there in our huge Infax database, so one day we should be able to link to archive footage in this way.

I hope this can give a sense of what can be achieved with the amazing data we have on offer, a few pre-prepared tools, a few different minds being brought together, and a sense of mischief.

Thanks of course to Ant Miller for organising the day, John Z from the R&D Archive research team and Tony Ageh for sponsoring various aspects of the day, and all the hackers who turned up!

My response to the Canvas PVT

I'm not sure whether I'm allowed / supposed to post a response to a Public Value Test by the BBC, but here's what I had to say about the Canvas PVT:

Speaking as someone who is considering starting up companies in this area, I think that this is a fantastic opportunity to create new markets and encourage a new wave of development and innovation in small businesses, in a similar way to how Facebook applications and the iPhone AppStore have created a new wave of companies exploiting those platforms by satisfying user needs.

In particular, the Canvas platform should offer a common payment mechanism so that people can download content and/or applications for the set-top box in a manner as simple as buying an application on the iTunes AppStore on an iPhone. eg you could pay 50p and get a great little TV-based application for seeing ski conditions on your favourite slopes, or a game, or an interactive TV-watching tool, etc. Either the Canvas JV itself could manage the payment interactions, or it could create a payment standard that is handled by the ISPs providing the broadband connections (users already have a financial relationship so this would be easy to manage, and would give ISPs a new revenue stream by taking a percentage of the fees paid).

Another area where innovation would be key is the EPG -- the Canvas system should allow for "pluggable EPGs" so people could choose to change their EPG, possibly using the payment model described above to purchase an EPG that exists in a 3D world, or whatever crazy ideas people think up!

BBC Backstage has taught us that without a commercial incentive, people building tools for BBC services are limited to hobbyists and dabblers, and user needs are not met in any mainstream way. Introduce a potential revenue model, and innovation flourishes. Canvas is a perfect opportunity to make this happen.

A thriving Canvas ecosystem could also encourage other countries to adopt the Canvas standards, giving the UK a lead in an important new market around the world.

[Also added in response to their questions about draft market analysis]

I don't believe that "freesat and freesat from skypenetration remain broadly constant" (annex 1.4.2.1 p28), wouldn't they switch over to Canvas Freesat?

You only seem to have covered threats/substitutes from pay-TV providers, what about Microsoft/XBox 360, Apple TV, IP-only boxes like Roku and Boxee using internet services such as the Netflix API in the US? They are already growing rapidly. Also of course there are the TVs with direct internet connections, most of which are coming out of Asia working with the US, eg embedded YouTube. If the UK doesn't respond to this emerging market, people may end up watching more US-originated YouTube content (much of it of dubious legal standing) rather than UK-originated services.

The radio market may be affected when people can use IP-connected TVs to listen to radio from around the world, or services such as Last.fm or Spotify through their TVs in surround sound etc. This is negative for incumbent radio stations but a huge opportunity in general.

QCon SF 2008 Day Three - nothing like a timely post

I mean that literally. This is nothing like a timely post. Still, better than no post at all, right?

Day Three was my volunteering exercise -- hey I was a starving student at the time, so I got a free student rego in exchange for helping out with conference organisation, collecting feedback forms, making sure the speakers were actually in the rooms speaking to people etc -- not a bad gig really if you choose the right room. I missed out on the "architectures you've always wondered about" track but I heard most of those guys in 2007 anyway, so I chose the Data Storage Rethinking: Document Oriented Distributed Databases track which I was very happy about -- it was fascinating and very useful for me.

Some notes only barely converted from my rough typing in between pressing the little clicker to count people going in and out of the room:

Hypertable

• A column-based bigtable clone
• GPLed
• Stores history of everytyhing– even deletes are just stored as new entries with a flag
• Splits tables automatically across machines if you need to
• Instrumentation for monitoring etc not there yet for 1.0 (jan/feb next year) (note as of blog posting date: it's at 0.9.2 right now, getting there...)
  
• In 1.1 master-slave communication will work much better, including intelligent resource allocation
• Keeps a write-ahead commit log as well as the data store, so can recover from failure if written to a distributed FS
• Has "Hyperspace" distributed lock manager – equivalent to “chubby” at google (whatever that is?! presumably somebody reading this knows...)
• -> currently a SPOF but will have “some form of replication” by release
• Can run on any distributed FS: hadoop HDFS, KFS (Kosmos FS) etc
• All communication is asynchronous
• Languages: C++ plus Thrift bindings which will expose java, python, PHP etc... Release containing this stuff will come out in a few weeks
• Concurrency: “it uses MVCC”, he skipped it.. What does this mean?? (Wikipedia tells me it's "multi-version concurrency control" which is used by CouchDB, BerkeleyDB, MySQL/InnoDB etc)
• Achieved over 1m inserts/sec on AOL test data (1TB of 30-byte query log rows -- ie almost pathological but good for certain use cases)
• Google has “megatable”, abstraction layer on top of bigtable — hypertable will have an equivalent eventually
• Have their own communication protocol
  

Atomserver

• From the guys behind homeaway.com – Bryon Jacob and chris Berry
  
• Took Abdera from apache to build their own framework
• Added Atom Publishing Protocol extensions, eg
• open search - google
• paging – mark nottingham - rfc5005
• “atom store” - get, put, edit, search via APP – canonical example is gdata
• Uses Abdera which graduated from the Apache incubator this week and will go 1.0 very soon
• Provides a solid, scalable, etc implementation on top of Abdera
• APP Spec doesn’t force you to make services and workspaces first-class objects with own RESTful interfaces and URIs, but they do anyway
• POST for new content where you let the server assign the ID, or PUT if you know the URI you want
• Uses model of starting at the beginning and following next links to get everything (a la GData, I think..?)
• Incrementing index numbers for all changes, so you can see things twice, as each changes gives the item a higher inde number, good for syncing eg queues
• Has APP categories (aka tags)
• Can create tags specifically for items using category docs
• Can create hooks for auto-categorisers
• xpath one built in, can use to extract standard tags from custom XML into category tags for querying later
• view feeds by category, atomserver specific but based on gdata implementation
• can do boolean ANDs and ORs of tags, to do a vague equyivalent of SQL SELECT queries
• Concurrency for edits: each edit must have the revision number appended to the URI for optimistic locking – if that’s not the correct revision, it is rejected (409 CONFLICT)
• link rel=”edit” URI has the revision number built in
• Example of “atom-based service architecture” using objects with states — eg could do a moderation service by querying for a feed of objects with “UNMODERATED” tags
• “etags are the preferred way to pass query parameters to atom” -- I wrote it down but I don't really know what they mean?! I thought etags were about caching?!
  
• Aggregate feeds ability (in AtomServer only) - “we join on categories in the same way that SQL would join tables based on a column”
• Batch updates via one feed doc — (I thought mime multipart was supposed to be used for that??)
• Custom (pluggable) content storage – only supports RDBMS now but planning to support key-value stores (eg couchdb) later (we at the BBC are very keen to see this happen!)
  
• Scales with multiple front-ends using one database – can replicate etc but still requires one sql database (for now)
• I think I heard them say at the end that they don’t support mysql because they use transactions!?!? would be good to know more about that...
  

neo4j

• it's a graph database
• started off talking about growth in connected data, eg Facebook’s MySQL store: “facebook has hundreds of machines with 1TB RAM to keep their entire database in memory”
• At first it sounded silly and just a replacement for RDF but when I could see that they can do depth-X pathExists() searches eg (friends of friends of people), 2ms for 1m people with average 50 connections, ie 25m connections! eg haven't you always wondered how LinkedIn could always say how many degrees away from each person you are when you do searches? that's hard! (I'm not saying that they use neo4j at LinkedIn, but they must use some similar algorithms -- I think they keep most of their social graph state in memory as well, from what I remember hearing at the QCon architectures track in 2007)
  
• Has a NeoMock in-memory implementation for testing, but you can just put lots of RAM to your JVM and it uses memory for you
• neo4j now has sparql support! v interesting
• they are working on NeoRDF – have two customers but haven’t released as a product yet, it's coming in 1.1
• They use the OSGi architecture for plugins -- it seems to be becoming a real standard now
• They are thinking about releasing a standalone server, REST API etc
• but they say that exposing domain-oriented services is better than exposing the database over the wire – as Ian yesterday was describing, Eran calls it “terrorist-oriented architecture” -- ie independent cells all capable of surviving on their own -- the extreme case of "small pieces loosely joined"
• "it's hard to think of a good REST API for something as chatty as we are"
• what's coming in v2.0? they are thinking of sharding (aka partitioning) on top of newton (infiniflow) from paremus
• based on CAP theorem, BASE rather than ACID -- ie everything is synchronised eventually -- if you don't get this then google it, there are loads of presos about it
  
• licensed under AGPLv3 – if you develop software with it it’s free, but if you use it to store more than 1m primitives, you have to pay
  

Couchdb
jan h, jan@apache.org

• has the same optimistic locking approach as atomserver, keeps all revisions – ie nothing is locked, ever
• uses mapreduce (with javascript as the scripting language!) for views, aggregations etc rather than inventing a new query language
  
• is slow the first time, as it parses the javascript etc, builds a btree index
• next time you query, it checks if anything updated and if so, gives the diff to the view server and builds a new diff with just the new data
• therefore inserts are cheap, you only rebuild views when they are queried again (and even then only incrementally)
• does syncing between DBs (based on lotus notes?!), one direction or bi-directional
• books.couchdb.org/relax – drafts coming out in jan (looking now, they might be running a bit behind... but some intro chapters are up at least)
  

Couchdb in the real world
Jan again (who is available for consulting BTW, apparently we at the BBC have already employed him at least once!)

• lots of standard storage design patterns change in a couchdb world...
• eg views (sort of like stored procedures, but using map/reduce)
• views are saved in “design documents” where your javascript goes
• you can have multiple views in the same design document, but realise that they’re all updated each time any data changes
• also put validation rules, authorisation, and more into a design document
• CouchDB has no such thing as sequences, but you wouldn’t want to use sequences in a distributed env anyway – you have a system field called _id but that’s not a guaranteed sequence
• if you do want to order your results, do it by a natural key such as time rather than some sequence id
• CouchDB provides no transactions, no roundtripping, no multi-node transactions (they would be too expensive) - "use an http proxy if you need redundancy"!!? (syncing helps with that I guess? but how resilient is it really?)
• Can have master-slave or master-master replication setups, eventually consistent but BASE, not ACID (see notes above)
• You can add as many masters as you like, unlike mysql
• Replication communication also happens over HTTP so you can use caches, proxies etc
• because it’s all asynchronous, you can actually call couchdb directly via ajax

Phew, long and disjointed, sorry about that, but it's enough to get my notes down, I might expand on these topics later as we explore these technologies some more!

Hope it's useful to somebody.

Interview with Vinod Khosla!

Presumably those reading this blog already know that I just returned from an exchange term at the Haas School of Business, UC Berkeley. One of the more fun subjects I studied was Venture Capital and Private Equity, taught by the inimitable trio: Jerry Engel of Monitor Venture Partners, Terry Opdendyk of ONSET Ventures, and Sean Foote of Labrador Ventures, who after seven years teaching together had all their lines honed to a tee but still played off each other, like a good standup act.

We were given an assignment to go and find a real VC or two, an interview them. We were quite proud of ourselves by managing to interview Silicon Valley scion Vinod Khosla, co-founder of Sun Microsystems, partner at Kleiner Perkins during their glory years of the dot-com boom, and proponent of all things clean-tech at his current venture, Khosla Partners.

For posterity, I thought I would cut and paste our assignment as I think we asked some pretty good questions, considering we only had five minutes with him!

Interview date: 3 October 2008 (a brief 5 minute chat before Khosla's presentation at Berkeley Labs)

How do you think the venture capital industry will change as a result of the flattening world?

I think the emerging world gives us more points of innovation, there are many smart people in the world and we can be open to all of them — the pool of talent is larger. We can cultivate local venture capitalists all around the world.

We noticed that most of your investments have been in the US — why is this?

Venture capital is a high-touch industry. You need to be close to your businesses to mentor them properly.

How do you reconcile that view with the idea that innovation is coming from everywhere? Are you saying that, to be successful, an international business has to start an operation in the US?

Being in the US increases your probability of success by ten times. The opportunities are all here; the venture industry is all here. You have to be here to be a part of it.

Do you think that this will change as a result of the financial crisis and the changing economy?

The VC industry won’t change much, it might be smaller in a few years but in the short term nothing will change. Hopefully the economic situation will mean that we move away from synthetic goods, and back towards physical goods that make a difference.

What do you mean by synthetic goods? Do you mean moving away from software?

I mean financial instruments, investment bankers making things up rather than creating things that actually add value to the world.

Some Agile notes from QCon SF 2008

I'm not the most excited person about Agile (hey if it gets the job done better and quicker then great, but that's not going to turn me into a religious fanatic) but I saw some interesting stuff about Agile today so I should report.

Some guys from salesforce.com spoke about how they migrated 50 teams, over 700 people, to Scrum -- all at once! Their rationale was that it was "like burning their boat after they had rowed to the other shore", and if some teams moved and others didn't, it would be a recipe for finger-pointing, missed dependency deadlines, and recriminations. Coupled with the fact that they all commit to the same codebase (!!) it sort of makes sense.

Notes:

• Had to spend a lot of time training product owners
• Now 75% test coverage
• They have an “emergency brake” feature — if the build breaks, fixing it becomes number one priority, they don’t just roll back or deal with less coverage (this comes straight from Toyota's production system, where their assembly-line staff can stop the whole production line if they detect a problem coming from the previous workers, and all focus shifts to fixing the problem -- which usually only delays the production process by a minute or so)
• They had lots of pain, people complained a lot — but they got over it in a few months
• Phases:
  • rollout (introduced tools – own app on force.com; office hours idea for UX/documentation people)
  • adoption( release planning, sustainable velocity reducing overcommitting)
  • excellence (moved systems testing to iterative approach, scrum of scrums as issue discussions rather than status reports, dependency tracking between teams)
  • expansion
• Focus this year is working with customers and partners and helping them go agile
• Moving to IT and operations going agile..!
• Some teams use TDD, some have QA people, no one fixed type
  
• Systems testing people work with high-risk code first (which is identified at architecture stage)
  
• They have short sprints with "releases" every month, but they only do a release three times a year, and they have a non-agile month at the end of each cyclefor testing and deployment (a "release sprint")
  
• They use continuous integration even for db schema changes

What they would do differently:

• involve more individuals earlier: eg openspace/unconference meeting
• earlier, more intense training
• more coaching
• giving concrete deliverables to executives during the rollout, get them engaged

Keys to success

• exec sponsorship and commitment
• focus on principles rather than mechanics (eg not necessarily daily scrum standups etc)
• focus on making okay teams excellent to get standard-bearers, rather than working with worst teams - “know which fires to let burn and which to fight”
• radical transparency – trust.salesforce.com has uptime figures – accountable to customers
• what the heat is on, stick to your principles
• we failed – all along the way – we experimented, were patient and expected to make mistakes
• having multiple teams helped — it’s not always agile that makes teams fail, so higher number of teams gives more chance of success
  

Slides (but if you search slideshare.net for "salesforce adm" you'll see many many more presentations -- they've obviously been milking this one for a while!)

Also on Agile, I'll report a few quotes I heard from the silicon valley agile users group, which had some acronym I missed. They were having a meeting in the main room while I was doing my email, and although it started off pretty dull, they soon got animated and were saying some very intelligent things about how people were getting too caught up in the ceremonies of scrum (chickens and pigs, standups, only three questions etc) and they were forgetting that the whole point of the manifesto is to be agile, to change things when you need to, and to deliver customer value and business value. It gladdened my heart!

Some choice quotes:

• (When a prominent agile coach was asked how to sell agile to upper management:) "I ran a large agile project for (a very large organisation) for two and a half years... the whole time my boss thought it was a waterfall project"
• “agile can be gamed, just like anything else”
• "If you measure velocity by how many tasks you complete, people will complete all the easy but useless tasks. We shouldn't be measuring task velocity, we should be measuring business-value velocity... people do what they are measured by" [the mantra our VC professor Terry has drilled into us: structure drives behaviour]
  
• “we’ve gone from one dogma [waterfall] to another dogma [scrum and XP] -- I thought the whole point was that we were supposed to be agile. What happened to thinking?”
  
  

Mark Nottingham on "HTTP Status"

From the author of XML and AtomPub specs to the author of the other half of the picture, the Atom Syndication Format spec-- Mark Nottingham. Seems like a nice guy, he apparently lives in Melbourne now! Smart guy.

Quick notes...

HTTP/1.1 was basically only written to “contain the damage” of 0.9 and 1.0 (vhosting, persistence, caching)
Mark was involved with the WS-* stack -- but he graciously apologised to the room for his sins ;-) An interesting comment regarding SOAP etc was that “having that much extension available in a protocol is socially irresponsible - protocols are all about agreement" and you need to draw lines to make soething useful. He was basically saying that WS-* allows you to do too much, giving you enough rope, and making the normal case hard just to make an extreme case possible. (Or something like that, if there's a blog post where he explains himself I'll gladly link to it instead of badly paraphrasing him)

Mark had a neat way of saying that RESTful APIs "use HTTP as protocol construction toolkit”. They're not built on top of HTTP, they're build as part of HTTP (in a way).

HTTP 1.1 bis: With Roy Fielding and others, Mark is working on "HTTP 1.1bis", a rewrite of the HTTP spec to make it much easier to read, to resolve ambiguities, and to define edge cases that were missed in the first version (eg "what happens when you put an ETag on the response to a PUT"?!). All this sounds very esoteric but people are really pushing the boundaries of HTTP these days with streaming services, Comet and Ajax, etc, so it's best to resolve the differences now rather than wait for implementations to define behaviour (and possibly have two versions of what happens in these scenarios)

One question I was wondering is how they will market the new spec: if it's being sold as "just a rewrite to make it easier to understand", then people won't pay much attention, but if people start creating new web servers that are "HTTP/1.1bis compliant" then it's a new standard, not a rewrite, and might as well have some new functionality as well! It's not obvious how this will work.

Compatibility: Mark mentioned an interesting point in passing: that “an http/1.0 server can still possibly take 1.1 directives” -- with squid as the canonical example. Squid officially doesn't support HTTP/1.1 yet, but it actually supports most 1.1 directives and commands.

HTTP methods: convention wisdom says that intermediaries might reject PUT and DELETE verbs due to security concerns, old gateways etc, but Mark asserted that it doesn’t really happen in practice. Google created a workaround whereby they send everything as a POST and have an extra HTTP header, “X-HTTP-Request” (I think that was right?) to "pretend" to do a PUT or DELETE. A bit silly really, signs that things need to change!

URI length: IE still limits URIs to 2k in length. Squid limits headers to 20k. HTTPbis is going to recommend at least 8k.

Cache testing: Coadvisor is a test suite for intermediaries

Headers/trailers: Most web programmers know how annoying it is to have to set all HTTP headers before you output any text. So they're thinking of “trailers” as well as headers to the envelope of your payload. This could be really useful.

Something about a 307 redirect for POST – not handled by safari... I kinda missed that bit?

Request-side cache control isn’t well supported – eg the act of posting to a blog should be able to invalidate the cache. Http currently has request cache control: eg “I’m okay with this being up to X seconds old”

Request pipelining – not supported except in safari, but would be v useful if it worked

Data ranges: need to be better supported. We should be able to jump to a section of a video etc without putting query params in the URI (although one thing you get from a URI is addressability, which shouldn't be overlooked in the quest to make things neat from an architecture perspective...)

OAuth: the IETF/OAuth BOF working group the other day went well, it could have been a culture clash but the "grey hairs" were visibly excited by the enthusiasm and drive of the OAuth guys (I guess this was Chris Messina, Eran Hammer-Lahav etc) and it ended up being "a bit of a love fest". So OAuth looks like becoming an IETF standard. Let's hope that means HTTP authentication improves a lot as a result.

New transport protocols: Looking at HTTP over SCTP, a streaming protocol I am not really familiar with. Mark is thinking of proxy-to-proxy overlays: one point-to-point many-streamed SCTP connection being muxed/demuxed to TCP at the edges.

Prefer header: is in internet draft now. More than content negotiation that lets you choose languages, encodings etc, Perfer lets you ask for semantically different content, eg only summaries or only pictures.

Typed links: making a comeback: “this invalidates X, the previous one is Y, edit this at Z” - similar to what Atom does with prev/next/edit etc. There will be a controlled list of types based on URIs (very semweb, which is nice), using the registry which already exists for Atom. (Mark didn't mention that he is the author of the internet-draft!)

What does the future hold...?

• libraries of “higher-level but still RESTful abstractions,” ie systems that let users - webmachine is an example
  
• Rack::Cache extends HTTP libraries to provide a better cache implementation built in to Ruby, he hopes to see them in other languages and frameworks soon
  
• Building blocks for intermediaries, so people don't have to extend Squid every time they want to build some kind of intermediary system – eg xLightweb (Java)
• The “O2.0” stack -- openid, oauth, comet, html5, gears etc -- “fail to consider overall architecture” - “cowboy development on the web” - “new pseudo-standards” - basically he wasn't very friendly to them! But I think he is much happier now that Messina etc are working with the IETF.
  

Link: http://tools.ietf.org/wg/httpbis/

QCON SF 08: Tim Bray on storage and persistence trends

The keynote this morning was Tim Bray, a bit of a guru in the unix and web development scene, who helped write the original XML spec among many many other things.

Some quick notes before getting to the meat of his talk:

• The Drizzle db project is worth keeping an eye on – a key mysql committer forked mysql's code to focus on the most minimal sql engine possible — one date type, one float type, no triggers, as simple as possible, but fast and reliable. The idea is so compelling that people are apparently running it in production even though it's barely in alpha!


• “Column oriented databases” are about to see their day – BigTable in Google AppEngine is probably the best indicator at the moment


• CouchDB has just become a top-level apache project – the author is now employed by amazon, and the project is going really well. I know Dirk and the guys back at the ranch have been looking at the product so this is good news. Some quick CouchDB facts:
  • REST-based, built in Erlang
  • uses the “eventually consistent” model
  • it has a nifty way of using MapReduce functions on the server to do views! (which could even be adapted to do "stored procedure" type functionality I guess)
  • HTTP is only access protoccol - “the most debugged protocol on the internet”
  So it sounds like CouchDB is here to stay. Good news.


• Atompub vs WebDAV: performance is always questioned, but Bray is building an atompub server apache module, mod_atom that seems to perform pretty well, and he hasn't even started optimising it yet. Sounds like mod_atom is something else to keep an eye on.


• Facebook gets 90,000 transactions/sec using memcached! (that's good... very good)

The main guts of the talk was a walkthrough the different layers of storage required by modern computer systems, in order of performance:

1. registers on a CPU,
2. Local cache (l-cache) in the processor,
3. DRAM on the server,
4. distributed hash table accessed over a network (eg memcached),
5. solid-state storage (ie Flash memory),
6. magnetic disk (or as Tim called it, "spinning rust"),
7. tape (which as Tim reminded us is used more than ever due to regulations like Sarbanes-Oxley requiring everyone to keep everything practically forever)

The news here is (a) a validation of our approach at the BBC's Forge project, where we use memcached as a critical part of the scaling infrastructure for dynamic publishing, just like a growing set of people, like Facebook, Yahoo, and many more, and (b) the introduction of solid-state storage to the list -- and so high up in the list!

But the thing that really got Tim excited was not just his impressive figures on how much faster solid state could be on the right filesystem (which was a bit of an ad for a new server released by his employer, Sun), but the fact that SSD has Moore's law on its side: as opposed to "spinning rust", SSD is all silicon, so it will only increase in price/performance over time.

As Tim says, "Ladies and gentlemen, you are looking at the future."

Note: For the business students that might stumble upon this blog, here's a reward for reading through all that techy stuff: Sandisk own many many patents in solid-state storage and they were strong enough to shrug off Samsung's offer a couple of months ago, so they could be an interesting stock to watch as solid-state disks become a key part of more and more high-end computer systems... but they're going down right now, and they might not have hit bottom yet as it looks like all the analysts are downgrading them one by one (not to be taken as investment advice blah blah)

Tim's slides (warning: the /tmp/ in the URL gives the indication that they may not be there forever...)

QCon SF 2008, day two

It's a shame that I missed the first day of QCon SF, especially as I enjoyed it so much last year. I was a paid delegate in 07 but this year I qualify for "starving student" rates (ie free), in exchange for volunteering by manning one of the session rooms tomorrow. Luckily that's the room about distributed databases so I probably would have gone to a lot of those sessions anyway. (For people looking for where to go, the "how did they build that?" architectures track is always great as well, but many of the people speaking were there last year as well so I don't really mind not going to that track, but I recommend it if you're into that stuff.)

I couldn't go to day one as I had classes yesterday, but from talking to a few people it seems that the better sessions were all today anyhow. The logistics were all great as usual, the wireless came thick and fast, and the food was pretty good -- ya gotta love that mid-afternoon ice cream run to keep the sugar levels high!

I'll spread the news over a few posts so if people want to comment on a particular section they can do so.

PS I might try to dig up my notes from last year's conference, as they seemed to go down well when I sent them around internally last year. That will probably qualify as the most out-of-date blog post ever!

Reviving the blog

An even two years after my last post, I figure it's about time to get this thing started again. Especially while I'm studying in Berkeley, I have a lot to say -- too bad I've left it so late that I only have about two weeks left... well I'd better hurry up and post some stuff then.

Apologies that there will be a mix of current stuff (eg QCon this week) and old stuff in no particular date order. I'm sure you can work things out, and anyway I don't think I'll be saying anything so cutting edge that the dates actually matter.

Enjoy, here's two at least a few posts before the next two-year gap...

Innovating our way around the country

I'm lucky enough to be involved with the BBC Innovation Labs, run by Matt Locke and the BBC Creative R&D team (er I think they're still called that?)

The Labs are basically a way to involve the new media "indie" community in developing new ideas that meet various strategic goals of the BBC (eg how to reach an under-served audience, or how to exploit a particular type of content we own, etc), giving them a bit of cash to sit with us for five days and work through an idea, to which we get first rights but non-exclusive access.

Along with Mark, Ian and Matt, I've been asked to come along to the intro sessions to talk about the BBC's web infrastructure. The idea is that agencies will have an idea of how their pitches will run on our architecture and what they will have to integrate with. Going by some of the demos we've seen recently, a lot of agencies don't seem to think they would have to integrate with anything, so getting the word out has to be a good thing.

It also helps that we can talk about what will be coming in the next year or so as the "BBC 2.0" project kicks off... we will have a nice scalable service-oriented architecture for people to build upon. Well, that's the plan anyway.

It raises a few interesting points:

• Where are the techy agencies?
  It seems that most of the people pitching to us have been "new meedja" agencies in the sense that they are used to building corporate web sites for corporates, with minimal technical input. But if we want some solid recommendation software either built or integrated, or a message queuing system integrated and supported, where do we go? We've got the money to spend... (well, we will have soon, if all goes to plan..!)


• How to make it worth an agency's while to build for our architecture?
  We want people to build tools that will integrate with our single sign-on, ratings systems, file stores, social networking / friends systems, etc etc... but if they build exclusively for our needs, they won't be able to re-sell their products to others. So either (a) that's fine but we have to pay more for the work, or (b) we have to come up with a compelling offer to these guys. One way might be to create open source implementations of each of our components, and make them all freely available to be integrated into other products (under, say, the Apache license). But as at least part-funders of these products, wouldn't licence fee payers expect to see a share in any future proceeds of products built on our shared infrastructure?
  
  Another way would be to exclusively use shared standards and make sure an open source product already exists for every component we use. That might be fine for SSO (using, say, JAAS) but I can't see it working for some of our more obscure components.

Lots of food for thought... which is after all what the innovation labs days are all about.

Fame of sorts

Heh -- last night I discovered that Yahoo are using My NewsML modules to power their news.yahoo.com site. The NewsML and NITF modules, the ones I haven't updated in over three years, are running on dozens of machines and their news site would fail if they didn't work!

I feel humble and proud ;-)

Although more on the humble side when I look at the code and realise what a rudimentary state I left it in...!

Weird.

If anyone else out there is using it, please let me know... who knows, I may actually put out another version one day!

Economist on simplicity in technology

This week's Economist has a special on designing for simplicity. It's described in terms a business manager can understand, with a few strained metaphors such as the journey of a byte from disk to user, but the point it raises is a good one.

I worry when I hear people say that simplicity is the way to go in the future, with the not-very-subtly-hidden subtext that all we have to do is get rid of layers of technology and complexity, and a simple interface will magically present itself.

The set of articles in the Economist uses the examples of cars and telephone networks to show that making things simple to use is actually very difficult, and could well require more layers of complexity to provide the interoperability and intuitive interface that we need if we are going to make sense of the changing world of technology in the next decade or so.

Update to RDF Data Access Use Cases

RDF Data Access Use Cases and Requirements

Somehow I feel the combined smarts of the W3C should be able to come up with more exciting use cases than finding the top 10 movies which also had soundtracks that came top 10 in the charts..? How is that supposed to energise a new industry to make its data available in RDF?

The use cases don't come from the real world that I live in... random quotes:

"José sends a query to the Census Bureau's new RDF storage server and requests that his client pass the query results to an XSLT transformation service so that he can print the resulting XHTML."

"Zoe wrote an IRC bot that they use to make assertions -- which the bot stores as RDF."

I hope someone out there is thinking about this stuff in a bit more end-user-friendly manner!

Events ML

Something I hadn't noticed before... IPTC EventsML Working Group.

Right now "The project scope is limited to the definition of an information interchange standard for newsworthy event information", but I'm sure they will expand the scope eventually to handle other types of events... I haven't looked into it yet but I hope they're thinking about integration with iCalendar...

Publishing our metadata in HTML

In the BBC English Regions web CMS I've been involved with building, we have six controlled vocabularies for descriptive metadata (location, name, subject, audience, BBC brand, time period), and we manage about 85,000 terms in those CVs. We sourced some of the CVs from external suppliers and modified them, some we built from scratch. We also have a team that maintains the existing terms and allows our journalists to suggest terms and have them vetted and finished off by the metadata specialists.

A few of us have some ideas around publishing out this metadata, we just have to run it past search to check what format we should use. I’m hoping we can just do

<meta name=”location” value=”BBC/C/Devon,BBC/C/London”>

etc. That would be very useful, in the sense that people could pick it up and do amazing things with it... maybe we can do that stuff ourselves, but even if we don’t, the data is there ready for someone else to play with.

As well as the text-based names, we have the lat/long data for those locations in an XML file somewhere, I might be able to get that extracted and put into the HTML as well. We aren't allowed to publish the complete location CV due to licensing restrictions, but we can extract pieces of it for certain purposes.

We haven’t really decided on the format yet, it might be worth making it RDF or something from the start -- although I’m not sure how useful that would be unless our content is all well-formed XML, and while we’re pretty close, I don't think we've sorted out all the issues of & in URLs etc.

Thoughts on a BBC API

After seeing some cool newsy metadataey things recently like aaronland's New York Times widgets and Stef's BBC News wikipedia thing, and talking to Tom L about some of his grand ideas, I've been thinking more about where the BBC should be in the semantic web, the lazyweb, etc etc... and in the end I figure we at the BBC should be able to build the content equivalent of Google API or Amazon API.

Basically, Google provides a bunch of web services (via SOAP and REST) and exposes their core functionality to developers around the world. They provide the engine, developers can build their own interface, as long as they follow a few rules. Amazon does the same thing, although of course they're not being nearly as nice as Google because they're really just opening up their shopping cart to others, they still get the money from any sales.

So if Google can provide the definitive search API that everyone can use, and Amazon can provide a book buying API, what could the BBC offer? The list is endless...

• GIS services: postocde<->location name<->lat/long mapping


• TV/radio listings services: what's on today, what's on this week, in a particular genre, on a paricular network, what's on right now


• Search services: BBC search via API


• metadata services: classification engine??, list of terms, integrate with BBC Search to show stories/pages that match metadata terms


• information services: weather, traffic info, financial data...

A simple application of this would be something that takes a person's postcode, uses the postcoder API to translate it to a lat/long, then uses the BBC search to find all content marked with locations within 10 miles, and presents a custom page for that user. Of course that's just the beginning!

Content Management Professionals organisation launched

After about six months in gestation by a group of Content Management gurus and me, CM Professionals - The Content Management Community of Practice was launched in September.

We hope it to be a place where the sort of people who make content management actually work can gather and share information. Vendors are not allowed to join as an organisation (although people who work for vendors are welcome to join as an individual), and we don't really care too much about the technology. We know that the problems of content modelling, change management, business process modelling and actually getting reasonable requirements for a CM project are much harder than merely building some software to manage content.

If that sounds interesting to you, please check out the site and maybe even sign up. We're having our first official meeting, the CM Pros Summit at Boston in November, co-located with this year's Gilbane Conference. You're welcome to join us!

Graf review on content management

The Graf review on the BBC's online offering was finally released today, after 18 months of development.

So what does it have to say about content management at the BBC? Some interesting points...

(Just notes-to-self for now, I'll come back and write about individual points later)

Page 20 of the main review: "The BBC has also employed a number of technical solutions to enable them to deliver news and information content quickly and effectively, over a number of platforms. The development of the News and Sport Content Production System (CPS) represents a significant investment in such technical solutions, with total costs amounting to £3.8m in 2003"

A constant theme is value for money, which resonate's well with Mark Thompson's recent prouncements to make all BBC services pass a "public value test". The term "cost per unique user", "CPUU" is referred throughout, obviously meaning the cost of content production divided by number of unique users. It will be interesting to see this statistic for a lot of our content. According to page 21, CPUU for weather is 1.8 pence and Nations and Regions (excluding Northern Ireland) is 20.5 pence. It will be interesting to see if this figure changes after our CMS has finished rolling out to the English Regions sites...

Page 21 again, "the BBC is currently developing the technology to track traffic exclusively to external sites." Well I thought the /go/ system would do that, but if you say so... anyway in the CMS, adapting our external weblinks to use a system like this (or to move from one such system to another) will take about 20 seconds of XSLT coding.

And more on page 21, "links from sites or pages are not... measured across the site." Another easy peasy thing for us to do with the CMS...

Page 22, "The total costs of the launch of the
search engine (worldwide and BBC site) were £414,268, and the current running costs of the
BBC Online search tool are £476,000."

Documentum specifically gets a mention on page 24!

Page 27, "Applications developed by the BBC, such as DNA have also enabled user-generated content to
be more stimulating for the user and more efficiently managed. The current growth in web log usage also allows users to contribute richer content (e.g. to news stories) in the form of text, pictures, and audio and video clips."

Page 32: "BBC Online’s ambition to syndicate online content to other providers (for example, free and non-exclusive arrangements for commercial websites such as www.streetmap.co.uk to carry BBC news headlines), and to other devices (for example, free and non-exclusive arrangements with mobile providers to ensure position of BBC Online on WAP versions), is again, a means to drive towards 100% reach. Reach is a key means to ensure that increasing numbers of licence fee payers can derive some value from the BBC’s online services. This strategic goal does, however, risk the BBC being perceived by commercial operators as an aggressive, and unfairly advantaged competitive force. Submitters to the review also argued that the BBC’s current
inconsistent approach to linking, the prominence of BBC Online results in its search engine, and the low level of joint venture or externally commissioned projects have compounded this
impression."

Page 35 has a graph of BBC Online Expenditure, charted over the years... the division with the most expenditure is of course News with 15.5m in 2003/04, second is Nations & Regions with 11.6m and third is Factual and Learning with 10.1m.

That'll do for now, more on the technology assessment appendix tomorrow :-)

Welcome!

Hello and welcome to what is currently called "Brendan's Braindump" -- Brendan Quinn's thoughts about the world of content management systems, the techniques of teasing these systems into doing something that's actually useful to human beings, and generally the practice of managing content in enterprises large and small.

Basically I figured after more than ten years on the web, it was about time I got myself a blog.

Hopefully this one will be updated more frequently than my horrendous efforts on Advogato (wow! over three years of inactivity, is that a record?) and one on livejournal I think I created once but can't remember for the life of me... and for that matter my own website which is lucky to see an update every six months.

So why did I never update my blog/s before? I know exactly why. It's simple really. I never thought I had anything interesting enough to say.

I'm not like a lot of bloggers out there who seem to have the web equivalent of verbal diahrroea, enjoying the sound of their own voice (or is it the glow of their own pixels? or something like that).

I suffer from indecision -- or is it insecurity? From the paralysing thought that what I am now writing might be proven wrong tomorrow, derided in other blogs, laughed at by posterity. No matter what I write, I know that unless I believe in it unequivocally I will never be happy with it. And being a scientifically-minded sceptic, it takes a *lot* for me to believe in something unequivocally.

The other reason, which in a way is the same reason I guess, is that I am far too much of a perfectionist. I will spend way too long going back and editing everything I said, trying to make it read more fluently, grimacing at my woeful use of what is after all the only language I speak, trying to cram way too many ideas into way too few lines, and becoming even more obsessed with spelling and punctuation. I know blogs are supposed to be free form, conversational, and mostly unedited, but that's just not my way. I like structure, I like correctness, and I take the time to make sure things are right. Already I've lost count of the number of times I've deleted chunks of this post and re-typed them (including this very sentence).

So it's really taken me ten years to realise that these qualities can be seen as virtues, and if nothing else they make me different from most other bloggers out there. Hopefully it will at least end up that I say things with meaning, things worth reading. (Not so far, I'll admit.)

It's taken me ten years to realise that I actually have things to say, but they are still swishing around my brain, waiting to be released in a more permanent, more structured way. And the only way to get them out is to start writing about them. And a blog is as good a way to do that as any.

In a way I've been forced into doing this, because my job is now requiring me to think some more, and write some more, about what content management means and where it will be in a few years' time... Where will the world of web services, rich metadata, and peer-to-peer applications leave the still fairly push-centric world of digital publishing? What lessons can the relatively established practice of web content management teach the "serious" media and broadcasting companies that are finally waking up to truly converged, truly digital audio and video production? What new standards will emerge to guide us through this difficult process of making so many different systems, processes and people work together?

Of course you would think that being a full-time, self-employed content management consultant for a year during all of 2001 and part of 2002 would have forced me into thinking about what I stood for, and what content management really meant, but at the time it was revolutionary enough just to be someone who asked the right questions.

Now I want to start thinking about some answers.