91Èȱ¬ Topics: How It Works
Last week, Matt McDonnell wrote about the new 91Èȱ¬ Topic Pages Beta. I'd now like explain how some of the many components that build those pages all work together.
The point of the Topic Pages is that they bring together content from all around bbc.co.uk. Obviously, many different systems produce all that content, and in general they don't tend to share content very well. Our challenge was to build a platform that could make sense of the different interfaces to those systems to make sharing that content easier.
The first thing to note is that the Topic Pages themselves are , unlike the vast majority of pages on bbc.co.uk. Essentially, this means that the HTML of the page isn't stored as a physical file on a hard-disk, but instead is built up dynamically when the page is requested.
This is done by the "Page Assembly Layer" or "PAL", a brand new component written in the . In the future, the intention is that most pages on bbc.co.uk will be produced dynamically using the PHP layer, and the Topic Pages system has blazed a trail, being the first released on this new platform.
The PAL itself does a fairly simple job, in principle. First, it receives a request for a Topic Page; it then looks up which modules (ie, the different blocks of content on the page, such as 91Èȱ¬ News, Programmes and Weather) it needs to build that page; it grabs all those different modules, which originate on various different systems and finally it assembles them before returning the page to the user.
The really important part here is that the PAL is grabbing all the useful content dynamically, and not storing any content itself (apart from a bit of , to help smooth out any spikes in load). This means that the PAL is a really generic system that can be used for building other sorts of pages as well.
The PAL actually requests the various content modules from another component, which itself routes these requests on to the underlying systems. This Module Routing system is implemented in , an open source framework, released by the . This way, the PAL can access content through a simple, uniform interface (which is based on , for those who are into such things), rather than having to deal directly with many complex interfaces to multiple systems.
This gives us two big benefits. First, if we want to change how a particular module is implemented, we can just reconfigure the Module Routing, and don't have to alter the PAL (which we don't want to tinker with too much, as it is busy serving pages); second, it makes it easier for any other system, or other page on bbc.co.uk, to reuse the modules. This model also makes it easier to add new content modules to the PAL, as most of the logic about how to access a new module can be handled in the Module Routing layer.
As Matt said, "Topics are automatically updated web pages, each one covering a different person, country or subject." We wanted to present content from across the various departments within the 91Èȱ¬. The Search Engines are the one place that all that content from bbc.co.uk (well, most of that content) is brought together (nearly 2 million documents from www.bbc.co.uk and the and sites). So it makes a lot of sense to create many of the modules from the Search indexes (the News module, and the Programmes module, for example, are both created like this). This is done using elaborate search queries, which are a bit fancier than the one- or two-keyword queries which the Search Engines normally receive - the query for the Fashion topic, for example, contains 364 terms! These queries are built using a variety of techniques by the Search Editorial Team.
We also use content from other sources than the Search indexes - for example, the Weather modules, and the information boxes about , which come from the 91Èȱ¬ News website. In order to make it as easy as possible to share and reuse these content modules, we are attempting to create some standards for the formats in which the data is passed around (rather than just allowing every system to specify its own format).
Having looked around a bit, we decided to base our data structure on the format, which was originally created to describe feeds from Blogs. Atom is a rich format, and one of its most attractive features is that you can add your own data elements. This means we can have a standard container for our data, but also include extra elements where appropriate (such as in the weather modules, to denote temperatures, for example).
This approach is similar to Google's , which is also based on ATOM. We have a prototype of this XML format - called 91Èȱ¬ Module XML - in use in the Topics pages, and in the next few months we will be looking at refining and improving it, and hopefully making it useful for other systems across bbc.co.uk. We use to transform from the XML to the nice modules which are placed on the page.
The only parts that I haven't touched upon so far are the admin systems for creating and maintaining the pages and also the modules that go on them. At the moment, these admin systems aren't as joined up as they should be (and are implemented in different programming languages in some cases). This is a headache for the editorial team that maintain the Topic Pages. Our next priority is figuring out how we can bring those systems closer together, and generally improve the workflow for the editorial staff, so that they can easily add more Topic Pages.
Once we've made their lives a little easier, we will look at more feature enhancements. These include providing feeds of the Topic Pages so that people can more easily stay up to date with their favourite topics. Additionally, we want to improve our systems for sharing metadata, so that it will be easier to automatically link to relevant Topic Pages from other pages on bbc.co.uk. And we will also add more types of content modules, to increase the range of content on the Topic Pages.
As you hopefully can see, the Topic Page project was pretty complex, and involved creating many new systems. Wherever possible, we have developed those systems to be generic and extensible, to provide not just Topic Pages, but also a platform for sharing and reusing content, and building other products in the future. This has all been possible thanks to some fantastic, far-sighted and occasionally frenetic work from the 91Èȱ¬ Search Team and colleagues in FM&T Journalism and FM&T Internet - thanks to all of them.
N.B. More information on /topics can be found here.
Stephen Betts is Search Technical Team Leader, 91Èȱ¬ Future Media & Technology
Comment number 1.
At 12th Jun 2008, Hymagumba wrote:Out of interest, during the MT4 upgrade a reason given for choosing MT was the inability to run PHP on the bbc.co.uk servers.
Is /topics using php the result of an upgrade to the main servers or is it running elsewhere? Will the main servers be upgraded at some point in the future to allow the rest of the site to modernise?
Also, slightly OT, I notice even the newer pages on the 91Èȱ¬ News site (like the Today site) have horrible table based HTML with massive chunks of white space. Are there any plans to upgrade the News CMS to allow better coded pages to be created?
Complain about this comment (Comment number 1)
Comment number 2.
At 14th Jun 2008, Briantist wrote:PHP is great for this kind of system: I have been using it for a few years now. You seem to have made really good use of it.
Complain about this comment (Comment number 2)
Comment number 3.
At 16th Jun 2008, kwdhinde wrote:Hymagumba, about the 91Èȱ¬ News HTML:
Yes, we know it's not great, and we do have plans to make big improvements next year.
We do test our not-beautiful HTML for speed of rendering and screenreader behaviour and make changes where we can. We know the way that the headers are nested is particularly annoying for screenreaders and we can tweak that soon.
Tweaks aside, we have a much bigger project coming up to improve the 91Èȱ¬ News HTML output. You are probably wondering what on earth could be taking us so long. The problem is we have a very adaptable content management system which can produce all kinds of page layout, but it was built (a long time ago) to deliver that flexibility using HTML tables; so to get it to do the same thing using CSS layout means changing quite a lot.
Kevin Hinde
91Èȱ¬ Future Media and Technology, Journalism
Complain about this comment (Comment number 3)