The UK Web Archive has a new user interface! Try it now: 

beta.webarchive.org.uk/

What's new?

• For the first time you can search both the 'Open UK Web Archive'¹ and the 'Legal Deposit Web Archive'² from the same search box
• We have improved the search and have included faceting so that it's easier to find what you are looking for
• A simple, clean design that (hopefully) allows the content to be the focus
• Easily browsable 'Special Collections' (curated groups of websites on a theme, topic or event)

 What next?

This is just the start of what will be a series of improvements, but we need your help! Please use the beta site and tell us how it went (good, bad or meh) by filling in this short 2 minute survey:

www.surveymonkey.co.uk/r/ukwasurvey01

Thank you!

by Jason Webber, Web Archive Engagement Manager, The British Library

¹ The Open UK Web Archive was started in 2005 and comprises of approximately 15,000 websites that can be viewed anywhere

² The Legal Deposit Web Archive was started in 2013 and comprises millions of websites but these can only be viewed in the Reading Rooms of UK Legal Deposit Libraries

By Dr Andrew Jackson, Web Archive Technical Lead, The British Library

The heart of the idea was simple. Rather than our traditional linear harvesting process, we would think in terms of annotating the live web, and imagine how we might use those annotations to drive the web-archiving process. From this perspective, each Target in the Web Curator Tool is really very similar to a bookmark on an social bookmarking service (like Pinboard, Diigo or Delicious1), except that as well as describing the web site, the annotations also drive the archiving of that site2.

In this unified model, some annotations may simply highlight a specific site or URL at some point in time, using descriptive metadata to help ensure important resources are made available to our users. Others might more explicitly drive the crawling process, by describing how often the site should be re-crawled, whether robots.txt should be obeyed, and so on. Crucially, where a particular website cannot be ruled as in-scope for UK legal deposit automatically, the annotations can be used to record any additional evidence that permits us to crawl the site. Any permissions we have sought in order to make an archived web site available under open access can also be recorded in much the same way.

Once we have crawled the URLs and sites of interest, we can then apply the same annotation model to the captured material. In particular, we can combine one or more targets with a selection of annotated snapshots to form a collection. These ‘instance annotations’ could be quite detailed, similar to those supported by web annotation services like Hypothes.is, and indeed this may provide a way for web archives to support and interoperate with services like that.3

Thinking in terms of annotations also makes it easier to peel processes apart from their results. For example, metadata that indicates whether we have passed those instances through a QA process can be recorded as annotations on our archived web, but the actual QA process itself can be done entirely outside of the tool that records the annotations.

To test out this approach, we built a prototype Annotation & Curation Tool (ACT) based on Drupal. Drupal makes it easy to create web UIs for custom content types, and we were able to create a simple, usable interface very quickly. This allowed curators to register URLs and specify the additional metadata we needed, including the crawl permissions, schedules and frequencies. But how do we use this to drive the crawl?

Our solution was to configure Drupal so that it provided a ‘crawl feed’ in a machine readable format. This was initially a simple list of data objects (one per Target), containing all the information we held about that Target, and where the list could be filtered by crawl frequency (daily, weekly, monthly, and so on). However, as the number of entries in the system grew, having the entire set of data associated with each Target eventually became unmanageable. This led to a simplified description that just contains the information we need to run a crawl, which looks something like this:

[
    {
        "id": 1,
        "title": "gov.uk Publications",
        "seeds": [
            "https://www.gov.uk/government/publications"
        ],
        "schedules": [
            {
                "frequency": "MONTHLY",
                "startDate": 1438246800000,
                "endDate": null
            }
        ],
        "scope": "root",
        "depth": "DEEP",
        "ignoreRobotsTxt": false,
        "documentUrlScheme": null,
        "loginPageUrl": null,
        "secretId": null,
        "logoutUrl": null,
        "watched": false
    },
    ...

This simple data export became the first of our web archiving APIs – a set of application programming interfaces we use to try to split large services into modular components4.

Of course, the output of the crawl engines also needs to meet some kind of standard so that the downstream indexing, ingesting and access tools know what to do. This works much like the API concept described above, but is even simpler, as we just rely on standard file formats in a fixed directory layout. Any crawler can be used as long as it outputs standard WARCs and logs, and puts them into the following directory layout:

/output/logs/{job-identifer}/{launch-timestamp}/*.log
/output/warcs/{job-identifer}/{launch-timestamp}/*.warc.gz

Where the {job-identifer} is used to specify which crawl job (and hence which crawl configuration) is being used, and the {launch-timestamp} is used to separate distinct jobs launched using the same overall configuration, reflecting repeated re-crawling of the same sites over time.

In other words, if we have two different crawler engines that can be driven by the same crawl feed data and output the same format results, we can switch between them easily. Similarly, we can make any kind of changes to our Annotation & Curation Tool, or even replace it entirely, and as long as it generates the same crawl feed data, the crawler engine doesn’t have to care. Finally, as we’ve also standardised the crawler output, the tools we use to post-process our crawl data can also be independent of the specific crawl engine in use.

This separation of components has been crucial to our recent progress. By de-coupling the different processes within the crawl lifecycle, each of the individual parts is able to be move at it’s own pace. Each can be modified, tested and rolled-out without affecting the others, if we so choose. True, making large changes that affect multiple components does require more careful management of the development process, but this is a small price to pay for the ease by which we can roll out improvements and bugfixes to individual components.

A prime example of this is how our Heritrix crawl engine itself has evolved over time, and that will be the subject of the next blog post.

1. Although, noting that Delicious is now owned by Pinboard, I would like to make it clear that we are not attempting to compete with Pinboard. ↩

2. Note that this is also a feature of some bookmarking sites. But we are not attempting to compete with Pinboard. ↩

3. I’m not yet sure how this might work, but some combination of the Open Annotation Specification and Memento might be a good starting point. ↩

4. For more information, see the Architecture section of this follow-up blog post ↩

What do you call the 5th of November? As a child of the 70s and 80s it was 'Guy Fawkes' night and my friends and I might make a 'guy' to throw on the bonfire. It is interesting to see through an analysis of the UK Web Archive SHINE service that the popularity of the term 'Guy Fawkes' was overtaken by 'Bonfire night' in 2009. I've included 'Fireworks night' too for comparison.

Is this part of a trend away from the original anti-catholic remembrance and celebration to a more neutral event?

Examine this (and other) trends on our SHINE service.

By Jason Webber, Web Archive Engagement Manager, The British Library

By Andy Jackson, Web Archive Technical Lead, The British Library - re-blogged from anjackson.net

Before I revisit the ideas explored in the first post in the blog series I need to go back to the start of this story…

Between 2003 and 2013 – before the Non-Print Legal Deposit regulations came into force – the UK Web Archive could only archive websites by explicit permission. During this time, the Web Curator Tool (WCT) was used to manage almost the entire life-cycle of the material in the archive. Initial processing of nominations was done via a separate Selection & Permission Tool (SPT), and the final playback was via a separate instance of Wayback, but WCT drove the rest of the process.

Of course, selective archiving is valuable in it’s own right, but this was also seen as a way of building up the experience and expertise required to implement full domain crawling under Legal Deposit. However, WCT was not deemed to be a good match for a domain crawl. The old version of Heritrix embedded inside WCT was not considered very scalable, was not expected to be supported for much longer, and was difficult to re-use or replace because of the way it was baked inside WCT.1

The chosen solution was to use Heritrix 3 to perform the domain crawl separately from the selective harvesting process. While this was rather different to Heritrix 1, requiring incompatible methods of set-up and configuration, it scaled fairly effectively, allowing us to perform a full domain crawl on a single server2.

This was the proposed arrangement when I joined the UK Web Archive team, and this was retained through the onset of the Non-Print Legal Deposit regulations. The domain crawls and the WCT crawls continued side by side, but were treated as separate collections. It would be possible to move between them by following links in Wayback, but no more.

This is not necessarily a bad idea, but it seemed to be a terrible shame, largely because it made it very difficult to effectively re-use material that had been collected as part of the domain crawl. For example, what if we found we’d missed an important website that should have been in one of our high-profile collections, but because we didn’t know about it had only been captured under the domain crawl? Well, we’d want to go and add those old instances to that collection, of course.

Similarly, what if we wanted to merge material collected using a range of different web archiving tools or services into our main collections? For example, for some difficult sites we may have to drive the archiving process manually. We need to be able to properly integrate that content into our systems and present them as part of a coherent whole.

But WCT makes these kind of things really hard.

If you look at the overall architecture, the Web Curator Tool enforces what is essentially (despite the odd loop or dead-end) a linear workflow (figure taken from here). First you sort out the permissions, then you define your Target and it’s metadata, then you crawl it (and maybe re-crawl it for QA), then you store it, then you make it available. In that order.

But what if we’ve already crawled it? Or collected it some other way? What if we want to add metadata to existing Targets? What if we want to store something but not make it available. What if we want to make domain crawl material available even if we haven’t QA’d it?

Looking at WCT, the components we needed were there, but tightly integrated in one monolithic application and baked into the expected workflow. I could not see how to take it apart and rebuild it in a way that would make sense and enable us to do what we needed. Furthermore, we had already built up a rather complex arrangement of additional components around WCT (this includes applications like SPT but also a rather messy nest of database triggers, cronjobs and scripts). It therefore made some sense to revisit our architecture as a whole.

So, I made the decision to make a fresh start. Instead of the WCT and SPT, we would develop a new, more modular archiving architecture built around the concept of annotations…

1. Although we have moved away from WCT it is still under active development thanks to the National Library of New Zealand, including Heritrix3 integration! ↩
2. Not without some stability and robustness problems. I’ll return to this point in a later post. ↩

By Jen Aggleton, PhD candidate in Education at the University of Cambridge

As part of my PhD placement at the British Library, I was asked to establish a special collection of webcomics within the UK Web Archive. In order to do so, it was necessary to outline the scope of the collection, and therefore attempt to define what exactly is and is not a digital comic. As anyone with a background in comics will tell you, comics scholars have been debating what exactly a comic is for decades, and have entirely failed to reach a consensus on the issue. The matter only gets trickier when you add in digital components such as audio and animation.

Due to this lack of consensus, I felt it was important to be very transparent about exactly what criteria have been used to outline the scope of this collection. These criteria have been developed through reference to scholarship on both digital and print comics, as well as my own analysis of numerous digital comics.

The scope of this collection covers items with the following characteristics:

• The collection item must be published in a digital format
• The collection item must contain a single panel image or series of interdependent images
• The collection item must have a semi-guided reading pathway¹

In addition, the collection item is likely to contain the following:

• Visible frames
• Iconic symbols such as word balloons
• Hand-written style lettering which may use its visual form to communicate additional meaning

The item must not be:

• Purely moving image
• Purely audio

For contested items, where an item meets these categories but still does not seem to be a comic, it will be judged to be a comic if it self-identifies as such (e.g. a digital picturebook may meet all of these criteria, but self-identifies as a picturebook, not a comic).

Where the item is an adaptation of a print born comic, it must be a new expression of the original, not merely a different manifestation, according to FRBR guidelines: www.loc.gov/cds/FRBR.html.

¹ Definition of a semi-guided reading pathway: The reader has autonomy over the time they spend reading any particular aspect of the item, and some agency over the order in which they read the item, especially the visual elements. However reading is also guided in the progression through any language elements, and likely to be guided in the order of movement from one image to another, though this pathway may not always be clear. This excludes items that are purely pictures, as well as items which are purely animation.

Alongside being clear about what the collection guidelines are, it is also important to give users information on the item acquisition process – how items were identified to be added to the collection. An attempt has been made to be comprehensive: including well known webcomics published in the UK and Ireland by award-winning artists, but also webcomics by creators making comics in their spare time and self-publishing their work. This process has, however, been limited by issues of discoverability and staff time.

Well known webcomics were added to the collection, along with webcomics discovered through internet searches, and those nominated by individuals after calls for nominations were sent out on social media. This process yielded an initial collection of 42 webcomic sites (a coincidental but nonetheless highly pleasing number, as surely comics do indeed contain the answers to the ultimate question of life, the universe, and everything). However, there are many more webcomics published by UK and Ireland based creators out there. If you know of a webcomic that should be added to our collection, please do nominate it at www.webarchive.org.uk/ukwa/info/nominate.

Jen Aggleton, PhD candidate in Education at the University of Cambridge, has recently completed a three month placement at the British Library on the subject of digital comics. For more information about what the placement has entailed, you can read this earlier blog.

By Hedley Sutton, Team Leader, Asian & African studies Reference Services

Here we are again, eavesdropping on a conversation among UK domain websites as to which one has the best claim to be recognized as the most extraordinary…

“Happy to start the ball rolling,” said the British Fantasy Society. “Clue in the name, you know.”

“Ditto,” added the Ghost Club.

“Indeed,” came the response. “However … how shall I put this? … don’t you think we need a site that’s a bit more … well, intellectual?” said the National Brain Appeal.

“Couldn’t agree more,” chipped in the Register of Accredited Metallic Phosphide Standards in the United Kingdom.

“Come off it,” chortled the Pork Pie Appreciation Society. “That would rule out lots of sites straightaway. Nothing very intellectual about us!”

“Too right,” muttered London Skeptics in the Pub.

Before things became heated the British Button Society. made a suggestion. “Perhaps we could ask the Witchcraft & Human Rights Information Network  to cast a spell to find out the strangest site?”

The silence that followed was broken by Campaign Bootcamp. “Come on – look lively, you ‘orrible lot! Hup-two-three, hup-two-three!”

“Sorry,” said the Leg Ulcer Forum. “I can’t, I’ll have to sit down. I’ll just have a quiet chat with the Society of Master Shoe Repairers. Preferably out of earshot of the Society for Old Age Rational Suicide.”

“Let’s not get morbid,” said Dream It Believe It Achieve It helpfully. “It’s all in the mind. You can do it if you really try.”

There was a pause. “What about two sites applying jointly?” suggested the Anglo Nubian Goat Society. “I’m sure we could come to some sort of agreement with the English Goat Breeders Association.”

“Perhaps you could even hook up with the Animal Interfaith Alliance,” mused the World Carrot Museum.

“Boo!” yelled the British Association of Skin Camouflage suddenly. “Did I fool you? I thought I would come disguised as the Chopsticks Club.

“Be quiet!” yelled the Mouth That Roars even louder. “We must come to a decision, and soon. We’ve wasted enough time as it is.”

The minutes of the meeting show that, almost inevitably, the site that was eventually chosen was … the Brilliant Club.

If there is a UK based website you think we should collect, suggest it here.

Jen Aggleton is a PhD candidate in Education at the University of Cambridge, and is completing a work placement at the British Library on the subject of digital comics. 

If you are a digital comics creator, publisher, or reader, we would love to hear from you. We’d like to know more about the digital comics that you create, find out links to add to our Web Archive collection, and find examples of comic apps that we could collect. Please email suggestions to [email protected]. For this initial stage of the project, we will be accepting suggestions until the end of August 2017.

I definitely didn’t apply for a three month placement at the British Library just to have an excuse to read comics every day. Having a number of research interests outside of my PhD topic of illustrated novels (including comics and library studies), I am always excited when I find opportunities which allow me to explore these strands a little more. So when I saw that the British Library were looking for PhD placement students to work in the area of 21st century British comics, I jumped at the chance.

Having convinced my supervisor that I wouldn’t just be reading comics all day but would actually be doing proper serious work, I temporarily put aside my PhD and came to London to read lots and lots of digital comics (for the purpose of proper serious work). And that’s when I quickly realised that I was already reading comics every day.

The reason I hadn’t noticed was because I hadn’t specifically picked up a printed comic or gone to a dedicated webcomic site every day (many days, sure, but not every day). I was however reading comics every day on Facebook, slipped in alongside dubiously targeted ads and cat videos. It occurred to me that lots of other people, even those who may not think of themselves as comics readers, were probably doing the same.

(McGovern, E. My Life As A Background Slytherin, https://www.facebook.com/backgroundslytherin/photos/a.287354904946325.1073741827.287347468280402/338452443169904/?type=3&theater Reproduced with kind permission of Emily McGovern.)

This is because the ways in which we interact with comics have been vastly expanded by digital technology. Comics are now produced and circulated through a number of different platforms, including apps, websites and social media, allowing them to reach further than their traditional audience. These platforms have made digital comics simultaneously both more and less accessible than their print equivalents; many webcomics are available for free online, which means readers no longer have to pay between £8 and £25 for a graphic novel, but does require them to have already paid for a computer/tablet/smartphone and internet connection (or have access to one at their local library, provided their local library wasn’t a victim of austerity measures).

Alongside access to reading comics, access to publishing has also changed. Anyone with access to a computer and internet connection can now publish a comic online. This has opened up comics production to many whose voices may not have often been heard in mainstream print comics, including writers and characters of colour, women, members of the LGBTQ+ community, those with disabilities, and creators who simply cannot give up the stability of full-time employment to commit the time needed to chase their dream of being a comics creator. The result is a vibrant array of digital comics, enormously varying in form and having a significant social and cultural impact.

But digital comics are also far more fragile than their print companions, and this is where the proper serious work part of my placement comes in. Comics apps are frequently removed from app stores as new platform updates come in. Digital files become corrupted, or become obsolete as the technology used to host them is updated and replaced. Websites are taken down, leaving no trace (all those dire warnings that the internet is forever are not exactly true. For more details about the need for digital preservation, see an earlier post to this blog). So in order to make sure that all the fantastic work happening in digital comics now is still available for future generations (which in British Library terms could mean ten years down the line, or five hundred years down the line), we need to find ways to preserve what is being created.

One method of doing this is to establish a dedicated webcomics archive. The British Library already has a UK Web Archive, due to the extension of legal deposit in 2013 to include the collection of non-print items. I am currently working on setting up a special collection of UK webcomics within that archive. This has involved writing collections guidelines covering what will (and won’t) be included in the collection, which had me wrestling with the thorny problem of what exactly a digital comic is (comics scholars will know that nobody can agree on what a print comic is, so you can imagine the fun involved in trying to incorporate digital elements such as audio and video into the mix as well). It has also involved building the collection through web harvesting, tracking down webcomics for inclusion in the collection, and providing metadata (information about the collection item) for cataloguing purposes (this last task may happen to require reading lots of comics).

Alongside this, I am looking into ways that digital comics apps might be preserved, which is very proper serious work indeed. Not only are there many different versions of the same app, depending on what operating system you are using, but many apps are reliant not only on the software of the platform they are running on, but sometimes the hardware as well, with some apps integrating functions such as the camera of a tablet into their design. Simply downloading apps will provide you with lots of digital files that you won’t be able to open in a few years’ time (or possibly even a few months’ time, with the current pace of technology). This is not a problem that can be solved in the duration of a three month placement (or, frankly, given my total lack of technical knowledge, by me at all). What I can do, however, is find people who do have technical knowledge and ask them what they think. Preserving digital comics is a complicated and ongoing process, and it is a great experience to be in at the early stages of exploration.

And you can be involved in this fun experience too! If you are a digital comics creator, publisher, or reader, we would love to hear from you. We’d like to know more about the digital comics that you create, find out links to add to our Web Archive collection, and find examples of comic apps that we could collect. Please email suggestions to [email protected]. For this initial stage of the project, we will be accepting suggestions until the end of August 2017. In that time, we are particularly keen to receive web addresses for UK published webcomics, so that I can continue to build the web archive, and do the proper serious work of reading lots and lots of comics.

 

Pictured: Official EU referendum campaign leaflets – Remain (left hand side) and Leave (right hand side). Do you see any similarities?

My name is Alexandra Bulat and I am a PhD student at the School of Slavonic and East European Studies, University College London. My research is on attitudes towards EU migrants in the UK, based on fieldwork in Stratford (London) and Clacton-on-Sea.

The 2016 EU referendum campaign represents an important period when attitudes towards the topical ‘uncontrolled EU migration’ were shaped, expressed, and passionately debated. In this context, websites and social media played a key role in presenting the public with arguments about EU migrants and migration. Can we find the same campaign information today by browsing web resources? Some campaign websites have since been amended, renamed, redesigned, or simply disappeared from the visible online space. Here is where the UK Web Archive can help researchers like me who analyse particular events in history, such as the EU referendum.

In June 2017, I started a three month placement with the British Library Contemporary British Collections. The project is titled Researching the EU Referendum through Web Archive and Leaflet Collections. I use the EU referendum web archive and 177 digitised leaflets and pamphlets (available in the LSE Digital Library ‘Brexit’ collection) to answer the following research question: Who is speaking about EU migration and how?

In the first stage of research, I created a spread sheet for the leaflets and pamphlets, recording basic information such as title, organisation, and their position in the campaign. I also included all the content about freedom of movement, migrants, refugees, and closely linked topics. Overall, almost two thirds of the materials supported remaining in the EU, with only five categorised as ‘neutral’ and the rest arguing for leaving the EU. Just under half of these materials mentioned immigration, with more ‘Leave’ than ‘Remain’ sources. About a fourth of the items were clearly targeted to a specific region or town/city, the most common being London, Cambridge and various locations in Wales.

The second stage involved using the UK Web Archive to search for the websites and social media (in particular, Twitter handles) that were explicitly mentioned in the printed material, or that I could easily infer from the information available. Only six leaflets did not mention an online presence and I was unable to find it any evidence of it. However, the large majority of them had website(s) or social media mentioned in the printed publication. I ended up with a list of 49 main websites and social media presence for over half of them. Almost all those websites were archived, so I could see the exact information which had been live during the referendum. Most websites were available in the UK Web Archive, but some archived copies were only found in the Internet Archive. For comparison purposes, I looked at the latest record each website had before June 23^rd. For some this was as close as 22 June, offering a real snapshot of the debate right before the polling day, but others were not archived in 2016 at all (but had earlier records).

There is a variety of websites, from the official Vote Leave (www.voteleavetakecontrol.org) and Britain Stronger In Europe (www.strongerin.co.uk), to less familiar campaigns such as University for Europe (www.universitiesforeurope.com) and The Eurosceptic (www.eurosceptic.org.uk). A majority of these websites are in the Library’s ‘EU Referendum’ special collection, which brings together a range of websites such as blogs, opinion polls, interest groups, news, political parties, research centres and think thanks, social media and Government sources, who all wrote about the Referendum. Nevertheless, some smaller campaigns, or websites that are not necessarily dedicated to the Referendum but included some content about it, were not included the special collection.

One example of the importance of archiving the web is www.labourinforbritain.org.uk . Although this is a rather well known campaign (which even has its own Wikipedia page, where this website is quoted), its website is not ‘live’ anymore.

Screen capture 1: ‘Live website’, 1 August 2017

The UK Web Archive only started making records of it in 2017, but it had already displayed an error message. However, the Internet Archive has snapshots from before it disappeared from the live web. The Labour In campaign is an important resource for my research – it is one amongst a small number of sources making a more positive case about EU migration, which is essential to compare and contrast to the less favourable arguments made by other campaigners. Although the main Labour Party website had a tab about the Referendum, it did not include the same content as this campaign website, entirely dedicated to referendum issues.

Screen capture 2: ‘Archived website’, 22 June 2016

In addition to finding information that is not ‘live’ anymore, the web archive helps to contextualise the leaflets and complement the information provided in those printed campaign materials. The Bruges Group webpage is a good example in this sense. The digitised leaflet collection has four different leaflets from them. However, a comprehensive list of viewable leaflets is available on the archived website. In this case, the information was still on the live web when I last checked (apart from a slight change in formatting). However, no one knows for how long it will remain there, particularly after ‘Brexit’ is not anymore in the public debate.

By helping recover seemingly ‘lost’ information, complementing other datasets, contextualising the research and possibly many other roles, web archives are valuable resources that researchers should be encouraged to explore in greater depth. To mark the end of my PhD placement, I am helping to put together a roundtable discussion at the British Library with EU referendum collection curators and academics from a number of institutions, to create the space for conversation around future use of web archives in academic research and beyond.

Alexandra Bulat, August 2017

Save

Save

Save

Save