By Jennie Grimshaw, Curator of Official Publications, The British Library

The vote to leave the EU on June 23^rd 2016 by 52% to 48% bitterly divided the nation. Remainers argued that the margin of victory was narrow, and that Leave voters had been misled. They saw Brexit as a self-inflicted wound and campaigned for a second referendum.  Leavers accused the Remain camp of seeking to undermine democracy and frustrate the clearly expressed will of the majority of voters. They saw Brexit as an opportunity to throw off what they viewed as the shackles of the EU and become a proud, independent sovereign nation once again.

The Brexit web archive collection follows on directly from the EU Referendum Collection and traces the course of the increasingly bitter battle to either overturn Brexit or “get it done” from June 24^th 2016 to Brexit Day on January 31^st 2020.  It was created by web archivists and curators at the British Library, the Bodleian Library, and the national libraries of Scotland and Wales working in collaboration over three years.  It seeks to achieve balance by including:

• News reports, indexed at the level of the individual article, from pro-Brexit, anti-Brexit and neutral newspapers and broadcasters. Pro-Brexit sources include the Sun, the Daily Mail and the Express newspapers. Overtly anti-Brexit sources are represented by the Guardian and the Independent.  Coverage of broadcast news sources focuses on the BBC, ITV and Sky News. The aim is to document every twist and turn of the unbelievably complex and convoluted process of delivering the EU Referendum result in order to guide enquirers through the labyrinth.
• A range of online only news and comment services, indexed at the site level, including Politics Home, Politics.co.uk, Huffingtom Post, CAPX, The Conversation, Spiked and Unherd. Unfortunately Politico had to be excluded as it operates outside of the UK web domain.
• Legislation and legal commentary, covering bills, Commons and Lords debates,  briefings on them by the Commons and Lords Libraries, legal challenges and case law, and comment in blogs such as EU Law Analysis and by professional groups such as Lawyers for Britain
• Government and Parliamentary action, offering Commons and Lords Select Committee reports, analysis by the Commons and Lords Libraries, and speeches, press notices, letters and guidance from GOV.UK.
• The EU view on developments as reported by UK-based news sources. The constraints of the UK Non-Print Legal Deposit regulations meant that we could not include EU institutional websites based in Brussels!
• The voices of politicians of all parties, heard through their websites, Twitter feeds, and blogs such as Conservative Home, Conservative Woman, Labour List and Liberal Democrat Voice.
• Comment by pro- and anti-Brexit pressure groups such as Best for Britain and the People’s Vote Campaign (anti-Brexit) and Brexit Central and Better Off Out (pro-Brexit)
• The voices of a wide range of trade and professional associations,  charities, business organisations and trade unions lobbying government and speaking up for and advising their members. Some voices are sadly silenced due to technical difficulties, such as the website of The3Million, which speaks for EU citizens in the UK, but which cannot be archived because it is Wix-based.  Others which are heard range from business associations MakeUK  and the CBI to charities like Age UK and professional bodies such as the British Veterinary Association.
• Research and analysis by pro- and anti- EU think tanks such as Centre for European Reform, the Federal Trust,  Policy Exchange, the Institute for Fiscal Studies, the Red Cell, Briefings for Brexit,  Demos, IPPR  and Centre for Policy Studies, the Institute for Government, Global Britain, Politeia,  etc.
• Sub-Collections on the impact of Brexit on Wales and Scotland built up energetically by colleagues at their respective National Libraries.
• Sub-collections on the impact on Northern Ireland and the Republic of Ireland, especially the vexed question of the Irish land border.

The path leading to Brexit day on January 31^st was long and winding. The debate polarised society and strained the UK constitution to breaking point as Remainers in a hung Parliament fought across party lines to delay or prevent Brexit  and Leavers went to equally extreme lengths to deliver it. The process saw the birth of Lawfare as Remainers used legal challenges to seek to block it.  Brexit also destabilised the devolution settlement and reignited calls for Scottish independence. We hope that by documenting the debate as it played out on the Internet and on social media we can help researchers to gain understanding in retrospect of this tumultuous period of our history.

By Nicola Bingham, Lead Curator of Web Archiving, The British Library

Note: This post was originally published on the Digital Preservation Coalition (DPC) blog.

The UK Web Archive, a partnership of the 6 UK Legal Deposit Libraries* (LDLs), has been collecting UK websites since the early 2000’s. As well as archiving snapshots of the whole UK Web Space we have dozens of curated collections focussing on a wide range of topics, themes and events reflecting all aspects of UK life.

Collections are instigated by a broad range of curators – in this context, ‘curator’ is not necessarily synonymous with job title - including LDL staff, academic researchers, various UK GLAM organisations (e.g., Jersey Heritage, Hampshire Archives and Local Studies, Wimbledon Lawn Tennis Museum) and local community groups. Collections may focus on a researcher’s area of interest, align with an institution’s collection policy or reflect diverse political, sporting or topical events such as the London Olympic Games, Brexit or Climate Change. Below are the members of the Web Archiving team at the British Library.

We have a particularly strong time-series of collections focusing on UK General Elections having archived every campaign since 2005. For each event we have used more or less the same categories – candidate’s web presence, national and local political party websites, online news and commentary, interest group manifestos and comment and analysis by think tanks.

Structuring the collections with consistent sub-categories enables curators to distribute web archiving more efficiently, as does dividing selection broadly along the lines of the geographical interest of the 3 National Libraries that belong to the UKWA.

We hope that our General Election collections will preserve the voices and illustrate the concerns and priorities of a wide spectrum of UK society and help to show how political parties and candidates engaged and responded at pivotal moments in UK history.

It is interesting to note how use of the Internet for political campaigning and communication has evolved over time. In 2005 very little social media existed and politicians were just beginning to explore its capabilities, whereas by 2019 campaigners were making little or no use of websites, concentrating almost exclusively on using social media.

The (somewhat) scheduled nature of UK General Elections, especially since the Fixed-term Parliaments Act of 2011, allows us to plan election web archiving strategies ahead of time. Having said this, we have been tested in recent years with snap elections in June 2017 and December 2019! And of course candidates are only announced a couple of week’s before polling day which means we have to react at that point to archive candidate’s websites, or official, publicly facing social media accounts.

Rapidly unfolding events such as natural disasters or terrorist attacks require a different approach. However, even here we have some experience, having archived collections about the London Terrorist Attack 2005, Grenfell Tower Fire, and Pandemic Outbreaks such as Avian Flu and Swine Flu over the years.

For the past few weeks we have been actively collecting the UK perspective of the Coronavirus (COVID-19) Pandemic. We are clearly facing one of the severest threats in our lifetimes, certainly one of the fastest and most clearly devastating, and while Librarians might not (yet) be members of the Emergency Services, we feel the act of recording the outbreak as it plays out online is a crucial one.

Websites are being selected by a cohort of curators across the LDLs and beyond. We have also been ably assisted by colleagues at the Royal College of Nursing Archives who are nominating health-related websites. However due to the unpredictable, fast paced nature of the outbreak and the consequent deluge in online information, it is more important for us to harness the crowd to elicit website nominations. For this reason, we will canvas for website nominations much more widely among our colleagues, the library and archive community and the general public when responding to rapidly unfolding events. We will also visit targeted websites much more frequently than we would usually to capture frequently edited web content.

The collection is not public yet while we concentrate on acquiring the websites. Once we’re finished, it will take time to prepare the collection for publication by performing quality assurance and clearing permissions for open access. In due course, the Coronavirus collection will be available here under the Pandemic Outbreaks Collection. The top-level heading reflects the fact that we have previously collected around Avian Flu and Swine Flu and acknowledges that, sadly, we will be collecting about future outbreaks.

In terms of getting involved, we welcome submissions from colleagues in the DPC community - and in fact from any member of the public. Details of how to nominate websites for inclusion are here: www.webarchive.org.uk/nominate. Alternatively, please email nominations to [email protected]
We’re also working on an international collection with the International Internet Preservation Consortium (IIPC). Details of how to contribute to this collection are here: netpreserveblog.wordpress.com/2020/02/13/cdg-collection-novel-coronavirus/ (non-English language websites are particularly welcome here).

If your organisation has not previously done any web archiving and you would like to capture your own institution’s or communities’ response to Coronavirus, plenty of tools exist that can be used remotely. Webrecorder is a good place to start as it can be used in a browser, free of charge up to a 5GB data limit. Of course web archives such as the UKWA and Internet Archive would also be very happy to preserve your websites free of charge (see details above).

*The UK Legal Deposit Libraries: Bodleian Libraries, Oxford University, British Library, Cambridge University Libraries, National Library of Scotland, National Library of Wales, Trinity College, Dublin

By Pete Hebden, Phd Student Placement, Newcastle University

One of the great features of the UK Web Archive is its series of curated collections, which can be found on the UKWA Topics and Themes page. Each collection centres on a specific topic, some responding to particular events, such as Brexit or the First World War Centenary, others drawing upon the knowledge of contributors to create a set of in-depth examples around a particular subject. During my time at the British Library, I spent some time contributing to the Poetry Zines and Journals collection, originally started in 2016 by previous PhD placement student Joe McCarthy. The collection contains an amazing range of UK-based online outlets for poetry, encompassing blogs, Twitter accounts, online journals, and the personal websites of some poets, where there is a significant amount of the poet’s creative work on the site.

Poetry writing

 Although the collection was already very well curated when I came to it, it had not been significantly updated in several years, so many newer publications were not included in the collection. The past few years have seen a serious increase in the number of high-quality online literary journals – a trend that this collection was very astutely responding to when it was first created – and so there were a number of recent but well-established poetry titles that I could add to the list. One example is perverse, an online-only poetry journal started by the poet Chrissy Williams in 2018.

Along with roughly a third of the Poetry Zines and Journals collection, the archived version of perverse is only accessible on-site at the reading rooms of the UK’s legal deposit libraries. The rest of the content, for which open access permission has been obtained, can be viewed from anywhere. The Rialto and Porridge magazine are two examples of recently added sites that are open access, and the links here lead to the archived versions of those sites.

My other choices for inclusion in the collection were guided by some of my own specific areas of knowledge and interest. I included several online journals that are based in, or focus on, the north of England and Scotland, as these are literary scenes that I am more familiar with. Butcher’s Dog and Another North are two relatively recent literary journals based in the north of England. Another North is entirely digital, while Butcher’s Dog is a print journal with a strong online presence. I also added several more websites for print magazines that feature a significant amount of poetry on their site. For example, Popshot and The Rialto, both print magazines, regularly feature poetry from their most recent issues on their websites and/or social media, which gives readers an idea of the journal’s editorial policy and marks a significant change in the way that poetry is distributed by these publications thanks to the internet.

One interesting problem that I encountered during the process was around the formats that some digital publications use for distribution. While most online poetry journals choose to publish in a standard website or blog form, some distribute each issue as a downloadable file, such as a PDF or EPUB. This method of delivery presents a problem when attempting to archive the content, as the web crawler is not necessarily able to access and download these files, meaning that the poetry itself goes unrecorded. For these journals, we had to use alternative ways of recording their poetry in the collection. For example, perverse (mentioned above), as well as distributing each issue as a PDF download, also posts each poem individually to Twitter, and so we set up a regular capture of their Twitter account in order to record all of the poetry. Many other journals use social media in a similar way, and so in these cases I was able to use this as a way of archiving the journal’s output.

Over the past few decades, the web has provided an exciting platform for a diverse range of poets and publishers to showcase their work and it has been a very enjoyable challenge to contribute to the cataloguing of this transformation. I hope that my work on the Poetry Zines and Journals collection will help other readers and researchers exploring the breadth and variety of UK poetry available online today. 

If you know of any websites that should be included in this collection or in the general UK Web Archive, please nominate it.

 

By Jason Webber, Web Archiving Engagement Manager, The British Library

Like many public places at the moment, the reading rooms of the UK Legal Deposit Libraries are going to be shut for some time. What does this mean for the UK Web Archive? Well as some of you might know we try to collect every UK website at least once a year and this is done under the provision of the Non-Print Legal Deposit Regulations 2013. A condition of these regulations are that content collected can only be viewed on library premises. Never fear though, we still have lots for you to do!

UKWA website home page

Discover millions of websites
At the end of 2018, we launched our new service for searching the whole UK Web Archive catalogue from anywhere. Go to our website: www.webarchive.org.uk and search for a web address (URL) or word/phrase. You will get results showing all of our resources that you can access from anywhere. Tick the box 'At Libraries' to see everything that we have collected. Access thousands of websites
Over the 15 years that we have been archiving websites we have frequently sought permission from owners to make their sites publicly viewable outside of library premises. In that time we have received permissions from over 15,000 website owners These websites have been selected because they relate to a  specific topics or event, for their importance, or because they were about to go offline. Lots to see!

Browse 'Topics and Themes'
You can browse over 100 different topics and themes. From the extensive 'Brexit' collection to 'Web Comics' there is something for everyone. As a starter, check out 'Online Enthusiasts' and discover many of the  hobbies and societies in the UK.

SHINE service
UK Web Archive holds a collection of all the .uk websites that were archived by the Internet Archive between 1996-2013.  The service includes a 'trends' feature that we highly recommend that you try.https://www.webarchive.org.uk/shine/graph

You can enter a word or phrase (in speech marks) to see the relative popularity in a given year. Enter different terms separated by a comma and you can compare popularity e.g. tom,jane. See who is 'best',  cat or dog or the emergence of words such as 'iphone', 'emoji' or phrases such as 'credit crunch'.

Do tell us what you find!

Trends - the use of the term 'loungewear' in the UK web

Nominate websites
We are still able to add websites to the archive and welcome nominations! We want to archive every single UK website and your help is invaluable. Make your suggestions here: www.webarchive.org.uk/nominate

Stay safe everyone!

@ukwebarchive

By Caio Mello, Doctoral Researcher at the School of Advanced Study, University of London

Recently, exploring the data available in the UK Web Archive related to London’s 2012 Olympic legacy, I found a very curious fact. Boris Johnson - Mayor of London during the games - told the BBC in 2013 that a baby boom in London that year was among the legacies of the event. According to Johnson, his team at City Hall had looked at the data and found a rise in birth rates that year not seen in the capital since 1967, the year after England won the FIFA World Cup. Moreover, Johnson said that even Kate Middleton and Prince William’s first baby could be considered a post-Olympic outcome.

In a recent blog post I briefly introduced my research on media coverage of the Rio and London Olympics and discussed the wide range of attributes to which the word legacy has been attached. As part of a Digital Humanities project, this study seeks to develop an interdisciplinary approach to the topic, combining both qualitative and quantitative methods. In order to do that, I have been looking at different repositories of news articles, including web archives. In addition to accessing content available in the UK Web Archive by going to the British Library, I have also searched for news articles through SHINE, a tool developed as part of the Big UK Domain Data for the Arts and Humanities project, to explore UK web content collected and stored by the Internet Archive. SHINE offers access to an open data repository that has allowed me to conduct multiple studies by writing Python scripts that return language and textual analysis. By scraping some of the news articles from SHINE and analysing word frequency in an initial exploratory study, I have been able to get a sense of how broad the concept of legacy might be.

Although often concerned with what could be described as ‘material legacy’, the articles and reportage go beyond physical infrastructure - such as stadiums - to describe expectations that more people will practise sports or even that a country might be more strongly recognised as open and welcoming. Legacy definitely seems to carry a very positive meaning per se and, when it refers to negative outcomes, it often seems to flirt with irony. On the other hand, words like gentrification appear in very dubious contexts: sometimes they refer to regeneration and development, at other times to an unsustainable process that leads to people’s exclusion from traditional areas affected by this sort of transformation.

While Johnson’s reference to the baby boom can be understood as a joke, it reveals how obsessive politicians can become in using the official narrative of Olympic legacy as it relates to their particular country or host city. A good legacy, as pointed out by MacRury and Poynter in Olympic Cities: 2012 and the remaking of London, is fundamentally important for managing tensions between Olympic dreams and huge economic investment.

In December 2019, Boris Johnson, now the Prime Minister, tweeted his desire to host the football World Cup in 2030. “I want it to show our national confidence as we get Brexit done”, wrote Johnson. Once again, immaterial legacy emerges at the heart of a political argument defending the choice to participate in such mega-events. Looking deeply into these multiple dimensions of legacy seems to be an important step to understand, through language usage, how narratives have been built around the Olympics and how different actors have appropriated the concept and disputed its meanings.

By Andy Jackson, Web Archiving Technical Lead, The British Library

My father used to joke about how he’d had his hammer his whole working life. He’d bought it when he’d started out as a joiner, and decades later it was still going strong. He’d replaced the shaft five times and the head twice, but it was still the same hammer! This simple story of maintenance and renewal springs to mind because a few days ago, we finally managed to replace the most important component of our data store. Our storage cluster has been running near-continuously for almost a decade, but as of now, every single hardware component has been renewed or replaced.

Andy's Dad's hammer

We use Apache Hadoop to provide our main data store, via the Hadoop Distributed File System (HDFS). We mostly like it because it's cheap, robust, and helps us run large-scale analysis and indexing tasks over our content. But we also like it because of how we can maintain it over time.

HDFS runs across multiple computers, all working together to ensure there are at least three copies of any data stored in the system, and that these copies are in separate machines and separate server racks. It runs like a beehive. The 'queen' is called the Namenode, and although it doesn't store any data, it keeps track of where all the data is and orchestrates the ingest and replication processes. The 'worker' nodes just store and maintain their own blocks of data, and send data back and forth between themselves as instructed by the Namenode. The Namenode also provides the interface we use to access the system, referring each client to the right set of worker nodes as files are accessed. All the time, the system calculates checksums of the chunks of data and uses this to verify the integrity of the files.

This architecture was designed to anticipate hardware failure and recover from it, which makes the system much easier to maintain. If a drive, or even a full server fails, we can simply remove it, replace it, and keep an eye on it as the data is re-distributed. As new, higher-capacity drives come along, we can upgrade the drives in each node one-by-one, in a rolling update that grows the capacity of the cluster.

Rear of the UKWA racks

Similarly, over time, we can upgrade the operating system and other supporting software on every node, to make sure we're up to date. Almost all of this can be done while the system is running, without interrupting access. But the exception is the Namenode – as a hive needs its Queen, HDFS needs its Namenode, so we avoid interrupting it unless absolutely required. It had been running on the same hardware all this time, but now it's happily running on a new bit of kit. At last.

Like the Ship of Theseus, every piece has been replaced, but it's still the same store, and the data is still safe. Of course, it's not as easy to manage and as transparently scaleable as Cloud storage, but for on-site storage it does a great job. Rather than having to shift between storage silos every few years, the data is in constant motion, and this design allows the components and support contracts for the different layers to move at different speeds and rates of renewal over the years. This is one of the advantages of open source systems – they can provide a stable interface for a service, decoupled from any particular vendor or hardware, allowing support methods, contracts and contractors to change over time.

But HDFS has strong competition these days. There's many other options, many of which are compatible with the defacto standard, S3 (Simple Storage Service).. Being able to work with the same interface whether storage is local or in the cloud might make all the difference. We're happy with HDFS for now, but we'll be preparing for the day when a new ship comes alongside and it's time to shift the cargo...

By Rachel Brett, Science Reference Team Leader, The British Library

The British Library is one of six UK legal deposit libraries.  We are a dual site based in London and Boston Spa, Yorkshire. At St. Pancras in London we have eleven different subject focused reading rooms comprising of:

• Humanities 1
• Humanities 2 - Sound & Vision
• Newsroom
• Science 1 & 2
• Social Sciences
• Business & Intellectual Property

Plus specialist collections readings rooms:

• Manuscripts
• World’s largest Maps collection
• Rare Books early pre 1850 print & Music
• Asian & African Studies

British Library reading room

All reading rooms are staffed by qualified Reference Specialists providing librarian support to all our readers. Access to the Reading Rooms is by free Reader Pass which everyone needs to register for first before they can enter the rooms. Online pre-register is available which must be completed in person with 2 forms of ID. It is advisable to check our website as only certain types of ID are suitable.

Once registration is complete readers are asked to leave large bags and outdoor coats in the cloak room and lockers. We are looking after the National collections, and therefore have some restrictions, including: No pens (pencils only) no food or drink or other items that may damage or mark collection items and no downloading (the only exception being BIPC1).

Depending on the subject and more specifically material type that is being consulted readers can select the room they wish to use. The exception being specialist collections can only be viewed in the respective reading rooms.

“Where are all the books?”
We hold millions of copies of collection items and as such you can’t come in and find all the books on the shelves as you would in public libraries. Our collection is in closed storage and can only be accessed via searching and ordering items from our online catalogues.  The main catalogue for the majority of the Library's collection, including digital items is called Explore the British Library and can be used remotely to discover and locate items, identify delivery times and order material. Items that are held in our basements at St. Pancras take 70 minute for delivery, while collections held at Boston Spa have a 48 hour+ delivery time, with no deliveries at weekends.  Readers can order up to 10 items per day and 6 can be kept on reserve for 3 working days on a rolling basis. The only material that is held on open shelves in the reading rooms is our reference collections including dictionary, encyclopaedia. National bibliographies, biographies etc.  Additionally, we also subscribe to databases, the majority of which can also only be accessed in the reading rooms.

More than just books
We are a hybrid library, and have multi-format collections covering all subjects, including

• Legal Deposit copies of items published within the UK & Ireland- which also now included digital items and ebooks
• Open access reference works
• World’s largest Maps and Rare Books collections
• Newspapers, Journals, comics, zines
• Manuscripts & Music scores, Theatre archives
• Indian Office records, photographs
• over 2 million sound recordings
• Stamps, Official publications
• 5 million reports, conference papers
• over 50m patents – the largest collection in the World

Oh yes and the UK Web Archive! 
Only some of the UK Web Archive is available to access anywhere, the majority is only available in Reading Rooms of the six UK Legal Deposit Libraries.

To access the web archive, take your readers pass to any of the reading rooms and find a free computer terminal. From there, go to www.webarchive.org.uk and you will be able to access all of the UK Web Archive collection.

Also note that there is FREE WIFI, and items can be scanned freely to email or USB, digital items cannot be downloaded, but photography is allowed with some restrictions. Copyright law applies.

Additionally we offer a Chat service and 1-2-1 sessions delivered by our reference teams to help readers navigate our procedures and collections.

Think back 15 years to the beginning of 2005. Future Prime Minister David Cameron wasn't yet Leader of the Conservative party and Google Maps, Twitter and the iPhone all had yet to be launched. It was, however, the year that we started collecting copies of UK published websites for permanent preservation and access.

First UKWAC website via the Internet Archive

Our Origins
In the beginning a group of interested UK institutions - The British Library, The National Archives, Wellcome Trust, National Library of Scotland, National Library of Wales and  JISC - formed a consortium (UKWAC) to implement a project to archive websites.

A multi-disciplinary team was formed to look into the many challenging technical and curatorial issues involved in archiving websites. The learning curve in this field can be steep and was especially so then. At the time only a few other organisations were in this field, including the Internet Archive and the National Library of Australia (NLA). In fact, initially, it was a tool developed by NLA - Pandas software - that we used in those early days. Later on we switched to using Heritrix to collect the web (and still do).

One of the special elements of archiving the web is that whilst it can be difficult, this very challenge encourages co-operation and partnership. We have been very grateful for all of the input and help along the way.  

What to archive?
In this early era of web archiving, website 'targets' were carefully selected by curators and the owners were asked for permission for us to archive and publicly display them. If the website owner refused, which is very rare, or didn't respond then the website couldn't be archived. 

Overall, the responses that we received from website owners were overwhelmingly positive and led to the formation of key early collections such as the 2005 General Election and the 7 July London terrorist attacks.

BBC News 2005 Election website

Where next?
By early 2005 the first websites were being captured and stored, however, it wasn't until May 2005 that we first displayed them to the public.

The evolution of the UK Web Archive website and how we developed a growing number of 'special collections' (as they were initially known) will be the subject of future blog posts over the next few months.

Watch this space!

#15YearsOfUKWA