By Nicola Bingham, Lead Curator of Web Archives, British Library

Home page of the ve-vjday80.gov.uk website

This year marks a significant national milestone: the 80th anniversary of the end of the Second World War. With Victory in Europe (VE) Day falling on 8th May and Victory over Japan (VJ) Day on 15th August, commemorations are planned across the UK to honour the conclusion of a conflict that reshaped the world.

To document this anniversary, the UK Web Archive is curating a special collection titled "VE / VJ Day 80", which will record how people and communities across the UK are commemorating the end of WWII, from national ceremonies to local grassroots events.

Collection Scope

This curated collection focuses on UK-based websites documenting commemorative events, public activities, and community involvement related to VE/VJ Day 80. Rather than a detailed historical retrospective, the collection aims to reflect contemporary responses and engagement with this anniversary.

Key Aspects of UK Commemorations

The collection includes a wide variety of commemorative themes and activities such as:

· National Events: Organised by groups like the Royal British Legion, including parades and memorials.

· Local Celebrations: Street parties, community gatherings, and regional events.

· Church Services: Remembrance services held nationwide.

· Beacon Lighting: Symbolic ceremonies at dusk.

· Remembrance Readings: Recitals of "The Tribute" and similar dedications.

· Veteran Involvement: Honouring the voices and presence of those who served.

· Contrasting voices or critical perspectives of the commemorations.

Why We Are Archiving This

By collecting these websites now, we’re creating a rich and enduring resource for future researchers, historians, educators, and the general public. This collection will preserve not only official narratives but also grassroots and personal perspectives, reflecting the diversity of the UK’s commemorative landscape.

One recent example of how the UK Web supports research is the work of Dr Liam Markey, whose blog post, published earlier this week, describes how he has used archived web content.

Between 2018 and 2023, Liam completed a PhD at the University of Liverpool in collaboration with the British Library, examining how remembrance practices in Britain, particularly the concept of military victimhood, shape national identity and reflect militaristic thinking. His work highlights the value of digital resources like the UK Web Archive in documenting contemporary remembrance culture.

How You Can Contribute

We welcome nominations of websites, blogs, and social media accounts that reflect VE/VJ Day 80 commemorations and perspectives.

Are you organising a public or community event?

Are you sharing your thoughts or experiences online?

If so, we’d love to hear from you.

Please email your suggestions to: [email protected] 

Although the UK Web Archive website is currently offline, our team is actively capturing web content using remotely hosted systems, ensuring this material is preserved for the future.

Here are a few examples of sites already being archived:

Royal British Legion – Remembering the End of WWII (https://www.britishlegion.org.uk/getinvolved/events/remembranceevents/rememberingtheendofthesecondworldwar)

VE Day 80 Community Events (https://www.veday80.org.uk/)

VE/VJ Day 80 (https://ve-vjday80.gov.uk/)

English Cathedrals – VE Day Services (https://www.englishcathedrals.co.uk/latestnews/veday808thmay2025asharedmomentofcelebration/)

Breckland Council – Remembrance Grants & Readings (https://www.breckland.gov.uk/article/24080/VEVJDay80AnniversaryGrants)

Royal Navy – WWII Veterans’ Stories (https://www.royalnavy.mod.uk/news/2025/january/06/20250106ww2veteransurgedtocomeforwardtomark80thanniversary)

Beacon Lighting Guide (Glinton Parish Council) (https://glintonpc.gov.uk/wpcontent/uploads/2024/07/VEDay80AnniversaryGuidev19.pdf)

VE Day Blog Posts from the British Library

This is one of multiple blog posts being published across the British Library blogs this week:

UK Web Archive: https://blogs.bl.uk/webarchive/2025/05/digital-memory-and-the-militarised-past.html 

European Studies: https://blogs.bl.uk/european/2025/05/remembering-sacrifice-celebrating-freedom.html

Newsroom: https://blogs.bl.uk/thenewsroom/2025/04/ve-day-in-the-news.html 

Social Science: https://blogs.bl.uk/socialscience/2025/05/ve-day-voices-from-history-.html 

Untold Lives: https://blogs.bl.uk/untoldlives/2025/04/children-in-war-time.html 

By Dr Liam Markey, University of Liverpool

This blog post will explore the immediate legacies of the First World War centenary in Britain, looking towards the culmination of the ongoing commemoration of the Second World War’s 80^th anniversary, with VE and VJ day being commemorated in May and August of this year respectively. It describes how discourse surrounding both world wars has shaped British attitudes and behaviours concerning conflict and military service over the last century, and how changing demographics may serve to consolidate these beliefs in the coming century. Special attention is paid to mixed media texts collected and held by the British Library, demonstrating the significance of the UK Web Archive (UKWA) in particular as a repository of counter-culture discourse in the context of British militarism.

A Second Century of Remembrance
As a response to the centenary of the First World War, between 2018 and 2023 I undertook a PhD at the University of Liverpool in collaboration with the British Library. My research cast a critical gaze upon the act of remembrance in Britain since the end of the First World War, with special attention paid to the concept of ‘military victimhood’ and its potential to mediate militaristic modes of thinking. The project was embarked upon in the wake of the national commemoration of the centenary of the First World War, a watershed moment in Britain which prompted the production of a myriad of state-funded cultural and educational events.

As David Cameron announced in 2012, British commemoration of the centenary would serve to,

“provide the foundations upon which to build an enduring cultural and educational legacy, to put young people front and centre in our commemoration and to ensure that the sacrifice and service of a hundred years ago is still remembered in a hundred years’ time.”

While such an important historical moment arguably provides an invaluable opportunity for critical reflection, my research ascertained that, largely, the centenary instead engendered a consolidation of, and recommitment to, traditional forms of remembrance. Ultimately, the foundations that the centenary provided were not ground-breaking, rather they had already been established during, and enacted since, the end of the First World War itself. This next century of commemoration, as envisioned by Cameron, would be cast in the image of the last, anchored upon the rituals and practices of what is referred to as the ‘1919 model’.

This in and of itself can be regarded as potentially problematic, as my research, and the work of many other scholars, demonstrates the proclivity of such forms of commemoration to perpetuate the core tenets of a militaristic ideology; seeing war glorified, justified, and normalised. This is largely achieved through the sanitisation of war’s power to victimise, with emphasis placed on an idealised vision of warfare and military service. ‘Official’ or ‘dominant’ narratives of commemoration also emphasise the unifying power the rituals of the 1919 model, such as the two-minute silence or the wearing of a poppy, have among the British population and the positive effect enactment has in relation to British victims of war. Commemorative discourse overwhelmingly emphasises notions of debt that the public are duty-bound to fulfil, while avoiding direct reference to war’s inherent violence and propensity to produce victims.

This depiction of warfare present in commemorative practices was chosen to serve a very specific purpose, as a way of alleviating the suffering of the bereaved by acknowledging that their loved ones died in service of a noble ideal. However, with much, if not all, of those for whom the 1919 model was created having now passed from the British population, sentiments of military service as being inherently glorious, core to dominant commemorative narratives, serve to sanitise war for generations of individuals with no personal experience of war’s traumatic reality.

Alongside overt references to war as glorious and necessary within dominant commemorative narratives, my research also uncovered the role of the ‘commemorative deviant’. These are individuals who choose to commemorate war in a manner outside of the official purview, and as such are vilified in the national mainstream media, encouraging others to condemn rather than replicate such behaviours. The majority of such depictions come from print texts taken from three mainstream British newspapers: The Daily Mail, The Daily Mirror, and The Times, collected for analysis from the British Library’s Newsroom. These newspaper texts serve to reinforce specific beliefs and behaviours concerning remembrance over the last century that ultimately perpetuate, rather than challenge, militaristic notions.

Mainstream narratives purport that since the end of the First World War commemoration has been static, with its enactment based on a general consensus, and those rare deviant individuals represent an anomaly rather than a pattern of behaviour visible throughout the last century. However, through access to the UKWA, and close collaboration with the UKWA team, I was able to create a unique digital dataset which challenged such notions and provided a far more expansive view of commemoration as enacted in Britain since 1918. Beyond official black and white narratives of morally righteous consensus and villainous deviance, digital texts demonstrated the complexity of British remembrance. They provided an insight into ‘ground-up’ commemorative initiatives, uncovering attitudes more often than not absent from the mainstream media due to their potential to undermine notions key to the proliferation of dominant commemorative narratives.

Websites collected by the UKWA demonstrated the rich variety of methods by which war has been commemorated in Britain since the First World War, with many serving to challenge and deprivilege assumptions inherent within dominant narratives. These ‘counter’-narratives illustrated the vastness of the category of military victims, many of whom, such as civilians or enemy soldiers, are absent from mainstream commemorative discourse, and whose existence serve to undermine notions of militarism. Many instances of ‘deviancy’ in mainstream thought became in this context simply an alternative perspective, which ultimately facilitated the broadening of knowledge concerning the enactment of British remembrance over the last century.

Take for instance the existence of the white poppy, a symbol denigrated by newspaper texts in the sample as disrespectful and a direct contributor to the suffering of military victims, such as disabled ex-servicemen or the bereaved. Digital texts provide expansive contextual information, highlighting that the white poppy was itself incepted by ex-servicemen and relatives of the war dead as a commitment to peace, remembrance of all victims of war, and as a direct challenge to a militaristic ideology. Digital texts also highlight the existence of otherwise marginalised individuals, such as dissenting ex-servicemen, conscientious objectors, soldiers ‘shot at dawn’, or soldiers severely disfigured as a result of their military service.

Alongside an expanded purview regarding representation of military victims, the digital texts collected from the UKWA also provided access to the thoughts and feelings of the average British citizen, many of which clash with mainstream declarations of consensus and unity. Message boards and amateur websites serve as a medium for dissenting viewpoints, exhibiting the democratising power of the internet. Ultimately, the UKWA provided a much fuller picture of remembrance than the one evident in mainstream media, providing a platform for individuals who have not featured at the forefront of commemoration over the last century, but are nevertheless integral components in wider British narratives of war.

Second World War 80^th Anniversary

Seven years on from the centenary of the First World War, we now find ourselves approaching the culmination of the first decade of the second century of British remembrance, and at the apex of the Second World War’s 80^th anniversary, concluding in the commemoration of the victories in Europe and Japan; VE Day on 8^th May, and VJ Day on 15^th August respectively. Preceding the commemoration of victory in 1945, we have also seen other major historical moments of the Second World War commemorated since 2019, such as the Battle of Britain and the D-Day landings.

Thus far, these tentpole national commemorative events have largely been celebrations of victory rather than meditations on the destructive nature of war. Unlike the First, which has largely been portrayed in popular culture as a futile endeavour, the Second World War stands apart as a just war, a struggle between good and evil. In recent years, it is through the lens of the Second World War that official narratives of war in Britain have been constructed, providing a useful template with which previous and later conflicts can be created in the guise of. Such a foregrounding of a single war in mainstream narratives can result in the depoliticising and decontextualising of conflict, providing an ahistorical view of war as a natural and inevitable continuum. While responses to the First World War during its centenary did in part deal with the ambiguous nature of its necessity in being fought, the Second World War is far more widely accepted as entirely justified, as a national struggle for survival. While there is no doubt that defeat of fascism is a cause worthy of celebration, it must not serve to enable a sanitisation of war’s reality by colouring our perception of conflict overall.

The 80^th anniversary of the Second World War may well enable dominant narratives of war to become further entrenched in the national psyche, particularly as more and more individuals with first-hand experience of total war pass from the population. For a new generation of Britons, whose primary connection to war is through the mass media, and indeed commemorative events, there is a real danger that a sanitised and depoliticised view of warfare will become the norm, especially through an ever more celebratory depiction adopted by mainstream commemorative initiatives.

As with the First World War centenary, this is where the vital role of repositories such as the UKWA can come into play, providing alternative viewpoints upon the topic of war and ensuring that a wide variety of voices are heard, rather than obscured by the fanfare of national enterprises. In light of the 80^th anniversaries of VE and VJ day, the UKWA will curate a special collection documenting events and activities relating to the end of the Second World War, and invite the public to directly submit relevant websites by emailing [email protected].

Through the creation of such a collection, the UKWA will secure an invaluable repository of digital texts, which will not only serve as a preservation of an important historical event, but also as a vital resource for future scholars. Provided will be a unique insight into national forms of commemoration alongside those enacted by individuals and local communities. Digital texts held in the UKWA collections were central to my own research, offering a window into otherwise marginalised and unseen discourses, demonstrating the vast breadth of public responses to and enactments of remembrance in Britain since the end of the First World War. I hope that, moving forwards into this second century of commemoration, the UKWA’s important work will continue, facilitating significant reflection on remembrance for future generations.

Dr Liam Markey is a Research Associate at the University of Liverpool’s Department of Sociology, Social Policy and Criminology. He completed his PhD in collaboration with the British Library in 2023 and is currently working on a British Academy funded project exploring ethical digital public histories of prisoners and the legacy of enslavement in Georgia, USA.

[email protected]

LastPosts.blog

By Helena Byrne, Curator of Web Archives

The Routledge Companion to Transnational Web Archive Studies Edited By Susan Aasman, Anat Ben-David, Niels Brügger

On Monday 28, April 2025, The Routledge Companion to Transnational Web Archive Studies was launched. The book “explores the untapped potential of web archives for researching transnational digital history and communication. It covers cross- border, cross- collection, and cross- institutional examination of web archives on a global scale”.

It is an interdisciplinary collaboration and one of the last outputs from the WARCnet research network, comprising  28 chapters  grouped into five sections. The last chapter in each section is a conversation which multiple authors contributed to by responding to questions set by the editors related to the theme of that section.

Lead editor Susan Aasman stated “The companion contains concrete examples on how to research national web domains through a transnational perspective; provides case studies with grounded explorations of the COVID- 19 crisis as a distinctly transnational event captured by web archives; offers methodological considerations while unpacking techniques and skill sets for conducting transnational web archive research; and critically engages the politics and power dynamics inherent to web archives as institutionalised collections”.

UK Web Archive curators, based at the British Library together with curators at University of Westminster contributed to chapters and conversations in the book. The editors stated that “The Routledge Companion to Transnational Web Archive Studies is an essential read for graduate students and scholars from internet and media studies, cultural studies, history, and digital humanities. It will also appeal to web archiving practitioners, including librarians, web curators, and IT developers”.

To celebrate the launch of the book, Routledge is offering a 20% discount with the code 25AFLY2 on http://www.routledge.com/. This code expires on 30th September 2025 and cannot be used with any other special offers.

By Helena Byrne, Curator of Web Archives

DHNB 2025 Conference Banner

The UK Web Archive was one of five web archive organisations represented in the Web Archive Collections as Data workshop held at the Digital Humanities in the Nordic and Baltic Countries (DHNB) 2025 conference held at the National Museum of Estonia in Tartu. The UK Web Archive has participated in the 2025, 2024 and 2023 DHNB conference. The workshop was organised by Olga Holownia, Senior Programme Officer at the International Internet Preservation Consortium (IIPC). It served as an introduction to web archives and web archives collections as data with a focus on use cases but also the challenges related to producing, sharing and publishing, collections as data.

The first stage of the workshop gave a brief overview of the collections as data movement within the GLAM sector, and introduced the Collections as Data Checklist developed by members of the GLAM Labs community. It also introduced what web archives are and where you can access them, how a selection of web archives are making their collections available as data as well as what are the potential research opportunities for these collections. The panel included Olga Holownia (IIPC), Gustavo Candela (University of Alicante), Helena Byrne (British Library), Jon Carlstedt Tønnessen (National Library of Norway), Anders Klindt Myrvoll (Royal Danish Library), Sophie Ham and Steven Claeyssens (KB, National Library of the Netherlands). 

The UK Web Archive presentation promoted the recently published Datasheets for Web Archives Toolkit and the new metadata data sets that are available through the British Library Research Repository. The presentation gave an overview of how the project started, the background to how the Toolkit was prepared and how it was implemented.

Web Archives Collections as Data Workshop at DHNB 2025. Photographer: Helena Byrne & Carmen Kurg.

 

The activity stage of the workshop focused on how we could adapt the Collections as Data Checklist for web archives. The participants were split into three groups. They reviewed the checklist through the lens of if it is applicable to web archives, how it could be adapted if it does not fit, what solutions can be developed to overcome some of the challenging sections of the checklist. There was a rich discussion amongst the groups which also benefited from having both researchers and library professionals involved in reviewing the checklist.

Web Archives Collections as Data Workshop at DHNB 2025. Photographer: Carmen Kurg.

The general consensus from the groups was that maybe more detail is needed to accompany the Checklist so that it could be applied to web archive collections. Some of the points on the Checklist are particularly difficult to apply to web archive collections. There was a lot of discussion on the first two points as they cover licensing and citation. These are particularly difficult for web archives due to national legislation; most web archives operate on a dark or grey access model and most onsite terminals used to access web archives have copy and paste functions disabled so citation can become problematic. However, the participants were positive about the potential to apply an annotated or adapted Collections as Data Checklist specifically for web archives. The brainstorming session at this workshop was the first step of starting a discussion about what resources are needed to improve the process of publishing web archive collections as data. The second of these discussions was picked up at the IIPC Web Archiving Conference in April 2025. 

For a more general report from the DHNB conference click the link to the Digital Scholarship blog to read the report: https://blogs.bl.uk/digital-scholarship/2025/04/dhnb-2025-digital-humanities-in-the-nordic-and-baltic-countries-conference-report.html 

By Helena Byrne, Curator of Web Archives

Datasheets for Web Archives Toolkit

Since autumn 2022, Emily Maemura from the University of Illinois and Helena Byrne from the UK Web Archive team at the British Library have been exploring how the Datasheets for Datasets framework, devised for machine learning by Gebru et. al, could be applied to web archives. In order to explore the research question “can we use datasheets to describe the provenance of web archives, supporting research uses?” a series of workshops were organised in 2023. 

These workshops included a card sorting exercise with expertise in web archives as well as general information management. After the card sorting exercise there was a general discussion about using this framework to describe web archive collections.

These workshops formed the core of the guidance documentation published in the Datasheets for Web Archives Toolkit published in the British Library Research Repository.

The Toolkit

This Toolkit provides information on the creation of datasheets for web archives datasets. The datasheet concept is based on past work from Gebru et al. at Microsoft Research. The datasheet template and samples here were developed through a series of workshops with web archives curators, information professionals, and researchers during Spring and Summer 2023. The toolkit is composed of several parts including templates, examples, and guidance documents. Documents in the toolkit are available at a single DOI (https://doi.org/10.22020/rq8z-r112) and include:

1. Toolkit Overview 
2. Datasheets Question Guide
3. Datasheet Blank Template

Implementation 

The UK Web Archive has implemented this framework to publish data sets from its curation software the W3 Annotation Curation Tool (ACT). These data sets are available to view in the UK Web Archive: Data folder in the British Library Research Repository. So far there are just a few collections published but this will grow over the coming months.

By Carlos Lelkes-Rarugal, Assistant Web Archivist

When people think of web archiving, social media is often overlooked as a content source. Although it's impossible to capture everything posted on social media platforms, at the UK Web Archive, we strive to archive the public profiles of key figures like politicians, journalists, athletes and industry leaders. However, archiving social media presents significant challenges for any institution trying to capture and preserve it. Recently, a new tool has helped us archive social media content with greater success.

This blog post outlines our approach to archiving social media for the 2024 General Election, highlighting what worked well and identifying areas for improvement.

Challenges of the Previous Workflow

In an earlier blog post, we discussed our efforts in collecting content for the 2024 General Election. While we updated the user nomination process, we still relied on the same website crawler, Heritrix. Here is a simplified version of the previous workflow:

•       Nominate a seed
•       Validate seed and create a metadata record
•       Send seed and metadata to the Heritrix crawler
•       Archive, process, and store the website
•       Make the archived website available for viewing

This workflow enabled us to archive thousands of websites daily, thanks to Heritrix’s robust capabilities. However, despite its effectiveness at archiving static websites, Heritrix is less adept at capturing dynamic content such as maps or social media. While we can archive video, UK Non-Print Legal Deposit regulations prevent us from archiving video-streaming platforms like YouTube or TikTok.

The Challenges of Archiving Dynamic Content

Dynamic content is notoriously difficult to archive. Automated crawlers like Heritrix struggle with elements that rely heavily on JavaScript, asynchronous loading, or user interactions—common features of social media platforms. Heritrix cannot simulate these browser-based interactions, meaning critical content can be missed.

The challenge for web archiving institutions is compounded by the rapid evolution of social media platforms, which continually update their designs and policies, often implementing anti-crawling measures. For example, X (formerly Twitter) once allowed open access to its API. In April 2023, however, the platform introduced a paid API and a pop-up login requirement to view tweets, essentially blocking crawlers. This shift mirrors a broader trend among social media platforms to protect user data from unauthorised scraping and repurposing of data, a practice often linked to training AI models.

While archiving dynamic content is a known problem, finding tools capable of managing these complexities has proven difficult. Webrecorder, an open-source tool, offers one potential solution. It allows users to record their interactions within a web browser, capturing the resources loaded during the browsing session. This content is then packaged into a file, enabling the recreation of entire web pages. While Webrecorder has evolved, it is only part of the solution.

Introducing Browsertrix

Heritrix and Browsertrix both offer valuable solutions for web archiving but operate on different scales. Heritrix’s strength lies in its ability to handle a high volume of websites efficiently, but it falls short with dynamic content. Browsertrix, by contrast, excels at capturing interactive, complex web pages, though it can require more manual intervention.

Despite the increased time and effort involved, Browsertrix offers several key advantages:

•       High-Fidelity Crawling: Browsertrix can accurately archive dynamic and interactive social media content.
•       Ease of Use: Its user-friendly interface and comprehensive documentation made Browsertrix relatively easy for our team to adopt. Plus, its widespread use within the International Internet Preservation Consortium (IIPC) means additional support is readily available.

Archiving Social Media: A New Approach

One of the most significant challenges in archiving social media is dealing with login authentication. Most social platforms now require users to log in to access content, making it impossible for Heritrix to proceed beyond the login page. Heritrix does not create a browser environment, let alone maintain cookies or browser sessions, so it cannot simulate user browser interactions that are sometimes necessary to view or download content.

This is where Browsertrix excels. Operating within a web browser environment, Browsertrix can handle login credentials, enable browser events like drop-down menus, and capture content that loads asynchronously, such as social media posts. Essentially, it records a user’s browsing session, capturing the resources that make the visible web page.

During the 2024 General Election, we ran Browsertrix alongside Heritrix. Heritrix handled the majority of the simpler website nominations, such as MP and party websites, while Browsertrix focused on more complex social media accounts.

Workflows and Resources for the 2024 General Election

Although we planned to integrate Browsertrix into our archiving efforts for the 2024 General Election, unforeseen delays meant that we only gained access to the tool on June 28th—just one week before polling day on July 5th. However, prior planning helped us decide on key social media accounts.

Key considerations for this workflow included:

•       Collaboration with Legal Deposit Libraries
•       Limited time frame
•       Archiving multiple social media accounts
•       Daily archiving schedules
•       Finite Browsertrix resources

We had an organisational account with five terabytes of storage and 6,000 minutes of processing time. However, as with any web archiving, the actual crawl times and data requirements were difficult to predict due to the variable size and complexity of websites.

Which is why we try to encapsulate our crawls with general parameters assigned to each seed, for example the frequency of a crawl or the data cap. In an ideal world, we would crawl them every minute with unlimited data, but there is a cost to everything, and so our strategy relies on the expertise of curators and archivists to determine the ideal parameters that will ensure a best-effort capture, whilst ensuring we utilise our hardware as efficiently as possible.

Using Browsertrix, the first task was to prioritise which social media platform to tackle first, depending on how many accounts were nominated for each platform. In total, we had 138 social media accounts to archive:

•       96 X accounts
•       25 Facebook accounts
•       17 Instagram accounts

X was by far the most active platform, making it a priority. After some trial and error, we found that a three-minute crawl time produced high-quality captures for most accounts. Here are some of the settings that were adjusted, in various combinations:

•       Start URL Scope
•       Extra URL Prefixes in Scope
•       Exclusions
•       Additional URLs
•       Max Pages
•       Crawl Time Limit
•       Crawl Size Limit
•       Delay After Page Load
•       Behaviour Timeout
•       Browser Windows
•       Crawler Release Channel
•       User Agent

For X specifically, we staggered crawls by 30 minutes to avoid triggering account blocks. This came with its own challenges, as we had no system in place to manage scheduling   and social media login details. For this reason, it was felt that the Browsertrix application should be solely managed by one experienced member of staff, rather than the curators who nominated the accounts in order to manage the social media account logins and the scheduling of crawl jobs. In practice, this meant that a spreadsheet was used, detailing the numerous social media accounts with their login and various crawling parameters.

Quality Assurance

Quality assurance (QA) is a crucial but time-consuming aspect of web archiving, especially when dealing with dynamic content. Browsertrix offers a QA tool that generates reports analysing the quality of individual crawl jobs, including screenshot comparisons and resource analysis. However, this feature can be resource-intensive; for instance, a QA report for a single Facebook capture required approximately 30 minutes of processing time. Given our limitation of 6,000 minutes of processing time and the large volume of crawl jobs, we had to selectively perform QA on key crawl jobs rather than generating reports for every one.

Browsertrix’s extensive documentation provides more details on its QA process, which we found valuable when managing our resources effectively during this large-scale archival effort. Users can run spot checks on crawl jobs, choosing those that might benefit from a QA report; this gives a sense of how healthy the capture is, and allows the user to adjust the Browsertrix settings. Another approach is to offload the quality assurance so that it is performed outside Browsertrix. The user can download the WACZ files and interrogate them to check their contents against the live website, again carrying out spot checks to see if certain significant resources were captured. 

Looking at the live website in a web browser, users can analyse the network traffic and view what resources are loading, usually through the browser developer tools. The resources that load during network analysis also have the exact URI of the resources, which can be searched for within the WACZ file. Bear in mind, this sort of comparison with the live website should be done soon after crawling has completed, otherwise you may be conducting a comparison on a URL where the content has changed significantly to that which was initially crawled.

Some of the QA considerations which we were guided by include:

•         If issues are found, what, if anything, can be realistically done to remedy them?
•         Is it an issue with the crawler or with the playback software?
•         How much time can you apportion to QA without it impacting other work?
•         Will the time given over to QA yield an appropriate benefit?
•         Can your QA scale?

Where to go from here?

The 2024 General Election marked the first time we used Browsertrix alongside Heritrix for social media archiving. While the process presented challenges, particularly around managing login authentication and processing constraints, Browsertrix proved to be an invaluable tool for capturing complex media. By refining our workflows and balancing the use of both crawl streams, we were able to archive a significant portion of relevant social media content. Looking forward, we will continue to develop and improve our tools and strategies; collaborating with partners and sharing our experience and knowledge by engaging with the wider web archiving community. 

By Carlos Lelkes-Rarugal, Assistant Web Archivist

We have data, lots and lots of data, which is of unique importance to researchers, but presents significant challenges for those wanting to interact with it. As our holdings grow by terabytes each month, this creates significant hurdles for the UK Web Archive team who are tasked with organising the data and for researchers who wish to access it. With the scale and complexity of the data, how can one first begin to comprehend what it is that they are dealing with and understand how the collection came into being? 

This challenge is not unique to digital humanities. It is a common issue in any field dealing with vast amounts of data. A recent special report on the skills required by researchers working with web archives was produced by the Web ARChive studies network (WARCnet). This report, based on the Web Archive Research Skills and Tools Survey (WARST), provides valuable insights and can be accessed here: WARCnet Special Report - An overview of Skills, Tools & Knowledge Ecologies in Web Archive Research.

At the UK Web Archive, legal and technical restrictions dictate how we can collect, store and provide access to the data. To enhance researcher engagement, Helena Byrne, Curator of Web Archives at the British Library, and Emily Maemura, Assistant Professor at the School of Information Sciences at the University of Illinois Urbana-Champaign, have been collaborating to explore how and which types of datasets can be published. Their efforts include developing options that would enable users to programmatically examine the metadata of the UK Web Archive collections.

Thematic collections and our metadata

To understand this rich metadata, we first have to examine how it is created and where it is held..

Since 2005 we have used a number of applications, systems, and tools to enable us to curate websites. The most recent being the Annotation and Curation Tool (ACT), which enables authenticated users, mainly curators and archivists, to create metadata that define and describe targeted websites. The ACT tool also serves  to help users build collections around topics and themes, such as the UEFA Women's Euro England 2022. To build collections, ACT users first input basic metadata to build a record around a website, including information such as website URLs, descriptions, titles, and crawl frequency. With this basic ACT record describing a website, additional metadata can be added, for example metadata that is used to assign a website record to a collection. One of the great features of ACT is its extensibility, allowing us, for instance, to create new collections.

These collections, which are based around a theme or an event, give us the ability to highlight archived content. The UK Web Archive holds millions of archived websites, many of which may be unknown or rarely viewed, and so to help showcase a fraction of our holdings, we build these collections which draw on the expertise of both internal and external partners.

Exporting metadata as CSV and JSON files

That’s how we create the metadata, but how is it stored? ACT  is a web application and the metadata created through it is stored in a Postgres relational database, allowing authenticated users to input metadata in accordance to the fields within ACT. As the Assistant Web Archivist, I was given the task to extract the metadata from the database, exporting each selected collection as a CSV and JSON file. To get to that stage, the Curatorial team first had to decide which fields were to be exported. 

The ACT database is quite complex, in that there are 50+ tables which need to be considered. To enable local analysis of the database, a static copy is loaded into a database administration application, in this case, DBeaver. Using the free-to-use tool, I was able to create entity relationship diagrams of the tables and provide an extensive list of fields to the curators so that they could determine which fields are the most appropriate to export.

I then worked on a refined version of the list of fields, running a script for the designated Collection and pulling out specific metadata to be exported. To extract the fields and the metadata into an exportable format, I created an SQL (Structured Query Language) script which can be used to export results in both JSON and/or CSV: 

Select

taxonomy.parent_id as "Higher Level Collection",

collection_target.collection_id as "Collection ID",

taxonomy.name as "Collection or Subsection Name",

CASE

     WHEN collection_target.collection_id = 4278 THEN 'Main Collection'

     ELSE 'Subsection'

END AS "Main Collection or Subsection",

target.created_at as "Date Created",

target.id as"Record ID",

field_url.url as "Primary Seed",

target.title as "Title of Target",

target.description as "Description",

target.language as "Language",

target.license_status as "Licence Status",

target.no_ld_criteria_met as "LD Criteria",

target.organisation_id as "Institution ID",

target.updated_at as "Updated",

target.depth as "Depth",

target.scope as "Scope",

target.ignore_robots_txt as "Robots.txt",

target.crawl_frequency as "Crawl Frequency",

target.crawl_start_date as "Crawl Start Date",

target.crawl_end_date as "Crawl End Date"

From

collection_target

Inner Join target On collection_target.target_id = target.id

Left Join taxonomy On collection_target.collection_id = taxonomy.id

Left Join organisation On target.organisation_id = organisation.id

Inner Join field_url On field_url.target_id = target.id

Where

collection_target.collection_id in (4278, 4279, 4280, 4281, 4282, 4283, 4284) And

(field_url.position Is Null Or field_url.position In (0))

JSON output example for the Women’s Euro Collection

Accessing and using the data

The published metadata is available from the BL Research Repository within the UK Web Archive section, in the folder “UK Web Archive: Data”. Each dataset includes the metadata seed list in both CSV and JSON formats, a data dictionary and a datasheet which gives provenance information about how the dataset was created as well as a data dictionary that defines each of the data fields. The first collections selected for publication were:

1. Indian Ocean Tsunami December 2004 (January-March 2005) [https://doi.org/10.23636/sgkz-g054]
2. Blogs (2005 onwards) [https://doi.org/10.23636/ec9m-nj89] 
3. UEFA Women's Euro England 2022 (June-October 2022) [https://doi.org/10.23636/amm7-4y46] 

By Ely Nott, Library, Information and Archives Services Apprentice

After another extended break, we return to a conversation between UK domain websites as they try to parse out who among them should be crowned the most extraordinary…

“Where should we start this time?” asked Following the Lights. “Any suggestions?”

“If we’re talking weird and wonderful, clearly we should be considered first.” urged Temporary Temples, cutting off Concorde Memorabilia before they could make a sound.

“We should choose a website with a real grounding in reality.” countered the UK Association of Fossil Hunters.

“So, us, then.” shrugged the Grampian Speleological Group. “Or if not, perhaps the Geocaching Association of Great Britain?”

“We’ve got a bright idea!” said Lightbulb Languages, “Why not pick us?”

“There is no hurry.” soothed the World Poohsticks Champsionships, “We have plenty of time to think, think, think it over.”

“This is all a bit too exciting for us.” sighed the Dull Men’s Club, who was drowned out by the others.

“The title would be right at gnome with us.” said The Home of Gnome, with a little wink and a nudge to the Clown Egg Gallery, who cracked a smile.

“Don’t be so corny.” chided the Corn Exchange Benevolent Society. “Surely the title should go to the website that does the most social good?”

“Then what about Froglife?” piped up the Society of Recorder Players.

“If we’re talking ecology, we’d like to be considered!” the Mushroom enthused, egged on by Moth Dissection UK. “We have both aesthetic and environmental value.”

“Surely, any discussion of aesthetics should prioritise us.” preened Visit Stained Glass, as Old so Kool rolled their eyes.

The back and forth continued, with time ticking on until they eventually concluded that the most extraordinary site of all had to be… Saving Old Seagulls.

Check out previous episodes in this series by Hedley Sutton - Part 1, Part 2, Part 3 Part 4 and Part 5