Nora McGregor, Digital Curator and manager of the Digital Scholarship Training Programme reflects on a year of delivering digital upskilling training to colleagues at British Library, part of the Digital Research Team's focus on Embedding Digital Humanities in the British Library | 39 | The Digital.

2024 was a strange and difficult year, to say the least, for us and all our lovely colleagues across the whole of the British Library as we contended daily with the ongoing effects of a cyber-attack disrupting just about every aspect of our work. Not to be cowed by criminality however, the Digital Research Team dug in and ensured the Digital Scholarship Training Programme (DSTP) continued without fail.

From our experience during the pandemic, we knew that in times of major disruption, British Library staff do not stand still. They focus on what they can do, including prioritising their upskilling and have come to count on the DSTP as a kind of refuge whilst temporarily separated from their collections and normal workload.

So it’s with gratefulness to my colleagues in the Digital Research Team, and to BL staff for their engagement, that I reflect proudly on a challenging year where we managed to deliver a whopping 39 individual training events with nearly 900 attendees!   

What we learned in 2024

Our training programme this year covered these topic priorities through a variety of talks, hands-on sessions, reading groups and formal workshops & courses: 

• State-of-the-art Automatic Text Recognition (ATR) technologies
• Useful data science, machine learning and AI applications for analysing and enhancing GLAM digital collections and data​
• The intersection of climate change + Digital Humanities
• Digital tools and methods to support the Library's Race Equality Action Plan
• WikiData, WikiSource, Wikimedia Commons
• OpenRefine for data-wrangling 
• Collections as Data
• Making the most of the IIIF standard

We’re especially thankful for all the academics & professionals who contributed to our learning throughout the year by sharing their projects, experience and expertise with us! If you’d like to be part of our programme in 2025 get in touch with us at [email protected] with your idea, we’d love to hear from you.

2024 Year in Review-External Infographic by Nora McGregor

My Personal Highlights 

In the coming months I will be interviewing my fellow Digital Curators to get their views on highlights from the 2024 Digital Scholarship Training Programme, either favourite events they attended or programmed in 2024 and topic areas they’re excited about this year. No easy ask actually, as I know they, like me, will have found every event spectacularly interesting and useful, but to highlight just a few for you...

21st Century Talks

Our 21st Century Curatorship talk series is looked after by Digital Curators Stella Wisdom and Adi Kienan-Schoonbaert. They are 1 hour invited guest lectures held once or twice a month where we learn about exciting, innovative, projects and research at the intersection of cultural heritage collections and new technologies. These talks are pitched for complete beginners – we try not to assume knowledge so that anyone from any department can come along! A few of my favourite talks in particular were from these projects:

• DE-BIAS - Detecting and cur(at)ing harmful language in cultural heritage collections | Europeana PRO
  Kerstin Herlt and Kerstin Arnold introduced us to the DE-BIAS project which aims to detect and contextualise potentially harmful language in cultural heritage collections. Working with themes like migration and colonial past, gender and sexual identity, ethnicity and ethno-religious identity, the project collaborates with minority communities to better understand the stories behind the language used - or behind the gaps apparent. We learned about the development of the vocabulary and the tools the project has created.
  
  
• The Print and Probability Project: From Restoration Era Printing to an Interim English Short Title Catalogue
  Nikolai Vogler gave us an entertaining view of a selection of findings from the University of California’s Print & Probability project, an interdisciplinary research group at the intersection of book history, computer vision, and machine learning that seeks to discover Restoration-era letterpress printers whose identities have eluded scholars for several hundred years. He also presented his work on creating an interim English Short Title Catalogue (ESTC) in response to the cyber-attack on the Library in 2023, a pursuit for which colleagues were incredibly grateful for!
  
  
• “Dark Matter: X%” - how many early modern Hungarian books disappeared without any trace?
  This was such a fascinating talk by Péter Király, software developer and digital humanities researcher at the Göttingen computation centre, Germany. Estimating the unknown is always an interesting endeavour. There is a registry of surviving books, and we have collective knowledge about lost books, but how many early Hungarian printings have been lost without any historical trace? Their research group transformed the analytical bibliography "Régi Magyarországi Nyomtatványok" (Early Hungarian Printings) into a database and the use of mathematical models from the toolbox of biologists were employed to help estimate it. The analysis of the database also highlights unknown or less investigated areas and enables them to extend previous research focusing on a particular time range to the whole period (such as religious trends during reformation and counter reformation, the changes of genres over times).

Hack & Yacks

I have the privilege of programming and leading this particular series of events and they are my favourite days in the calendar! These are our casual, 2hr monthly meet ups where we all take some time to have a hands-on exploration of new tools, techniques, and applications. No previous experience is ever needed, these are aimed at complete beginners (we’re usually learning something new too!) and we welcome colleagues from across the Library to come have a play! Some sessions are more "yack" than "hack", while others are more quiet hacking depending on the topic but no matter the balance they're always illuminating.

• Introduction to AI and Machine learning was great fun for me personally as I had the chance to give staff an interactive and hands-on introduction to concepts around AI and ML, as it relates to library work, and play around with some open machine learning tools. The session was based on much of the text and activities offered in this topic guide AI & ML in Libraries Literacies – Digital Scholarship & Data Science Essentials for Library Professionals and it was a useful way for me to test the content directly with its intended audience!
  
  
• Catalogues as Data was a session run by Harry Lloyd our Research Software Engineer Extraordinaire and Rossitza Atanassova, Digital Curator, as a two part guided exploration of printed Catalogues as data, working with OCR output and corpus linguistic analysis. In the first half we followed steps in a Jupyter Notebook to extract catalogue entries from OCR text, troubleshoot errors in the algorithm, and investigate Named Entity Recognition techniques. In the second half we explore catalogue entries using corpus linguistic techniques using AntConc, gaining a sense of how cataloguing practice and the importance of different terms changes over time.

Digital Scholarship Reading Group

These monthly discussions led by Digital Curators Mia Ridge and Rossitza Atanassova, are always open to any of our BL colleagues & students, regardless of job title or department. Discussions are regularly attended by colleagues from a range of departments including curators, reference specialists, technology, and research services.

My favourite session of the year by far was “No stupid questions, AI in Libraries”, a lovely meandering session we held in December and a great way to wrap up the year. Instead of discussing any particular reading, we all shared bits about what we had read or learned about independently on the topic of AI in Libraries and had some good-natured debate about where we believe it’s all headed for us on personal and professional levels. Though no readings were required, these were offered in case folks wanted to swot up:

• National Film and Sound Archive of Australia (NFSA) Principles for ML and AI
• Approaching AI at the National Library of Scotland
• Creative Australia Principles: Generative Artificial Intelligence and creative work
• (US) National Archives’ New Strategic Framework Emphasizes Building Capacity Through Responsible Use of Artificial Intelligence
• Smithsonian AI Values Statement -- Spring 2022
• Library of Congress (LC) Labs AI Planning Framework
• Generative AI Framework for HMG (which includes the BL!)
• Generative AI at the BBC

Formal Workshops

We also programme formal courses as needed and this year we focussed very much on building our knowledge of the Wikimedia Universe. I thoroughly enjoyed the lessons we got from Lucy Hinnie and Stuart Prior which covered nearly every aspect of Wikimedia, and we’ll doing much more with this new knowledge, particularly WikiData in 2025!

 

Machine learning (ML) is a hot topic, especially when it’s hyped as ‘AI’. How might libraries use machine learning / AI to enrich collections, making them more findable and usable in computational research? Digital Curator Mia Ridge lists some examples of external collaborations, internal experiments and staff training with AI / ML and digitised and born-digital collections.

Background

The trust that the public places in libraries is hugely important to us - all our 'AI' should be 'responsible' and ethical AI. The British Library was a partner in Sheffield University's FRAIM: Framing Responsible AI Implementation & Management project (2024). We've also used lessons from the projects described here to draft our AI Strategy and Ethical Guide.

Many of the projects below have contributed to our Digital Scholarship Training Programme and our Reading Group has been discussing deep learning, big data and AI for many years. It's important that libraries are part of conversations about AI, supporting AI and data literacy and helping users understand how ML models and datasets were created.

If you're interested in AI and machine learning in libraries, museums and archives, keep an eye out for news about the AI4LAM community's Fantastic Futures 2025 conference at the British Library, 3-5 December 2025. If you can't wait that long, join us for the 'AI Debates' at the British Library.

Using ML / AI tools to enrich collections

Generative AI tends to get the headlines, but at the time of writing, tools that use non-generative machine learning to automate specific parts of a workflow have more practical applications for cultural heritage collections. That is, 'AI' is currently more process than product.

Text transcription is a foundational task that makes digitised books and manuscripts more accessible to search, analysis and other computational methods. For example, oral history staff have experimented with speech transcription tools, raising important questions, and theoretical and practical issues for automatic speech recognition (ASR) tools and chatbots.

We've used Transkribus and eScriptorium to transcribe handwritten and printed text in a range of scripts and alphabets. For example:

• ‘Using Transkribus for Arabic Handwritten Text Recognition’, ‘Using Transkribus for automated text recognition of historical Bengali Books’.
• Investigating the legacies of curatorial voice in the descriptions of incunabula collections at the British Library and student work on Detecting Catalogue Entries in Printed Catalogue Data
• Handwritten Text Recognition of the Dunhuang manuscripts: the challenges of machine learning on ancient Chinese texts (eScriptorium)
• Reinventing the 'Convert-a-Card' crowdsourcing project as a semi-automated workflow: Convert-a-Card: Past, Present and Future of Catalogue Cards Retroconversion, Convert-a-Card: Helping Cataloguers Derive Records with OCLC APIs and Python; Convert-a-Card: Extracting Entities from Catalogue Cards to Create E-Records 

Creating tools and demonstrators through external collaborations

Mining the UK Web Archive for Semantic Change Detection (2021)

This project used word vectors with web archives to track words whose meanings changed over time. Resources: DUKweb (Diachronic UK web) and blog post ‘Clouds and blackberries: how web archives can help us to track the changing meaning of words’.

From blackberries to clouds... word associations change over time

Living with Machines (2018-2023)

Our Living With Machines project with The Alan Turing Institute pioneered new AI, data science and ML methods to analyse masses of newspapers, books and maps to understand the impact of the industrial revolution on ordinary people. Resources: short video case studies, our project website, final report and over 100 outputs in the British Library's Research Repository.

Outputs that used AI / machine learning / data science methods such as lexicon expansion, computer vision, classification and word embeddings included:

• T-Res: A Toponym Resolution Pipeline for Digitised Historical Newspapers
• MapReader: A computer vision pipeline for exploring and analyzing images at scale
• DeezyMatch: A Flexible Deep Neural Network Approach to Fuzzy String Matching
• The Living Machine: A Computational Approach to the Nineteenth-Century Language of Technology
• Generating metadata from digitised 'Mitchells' Press Directories
• Generating metadata from digitised Parliamentary Road Acts (with CogApp)
• Experimental pipelines for processing digitised pages with structured or semi-structured content

Tools and demonstrators created via internal pilots and experiments

Many of these examples were enabled by on-staff Research Software Engineers and the Living with Machines (LwM) team at the British Library's skills and enthusiasm for ML experiments in combination with long-term Library’s staff knowledge of collections records and processes:

• Identifying upside-down images in the Endangered Archives Project – projects within this important collection were often digitised under trying circumstances, so training machine learning to identify image attributes is useful.
• Languid: Language Identification Project (2020) – Metadata Services' Victoria Morris experimented with machine learning (and ‘human review’ from c40 enthusiastic language experts checking the results) and was able to add language codes to over 3 million catalogue records. Her project identified 471 languages in the records, 141 of which were not previously represented. Resources: short video, longer video, publication Automated Language Identification of Bibliographic Resources: Cataloging & Classification Quarterly: Vol 58, No 1.
• Flyswot (2021) – BL staff trained a machine learning model to find images of digitised manuscripts incorrectly labelled as ‘flysheets’.
• Trialling a book genre classification model (2022) - while the team concluded that the model worked well, but not well enough to use for creating catalogue data yet, they shared their model on Hugging Face and training data created by British Library staff on Zooniverse. Resources: blog post and tutorial.

British Library resources for re-use in ML / AI

Our Research Repository includes datasets suitable for ground truth training, including 'Ground truth transcriptions of 18th &19th century English language documents relating to botany from the India Office Records'. 

Our ‘1 million images’ on Flickr Commons have inspired many ML experiments, including:

• Mario Klingemann (AKA Quasimondo) used semi-automated image classification and machine learning to group images into thematic collections ((2014 - 16). See also: ‘BL Labs Awards (2015): Creative/Artistic category Award winning project’
• British Library & Flickr Commons: The many hands (and some machines) making light work
• University of Oxford's Visual Geometry Group (VGG) using ‘BL1M’; multi-modal image searchSherlockNet: Using Convolutional Neural Networks (CNNs) to automatically tag and caption the British Library Flickr collection

The Library has also shared models and datasets for re-use on the machine learning platform Hugging Face.

Oral History Archivist Charlie Morgan provides examples of how AI-based tools integrated into workflows might affect oral historians' consideration of orality and silence in the second of two posts on a talk he gave with Digital Curator Mia Ridge at the 7th World Conference of the International Federation for Public History in Belval, Luxembourg. His first post proposed some key questions for oral historians thinking about AI, and shared an example automatic speech recognition (ASR) tools in practice. 

While speech to text once seemed at the cutting edge of AI, software designers are now eager to include additional functions. Many incorporate their own chatbots or other AI ‘helpers’ and the same is true of ‘standard’ software. Below you can see what happened when I asked the chatbots in Otter and Adobe Acrobat some questions about other transcribed clips from the ‘Lives in Steel’ CD:

A composite image of chatbot responses to questions about transcribed clips

In Otter, the chatbot does well at answering a question on sign language but fails to identify the accent or dialect of the speaker. This is a good reminder of the limits of these models and how, without any contextual information, they cannot understand the interview beyond textual analysis. Oral historians in the UK have long understood interviews as fundamentally oral sources and current AI models risk taking us away from this.

In Adobe I tried asking a much more subjective question around emotion in the interview. While the chatbot does answer, it is again worth remembering the limits of this textual analysis, which, for example, could not identify crying, laughter or pitch change as emotion. It would also not understand the significance of any periods of silence. On our panel at the IFPH2024 conference in Luxembourg Dr Julianne Nyhan noted how periods of silence tend to lead speech-to-text models to ‘hallucinate’ so the advice is to take them out; the problem is that oral history has long theorised the meaning and importance of silence.

Alongside the chatbot, Adobe also includes a degree of semantic searching where a search for steel brings up related words. This in itself might be the biggest gift new technologies offer to catalogue searching (shown expertly in Placing the Holocaust) – helping us to move away from what Mia Ridge calls ‘the tyranny of the keyword’.

However, the important thing is perhaps not how well these tools perform but the fact they exist in the first place. Oral historians and archivists who, for good reasons, are hesitant about integrating AI into their work might soon find it has happened anyway. For example, Zencastr, the podcasting software we have used since 2020 for remote recordings, now has an in-built AI tool. Robust principles on the use of AI are essential then not just for new projects or software, but also for work we are already doing and software we are already using.

The rise of AI in oral history raises theoretical questions around orality and silence, but must also be considered in terms of practical workflows: Do participation and recording agreements need to be amended?​ How do we label AI generated metadata in catalogue records, and should we be labelling human generated metadata too? Do AI tools change the risks and rewards of making oral histories available online? We can only answer these questions through though critical engagement with the tools themselves.

Oral History Archivist Charlie Morgan shares some key questions for oral historians thinking about AI, and shares some examples of automatic speech recognition (ASR) tools in practice in the first of two posts...

Oral history has always been a technologically mediated discipline and so has not been immune to the current wave of AI hype. Some have felt under pressure to ‘do some AI’, while others have gone ahead and done it. In the British Library oral history department, we have been adamant that any use of AI must align practically, legally and ethically with the Library’s AI principles (currently in draft form). While the ongoing effects of the 2023 cyber-attack have also stymied any integration of new technologies into archival workflows, we have begun to experiment with some tools. In September, I was pleased to present on this topic with Digital Curator Mia Ridge at the 7th World Conference of the International Federation for Public History in Belval, Luxembourg. Below is a summary of what I spoke about in our presentation, ‘Listening with machines? The challenges of AI for oral history and digital public history in libraries’.

The ‘boom’ in AI and oral history has mostly focussed on speech recognition and transcription, driven by the release of Trint (2014) and Otter (2016), but especially Whisper (2022). There have also been investigations into indexing, summarising and visualisation, notably from the Congruence Engine project. Oral historians are interested in how AI tools could help with documentation and analysis but many also have concerns. Concerns include, but are not limited to, ownership, data protection/harvesting, labour conditions, environmental costs, loss of human involvement, unreliable outputs and inbuilt biases.

For those of us working with archived collections there are specific considerations: How do we manage AI generated metadata? Should we integrate new technologies into catalogue searching? What are the ethics of working at scale and do we have the experience to do so? How do we factor in interviewee consent, especially since speakers in older collections are now likely dead or uncontactable?

With speech recognition, we are now at a point where we can compare different automated transcripts created at different times. While our work on this topic at the British Library has been minimal, future trials might help us build up enough research data to address the above questions.

Robert Gladders was interviewed by Alan Dein for the National Life Stories oral history project ‘Lives in Steel’ in 1991 and the extract below was featured on the 1993 published CD ‘Lives in Steel’.

The full transcripts for this audio clip are at the end of this post.

Sign Language

We can compare three automatic speech recognition (ASR) transcripts of the first line:

• Human: Sign language was for telling the sample to the first hand, what carbon the- when you took the sample up into the lab, you run with the sample to the lab​
• Otter 2020: Santa Lucia Chelan, the sound pachala fest and what cabin the when he took the sunlight into the lab, you know they run with a sample to the lab​
• Otter 2024: Sign languages for selling the sample, pass or the festa and what cabin the and he took the samples into the lab. Yet they run with a sample to the lab.
• Whisper 2024: The sand was just for telling the sand that they were fed down. What cabin, when he took the sand up into the lab, you know, at the run with the sand up into the lab

Gladders speaks with a heavy Middlesbrough accent and in all cases the ASR models struggle, but the improvements between 2020 and 2024 are clear. In this case, Otter in 2024 seems to outperform Whisper (‘The sand’ is an improvement on ‘Santa Lucia Chelan’ but it isn’t ‘Sign languages’), but this was a ‘small’ version of Whisper and larger models might well perform better.

One interesting point of comparison is how the models handle ‘sample passer’, mentioned twice in the short extract:

• Otter 2020: Sentinel pastor / sound the password​
• Otter 2024: Salmon passer / Saturn passes​
• Whisper 2024: Santland pass / satin pass

While in all cases the models fail, this would be easy to fix. The aforementioned CD came with its own glossary, which we could feed into a large language model working on these transcriptions. Practically this is not difficult but it raises some larger questions. Do we need to produce tailored lexicons for every collection? This is time-consuming work so who is going to do it? Would we label an automated transcript in 2024 that makes use of a human glossary written in 1993 as machine generated, human generated, or both? Moreover, what level of accuracy we are willing to accept and how do we define accuracy itself?

 

Sample glossary terms

Drawing on work at and prior to the British Library, Digital Curator Mia Ridge shares a personal perspective on open cultural data for galleries, libraries, archives and museums (GLAMs) based on a recent lecture for students in Archives and Records Management…

Cultural heritage institutions face both exciting opportunities and complex challenges when sharing their collections online. This post gives common reasons why GLAMs share collections as open cultural data, and explores some strategic considerations behind making collections accessible.

What is Open Cultural Data?

Open cultural data includes a wide range of digital materials, from individual digitised or born-digital items – images, text, audiovisual records, 3D objects, etc. – to datasets of catalogue metadata, images or text, machine learning models and data derived from collections.

Open data must be clearly licensed for reuse, available for commercial and non-commercial use, and ideally provided in non-proprietary formats and standards (e.g. CSV, XML, JSON, RDF, IIIF).

Why Share Open Data?

The British Library shares open data for multiple compelling reasons.

Broadening Access and Engagement: by releasing over a million images on platforms like Flickr Commons, the Library has achieved an incredible 1.5 billion views. Open data allows people worldwide to experience wonder and delight with collections they might never physically access in the UK.

Deepening Access and Engagement: crowdsourcing and online volunteering provide opportunities for enthusiasts to spend time with individual items while helping enrich collections information. For instance, volunteers have helped transcribe complex materials like Victorian playbills, adding valuable contextual information.

Supporting Research and Scholarship: in addition to ‘traditional’ research, open collections support the development of reproducible computational methods including text and data mining, computer vision and image analysis. Institutions also learn more about their collections through formal and informal collaborations.

Creative Reuse: open data encourages artists to use collections, leading to remarkable creative projects including:

• A Malaysian band creating a music video using British Library images
• People experiencing an artistic commission with digitised book illustrations projected onto a massive building in central Leeds
• Experiments with image clustering and pose detection that provide new ways into collections

Screenshot from Hey There Young Sailor (Official Video) - The Impatient Sisters

 

'16 Very Sad Girls' by Mario Klingemann

 

The BookBinder, by Illuminos, with British Library collections

 

Some lessons for Effective Data Sharing

Make it as easy as possible for people to find and use your open collections:

• Tell people about your open data
• Celebrate and highlight creative reuses
• Use existing licences for usage rights where possible
• Provide data in accessible, sustainable formats
• Offer multiple access methods (e.g. individual items, datasets, APIs)
• Invest effort in meeting the FAIR, and where appropriate, CARE principles

Navigating Challenges

Open data isn't without tensions. Institutions must balance potential revenue, copyright restrictions, custodianship and ethical considerations with the benefits of publishing specific collections.

Managing expectations can also be a challenge. The number of digitised or born-digital items available may be tiny in comparison to the overall size of collections. The quality of digitised records – especially items digitised from microfiche and/or decades ago – might be less than ideal. Automatic text transcription and layout detection errors can limit the re-usability of some collections.

Some collections might not be available for re-use because they are still in copyright (or are orphan works, where the creator is not known), were digitised by a commercial partner, or are culturally sensitive.

The increase in the number of AI companies scraping collections site to train machine learning models has also given some institutions cause to re-consider their open data policies. Historical collections are more likely to be out of copyright and published for re-use, but they also contain structural prejudices and inequalities that could be embedded into machine learning models and generative AI outputs.

Conclusion

Open cultural data is more than just making collections available—it's about creating dynamic, collaborative spaces of knowledge exchange. By thoughtfully sharing our shared intellectual heritage, we enable new forms of research, inspiration and enjoyment.

 

AI use transparency statement: I recorded my recent lecture on my phone, then generated a loooong transcription on my phone. I then supplied the transcription and my key points to Claude, with a request to turn it into a blog post, then manually edited the results.

What do you do when you have millions of books and no record of the language they were written in? Collection Metadata Analyst Victoria Morris looks back to describe how she worked on this in 2020...

Context

In an age of online library catalogues, recording the language in which a book (or any other textual resource) is written is vital to library curators and users alike, as it allows them to search for resources in a particular language, and to filter search results by language.

As the graph below illustrates, although language information is routinely added to British Library catalogue records created as part of ongoing annual production, fewer than 30% of legacy records (from the British Library’s foundation catalogues) contain language information. As of October 2018, nearly 4.7 million of records were lacking any explicit language information. Of these, 78% were also lacking information about the country of publication, so it would not be possible to infer language from the place of publication.

The question is: what can be done about this? In most cases, the language of the resource described can be immediately identified by a person viewing the book (or indeed the catalogue record for the book). With such a large number of books to deal with, though, it would be infeasible to start working through them one at a time ... an automated language identification process is required.

Language identification

Language identification (or language detection) refers to the process of determining the natural language in which a given piece of text is written. The texts analysed are commonly referred to as documents.

There are two possible avenues of approach: using either linguistic models or statistical models. Whilst linguistic models have the potential to be more realistic, they are also more complex, relying on detailed linguistic knowledge. For example, some linguistic models involve analysis of the grammatical structure of a document, and therefore require knowledge of the morphological properties of nouns, verbs, adjectives, etc. within all the languages of interest.

Statistical models are based on the analysis of certain features present within a training corpus of documents. These features might be words, character n-grams (sequences of n adjacent characters) or word n-grams (sequences of n adjacent words). These features are examined in a purely statistical, ‘linguistic-agnostic’ manner; words are understood as sequences of letter-like characters bounded by non-letter-like characters, not as words in any linguistic sense. When a document in an unknown language is encountered, its features can be compared to those of the training corpus, and a predication can thereby be made about the language of the document.

Our project was limited to an investigation of statistical models, since these could be more readily implemented using generic processing rules.

What can be analysed?

Since the vast majority of the books lacking language information have not been digitised, the language identification had to be based solely on the catalogue record. The title, edition statement and series title were extracted from catalogue records, and formed the test documents for analysis.

Although there are examples of catalogue records where these metadata elements are in a language different to that of the resource being described (as in, for example, The Four Gospels in Fanti, below), it was felt that this assumption was reasonable for the majority of catalogue records.

Measures of success

The effectiveness of a language identification model can be quantified by the measures precision and recall; precision measures the ability of the model not to make incorrect language predictions, whilst recall measures the ability of the model to find all instances of documents in a particular language. In this context, high precision is of greater value than high recall, since it is preferable to provide no information about the language of content of a resource than to provide incorrect information.

Various statistical models were investigated, with only a Bayesian statistical model based on analysis of words providing anything approaching satisfactory precision. This model was therefore selected for further development.

The Bayesian idea

Bayesian methods are based on a calculation of the probabilities that a book is written in each language under consideration. An assumption is made that the words present within the book title are statistically independent; this is obviously a false assumption (since, for example, adjacent words are likely to belong to the same language), but it allows application of the following proportionality:

The right-hand side of this proportionality can be calculated based on an analysis of the training corpus. The language of the test document is then predicted to be the language which maximises the above probability.

Because of the assumption of word-independence, this method is often referred to as naïve Bayesian classification.

What that means in practice is this: we notice that whenever the word ‘szerelem’ appears in a book title for which we have language information, the language is Hungarian. Therefore, if we find a book title which contains the word ‘szerelem’, but we don’t have language information for that book, we can predict that the book is probably in Hungarian.

Szerelem: definitely a Hungarian word => probably a Hungarian title

If we repeat this for every word appearing in every title of each of the 12 million resources where we do have language information, then we can build up a model, which we can use to make predictions about the language(s) of the 4.7 million records that we’re interested in. Simple!

Training corpus

The training corpus was built from British Library catalogue records which contain language information, Records recorded as being in ‘Miscellaneous languages’, ‘Multiple languages’, ‘Sign languages’, ‘Undetermined’ and ‘No linguistic content’ were excluded. This yielded 12,254,341 records, of which 9,578,175 were for English-language resources. Words were extracted from the title, edition statement, and series title, and stored in a ‘language bucket’.

Language buckets were analysed in order to create a matrix of probabilities, whereby a number was assigned to each word-language pair (for all words encountered within the catalogue, and all languages listed in a controlled list) to represent the probability that that word belongs to that language. Selected examples are listed in the table below; the final row in the table illustrates the fact that shorter words tend to be common to many languages, and are therefore of less use than longer words in language identification.

౨	{Telugu: 0.750, Somali: 0.250}
aaaarrgghh	{English: 1.000}
aaavfleeße	{German: 1.000}
aafjezatsd	{German: 0.333, Low German: 0.333, Limburgish: 0.333}
aanbidding	{Germanic (Other): 0.048, Afrikaans: 0.810, Low German: 0.048, Dutch: 0.095}
نبوغ	{Persian: 0.067, Arabic: 0.200, Pushto: 0.333, Iranian (Other): 0.333, Azerbaijani: 0.067}
metodicheskiĭ	{Russian: 0.981, Kazakh: 0.019}
nuannersuujuaannannginneranik	{Kalâtdlisut): 1.000}
karga	{Faroese: 0.020, Papiamento: 0.461, Guarani: 0.010, Zaza: 0.010, Esperanto: 0.010, Estonian: 0.010, Iloko: 0.176, Maltese: 0.010, Pampanga: 0.010, Tagalog: 0.078, Ladino: 0.137, Basque: 0.029, English: 0.010, Turkish: 0.029}

Results

Precision and recall varied enormously between languages. Zulu, for instance, had 100% precision but only 20% recall; this indicates that all records detected as being in Zulu had been correctly classified, but that the majority of Zulu records had either been mis-classified, or no language prediction had been made. In practical terms, this meant that a prediction “this book is in Zulu” was a prediction that we could trust, but we couldn’t assume that we had found all of the Zulu books. Looking at our results across all languages, we could generate a picture (formally termed a ‘confusion matrix’) to indicate how different languages were performing (see below). The shaded cells on the diagonal represent resources where the language has been correctly identified, whilst the other shaded cells show us where things have gone wrong.

The best-performing languages were Hawaiian, Malay, Zulu, Icelandic, English, Samoan, Finnish, Welsh, Latin and French, whilst the worst-performing languages were Shona, Turkish, Pushto, Slovenian, Azerbaijani, Javanese, Vietnamese, Bosnian, Thai and Somali.

Where possible, predictions were checked by language experts from the British Library’s curatorial teams. Such validation facilitated the identification of off-diagonal shaded areas (i.e. languages for which predictions which should be treated with caution), and enabled acceptance thresholds to be set. For example, the model tends to over-predict English, in part due to the predominance of English-language material in the training corpus, thus the acceptance threshold for English was set at 100%: predictions of English would only be accepted if the model claimed that it was 100% certain that the language was English. For other languages, the acceptance threshold was generally between 95% and 99%.

Outcomes

Two batches of records have been completed to date. In the first batch, language codes were assigned to 1.15 million records with 99.7% confidence; in the second batch, a further 1 million language codes were assigned with 99.4% confidence. Work on a third batch is currently underway, and it is hoped to achieve at least a further million language code assignments. The graph below shows the impact that this project is having on the British Library catalogue.

The project has already been well-received by Library colleagues, who have been able to use the additional language coding to assist them in identifying curatorial responsibilities and better understanding the collection.

Further reading

For a more in-depth, mathematical write-up of this project, please see a paper written for Cataloging & Classification Quarterly, which is available at: https://doi.org/10.1080/01639374.2019.1700201, and is also in the BL research repository at https://bl.iro.bl.uk/work/6c99ffcb-0003-477d-8a58-64cf8c45ecf5.

From April to July this year an Assistant Statistician at the Cabinet Office and a Research Software Engineer at the British Library teamed up as mentee (Catherine Macfarlane, CO) and mentor (Harry Lloyd, BL) for the Data Science Accelerator. In this blog post we reflect on the experience and what it meant for us and our work.

Introduction to the Accelerator

Harry: The Accelerator has been around since 2015, set up as a platform to ‘accelerate’ civil servants at the start of their data science journey who have a business need project and a real willingness to learn. Successful applicants are paired with mentors from across the Civil Service who have experience in techniques applicable to the problem, working together one protected day a week for 12 weeks. I was lucky enough to be a mentee in 2020, working on statistical methods to combine different types of water quality data, and my mentor Charlie taught me a lot of what I know. The programme played a huge role in the development of my career, so it was a rewarding moment to come back as a mentor for the April cohort. 

Catherine: On joining the Civil Service in 2023, I had the pleasure of becoming part of a talented data team that has motivated me to continually develop my skills. My academic background in Mathematics with Finance provides me with a strong theoretical foundation, but I am striving to improve my practical abilities. I am particularly interested in Artificial Intelligence, which is gaining increasing recognition across government, sparking discussions on its potential to improve efficiency.

I saw the Data Science Accelerator as an opportunity to deepen my knowledge, address a specific business need, and share insights with my team. The prospect of working with a mentor and immersing myself in an environment where diverse projects are undertaken was particularly appealing. A significant advantage was the protected time this project offered - a rare benefit! I was grateful to be accepted and paired with Harry, an experienced mentor who had already completed the programme. Following our first meeting, I felt ready to tackle the upcoming 12 weeks to see what we could achieve!

With one of us based in England and the other in Scotland virtual meetings were the norm. Collaborative tools like screen sharing and Github allowed us to work effectively together.

The Project

Catherine: Our team is interested in the annual reports and accounts of Arm’s Length Bodies (ALBs), a category of public bodies funded to deliver a public or government service.  The project addressed the challenge my team faces in extracting the highly unstructured information stored in annual reports and accounts. With this information we would be able to enhance the data validation process and reduce the burden of commissioning data from ALBs on other teams. We proposed using Natural Language Processing to retrieve this information, analysing and querying it using a Large Language Model (LLM).

Initially, I concentrated on extracting five features, such as full-time equivalent staff in the organisation, from a sample of ALBs across 13 departments for the financial year 22/23. After discussions with Harry, we decided to use Retrieval-Augmented Generation (RAG), to develop a question-answering system. RAG is a technique that combines LLMs with relevant external documents to improve the accuracy and reliability of the output. This is done by retrieving documents that are relevant to the questions asked and then asking the LLM to generate an answer based on the retrieved material. We carefully selected a pre-trained LLM while considering ethical factors like model openness.

How a retrieval augmented generation system works. A document in this context is a segmented chunk of a larger text that can be parsed by an LLM.

The first four weeks focused on exploratory analysis, data processing, and labelling, all completed in R, which was essential for preparing the data for input into the language model. The subsequent stages involved model building and evaluation in Python, which required the most time and focus. This was my first time using Python, and Harry’s guidance was extremely beneficial during our pair coding sessions. A definite highlight for me was seeing the pipeline start to generate answers!

To bring all our results together, I created a dashboard in Shiny, ensuring it was accessible to both technical and non-technical audiences. The final stage involved summarising all our hard work from the past 12 weeks in a 10 minute presentation and delivering it to the Data Science Accelerator cohort.

Harry: Catherine’s was the best planned project of the ones I reviewed, and I suspected she’d be well placed to make best use of the 12 weeks. I wasn’t wrong! We covered a lot of the steps involved in good reproducible analysis. The exploratory work gave us a great sense of the variance in the data, setting up quantitative benchmarks for the language model results drove our development of the RAG system, and I was so impressed that Catherine managed to fit in building a dashboard on top of all of that.

Our Reflections

Catherine: Overall this experience was fantastic. In a short amount of time, we managed to achieve a considerable amount. It was amazing to develop my skills and grow in confidence. Harry was an excellent mentor; he encouraged discussion and asked insightful questions, which made our sessions both productive and enjoyable. A notable highlight was visiting the British Library! It was brilliant to have an in-person session with Harry and meet the Digital Research team.

A key success of the project was meeting the objectives we set out to achieve. Patience was crucial, especially when investigating errors and identifying the root problem. The main challenge was managing such a large project that could be taken in multiple directions. It can be natural to spend a long time on one area, such as exploratory analysis, but we ensured that we completed the key elements that allowed us to move on to the next stage. This balance was essential for the project's overall success.

Harry: We divided our days between time for Catherine to work solo and pair programming. Catherine is a really keen learner, and I think this approach helped her drive the project forward while giving us space to cover foundational programming topics and a new programming language. My other role was keeping an eye on the project timeline. Giving the occasional steer on when to stick with something and when to move on helped (I hope!) Catherine to achieve a huge amount in three months. 

A page from the dashboard Catherine created in the last third of the project.

Ongoing Work

Catherine: Our team recognises the importance of continuing this work. I have developed an updated project roadmap, which includes utilising Amazon Web Services to enhance the speed and memory capacity of our pipeline. Additionally, I have planned to compare various large language models, considering ethical factors, and I will collaborate with other government analysts involved in similar projects. I am committed to advancing this project, further upskilling the team, and keeping Harry updated on our progress.

Harry: RAG, and the semantic rather than key word search that underlies it, represents a maturation of LLM technology that has the potential to change the way users search our collections. Anticipating that this will be a feature of future library services platforms, we have a responsibility to understand more about how these technologies will work with our collections at scale. We’re currently carrying out experiments with RAG and the linked data of the British National Bibliography to understand how searching like this will change the way users interact with our data.

Conclusions

Disappointingly the Data Science Accelerator was wound down by the Office for National Statistics at the end of the latest cohort, citing budget pressures. That has made us one of the last mentor/mentee pairings to benefit from the scheme, which we’re both incredibly grateful for and deeply saddened by. The experience has been a great one, and we’ve each learned a lot from it. We’ll continue to develop RAG at the Cabinet Office and the British Library, and hope to advocate for and support schemes like the Accelerator in the future!

This post was originally written by Giorgia Tolfo in early 2023 then lightly edited and posted by Mia Ridge in late 2024. It describes work undertaken in 2019, and provides context for resources we hope to share on the British Library's Research Repository in future.

The Living with Machines project used a range of diverse sources, including newspapers to maps and census data.  This post discusses the Road Acts, 18th century Acts of Parliament stored at the British Library, as an example of some of the challenges in digitising historical records, and suggests computational methods for reducing some of the overhead for cataloging Library records during digitisation.

What did we want to do?

Before collection items can be digitised, they need a preliminary catalogue record - there's no point digitising records without metadata for provenance and discoverability. Like many extensive collections, the Road Acts weren't already catalogued. Creating the necessary catalogue records manually wasn't a viable option for the timeframe and budget of the project, so with the support of British Library experts Jennie Grimshaw and Iris O’Brien, we decided to explore automated methods for extracting metadata from digitised images of the documents themselves. The metadata created could then be mapped to a catalogue schema provided by Jennie and Iris. 

Due to the complexity, the timeframe of the project, the infrastructure and the resources needed, the agency Cogapp was commissioned to do the following:

• Export metadata for 31 scanned microfilms in a format that matched the required field in a metadata schema provided by the British Library curators
• OCR (including normalising the 'long S') to a standard agreed with the Living with Machines project
• Create a package of files for each Act including: OCR (METS + ALTO) + images (scanned by British Library)

To this end, we provided Cogapp with:

• Scanned images of the 31 microfilm reels, named using the microfilm ID and the numerical sequential order of the frame
• The Library's metadata requirements
• Curators' support to explain and guide them through the metadata extraction and record creation process 

Once all of this was put in place, the process started, however this is where we encountered the main problem. 

First issue: the typeface

After some research and tests we came to the conclusion that the typeface (or font, shown in Figure 1) is probably English Blackletter. However, at the time, OCR software - software that uses 'optical character recognition' to transcribe text from digitised images, like Abbyy, Tesseract or Transkribus - couldn't accurately read this font. Running OCR using a generic tool would inevitably lead to poor, if not unusable, OCR. You can create 'models' for unrecognised fonts by manually transcribing a set of documents, but this can be time-consuming. 

Figure 1: Page showing typefaces and layout. SPRMicP14_12_016

Second issue: the marginalia

As you can see in Figure 2, each Act has marginalia - additional text in the margins of the page. 

This makes the task of recognising the layout of information on the page more difficult. At the time, most OCR software wasn't able to detect marginalia as separate blocks of text. As a consequence these portions of text are often rendered inline, merged with the main text. Some examples showing how OCR software using standard settings interpret the page in Figure 2 are below.

Figure 2 Printed page with marginalia. SPRMicP14_12_324

 

OCR generated by ABBYY FineReader:

Qualisicatiori 6s Truitees;

Penalty on acting if not quaiified.

Anno Regni septimo Georgii III. Regis.

9nS be it further enaften, Chat no person ihali he tapable of aftingt ao Crustee in the Crecution of this 9ft, unless be ftall he, in his oton Eight, oj in the Eight of his ©Btfe, in the aftual PofTefli'on anb jogment oj Eeceipt of the Eents ana profits of tanas, Cenements, anb 5)erebitaments, of the clear pearlg Oalue of J?iffp Pounbs} o? (hall be ©eit apparent of some person hatiing such estate of the clear gcatlg 5ia= lue of ©ne hunb?eb Pounbs; o? poffcsseb of, o? intitieb unto, a personal estate to the amount o? Oalue of ©ne thoufanb Pounbs: 9nb if ang Person hcrebg beemeo incapable to aft, ihali presume to aft, etierg such Per* son (hall, so? etierg such ©ffcnce, fojfcit anb pag the @um of jTiftg pounbs to ang person o? 

 

OCR generated by the open source tool Tesseract:

586 Anno Regni ?eptimo Georgi III. Regis.

Qualification

of Truttees;

Penalty on

Gnd be it further enated, That no. Per?on ?hall bÈ

capable of ating as Tru?tËe in the Crecution of thig

A, unle?s he ?hall be, in his own Right, 02 in the

Right of his Wife, in the a‰ual Pofe??ion and En. |

joyment 02 Receipt of the Rents and P2zofits of Lands,

Tenements, and hereditaments, of the clear pearly

Ualue of Fifty Pounds z o? hall be Deir Apparent of

?ome Per?on having ?uch Cfitate of the clear yearly Uga-

lue of Dne Hundred Pounds ; 02 po??e?leD of, 02 intitled

unto, a Per?onal E?tate to the Amount 02 Ualue of One

thou?and Pounds : And if any Per?on hereby deemed

acting if not incapable to ai, ?hall p2e?ume to ait, every ?uch Perz

qualified.

 

OCR generated by Cogapp (without any enhancement)

of Trusteesi

586

Anno Regni ſeptimo Georgii III. Regis.

Qualihcation and be it further enałted, That no perſon thall be

capable of aging as Trulltee in the Erecution of this

ad, unlefs he thall be, in his own Right, of in the

Right of his Wife, in the ađual Polellion and En:

joyment or Receipt of the Rents and Profits of Lands,

Tenements, and hereditaments, of the clear pearly

Ualue of ffifty pounds : oi thall be peir apparent of

ſome Perſon having ſuch Etate of the clear yearly Ua:

lue of Dne hundred Pounds; ou podeled of, od intitled

unto, a Perſonal Elate to the amount ou Ualue of Dne

Penalty on thouſand Pounds : and if any perſon hereby deemed

acting if not incapable to ad, thall preſume to ađ, every ſuch Per-

Qualified.

 

As you can see, the OCR transcription results were too poor to use in our research.

Changing our focus: experimenting with metadata creation

Time was running out fast, so we decided to adjust our expectations about text transcription, and asked Cogapp to focus on generating metadata for the digitised Acts. They have reported on their process in a post called 'When AI is not enough' (which might give you a sense of the challenges!).

Since the title page of each Act has a relatively standard layout it was possible to train a machine learning model to recognise the title, year and place of publication, imprint etc. and produce metadata that could be converted into catalogue records. These were sent on to British Library experts for evaluation and quality control, and potential future ingest into our catalogues.

Conclusion

This experience, although only partly successful in creating fully transcribed pages, explored the potential of producing the basis of catalogue records computationally, and was also an opportunity to test workflows for automated metadata extraction from historical sources. 

Since this work was put on hold in 2019, advances in OCR features built into generative AI chatbots offered by major companies mean that a future project could probably produce good quality transcriptions and better structured data from our digitised images.

If you have suggestions or want to get in touch about the dataset, please email [email protected]