Web Archives for Social Sciences Datathon

Bristol Digital Futures Institute, University of Bristol, November 2025

On 27–28 November 2025 a Web Archives for Social Sciences Datathon was organised at the University of Bristol. This was in collaboration between the Common Crawl Foundation and UK Web Archive at The British Library. The Datathon took place at the BDFI Neutral Lab.

This two-day event aimed to build capacity in the social science research community to use large-scale Web Archive data for policy-relevant, socio-economic research. Participants worked in teams with curated data extracts from Common Crawl to address real-world research challenges, supported by expert facilitators.

This Datathon was part of the Atlas of Economic Activities project and was funded by SDR UK.

Facilitators: Emmanouil Tranos, Leonardo Castro Gonzalez, Jon Reades, Laurie Burchell and Thom Vaughan.

Problems

1. This is a cache of web data containing all commercial websites (.co.uk, landing webpages only) archived by Common Crawl (2021) that (1) include at least one UK postcode in their web text, and (2) we believe represent Financial Services. Identify the different sub-classes (industries) within Financial Services and highlight the websites that provide Fintech services.

2. This is a cache of web data containing all commercial websites (.co.uk, landing webpages only) archived by Common Crawl (2021) that (1) include at least one UK postcode in their web text, and (2) we believe represent the Creative Industries. Identify the different sub-classes (industries) within the Creative Industries and highlight the CreaTech websites.

3. This is a cache of web data containing all commercial websites (.co.uk, landing webpages only) archived by Common Crawl (all of 2021 and 2024) that include at least one postcode from Manchester and Birmingham in their web text. The 2021 commercial websites are classified by economic activity, while the 2024 websites are not. Use the 2021 data to classify the 2024 data for both cities. Compare the industrial structure of these cities and analyse how it evolved over time. The data include Manchester for 2021 and 2024 and Birmingham for 2021 and 2024.

4. This is a cache of web data (provided in two parts) containing all the UK governmental webpages (.gov.uk) that have been archived by the Common Crawl (February to March 2024). Identify the key policy areas Local Authorities in the UK focus on. Are there any Local Authorities that have implemented distinct policies or actions?

5. This is a cache of web data, which contains all the UK governmental webpages (.gov.uk) that have been archived by Common Crawl in two points in time: February to March 2024 (same data as Problem 4) and October 2025. Identify changes in a specific policy domain associated with the new government elected in July 2024.

Data files and checksums

Data description

You will find the following columns in the different data packages. Please be aware that not all columns are present in all data packages.

• id : An ID. Not linked to any other external data
• content : the web text from the landing page of each website
• summary : an LLM-generated summary of the web text
• explanation : an LLM-generated explanation of how the summary of was conducted
• clean_content : LLM-cleaned web text from the landing page of each website
• url : the website URL
• parent_url : the website domain
• postcodes : UK postcodes found in the web text

The rest of the fields were produced by our TNT-LLM inspired pipeline that classifies websites into a two-level typology (for now) of economic activities (high level clusters and low level clusters) using LLMs. These fields are:

• partition_id : high level cluster ID (imagine something like sector)
• label_id : low level cluster ID (imagine something like industry)
• label_name : low level cluster name (imagine something like industry)
• label_description : low level cluster description (imagine something like industry)

Results

The results can be found in the following GitHub repositories in order of the problems:

1. https://github.com/kellyyubini/datathon_bristol
2. https://github.com/JGIBristol/team2-createch
3. https://github.com/jatonline/common-crawl-datathon-group-3
4. https://github.com/laurieburchell/datathon_problem4
5. https://github.com/eshasadia/G5-CommonCrawl

You can download an archive of the presentations from the event here.

Contributors

• Aditi Dutta
• Camilo Andrés López Barra
• Céline Van Migerode
• Christina Palantza
• Do Ngoc Thao
• Esha Sadia Nasir
• Fanqi Zeng
• Filippo Dionigi
• Gabriel A. Pierzynski
• Giovanni Maria Pala
• Helena Byrne
• James Thomas
• Jia Zhao
• Jo Kent
• Kelly Yubini Yubini
• Mariam Cook
• Meihui He
• Meng Le Zhang
• Nirat Rujimora
• Nora Ramsey
• Paddy Smith
• Rita Rasteiro
• Thomas Carey-Wilson
• Timothy Monteath
• Wander Demuynck
• Wong E. Chern