Bulk data availability and access to case law

Request

As follows from the Justice Committee report regarding Open Justice
(https://publications.parliament.uk/pa/cm5803/cmselect/cmjust/1040/report.html?utm_source=chatgpt.com), in April 2022, The National Archives became responsible for the storage and publication of judgments and decisions from courts and tribunals in England and Wales, increasing access to case law. The “Find Case Law” section of the website includes an option to export individual legal documents to a machine-readable XML format for, as stated, “developers, data scientists and researchers.”

However, according to the Terms of Service published by The National Archives (https://www.nationalarchives.gov.uk/legal/use-of-bulk-downloads-and-web-crawlers/), bulk downloads are strictly prohibited. This limitation makes it impossible for data scientists to conduct research that requires aggregation of multiple documents to produce insights.

I would like to request the following information:

1. Bulk Data Availability: Does The National Archives provide any form of bulk data export or dataset dumps for case law, legislation, or other public domain materials
2. Access to Case Law Data: What provisions exist for researchers and data scientists who require access to large-scale case law datasets for analysis?
3. Alternative Access Methods: If direct bulk downloads are restricted, does The National Archives offer an API, secure research environment, or other alternative means to access large volumes of data?
4. Third-Party Agreements: Are there any existing agreements or programs under which institutions, researchers, or organisations can request and obtain bulk data access?

I would appreciate any documentation or guidelines related to these questions. If any of the requested information is exempt from disclosure, please provide the specific exemptions applied and any relevant public interest considerations.

Outcome

Some information provided.

Response

1. Bulk Data Availability:

Does The National Archives provide any form of bulk data export or dataset dumps for case law, legislation, or other public domain materials?

Researchers can export up to 10,000 search results as a CSV from The National Archive’s online catalogue, Discovery. The UK Government Web Archive contains downloadable, archived documents and datasets. The availability of these datasets will vary over time depending on how departments have used the web to disseminate information. However, not all datasets would have been successfully archived due to technical limitations of web crawlers.
The EU Exit Web Archive contains a wide selection of documents taken from EUR-Lex (the European legislation website) archived on 31 Dec 2020. This data is searchable but not downloadable in bulk. Beyond this, we do not currently provide a data (or metadata) download service for our web archives or social media archives.

The www.legislation.gov.uk API is an integral part of the website and is available to all users. The documentation can be found in our GitHub repository. Additionally, the Legislation Team are aware that there is significant user interest in being able to download all documents on the website. Over the past few years, we have been developing a Legislation Research website which will act as a companion site to the main legislation website. This new service has been designed for researchers, whether academic, commercial or within government. The aim is to provide access to data, tools, and enhanced search functions that allow in-depth analysis of legislation data that can’t be carried on the main legislation website. This includes a set of pre-packaged bulk data downloads of the documents on the website organized by format, version, document type and year. The live research website is in the final stages of testing and not quite ready for general use, but we have been providing pre-release access for beta testers. Access may be given upon request.

Find Case Law is still a Beta service. The service is being developed following an iterative Agile approach. The service has been released into Beta with known constraints. We are planning on carrying out research with data users in the next financial year with a view to improving programmatic access to the collection via our API and Atom feed functionality. All service enhancements are planned and implemented in response to evidenced user need.

You may find the following links informative:
Researchers can export up to 10,000 search results as a CSV from The National Archive’s online catalogue, Discovery. There is also an API for this service, please see:

Discovery for developers: about the application programming interface (API) – Website help and terms of use. And further information on using the API can be found on discovery, our catalogue and on this blog post.
More information on the limitations of the UK government web archive can be found here.

2. Access to Case Law Data:

What provisions exist for researchers and data scientists who require access to large-scale case law datasets for analysis?

Information on how to re-use Case Law records is published here.

This includes links to Find Case Law’s API documentation and licence application process if that is applicable to the particular context of re-use.

3. Alternative Access Methods:

If direct bulk downloads are restricted, does The National Archives offer an API, secure research environment, or other alternative means to access large volumes of data?

As above, API and Atom feed documentation is linked to from this page.

4. Third-Party Agreements:

Are there any existing agreements or programs under which institutions, researchers, or organisations can request and obtain bulk data access?

There are no agreements in place for third parties to request and obtain bulk data exports. Reasonable use of content from our web archives is acceptable, but users should be aware of the re-use conditions of the information. Most of it falls under the Open Government License, but there are exceptions to this. These conditions should be made clear on pages linking to resources, or in the Terms and Conditions of the archived websites. www.legislation.gov.uk was created with the principle of open data in mind. Accordingly, we make all information that held on www.legislation.gov.uk available in human-readable form for regular users and in machine-readable form for people looking to automatically index or extract legislation data for reuse. All www.legislation.gov.uk content is available under the Open Government Licence v3.0 except where otherwise stated.