Identifying cloud files with DROID

What is DROID?

DROID is a free software tool developed by The National Archives that supports digital archivists in automatically profiling a wide range of file formats. It can identify which versions of a file you have, its age and size, and when it was last changed. It can also help locate duplicates.

Profiling file formats can help archivists in several ways, including:

• Managing your information more effectively
• Identifying risks (and plan mitigating actions)
• Reducing storage and processing costs through data reduction

Why was an update needed?

Many organisations store their files in the cloud, but DROID was previously unable to access data outside of local storage (where data is physically stored on a device). Due to this, files would have to be downloaded by a user to then be identified using DROID.

This presents many issues including:

• Downloading files risks accidental corruption or trimming of data. This can lead to incorrect identification.
• Difficulty in using DROID in a workflow running in the cloud, especially when a user’s disk space is limited.
• Cloud providers often charge users for moving data away from the cloud, creating additional expenses.
• DROID needs to download the entire file from the cloud to review it, even if the user only wanted to identify a small part of the file format. This could waste disk space and could be time consuming to download.

To address these challenges, we have updated DROID to work with cloud-based data.

What is the update?

There are two ways to identify files in a cloud environment using DROID. In both cases, DROID only accesses the data needed to complete the format characterisation. This can be done through HTTP (HyperText Transfer Protocol) and S3 (Simple Storage Service) identification.

HTTP identification

This option allows users to identify files from any HTTP web address. It can be used for files on the public internet or for files in a private cloud environment where the authentication is built into the URL. For example, a URL from a cloud software can be used by DROID to identify a file.

Advantages of HTTP

This solution works on most cloud software. This means that if there is a file available on the internet, without needing authentication in the browser session, DROID can identify it.

Disadvantages of HTTP

Due to the way files are delivered over the internet, it’s not possible to identify multiple files at a time using HTTP identification. This makes HTTP unsuitable for identifying large quantities of files.

Additionally, a limitation of HTTP identification is that the web server delivering files needs to accept a specific set of headers and return another specific set. These are common but not implemented in all cloud software, which means DROID will not work with some systems.

S3 identification

S3 identification identifies files within an AWS (Amazon Web Sever) S3 bucket. This method requires valid AWS credentials with access to the files in the S3 bucket. This can be done through environment variables, a credentials file, or SSO (Single Sign-On) login.

Advantages of S3 identification

When using S3 identification, you can identify more than one file at a time. This is done by providing a folder path, which will allow DROID to identify all files that sit within that folder and its sub-folders.

Additionally, as this approach uses AWS credentials, it doesn’t require a publicly accessible URL to identify private files.

Disadvantages of S3 identification

Currently, this method only works for AWS S3 buckets and will not work for other cloud providers such as Azure blob storage or Google storage.

Examples

The following examples highlight how DROID can be used with cloud-hosted files. These examples can only be run using the DROID's command line version (DROID CLI).

HTTP identification

Here is an HTML example from Discovery, The National Archives’ online catalogue.

\.droid https://www.nationalarchives.gov.uk

URI: https://discovery.nationalarchives.gov.uk

SIZE: 48911

TYPE: File

LAST_MODIFIED: 2026-02-23T13:10:48

EXTENSION_MISMATCH: TRUE

PUID: fmt/471

FORMAT_NAME: Hypertext Markup Language

Key points from this output are:

• The file size is correct and is provided by the web server.
• The last modified date is not correct. Whether we can calculate the last modified date depends on certain headers we get in the response. This is discussed in more detail below.

S3 identification

This command was run against a folder path in S3:

\.droid s3://name-of-s3-bucket/files

URI: s3://name-of-s3-bucket/files
SIZE:
TYPE: Folder
LAST_MODIFIED: 1970-01-01T01:00:00
EXTENSION_MISMATCH: FALSE
PUID:
FORMAT_NAME:

URI: s3://name-of-s3-bucket/files/file3.zip
SIZE: 163
TYPE: Container
LAST_MODIFIED: 2026-02-23T12:37:36
EXTENSION_MISMATCH: FALSE
PUID: x-fmt/263
FORMAT_NAME: ZIP Format

URI: s3://name-of-s3-bucket/files/test.zip
SIZE: 5953
TYPE: Container
LAST_MODIFIED: 2026-02-23T12:43:24
EXTENSION_MISMATCH: FALSE
PUID: x-fmt/263
FORMAT_NAME: ZIP Format

URI: s3://name-of-s3-bucket/files/file2.doc
SIZE: 34304
TYPE: File
LAST_MODIFIED: 2026-02-23T12:37:36
EXTENSION_MISMATCH: FALSE
PUID: fmt/40
FORMAT_NAME: Microsoft Word Document

URI: zip:s3://name-of-s3-bucket/files/test.zip!/untitled.docx
SIZE: 6462
TYPE: File
LAST_MODIFIED: 2025-10-10T08:45:56
EXTENSION_MISMATCH: FALSE
PUID: fmt/412
FORMAT_NAME: Microsoft Word for Windows

URI: s3://name-of-s3-bucket/files/file1.tif
SIZE: 13898630
TYPE: File
LAST_MODIFIED: 2026-02-23T12:37:36
EXTENSION_MISMATCH: FALSE
PUID: fmt/353
FORMAT_NAME: Tagged Image File Format

The results are identical to those from HTTP identification but DROID has identified multiple files at the same time using this method.

Technical implementation

General approach

In local systems, DROID will extract metadata from a file including the last modified date and the size of the file. It will then try to identify the file.

To do this, it reads the bytes (data) of the file in chunks of a configurable size and compares these to the patterns of each file type’s unique byte sequence. If the bytes match, then DROID will identify the file as the format attached to that signature. If not, it will move on to the next byte range and try again.

DROID will cache each of these byte ranges, so they are only read from the file once for the first signature checks. For every subsequent check, they are read from memory. This speeds up DROID’s identification, saving users’ time and reducing data usage.

The method for metadata extraction and for reading chunks from the file is different for identification from cloud systems.

For S3 and HTTP identification, the default chunk size is set to 4kB. This is because reading the bytes over a network is much slower than reading from local disk, where the default chunk size is set to 4MB. This speeds up S3 and HTTP identification for large files, however it requires higher memory use.

HTTP metadata and identification

DROID outputs metadata for the files it identifies. For HTTP identification, this metadata comes from the HTTP headers sent in the response from the web server delivering the file. If the web server doesn’t provide the header, then DROID will use a default value.

If the server provides a Last-Modified header in the RFC-1123 format (which displays dates in a standardised format), DROID will return this in the response. If the header is missing or is in an unexpected format, DROID will default to the current date.

If the server provides Content-Range headers, DROID will use this as the file size. If this is not returned, it will default to zero.

DROID will send a Range request header to the server with a value of bytes=<startByte>-<endByte>

If the server can process this header, it will send back only the bytes contained in that range. If the server cannot process this header, DROID will return an error as there is no way to identify a file if the server doesn’t support this request header.

S3 metadata and identification

DROID will use the AWS software development kit (SDK) for Java to get metadata for a file in an S3 bucket. When DROID is run, it will run ListObjects on the S3 file path provided by the user. The response for this call contains the last modified date and file size of each object which is then returned by DROID.

S3 does not store folder information. Every object in a bucket has a single path, although the S3 user interface (UI) will split this into folders if the path contains a /

For example, given the following objects in S3:

/folder1/object.txt  

/folder1/object.pdf  

/folder11/object.csv  

/folder2/object.docx

If you make the request

droid s3://bucket.name/folder1

DROID will return results for /folder1/object.txt, /folder1/object.pdf and /folder11/object.csv

Once it has listed all objects in the bucket with the provided path prefix and collected the metadata, it will run the file identification. This uses the standard GetObject command but with a byte range provided of bytes=<startByte>-<endByte>

Future improvements

Improving DROID is an ongoing project for our team. These improvements may include:

• Adding support for other cloud providers such as Azure blob storage, Google storage and SharePoint.
• Building in support for cloud identification through UI.
• Creating tools, such as Docker images, which will enable DROID to be easily used in native cloud environments.

Digital blog

Introducing export templates: Customise DROID exports without external tools

Explore more DROID features