Before digital records can be transferred to The National Archives, they must be appraised and selected for permanent preservation and reviewed for sensitivity. Once these stages have been completed and the records prepared, the records may be delivered to The National Archives.
For many of these steps you will need support from your ICT supplier or ICT colleagues and we recommend that you share this guidance with them.
Appraisal and Selection (steps 1 and 2): File formats
Records appraisal is a format-neutral process which considers only the content and value of the records. See our guidance on records appraisal for more information about how to appraise records. When you appraise digital records, take care that this work does not unduly alter the metadata of the records. Moving records around can alter the context of the records by removing them from their original folder structure. This, in turn, can make it much harder to review them for sensitivity (see step 3 Sensitivity review).
During the records selection stage, you must consider the file formats of your digital records. Our guidance on suitable file formats for transfer (see our guidance and forms page) explains how the format of digital records affects permanent preservation and lists current suitable file formats for transfer. If records that are selected for permanent preservation include formats which do not appear on this list, please discuss this with us at the earliest opportunity. Formats that are not listed will need to be assessed on an individual basis to determine whether they should be selected and, if so, whether they can be transferred at this stage or whether you (or your sponsoring department) will need to retain and manage them until acceptable arrangements for transfer can be made. Factors such as the format, version, content of the records and number of files will influence this discussion.
If some of your records are duplicated in different formats you should also discuss this with us to agree which versions should be transferred for the purposes of preservation and long-term access.
Appraisal and Selection (steps 1 and 2): Transfer size
It is important for us to know the size of the transfer - both the volume (usually in gigabytes) and the total number of individual files and folders. This helps us plan for the transfer and will inform the choice of transfer media. You will be able to tell us this once the selection has been agreed.
Sensitivity review (step 3): Applying for closure on transfer
All digital records must be reviewed for sensitivity before they can be transferred. The National Archives is unable to accept digital records classified above RESTRICTED/IL3. When such records are selected for permanent preservation, you (or your sponsoring department) must retain and manage them until their sensitivity diminishes.
Unless all records are open on transfer you will be asked to complete an application for closure on transfer form (see our guidance and forms page).
This form specifies the closure status of any records that are not open on transfer and provides information about their sensitivity and the Freedom of Information (FOI) exemptions under which they have been closed. If you transfer multiple series at one time, you will usually make a single application for closure for the entire transfer.
Closure should be asserted at the highest applicable level. For example, if all records within a folder are closed for the same period under the same FOI exemption your application need only reference that folder. However if the files within a folder have different closure periods and/or FOI exemptions, your application will need to reference each file. An application for closure form may cover multiple series, however, you may only list one series, folder or file per row on the form.
Bear in mind that the identifiers you give on this form must correspond with the identifiers you provide in the metadata file for the same records (see guidance on Metadata below).
If you send us redacted records, you will need to transfer the original (un-redacted) versions too, provided these are not classified above RESTRICTED. The un-redacted records should be referenced on the application for closure form. Please consult our redaction toolkit for further guidance on how to redact digital records (see our guidance and forms page).
Closed records will be released on a date calculated from the 'closure period' specified on your application for closure form. We will usually ask you to review this decision prior to release.
For more clarity on the publication of your record descriptions, read our policy below:Policy on the display of closed digital record descriptions (PDF, 0.03Mb)
You need a documented, repeatable process for exporting records and metadata from your system(s) so that they can be transferred. We will ask you how the digital records you intend to transfer are stored and managed and, early in the process, we will ask you to send us a sample export of records and their associated metadata. This allows you to test your export method and will enable us to verify that your records and metadata comply with our requirements. You may need to consult your IT team or service provider at this stage.
For ease of handling, your sample records and metadata should be open (UNCLASSIFIED). We will dispose of the sample securely once we have assessed it, and it will not form part of a formal transfer of records to The National Archives.
As part of the test, you should check that the exported records are not corrupted or changed by the processes. For example the date of the records (a vital piece of metadata) might be reset to the current date if records are copied to a holding area prior to export.
When you are ready to transfer, you will need to examine your file formats in detail to establish the precise versions of each format, verify that your files are free of corruption, and confirm that the formats are as expected. Our free, open source file format identification tool, DROID, can assist you with this. Guidance on how to use DROID is available or we can arrange a face-to-face demonstration of the tool.
We will run DROID on any sample records you send us, as well as running it on any records you transfer. For this reason please ensure that your test sample is representative of the range of formats that will be present in the final transfer.
In addition to reporting file formats and versions, DROID generates all the mandatory metadata fields we require, except copyright and closure status (see Metadata below).
At this stage, you should remove any files that were not included in your agreed selection (for example, temporary .tmp files or system files such as .bat .log .prn or .inf), and check that you can open all of the files and view them fully without needing to enter a password or supply a cryptographic key. You should also scan all files for viruses.
Preparation for transfer (step 4): Metadata
Metadata is additional information describing the records being transferred. It describes the structure and format of the records as well as their content, context, provenance and sensitivity. We use this metadata to help us interpret the structure of the records, to understand how to handle them (for example: are they open or closed? what is their copyright status?) and to list them on our catalogue to facilitate access when appropriate.
Some of the high-level metadata about the records series you are transferring comes to us via the transfer and delivery forms that we will ask you to complete (see stage 5 Packaging and delivery below). Additional metadata about each record is supplied via a metadata file that must accompany the records and the application for closure form for sensitive records.metadata.csv
This file contains contextual and technical information about the folders and files that make up the transfer. You must include a metadata file for every series (or partial series) of digital records being transferred.
The six mandatory metadata fields required for digital records at folder and file level are listed below. We offer a spreadsheet tool to help you generate this metadata for records held on a file-system. The outputs of this tool, or a report from DROID, will provide fields 1-5 below. We will also accept any additional metadata elements that add context or description to the records.
The metadata file must be sent as a .CSV file, encoded as UTF-8. DROID and our metadata tool both provide this. If you use Excel to create your metadata please set it to UTF-8 (under 'Excel Options' > 'Advanced' > 'Web Options' > 'Encoding') and then save as csv (comma separated values).
If you change or remove any files after this stage you must edit the metadata file(s) to reflect the changes. It may be easier to regenerate the metadata file(s) than to amend them.
This should be a meaningful folder or file name.
This is not a system-generated ID number but the file path represented as a URI - it helps supply context for the record. Please ensure that your identifiers match those referenced in the closure file. If you are using a Windows operating system and your filepaths are more than 255 characters in length (including spaces and back slashes) you will likely find it difficult to copy the files onto media for transfer. If your records are in folders nested this deep please contact us to discuss.
This should be the date of the record, not the date it was copied to its current location. For records generated in Windows the date last modified is often the most reliable.
4. Folder or file
State whether the digital object is a folder or a file - this is required to resolve ambiguous file names.
This should be generated using the SHA-2(256) algorithm. We use it to verify that the file we receive has not been corrupted or changed during transfer.
This is only required if the record is not Crown Copyright.
Packaging and delivery (step 5): Transfer methods and media
We offer two methods for transferring digital records to The National Archives:
- physical transfer of encrypted media via a secure courier
- electronic transfer of an encrypted Electronic Assembly via secure file transfer
You must always encrypt all physical media you send to The National Archives. This includes transfers of Open (UNCLASSIFIED) and test data. We accept USB hard drives and DVDs. We will usually loan you an encrypted hard drive for this purpose but if you supply your own drives, your choice of media and encryption technology must conform to our standards. We will discuss these with you in detail when we agree the transfer.
Digital records are transferred as a set of folders and files with accompanying metadata files.
Within each transfer, we expect the folders and files to be structured in a particular way. They may then be delivered to us either on physical media or as electronic assemblies.
Hard drives must contain a single, NTFS formatted, file-system. The file-system volume label(s) provide an identifier for the physical media and you will also need to record these on the accompanying Delivery and Transfer forms. We will pre-set these if you use one of our drives; if you supply your own media we will specify the volume label(s) for you to use.
- If you are sending closed records, save a copy of your application for closure form to the root of the file system on every drive that contains closed records referenced on that form. There is no need to sub-divide the closure form if a transfer spans several series or hard drives
- At the root of the file-system, create a folder representing the series to which the transferring records belong. Use the series code as the folder name, but with an underscore character in place of the space
- Put the metadata.csv file directly into this series folder. You also need to send us a checksum for this metadata file - you should create it exactly as you created the checksums for the records, and save it in the same folder as the metadata.csv file. The checksum should be in a simple text file called 'metadata.sha256'
- Create a further folder called 'content' inside the series folder. This will act as a container for the records themselves. We have not defined the structure within the content folder as this will depend on the records you are transferring. We assume that the contents of the folder will be described by the metadata files you supply
- If you need to transfer records from multiple series, repeat these steps to create additional series folders at the root of the file system
- Do not delete any original records until you receive confirmation that we have safe custody of your records. This is usually within two weeks, but may take longer if you send a large transfer spanning multiple drives
Our hard drives currently have a 2TB capacity. If a series will not fit on a single drive you should divide the records logically between two or more drives. You will also need to divide the metadata file so that each set of records remains with its associated metadata (and the checksum for the metadata file). You may find it easier to generate the metadata for the two parts of the series separately rather than dividing the file. There is no need to sub-divide the closure file in this way.
If you wish to send the transfer electronically by secure ftp, you should structure the files in exactly the same way before creating an encrypted electronic assembly for transfer. We can send you detailed instructions for doing this.
Packaging and delivery (step 5): How to send us the records
Before sending us any digital records, you must complete a Transfer form and send it to us by email (see our guidance and forms page). This formally authorises the transfer of custody, confirms that the selection has been agreed, confirms that you have identified sensitive records and non-Crown Copyright records, and assures us that you have checked and virus scanned the files. Once we confirm both that we have received this form and that the information contained within it is correct, you can send us your records.
Please then complete a Delivery form (see our guidance and forms page). For a transfer of physical media, you should enclose a paper copy of the form within the transfer package itself to help us identify the records easily upon receipt. Send the package by secure courier to the address specified on the delivery form.
For both physical delivery and secure file transfer please also send us a copy of the delivery form by email (to the addresses stated on the form) to tell us that the package has been dispatched.
If you wish to transfer an encrypted electronic assembly by sftp, please contact us for further details of the process.
Remember, you must not delete the records from your systems until we confirm custody. This usually takes two weeks but may take longer if you transfer a large volume of records.