Intelligence Records in The National Archives

Military Records Information 26

1. Introduction

This research guide describes records of intelligence sections of the service departments and some central government departments which are available for consultation at The National Archives. Further information on intelligence and security work in modern government will be found in the major series of departmental files.

Researchers working in this field should note the existence of the inter-departmental Advisory Group on Security and Intelligence Records. Under its terms of reference, this group provides expert guidance on intelligence-related records to the official, archival and academic communities. Researchers can submit requests for guidance to the Advisory Group.

2. War Office

For a large part of the nineteenth century several separate departments within the War Office were given responsibility for different aspects of intelligence related work, particularly with maps. Centralisation was first carried out in 1873 when the Intelligence Branch was established and this became the Directorate of Military Intelligence (DMI) in 1888. In 1904 DMI became part of the Directorate of Military Operations (DMO); it was made independent in 1915 and then reunited with DMO in 1922. The two departments were again separated in 1939.

The various sections of Military Intelligence (MI) and Military Operations (MO) were kept distinct even when under a single head. DMO was responsible for outline operational planning up to the time when an operation Commander was appointed. It also collected information about British forces and the armed forces of close allies. DMI was concerned with the armed forces of enemy countries, distant allies and neutral countries. It was in close touch with military attaches and missions abroad and was interested not only in military details but also in more general historical, topographical and economic information. Until 1940, when a separate department was established for the purpose, DMI was also responsible for censorship. In addition to the departments in London, military intelligence had staff attached to missions overseas and field headquarters gathering local intelligence.

3. Admiralty

Long before intelligence was an organised function, officers of the Royal Navy were expected to report any information about the whereabouts and strength of hostile shipping. The first proposal for a specialist staff to co-ordinate this activity was made in 1879, but it was not until December 1882 that the Foreign Intelligence Committee was set up. This was replaced in 1887 by a new department of the Admiralty under a Director of Naval Intelligence (DNI).

The DNI's department quickly became efficient and highly effective, and thus an influential voice within the Admiralty. It was concerned with all aspects of enemy and allied shipping, including the number of vessels, their types, armaments and so on. It also plotted shipping movements, particularly of enemy surface cruisers and submarines. Information on the topography of foreign countries, particularly coasts, and on coastal defences was also collected. The department played a major part in signals intelligence work in both world wars (see section 5. Signals Intelligence below).

No peace time files of the Naval Intelligence Department (NID) have yet been released, although there are printed reports in ADM 231 .

4. Air Ministry

Until 1918 responsibility for aerial warfare was divided between the War Office and the Admiralty. Only when the Air Ministry was established in that year was a separate Directorate of Air Intelligence created. This became the Directorate of Operations and Intelligence in 1920 and it was not until 1939 that intelligence was made independent again when the work on operations passed to a new Directorate of Plans.

Air Intelligence were interested in allied and enemy aircraft, their fuel systems and weaponry, and airfields. They also prepared analyses of bombing targets and appreciations of raids, much information coming from aerial reconnaissance, and studied enemy and allied air activity. The study of operational research was invented by the RAF, and this involved the creation of models (for example, theoretical or statistical) to discover the most effective solution to a problem. Much intelligence was also gained from prisoners of war in enemy hands, either by coded letters or by interrogation after escapes.

5. Signals Intelligence

On 5 August 1914, the cable ship Telconia lifted from the bed of the North Sea the German overseas telegraph cables. Thereafter German diplomatic communications had to go by wireless, as did signals to the High Seas Fleet and the U boats. These could be intercepted and so were sent by cypher. Cryptography had been subject to a lot of study in Britain before the War, particularly at Naval Intelligence Department, and as a result, specialists at NID were able to read many of Germany's diplomatic and operational signals. The knowledge thus gained gave NID much influence and the work was at times of major significance, leading, for example, to the entry of America into the War because of the interception of the notorious Zimmerman telegram.

In 1919 the Cabinet established the Government Code and Cypher School (GCCS) to advise on the security of British codes and cyphers and to study the methods of cypher communications used by foreign powers. It was an inter-service organisation and was finally placed under the Foreign Office for administrative purposes in 1922. During the Second World War the School was based at Bletchley Park. Many German signals during this period were encyphered by a machine called Enigma, which had originally been developed for commercial use. The Germans improved its security for military use. At Bletchley Park, using the 'Bombe' (and not the computer called 'Colossus'), GCCS decrypted the intercepted signals. The resulting intelligence was called 'X' source or 'Ultra'. The initial work of radio interception was the responsibility of 'Y' service.

6. Foreign Office

Funds for the collection of intelligence abroad had been voted by Parliament for much of the early modern period and were administered by the Secretaries of State. From 1782 the Foreign Secretary took on this role, with administrative responsibility being passed to one of the two Under Secretaries at the Foreign Office in 1825. The Fund was used during the nineteenth century for a number of purposes, not all of them intelligence related. Records relating to the administration of the secret Service Fund, 1791-1909, can be found in the following:

Records relating to the British Secret Service during the Napoleonic Era can be found in FO 38 . These records contain secret communications, originally from agents on the frontier of Holland and France, but later from agents in Northern Germany and Sweden too. The FO 1093 series consists of a collection of miscellaneous unregistered papers originating in the Permanent Under-Secretary's Department (PUSD) and covering the period 1873 to 1946. From 1909 the PUSD was responsible for liaison with the intelligence services and the administration of the secret service fund. This series of records contains minutes of meetings for the Secret Service Bureau formed in 1909 from which both MI5 and MI6 originate. The files also contain correspondence dealing with Secret Service matters such as estimates, expenditure and accounts as well as pensions, foreign espionage in England, Secret Intelligence Service (SIS) activity during the late 1930s including details of the 'Noulens case' and correspondence arising from Secret Service activity in Turkey during the First World War, in particular the activities of the chairman of the Ottoman Public Debt Committee Sir Vincent Calliard. The earlier part of the series, dealing with the period 1939 to 1946 relates to Rudolf Hess during his time as a prisoner of war, and the activities of the supposed pro-Nazi Duke of Windsor during the Second World War.

A fuller description is available in L Atherton, Top secret: An Interim Guide to Recent Releases of Intelligence Records at the Public Record Office (London, 1993).

7. Records of the Special Operations Executive (SOE)

The Special Operations Executive (SOE) was formed in 1940 from three separate organisations to assist local resistance groups and to promote sabotage and subversion throughout occupied Europe; in Winston Churchill's memorable phrase 'to set Europe ablaze.' The chief executive of SOE was responsible to Hugh Dalton at the Ministry of Economic Warfare. It was divided into three organisational branches reflecting its origins; SO1 (propaganda), SO2 (active operations - sub-divided geographically) and SO3 (planning). Its broad remit often led to confusion and inter-departmental disputes with the War Office, the Foreign Office and MI6 regarding its responsibilities and priorities.

In 1941, following a dispute between the Ministry of Information and the Foreign Office, the bulk of SO1 was transferred to the newly created Political Warfare Executive (PWE) under Foreign Office control, whereupon it was amalgamated with the Ministry of Information, Foreign Publicity Department and the BBC European Section. Thereafter the remainder of SOE continued as a planning and operational entity until it was disbanded in 1946. When the Ministry of Economic Warfare was wound up in May 1945 responsibility for SOE was transferred to the Economic Warfare Department of the Foreign Office.

Unfortunately, the vast majority of SOE operational files have not survived. Many were destroyed in a fire at SOE's headquarters shortly after 1945 and some files, particularly personnel files relating to administrative staff seconded from the armed services were destroyed at the end of the war. As there was no central registry and no indication of the file series it is difficult to estimate overall losses, though these have been estimated as being as high as 80 percent. However, surviving SOE records can be found in department code HS.

HS 9 - SOE Personnel Files (PFs) were transferred to The National Archives in 2003. As with SOE operational records many PFs were destroyed, particularly those of junior SOE staff, whilst others were damaged, or contain extracts that continue to be retained under section 3 (4) of the Public Records Act (1958). The aforementioned fire at SOE's offices is thought to have destroyed a further 3000 PFs. Other absences are explained by the fact that many PFs were transferred to the personal files maintained of other government agencies for which an individual agent subsequently worked.

Because of the sensitive personnel nature of many of these files the Lord Chancellor has stated that SOE personnel files remain closed for the lifetime of the individual concerned (i.e. for 100 years from 1930). If a researcher can demonstrate that the person whose file they wish to examine was in fact born more than 100 years ago, or is deceased, or indeed if you are a living SOE agent and wish to examine your own file, then The National Archives will make the relevant record available. Please note, however, that application for disclosure, enclosing any copies of evidence of date of both, or of death, of the subject, must be made in writing to The National Archives and, because of the physical arrangement of the records, an estimated two weeks should be allowed after the approval of the Lord Chancellor before the record can be made available.

Enquiries should be addressed in the first instance to Records Management Department, The National Archives, Kew, Richmond, Surrey, TW9 4DU. (Email: recordsmanagement@nationalarchives.gov.uk).

All surviving SOE records have now been transferred to the National Archives.

MI5 investigations of several suspected renegade SOE agents can be found in KV 1 and KV 5 . There are also many scattered references to the search for suspected renegades in the diaries of Captain Guy Liddell (see below).

Another further useful source of information regarding SOE are the original SOE general file indexes transferred to The National Archives by the Secret Intelligence Service (SIS) in 2004. These contain for each entry the subject original SOE file references. Since the reorganisation of the SOE archive in the 1960s, it is no longer possible, without extensive investigation and speculation, to identify current piece numbers from these file references. Nevertheless, these index cards give the last remaining references to headquarters files that no longer survive. The cards include occasional post-war additions by successive SOE Advisers, including indications that the file referred to was now 'missing', and sometimes, for individuals, dates of death or other biographical information. The two types of card, subject and nominal, use for the most part different forms of printed card, and it seems likely that this index was originally two separate indices which were, at some unknown time in the past, merged. These SOE card indexes can only be seen under supervision. Depending on the subject (indicated below) and the circumstances in which the information on them was compiled, these index cards may list personal details of agents and contacts, pseudonyms, details of the circuits with which they were involved, post-war tracing details and confirmation of death, First Aid Yeomanries (FANYs) attached to SOE with details of postings, contact addresses and in rare instances photographs. They also frequently contain references to the status of contacts known to be in enemy hands, safe houses, enemy intelligence officers, collaborators and traitors.

Records relating to SOE operations can also be found in the files of the Air Ministry (AIR), War Office (WO), Foreign Office (FO) and Prime Minister's Office (PREM). There are over 200 records in these record series, each of which can located using a key word search on the Catalogue, the use of which now supersedes the old SOE 'Source Sheet No.11'. Several research guides have been published relating to SOE operational records (see further reading).

8. Joint Intelligence Committee (JIC)

The Joint Intelligence Committee (JIC), now a part of the Cabinet Office, was established in 1936 to provide the Chiefs of Staff, Ministers and senior officials with intelligence assessments on a range of immediate and long-term issues (i.e. security, defence and foreign affairs). The JIC co-ordinates the work of the separate intelligence services, whose requirements and priorities it sets. The JIC comprises of senior officials drawn from the Foreign Office, Ministry of Defence, Home Office, Department of Trade and Industry, Treasury and Cabinet Office, as well as the heads of the MI5, MI6 and GCHQ.

9. The Security Service (MI5) and the Secret Intelligence Service (MI6)

MI5 started life in March 1909 when, following a recommendation of the Committee of Imperial Defence, the Secret Service Bureau was founded by Captain Vernon Kell (K) and Captain Mansfield Cumming (C) who were responsible for counter-espionage and gathering overseas intelligence, respectively. The extreme secrecy surrounding its operations is reflected by the fact that the sole copy of the sub-committee report regarding its foundation was placed in custody of the Director of Military Operations at the War Office: CAB 16/232 . During August 1914 Kell and his small staff were absorbed into the Directorate of Military Operations as MO5 (g). The following year MO5 became part of the Directorate of Military Intelligence, assumed the title MI5, and was made responsible for upholding the provisions of the Defence of the Realm Regulations and the Aliens Restriction Act in the face of German espionage. Following the Bolshevik revolution in 1917 MI5 began to concentrate on the perceived threat of Communist subversion, which (together with Irish terrorism) was to remain a principal theatre of operations until 1989.

The majority of MI5 records are retained under section 3 (4) of the Public Records Act (1958). They will remain exempt from disclosure under the Freedom of Information Act (2000). However, since 1999 approximately 2000 'historical' files have been transferred to The National Archives. These cover a wide range of subjects and individuals that have fallen under the purview of MI5 since its inception. These include files on MI5 operations during World War One (KV 1 ); German spies and intelligence agents, renegades, double agent operations, espionage cases, Japanese, Hungarian and Italian intelligence agents, SOE agents, right-wing extremists and fascists, Soviet intelligence officers, communist 'front' organisations, pacifists and refugees (KV 2 ); German and Soviet intelligence operations, the ARCOS raid and British fascism (KV 3 ); the History of the Security Service, the 'Curry Report' and MI5 section history's (KV 4 ); Soviet, Pro-Nazi and Zionist organisations (KV 5 ); Jeffrey Hamm, compromised SOE agents and investigation of leaks of information to German intelligence (KV 6 ).

One particularly interesting source on MI5 held by The National Archives are the Second World War diaries (See KV 4/185-196 ) of Captain Guy Liddell, head of MI5's B Division (counter-subversion). Volumes 1 and 12 are available on DocumentsOnline (See www.nationalarchives.gov.uk/documentsonline/).

The KV series is accruing; additions are made bi-annually. However, the records of Security Service are necessarily incomplete. Between 1909 and the early 1970s more than 175,000 files were destroyed 'as obsolete or because of major contractions in the service,' primarily in the aftermath of the two world wars though some were microfilmed prior to destruction, and where these records have now been released, the records transferred to The National Archives are paper print outs (of varying quality) from those films. Following the realisation that the destruction of relevant records was hampering its investigations there followed a brief interregnum during which all files were preserved 'indefinitely'. However, following the fall of Soviet Communism in the early 1990s and the associated decline of 'the threat from subversion' a policy of review and destruction was again instituted leading to the destruction of a further 180,000 files. For more information see www.mi5.gov.uk.

For further information on how the records of the Security Service are appraised and selected see the 'Operational Selection Policy OSP8: The Security Service' available online at: www.nationalarchives.gov.uk/documents/osp8.pdf.

For more on this topic see Gill Bennett, 'Declassification and Release Policies of the UK's Intelligence Agencies' and Sir Stephen Lander 'British Intelligence in the Twentieth Century' both in Intelligence and National Security, Vol. 17 No. 2, 2002, pp.7-32.

The records of the Secret Intelligence Service (SIS), more commonly known as MI6 are not in the public domain nor open to public scrutiny. However, a number of MI6 'CX' reports can be found in HW1 (see below) which consist of signals intelligence passed by Government Code and Cypher School (GC&CS) to 'C' the head of SIS and thence to the Prime Minister during the course of the Second World War. FO 1093 also provides an illuminating source on the activities and funding of SIS. Much SIS material can also be found throughout the files of its sister agency MI5.

10. Government Code and Cypher School (GC&CS) and its successors

The Government Code and Cypher School (GC&CS) was formed in 1919 with overall responsibility for Signals intelligence (SIGINT); the interception of enemy communications as well as the security of the Government's own communications. Most famously GC&CS scored a resounding success against Nazi Germany when its code-breakers cracked the enciphered codes to the ENIGMA machine in World War Two. In 1946 the GC&CS was renamed the Government Communications Headquarters (GCHQ). GCHQ is also responsible for the security of the government's electronic communications through its Communications Electronic Security Group. Records of GC&CS (as well as the Far East Combined Bureau and the interception and direction finding station at RAF Cheadle) can be found in the HW record series. There are 74 series in the HW class, some highlights of which include:

For further information on how the records of the GCHQ are appraised and selected see 'Operational Selection Policy OSP28: Government Communication Headquarters and Predecessors' (www.nationalarchives.gov.uk/recordsmanagement/selection/osp-subject.htm).

11. Defence Intelligence Staff (DIS)

The Defence Intelligence Staff (DIS) was formed in 1964 following the merger of all three intelligence staffs and the civilian Joint Intelligence Bureau (JIB) created in 1946 under the direction of General Eisenhower's British wartime intelligence chief, General Keith Strong. The creation of the DIS was intended to form an integrated service to analyse and collate raw intelligence and produce measured intelligence assessments on a range of strategic issues that would serve the policy makers and planners of the Ministry of Defence, the Armed Forces and other Government Departments.

The files of the Defence Intelligence Staff Sub-Committee for the years 1966-1974 can be found in DEFE 27 . This series was created for registered files of major Ministry of Defence committees, although at present only the Headquarters Organisation committee and its Defence Intelligence Staff subcommittee is currently represented. Files relating to the work of R.V. Jones, the Director of Scientific Intelligence from 1939-54, can be located in DEFE 40 .

Further Reading

(ri2026) Last updated: 7 October 2008 | © Crown Copyright 2005