How the Daily Mail invented Britain's bungling-est spy-agency

Adam Curtis's latest piece for the BBC starts out as a strange history of the role that the Daily Mail had in the formation of the British MI5 spy-agency, but then veers into an amazing history of MI5 brutal, awful, terrible record of incompetence, foolishness, self-sabotage, and waste. It turns out that the MI5 owes its origins to a German spy-scare the Mail whipped up 1910 by publishing a serialized novel about a fictional German invasion of England (the route of the invasion was tailored to pass through towns with large populations of Mail subscribers). This led to thousands of impressionable Mail readers writing in, saying they'd seen German spies out and about, and they demanded that Parliament Do Something. And so, MI5 was born.

But as I said, this is just the start of the story. Following on from its weird origin, MI5 spent generations cocking up, framing people, missing double-agents in their ranks, and generally wasting tons of money on paranoid losers who never caught a spy. Curtis is brutal in documenting the depth of MI5's failures, from its storied roundup of a "German spy ring" in 1914 (decades later, it emerged that none of the 21 "spies" were actually spies) to its failure to spot the incipient demise of the Soviet Union, to its hilariously evil false espionage accusations against 33 Iraqi students in 1991, none of whom were spies.

Curtis builds up a picture of spooks as neither evil masterminds nor brave sleuths, but as banal incompetents. As John Le Carre said of his own service in the spy-world: "the reality was the mediocrity. Ex-colonial policemen mingling with failed academics, failed lawyers, failed missionaries and failed debutantes gave our canteen the amorphous quality of an Old School outing on the Orient express. Everyone seemed to smell of failure."

Bettaney came back to London a changed man. He decided that MI5 was both corrupt and incompetent. He started drinking heavily and told his colleagues loudly that he was no longer a fascist – but he had become a communist.

So MI5 decided to promote him. He was positively vetted again – found to be perfect MI5 material, and sent to the Russian desk.

Bettaney became more and more unstable. In October 1982 he was convicted of being drunk and disorderly. The next week he was convicted for fare-dodging. Finally MI5 did begin to notice – and two separate inquiries were set up to look into Bettaney's behaviour. But each was unaware of the other's existence.

Neither of them noticed that he had been stealing a huge amount of MI5 top secret documents and stashing them at his home. Bettaney was only caught when he took some of the best of these secrets and tried to stuff them into the letter box of the Second Secretary of the Russian Embassy – Mr Gouk.

Mr Gouk was so confused by this that, instead of passing them on to the KGB, he went round to MI5 and gave them back, and told them where they had come from. MI5 arrested Bettaney and he was put on trial.

The man who was in charge of the vetting of government employees – like Michael Bettaney – was then allowed to vet the members of the jury at Bettaney's trial. Luckily this time he got it right – and Bettaney was sent to prison on the Isle of Sheppey for 23 years.