Category Archives: Legal issues

Security-related legislation, government initiatives, court cases

History of the Crypto Wars in Britain

Academic papers, Legal issues, News coverage, Politics, Security economics

Back in March I gave an invited talk to the Cambridge University Ethics in Mathematics Society on the Crypto Wars. They have just put the video online here.

We spent much of the 1990s pushing back against attempts by the intelligence agencies to seize control of cryptography. From the Clipper Chip through the regulation of trusted third parties to export control, the agencies tried one trick after another to make us all less secure online, claiming that thanks to cryptography the world of intelligence was “going dark”. Quite the opposite was true; with communications moving online, with people starting to carry mobile phones everywhere, and with our communications and traffic data mostly handled by big firms who respond to warrants, law enforcement has never had it so good. Twenty years ago it cost over a thousand pounds a day to follow a suspect around, and weeks of work to map his contacts; Ed Snowden told us how nowadays an officer can get your location history with one click and your address book with another. In fact, searches through the contact patterns of whole populations are now routine.

The checks and balances that we thought had been built in to the RIP Act in 2000 after all our lobbying during the 1990s turned out to be ineffective. GCHQ simply broke the law and, after Snowden exposed them, Parliament passed the IP Act to declare that what they did was all right now. The Act allows the Home Secretary to give secret orders to tech companies to do anything they physically can to facilitate surveillance, thereby delighting our foreign competitors. And Brexit means the government thinks it can ignore the European Court of Justice, which has already ruled against some of the Act’s provisions. (Or perhaps Theresa May chose a hard Brexit because she doesn’t want the pesky court in the way.)

Yet we now see the Home Secretary repeating the old nonsense about decent people not needing privacy along with law enforcement officials on both sides of the Atlantic. Why doesn’t she just sign the technical capability notices she deems necessary and serve them?

In these fraught times it might be useful to recall how we got here. My talk to the Ethics in Mathematics Society was a personal memoir; there are many links on my web page to relevant documents.

Compartmentation is hard, but the Big Data playbook makes it harder still

Legal issues, News coverage, Operating systems, Security engineering, Usability

A new study of Palantir’s systems and business methods makes sobering reading for people interested in what big data means for privacy.

Privacy scales badly. It’s OK for the twenty staff at a medical practice to have access to the records of the ten thousand patients registered there, but when you build a centralised system that lets every doctor and nurse in the country see every patient’s record, things go wrong. There are even sharper concerns in the world of intelligence, which agencies try to manage using compartmentation: really sensitive information is often put in a compartment that’s restricted to a handful of staff. But such systems are hard to build and maintain. Readers of my book chapter on the subject will recall that while US Naval Intelligence struggled to manage millions of compartments, the CIA let more of their staff see more stuff – whereupon Aldrich Ames betrayed their agents to the Russians.

After 9/11, the intelligence community moved towards the CIA model, in the hope that with fewer compartments they’d be better able to prevent future attacks. We predicted trouble, and Snowden duly came along. As for civilian agencies such as Britain’s NHS and police, no serious effort was made to protect personal privacy by compartmentation, with multiple consequences.

Palantir’s systems were developed to help the intelligence community link, fuse and visualise data from multiple sources, and are now sold to police forces too. It should surprise no-one to learn that they do not compartment information properly, whether within a single force or even between forces. The organised crime squad’s secret informants can thus become visible to traffic cops, and even to cops in other forces, with tragically predictable consequences. Fixing this is hard, as Palantir’s market advantage comes from network effects and the resulting scale. The more police forces they sign up the more data they have, and the larger they grow the more third-party databases they integrate, leaving private-sector competitors even further behind.

This much we could have predicted from first principles but the details of how Palantir operates, and what police forces dislike about it, are worth studying.

What might be the appropriate public-policy response? Well, the best analysis of competition policy in the presence of network effects is probably Lina Khan’s, and her analysis would suggest in this case that police intelligence should be a regulated utility. We should develop those capabilities that are actually needed, and the right place for them is the Police National Database. The public sector is better placed to commit the engineering effort to do compartmentation properly, both there and in other applications where it’s needed, such as the NHS. Good engineering is expensive – but as the Los Angeles Police Department found, engaging Palantir can be more expensive still.

National Audit Office confirms that police, banks, Home Office pass the buck on fraud

Banking security, Cybercrime, Legal issues, News coverage, Security economics, Security psychology

The National Audit Office has found as follows:

“For too long, as a low value but high volume crime, online fraud has been overlooked by government, law enforcement and industry. It is now the most commonly experienced crime in England and Wales and demands an urgent response. While the Department is not solely responsible for reducing and preventing online fraud, it is the only body that can oversee the system and lead change. The launch of the Joint Fraud Taskforce in February 2016 was a positive step, but there is still much work to be done. At this stage it is hard to judge that the response to online fraud is proportionate, efficient or effective.”

Our regular readers will recall that over ten years ago the government got the banks to agree with the police that fraud would be reported to the bank first. This ensured that the police and the government could boast of falling fraud figures, while the banks could direct such fraud investigations as did happen. This was roundly criticized by the Science and Technology Committee (here and here) but the government held firm. Over the succeeding decade, dissident criminologists started pointing out that fraud was not falling, just going online like everything else, and the online stuff was being ignored. Successive governments just didn’t want to know; for most of the period in question the Home Secretary was one Theresa May, who so impressed her party by “cutting crime” even though she’d cut 20,000 police jobs that she got a promotion.

But pigeons come home to roost eventually, and over the last two years the Office of National Statistics has been moving to more honest crime figures. The NAO report bears close study by anyone interested in cybercrime, in crime generally, and in how politicians game the crime figures. It makes clear that the Home Office doesn’t know what’s going on (or doesn’t really want to) and hopes that other people (such as banks and the IT industry) will solve the problem.

Government has made one or two token gestures such as setting up Action Fraud, and the NAO piously hopes that the latest such (the Joint Fraud Taskforce) could be beefed up to do some good.

I’m afraid that the NAO’s recommendations are less impressive. Let me give an example. The main online fraud bothering Cambridge University relates to bogus accommodation; about fifty times a year, a new employee or research student turns up to find that the apartment they rented doesn’t exist. This is an organised scam, run by crooks in Germany, that affects students elsewhere in the UK (mostly in London) and is netting £5-10m a year. The cybercrime guy in the Cambridgeshire Constabulary can’t do anything about this as only the National Crime Agency in London is allowed to talk to the German police; but he can’t talk to the NCA directly. He has to go through the Regional Organised Crime Unit in Bedford, who don’t care. The NCA would rather do sexier stuff; they seem to have planned to take over the Serious Fraud Office, as that was in the Conservative manifesto for this year’s election.

Every time we look at why some scam persists, it’s down to the institutional economics – to the way that government and the police forces have arranged their targets, their responsibilities and their reporting lines so as to make problems into somebody else’s problems. The same applies in the private sector; if you complain about fraud on your bank account the bank may simply reply that as their systems are secure, it’s your fault. If they record it at all, it may be as a fraud you attempted to commit against them. And it’s remarkable how high a proportion of people prosecuted under the Computer Misuse Act appear to have annoyed authority, for example by hacking police websites. Why do we civilians not get protected with this level of enthusiasm?

Many people have lobbied for change; LBT readers will recall numerous articles over the last ten years. Which? made a supercomplaint to the Payment Services Regulator, and got the usual bland non-reassurance. Other members of the old establishment were less courteous; the Commissioner of the Met said that fraud was the victims’ fault and GCHQ agreed. Such attitudes hit the poor and minorities the hardest.

The NAO is just as reluctant to engage. At p34 it says of the Home Office “The Department … has to influence partners to take responsibility in the absence of more formal legal or contractual levers.” But we already have the Payment Services Regulations; the FCA explained in response to the Tesco Bank hack that the banks it regulates should make fraud victims good. And it has always been the common-law position that in the absence of gross negligence a banker could not debit his customer’s account without the customer’s mandate. What’s lacking is enforcement. Nobody, from the Home Office through the FCA to the NAO, seems to want to face down the banks. Rather than insisting that they obey the law, the Home Office will spend another £500,000 on a publicity campaign, no doubt to tell us that it’s all our fault really.

Regulatory capture

Banking security, Cybercrime, Legal issues, News coverage, Politics, Security economics, Security psychology

Today’s newspapers report that the cladding on the Grenfell Tower, which appears to have been a major factor in the dreadful loss of life there, was banned in Germany and permitted in America only for low-rise buildings. It would have cost only £2 more per square meter to use fire-resistant cladding instead.

The tactical way of looking at this is whether the landlords or the builders were negligent, or even guilty of manslaughter, for taking such a risk in order to save £5000 on an £8m renovation job. The strategic approach is to ask why British regulators are so easily bullied by the industries they are supposed to police. There is a whole literature on regulatory capture but Britain seems particularly prone to it.

Regular readers of this blog will recall many cases of British regulators providing the appearance of safety, privacy and security rather than the reality. The Information Commissioner is supposed to regulate privacy but backs away from confronting powerful interests such as the tabloid press or the Department of Health. The Financial Ombudsman Service is supposed to protect customers but mostly sides with the banks instead; the new Payment Systems Regulator seems no better. The MHRA is supposed to regulate the safety of medical devices, yet resists doing anything about infusion pumps, which kill as many people as cars do.

Attempts to fix individual regulators are frustrated by lobbyists, or even by fear of lobbyists. For example, my colleague Harold Thimbleby has done great work on documenting the hazards of infusion pumps; yet when he applied to be a non-executive director of the MHRA he was not even shortlisted. I asked a civil servant who was once responsible for recommending such appointments to the Secretary of State why ministers never seemed to appoint people like Harold who might make a real difference. He replied wearily that ministers would never dream of that as “the drug companies would make too much of a fuss”.

In the wake of this tragedy there are both tactical and strategic questions of blame. Tactically, who decided that it was OK to use flammable cladding on high-rise buildings, when other countries came to a different conclusion? Should organisations be fined, should people be fired, and should anyone go to prison? That’s now a matter for the public inquiry, the police and the courts.

Strategically, why is British regulators so cosy with the industries they regulate, and what can be done about that? My starting point is that the appointment of regulators should no longer be in the gift of ministers. I propose that regulatory appointments be moved from the Cabinet Office to an independent commission, like the Judicial Appointments Commission, but with a statutory duty to hire the people most likely to challenge groupthink and keep the regulator effective. That is a political matter – a matter for all of us.

Camouflage or scary monsters: deceiving others about risk

Academic papers, Cybercrime, Legal issues, Politics, Security economics, Security psychology

I have just been at the Cambridge Risk and Uncertainty Conference which brings together people who educate the public about risks. They include public-health doctors trying to get people to eat better and exercise more, statisticians trying to keep governments honest about crime statistics, and climatologists trying to educate us about global warming – an eclectic and interesting bunch.

Most of the people in this community see their role as dispelling ignorance, or motivating the slothful. Yet in most of the cases we discussed, the public get risk wrong because powerful interests make a serious effort to scare them about some of life’s little hazards, or to reassure them about others. When this is put to the risk communication folks in a question – whether after a talk or in the corridor – they readily admit they’re up against a torrent of misleading marketing. But they don’t see what they’re doing as adversarial, and I strongly suspect that many risk interventions are less effective as a result.

In my talk (slides) I set this out as simply and starkly as I could. We spend too much on terrorism, because both the terrorists and the governments who’re supposed to protect us from them big up the threat; we spend too little on cybercrime, because everyone from the crooks through the police and the banks to the computer industry has their own reason to talk down the threat. I mentioned recent cases such as Wannacry as examples of how institutions communicate risk in self-serving, misleading ways. I discussed our own study of browser warnings, which suggests that people at least subconsciously know that most of the warnings they see are written to benefit others rather than them; they tune out all but the most specific.

What struck me with some force when preparing my talk, though, is that there’s just nobody in academia who takes a holistic view of adversarial risk communication. Many people look at some small part of the problem, from David Rios’ game-theoretic analysis of adversarial risk through John Mueller’s studies of terrorism risk and Alessandro Acquisti’s behavioural economics of privacy, through to criminologists who study pathways into crime and psychologists who study deception. Of all these, the literature on deception might be the most relevant, though we should also look at politics, propaganda, and studies of why people stubbornly persist in their beliefs – including the excellent work by Bénabou and Tirole on the value people place on belief. Perhaps the professionals whose job comes closest to adversarial risk communication are political spin doctors. So when should we talk about new facts, and when should we talk about who’s deceiving you and why?

Given the current concern over populism and the role of social media in the Brexit and Trump votes, it might be time for a more careful cross-disciplinary study of how we can change people’s minds about risk in the presence of smart and persistent adversaries. We know, for example, that a college education makes people much less susceptible to propaganda and marketing; but what is the science behind designing interventions that are quicker and cheaper in specific circumstances?

When safety and security become one

Academic papers, Legal issues, Open-source security, Operating systems, Security economics, Security engineering

What happens when your car starts getting monthly upgrades like your phone and your laptop? It’s starting to happen, and the changes will be profound. We’ll be able to improve car safety as we learn from accidents, and fixing a flaw won’t mean spending billions on a recall. But if you’re writing navigation code today that will go in the 2020 Landrover, how will you be able to ship safety and security patches in 2030? In 2040? In 2050? At present we struggle to keep software patched for three years; we have no idea how to do it for 30.

Our latest paper reports a project that Éireann Leverett, Richard Clayton and I undertook for the European Commission into what happens to safety in this brave new world. Europe is the world’s lead safety regulator for about a dozen industry sectors, of which we studied three: road transport, medical devices and the electricity industry.

Up till now, we’ve known how to make two kinds of fairly secure system. There’s the software in your phone or laptop which is complex and exposed to online attack, so has to be patched regularly as vulnerabilities are discovered. It’s typically abandoned after a few years as patching too many versions of software costs too much. The other kind is the software in safety-critical machinery which has tended to be stable, simple and thoroughly tested, and not exposed to the big bad Internet. As these two worlds collide, there will be some rather large waves.

Regulators who only thought in terms of safety will have to start thinking of security too. Safety engineers will have to learn adversarial thinking. Security engineers will have to think much more about ease of safe use. Educators will have to start teaching these subjects together. (I just expanded my introductory course on software engineering into one on software and security engineering.) And the policy debate will change too; people might vote for the FBI to have a golden master key to unlock your iPhone and read your private messages, but they might be less likely to vote them a master key to take over your car or your pacemaker.

Researchers and software developers will have to think seriously about how we can keep on patching the software in durable goods such as vehicles for thirty or forty years. It’s not acceptable to recycle cars after seven years, as greedy carmakers might hope; the embedded carbon cost of a car is about equal to its lifetime fuel burn, and reducing average mileage from 200,000 to 70,000 would treble the car industry’s CO2 emissions. So we’re going to have to learn how to make software sustainable. How do we do that?

Our paper is here; there’s a short video here and a longer video here. The full report is available from the EU here.

RIP smart meters

Legal issues, News coverage, Politics, Security economics

The Telegraph has just run an op-ed they asked me to write over the weekend, after I pointed out here on Friday that the Conservative manifesto had quietly downgraded the smart meter programme to a voluntary one.

Regular readers of Light Blue Touchpaper will have followed the smart meter story for almost a decade, back through the dishonest impact assessment to the fact that they pose a threat to critical infrastructure.

Manifestos and tech

Authentication, Cybercrime, Internet censorship, Legal issues, News coverage, Politics, Security economics

The papers went to town yesterday on the Conservative manifesto but missed some interesting bits.

First, no-one seems to have noticed that the smart meter programme is being quietly put to death. We read on page 60 that everyone will be offered a smart meter by 2020. So a mandatory national programme has become voluntary, just like that. Regular readers of this blog will recall that the programme was sold in 2008 by Ed Milliband using a dishonest impact assessment, yet all the parties backed it after 2010, leaving no-one to point out that it was going to cost us all a fortune and never save any carbon. May says she wants to reduce energy costs; this was surely a no-brainer.

That was the good news for England. The good news for friends in rural Scotland is high-speed broadband for all by 2020. But there are some rather weird things in there too.

What on earth is “the right of businesses to insist on a digital signature”? Digital signatures are very 1998, and we already have the electronic signature directive. From whom will businesses be able to insist on a signature, and if I’m one of the legislated victims, how much do I have to pay to buy the apparatus?

All digital businesses will have “to support new digital proofs of identification”. That presumably means forcing firms to use Verify, a dysfunctional online authentication service whose roots lie in Blair’s obsession with identity. If a newspaper currently identifies its subscribers via a proprietary logon, will they have to offer Verify as an option? Will it have to be the only option, displacing Facebook and Twitter? The manifesto also says that local government will have to use Verify; and elsewhere that councils must publish planning applications and bus routes “without the hassle and delay that currently exists.” OK, so some councils could so with more competent webmasters, but don’t worry: “hundreds of leaders from the world of tech can come into government to help deliver better public services.”

The Land Registry, the Ordnance Survey and other quangos that do geography (our leader’s degree subject) will all band together to create the largest open repository of land data in the world. So where will the Ordnance Survey get its money from then? That small question killed the same idea in 2010 after Tim Berners-Lee sold it to Cameron.

There will be a levy on social media companies, like on gambling companies, to support awareness and preventive activity. And they must not direct users, even unintentionally, to hate speech. So will Facebook be fined whenever they let users like a xenophobic article in the Daily Mail?

No doubt in view of the delicacy of such regulatory decisions, Leveson II is killed; there will be a Data Use and Ethics Commission instead. It will advise regulators and develop the principles and rules that will give people confidence their data are being handled properly. Wow. We now have the Charter of Fundamental Rights to give us principles, the GDPR to give us rules, and the ECJ to hammer out the case law. Now the People don’t have confidence in such experts we’re going to let the Prime Minister of the day appoint a different lot.

The next government will further strengthen cyber security standards for government and public services, so presumably all such services will have to use expensive networks such as the NHS-wide network from BT which will expect them to manage their own firewalls without telling them how to.

But don’t worry. It will become “as difficult to commit a crime digitally as it is physically”. There is text about working “with international law enforcement agencies to ensure perpetrators are brought to justice” but our local police force isn’t allowed to do anything effective about online accommodation fraud committed by a gang in Germany. They have to work through the NCA – who don’t care. The manifesto signals more of the same: the NCA will get to eat the SFO, which does crimes over £100m, leaving them even less interested in online crooks who steal a thousand pounds of deposit from dozens of students a year.

In fact there is no signal anywhere in the manifesto that May understands the impact of volume cybercrime, even though it’s now most of the property crime in the UK. She rather prefers to boast of the falling crime over the past seven years, as if it were her achievement as Home Secretary. The simple fact is that crime has been going online like everything else, and until 2015 the online part of it wasn’t recorded properly. This was not the doing of Theresa May, but of Margaret Hodge.

The manifesto rather seems to have been drafted in a geek-free room. And let’s not spoil the party by mentioning the impact that tight immigration targets will have on the IT industry, or for that matter on higher education. Perhaps they want us to hope that they don’t really mean that part of it, but perhaps we’d better make a plan to open a campus in India or Canada, just in case.

Bad malware, worse reporting

Cybercrime, Legal issues, News coverage, Security economics, Security psychology

The Wannacry malware that has infected some UK hospital computers should interest not just security researchers but also people interested in what drives fake news.

Some made errors of fact: the Daily Mail inititally reported the ransom demand as 300 bitcoin, or £415,000, rather than $300 in bitcoin. Others made errors of logic: the Indy, for example, reported that “Up to 90 percent of NHS computers still run XP, released in 2001”, citing as its source a BMJ article which stated that 90% of trusts run this version of Windows. And some made errors of concurrency. After dinner I found inquiries from journalists about my fight with the Prime Minister. My what? Eventually I found that the Guardian had followed something Mrs May’s spokesman had said (“not aware of any evidence that patient data has been compromised”) with something I’d said a couple of hours earlier (“The NHS are saying that patient privacy hasn’t been compromised, but if significant numbers of hospitals have been negligently running unpatched computers for two months after the patch came out, how do they know?”). The Home Secretary later helpfully glossed the PM’s stonewall as “No patient data has been accessed or transferred in any way” but leaving the get-out-of-jail card “that’s the information we’ve been given.”

Many papers caught the international political aspect: that the vulnerability was discovered by the NSA, kept secret rather than fixed (contrary to the advice of Obama’s NSA review group), then stolen from the CIA by the Russians and published via wikileaks. Scary stuff, eh? And we read of some surprising overreactions, such as the GP who switched off his networking as a precaution and found he couldn’t access any of his patients’ records.

As luck would have it, yesterday was the day that I gave my talk on entomology – the classification of software bugs and other security vulnerabilities – to my first-year security and software engineering class. So let’s try to look at it calmly as I’d expect of a student writing an assignment.

The first point is that there’s not a really lot of this malware. The NHS has over 200 hospitals, and the typical IT director is a senior clinician supported by technicians. Yet despite having their IT run by well-meaning amateurs, only 16 NHS organisations have been hit, according to the Register and Kaspersky – including several hospitals.

So the second point is that when the Indy says that “The NHS is a perfect combination of sensitive data and insecure storage. And there’s very little they can do about it” the answer is simple: in well over 90% of NHS organisations, the well-meaning amateurs managed perfectly well. What they did was to keep their systems patched up-to-date; simple hygiene, like washing your hands after going to the toilet.

The third takeaway is that it’s worth looking at the actual code. A UK researcher did so and discovered a kill switch.

Now I am just listening on the BBC morning news to a former deputy director of GCHQ who first cautions against alarmist headlines and argues that everyone develops malware; that a patch had been issued by Microsoft halfway through March; that you can deal with ransomware by keeping decent backups; and that paying ransom will embolden the bad guys. However he claims that it’s clearly an organised criminal attack. (when it could be one guy in his bedroom somewhere) and says that the NCSC should look at whether there is some countermeasure that everyone should have taken (for answer see above).

So our fourth takeaway is that although the details matter, so do the economics of security. When something unexpected happens, you should not just get your head down and look at the code, but look up and observe people’s agendas. Politicians duck and weave; NHS managers blame the system rather than step up to the plate; the NHS as a whole turns every incident into a plea for more money; the spooks want to avoid responsibility for the abuse of their stolen cyberweaponz, but still big up the threat and get more influence for a part of their agency that’s presented as solely defensive. And we academics? Hey, we just want the students to pay attention to what we’re teaching them.

Hope this helps!