Evidence based policing (of booters)

Cybercrime, Security economics

“Booters” (they usually call themselves “stressers” in a vain attempt to appear legitimate) are denial-of-service-for-hire websites where anyone can purchase small scale attacks that will take down a home Internet connection, a High School (perhaps there’s an upcoming maths test?) or a poorly defended business website. Prices vary but for around $20.00 you can purchase as many 10 minute attacks as you wish to send for the next month! In pretty much every jurisdiction, booters are illegal to run and illegal to use, and there have been a series of Law Enforcement take-downs over the years, notably in the US, UK, Israel and the Netherlands.

On Wednesday December 14th, in by far the biggest operation to date, the FBI announced the arrest of six booter operators and the seizure of 49 (misreported as 48) booter domain names. Visiting those domains will now display a “WEBSITE SEIZED” splash page.

FBI website seizure splash page

The seizures were “evidence based” in that the FBI specifically targeted the most active booters by taking advantage of one of the datasets collected by the Cambridge Cybercrime Centre, which uses self-reported data from booters.
Continue reading Evidence based policing (of booters) →

Hiring for AP4L

Cybercrime, Jobs, Privacy technology

I’m hiring a Research Assistant/Associate to work on the EPSRC-funded Adaptive PETs to Protect & emPower People during Life Transitions (AP4L) project. The project is being undertaken with the Universities of Surrey, Queen Mary, Strathclyde, Edge Hill, and Edinburgh.

AP4L is a program of interdisciplinary research, centring on the online privacy & vulnerability challenges that people face when going through major life transitions. The four transitions we are considering in the scope of this project are relationship breakdowns; LBGT+ transitions or transitioning gender; entering/ leaving employment in the Armed Forces; and developing a serious illness or becoming terminally ill. Our central goal is to develop privacy-by-design technologies to protect & empower people during these transitions.

We are looking for a researcher with experience in quantitative data analysis, threat assessment, data science, machine learning and/or natural language processing, as well as excellent programming and technical writing skills. Expertise in cybercrime or privacy enhancing technologies (PETs) research is desirable, but not essential. Successful applicants will review the relevant literature, design research projects, develop tools, collect and analyse data, and write research outputs.

The role will analyse life transitions from the attacker’s perspective, such as how and where they gather information about their victims. This will require the analysis of cybercrime forums and similar data at scale. Furthermore, the tools we develop are designed for an adversarial context. Adversaries include those known to individuals, such as interfamilial abuse, as well as targeted and indiscriminate attacks. The researcher will also undertake a rigorous threat analysis for each of the tools developed within the overall project.

The full details are available here.

Chatcontrol or Child Protection?

Academic papers, Cybercrime, Internet censorship, Legal issues, News coverage, Politics, Privacy technology

Today I publish a detailed rebuttal to the argument from the intelligence community that we need to break end-to-end encryption in order to protect children. This has led in the UK to the Online Safety Bill and in the EU to the proposed Child Sex Abuse Regulation, which has become known in Brussels as “chatcontrol”.

The intelligence community wants to break WhatsApp, as that carries everything from diplomatic and business negotiations to MPs’ wheeling and dealing. Both the UK and EU proposals will take powers to mandate scanning of both text and images in your phone before messages are encrypted and sent, or after they are received and decrypted.

This is justified with arguments around child protection, which require careful study. Most child abuse happens in dysfunctional families, with the abuser typically being the mother’s partner; technology is often abused as a means of extortion and control. Indecent images get shared with outsiders, and user reports of such images are a really important way of alerting the police to new cases. There are also abusers who look for vulnerable minors online, and here too it’s user reporting that does most of the work.

But it costs money to get moderators to respond to user reports of abuse, so the tech firms’ performance here is unimpressive. Facebook seems to be the best of a bad lot, while Twitter is awful – and so hosts a lot more abuse. There’s a strong case for laws to compel service providers to manage user reporting better, and the EU’s Digital Services Act goes some way in this direction. The Online Safety Bill should be amended to do the same, and we produced a policy paper on this last week.

But details matter, as it’s important to understand the many inappropriate laws, dysfunctional institutions and perverse incentives that get in the way of rational policies around the online aspects of crimes of sexual violence against minors. (The same holds for violent online political extremism, which is also used as an excuse for more censorship and surveillance.) We do indeed need to spend more money on reducing violent crime, but it should be spent locally on hiring more police officers and social workers to deal with family violence directly. We also need welfare reform to reduce the number of families living in poverty.

As for surveillance, it has not helped in the past and there is no real prospect that the measures now proposed would help in the future. I go through the relevant evidence in my paper and conclude that “chatcontrol” will not improve child protection, but damage it instead. It will also undermine human rights at a time when we need to face down authoritarians not just technologically and militarily, but morally as well. What’s the point of this struggle, if not to defend democracy, the rule of law, and human rights?

Edited to add: here is a video of a talk I gave on the paper at Digitalize.

ML models must also think about trusting trust

Academic papers, Programming languages, Security engineering

Our latest paper demonstrates how a Trojan or backdoor can be inserted into a machine-learning model by the compiler. In his Turing Award lecture, Ken Thompson explained how this could be done to an operating system, and in previous work we’d shown you you can subvert a model by manipulating the order in which training data are presented. Could these ideas be combined?

The answer is yes. The trick is for the compiler to recognise what sort of model it’s compiling – whether it’s processing images or text, for example – and then devising trigger mechanisms for such models that are sufficiently covert and general. The takeaway message is that for a machine-learning model to be trustworthy, you need to assure the provenance of the whole chain: the model itself, the software tools used to compile it, the training data, the order in which the data are batched and presented – in short, everything.

Assistant/Associate Professor in Security and Privacy

Jobs

The Department of Computer Science and Technology is hiring six new faculty members, including an Assistant or Associate Professor in the area of Privacy and/or Security.


The Department is one of the world leaders in computer security, with outstanding historic contributions (such as the Needham-Schroeder protocol and the economics of computer security), as well as vibrant current research (the Cambridge Cybercrime Centre, CHERI processor architecture, and hardware tamper lab). Security is one of the ten core research themes in the department. We take a holistic and interdisciplinary view of the topic, so while we look in detail at many of the technical areas, we also work across traditional subject boundaries to tackle major challenges.


We are looking for someone who can demonstrate they are capable of world-class research which will complement existing expertise in the Department. Given the fast-moving nature of the field, evidence of breadth and flexibility in research is expected.


We aim to substantially broaden coverage of security-related research and teaching in the Department and we welcome applications relating to a wide range of security and privacy topics, including cryptography, cryptographic protocols and verification, distributed-systems security, malware analysis, forensics, machine learning, privacy, software security, computer hardware security, human factors, ledger technologies, and security economics.


The full details are available here.

The Online Safety Bill: Reboot it, or Shoot it?

Academic papers, Cryptology, Cybercrime, Internet censorship, Legal issues, News coverage, Politics, Security economics, Social networks

Yesterday I took part in a panel discussion organised by the Adam Smith Institute on the Online Safety Bill. This sprawling legislative monster has outlasted not just six Secretaries of State for Culture, Media and Sport, but two Prime Ministers. It’s due to slither back to Parliament in November, so we wrote a Policy Brief that explains what it tries to do and some of the things it gets wrong.

Some of the bill’s many proposals command wide support – for example, that online services should enable users to contact them effectively to report illegal material, which should be removed quickly. At present, only copyright owners and the police seem to be able to get the attention of the major platforms; ordinary people, including young people, should also be able to report unlawful things and have them taken down quickly. Here, the UK government intends to bind only large platforms like Facebook and Twitter. We propose extending the duty to gaming platforms too. Kids just aren’t on Facebook any more.

The Bill also tries to reignite the crypto wars by empowering Ofcom to require services to use “accredited technology” (read: software written by GCHQ contractors) to scan your WhatsApp messages. The idea that you can catch violent criminals such as child abusers and terrorists by bulk text scanning is entirely implausible; the error rates are so high that the police would swamped with false positives. Quite apart from that, bulk intercept has always been illegal in Britain, and would also contravene the European Convention on Human Rights, to which we are still a signatory despite Brexit. This power to mandate client-side scanning has to be scrapped, a move that quite a few MPs already support.

But what should we do instead about illegal images of minors, and about violent online political extremism? More local policing would be better; we explain why. This is informed by our work on the link between violent extremism and misogyny, as well as our analysis of a similar proposal in the EU. So it is welcome that the government is hiring more police officers. What’s needed now is a greater focus on family violence, which is the root cause of most child abuse, rather than using child abuse as an excuse to increase the central agencies’ surveillance powers and budgets.

In our Policy Brief, we also discuss content moderation, and suggest that it be guided by the principle of minimising cruelty. One of the other panelists, Graham Smith, discussed the legal difficulties of regulating speech and made a strong case that restrictions (such as copyright, libel, incitement and harassment) should be set out in primary legislation rather than farmed out to private firms, as at present, or to a regulator, as the Bill proposes. Given that most of the bad stuff is illegal already, why not make a start by enforcing the laws we already have, as they do in Germany? British policing efforts online range from the pathetic to the outrageous. It looks like Parliament will have some interesting decisions to take when the bill comes back.

Talking Trojan: Analyzing an Industry-Wide Disclosure

Academic papers, Open-source security, Programming languages, Security economics, Security engineering, , ,

Talking Trojan: Analyzing an Industry-Wide Disclosure tells the story of what happened after we discovered the Trojan Source vulnerability, which broke almost all computer languages, and the Bad Characters vulnerability, which broke almost all large NLP tools. This provided a unique opportunity to measure software maintenance in action. Who patched quickly, reluctantly, or not at all? Who paid bug bounties, and who dodged liability? What parts of the disclosure ecosystem work well, which are limping along, and which are broken?

Security papers typically describe a vulnerability but say little about how it was disclosed and patched. And while disclosing one vulnerability to a single vendor can be hard enough, modern supply chains multiply the number of affected parties leading to an exponential increase in the complexity of the disclosure. One vendor will want an in-house web form, another will use an outsourced bug bounty platform, still others will prefer emails, and *nix OS maintainers will use a very particular PGP mailing list. Governments sort-of want to assist with disclosures but prefer to use yet another platform. Many open-source projects lack an embargoed disclosure process, but it is often in the interest of commercial operating system maintainers to write embargoed patches – if you can get hold of the right people.

A vulnerability that affected many different products at the same time and in similar ways gave us a unique chance to observe the finite-impulse response of this whole complex system. Our observations reveal a number of weaknesses, such as a potentially dangerous misalignment of incentives between commercially sponsored bug bounty programs and multi-vendor coordinated disclosure platforms. We suggest tangible changes that could strengthen coordinated disclosure globally.

We also hope to inspire other researchers to publish the mechanics of individual disclosures, so that we can continue to measure and improve the critical ecosystem on which we rely as our main defense against growing supply chain threats. In the meantime, our paper can be found here, and will appear in SCORED ‘22 this November.

ExtremeBB: Supporting Large-Scale Research into Misogyny and Online Extremism

Academic papers, ,

Online anonymous platforms such as forums enable freedom of speech, but also facilitate misogyny, extremism, and political polarisation. We have collected tens of millions of postings to such forums and created a new tool for social scientists to study how these phenomena are linked.

Far-right extremism has been associated with a growing number of mass killings, overtaking Islamist terrorism in about 2018. Examples include the Wisconsin Sikh temple shooting (2012), the riots in Charlottesville (2017), the Pittsburgh synagogue shooting (2018), the Christchurch mosque shootings (2019), the US Capitol riots (January 2021), and recently the Buffalo shooting (May 2022). Misogyny has been explicitly linked with terror attacks including the Isla Vista killings (2014), the Toronto Van attack (2018), the Hanau shootings (early 2020), and most recently, the Plymouth shooting in the UK (August 2021).

Are extremism and misogyny linked? Joan Smith documented how the great majority of the men who committed terrorist killings in Europe since 9/11, whether far-right or Islamist, display strongly misogynistic attitudes. Most also have a history of physically abusing women — often in their own families — before committing acts of violence against strangers. The Womanstats database, created by Val Hudson and colleagues, has uncovered many statistically significant relationships between the physical security of women and the security of states: authoritarian patriarchal attitudes undermine good government in multiple ways.

Social scientists — who often have limited technical skills to deal with complicated collection techniques to compile a reasonably meaningful database — lack quantitative measurements at a finer granularity. The case studies collected by Smith and the macroeconomic data collected in Womanstats are compelling in their own ways. However, there are not many high-quality datasets that support quantitative analysis at scales in between individuals and whole societies. The existing resources tend to be small, difficult to access, or not well-maintained.

We have therefore created ExtremeBB, a longitudinal structured textual database of nearly 50M posts made by around 400K registered active members on 12 online extremist forums that promote misogyny and far-right extremism (as of September 2022). Its goal is to facilitate both qualitative and quantitative research on historical trends going back two decades. Our data can help researchers trace the evolution of extremist ideology, extremist behaviours, external political movements and relationships between online subcultures; measure hate speech and toxicity; and explore links between misogyny, far-right extremism, and their correlation. A better understanding of extremist subcultures may lead to more effective interventions, while ExtremeBB may also help monitor the effectiveness of any interventions that are undertaken.

This database is being actively maintained and developed with special attention to ensuring data completeness and making it a reliable resource. Academic researchers can request access through the Cambridge Cybercrime Centre, subject to a standard license to ensure lawful and ethical use. Since the database was first opened to external researchers in 2021, access has been granted to 49 researchers from 16 groups in 12 universities. The paper describing this powerful new resource and describing some of the things we have so far discovered using it can be found here.

The Dynamics of Industry-wide Disclosure

Academic papers, Open-source security, Programming languages, Security economics, Usability

Last year, we disclosed two related vulnerabilities that broke a wide range of systems. In our Bad Characters paper, we showed how to use Unicode tricks – such as homoglyphs and bidi characters – to mislead NLP systems. Our Trojan Source paper showed how similar tricks could be used to make source code look one way to a human reviewer, and another way to a compiler, opening up a wide range of supply-chain attacks on critical software. Prior to publication, we disclosed our findings to four suppliers of large NLP systems, and nineteen suppliers of software development tools. So how did industry respond?

We were invited to give the keynote talk this year at LangSec, and the video is now available. In it we describe not just the Bad Characters and Trojan Source vulnerabilities, but the large natural experiment created by their disclosure. The Trojan Source vulnerability affected most compilers, interpreters, code editors and code repositories; this enabled us to compare responses by firms versus nonprofits and by firms that managed their own response versus those who outsourced it. The interaction between bug bounty programs, government disclosure assistance, peer review and press coverage was interesting. Most of the affected development teams took action, though some required a bit of prodding.

The response by the NLP maintainers was much less enthusiastic. By the time we gave this talk, only Google had done anything – though we now hear that Microsoft is now also working on a fix. The reasons for this responsibility gap need to be understood better. They may include differences in culture between C coders and data scientists; the greater costs and delays in the build-test-deploy cycle for large ML models; and the relative lack of press interest in attacks on ML systems. If many of our critical systems start to include ML components that are less maintainable, will the ML end up being the weakest link?

Morello chip on board

Formal CHERI: rigorous engineering and design-time proof of full-scale architecture security properties

Academic papers, CHERI, CL frontpage, Programming languages, Security engineering

Memory safety bugs continue to be a major source of security vulnerabilities, with their root causes ingrained in the industry:

  • the C and C++ systems programming languages that do not enforce memory protection, and the huge legacy codebase written in them that we depend on;
  • the legacy design choices of hardware that provides only coarse-grain protection mechanisms, based on virtual memory; and
  • test-and-debug development methods, in which only a tiny fraction of all possible execution paths can be checked, leaving ample unexplored corners for exploitable bugs.

Over the last twelve years, the CHERI project has been working on addressing the first two of these problems by extending conventional hardware Instruction-Set Architectures (ISAs) with new architectural features to enable fine-grained memory protection and highly scalable software compartmentalisation, prototyped first as CHERI-MIPS and CHERI-RISC-V architecture designs and FPGA implementations, with an extensive software stack ported to run above them.

The academic experimental results are very promising, but achieving widespread adoption of CHERI needs an industry-scale evaluation of a high-performance silicon processor implementation and software stack. To that end, Arm have developed Morello, a CHERI-enabled prototype architecture (extending Armv8.2-A), processor (adapting the high-performance Neoverse N1 design), system-on-chip (SoC), and development board, within the UKRI Digital Security by Design (DSbD) Programme (see our earlier blog post on Morello). Morello is now being evaluated in a range of academic and industry projects.

Morello desktopMorello chip on board

However, how do we ensure that such a new architecture actually provides the security guarantees it aims to provide? This is crucial: any security flaw in the architecture will be present in any conforming hardware implementation, quite likely impossible to fix or work around after deployment.

In this blog post, we describe how we used rigorous engineering methods to provide high assurance of key security properties of CHERI architectures, with machine-checked mathematical proof, as well as to complement and support traditional design and development workflows, e.g. by automatically generating test suites. This is addressing the third problem, showing that, by judicious use of rigorous semantics at design time, we can do much better than test-and-debug development.
Continue reading Formal CHERI: rigorous engineering and design-time proof of full-scale architecture security properties →