Category Archives: Academic papers

Three Paper Thursday: Attacking Machine Vision Models In Real Life

This is a guest post by Alex Shepherd.

There is a growing body of research literature concerning the potential threat of physical-world adversarial attacks against machine-vision models. By applying adversarial perturbations to physical objects, machine-vision models may be vulnerable to images containing these perturbed objects, resulting in an increased risk of misclassification. The potential impacts could be significant and have been identified as risk areas for autonomous vehicles and military UAVs.

For this Three Paper Thursday, we examine the following papers exploring the potential threat of physical-world adversarial attacks, with a focus on the impact for autonomous vehicles.

Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world, arXiv:1607.02533 (2016)

In this seminal paper, Kurakin et al. report their findings of an experiment conducted using adversarial images taken from a phone camera as input for a pre-trained ImageNet Inceptionv3 image classification model. Methodology was based on a white-box threat model, with adversarial images crafted from the ImageNet validation dataset using the Inceptionv3 model.
Continue reading Three Paper Thursday: Attacking Machine Vision Models In Real Life

SHB Seminar

The SHB seminar on November 5th was kicked off by Tom Holt, who’s discovered a robust underground market in identity documents that are counterfeit or fraudulently obtained. He’s been scraping both websites and darkweb sites for data and analysing how people go about finding, procuring and using such credentials. Most vendors were single-person operators although many operate within affiliate programs; many transactions involved cryptocurrency; many involve generating pdfs that people can print at home and that are good enough for young people to drink alcohol. Curiously, open web products seem to cost twice as much as dark web products.

Next was Jack Hughes, who has been studying the contract system introduced by hackforums in 2018 and made mandatory the following year. This enabled him to analyse crime forum behaviour before and during the covid-19 era. How do new users become active, and build up trust? How does it evolve? He collected 200,000 transactions and analysed them. The contract mandate stifled growth quickly, leading to a first peak; covid caused a second. The market was already centralised, and became more so with the pandemic. However contracts are getting done faster, and the main activity is currency exchange: it seems to be working as a cash-out market.

Anita Lavorgna has been studying the discourse of groups who oppose public mask mandates. Like the antivaxx movement, this can draw in fringe groups and become a public-health issue. She collected 23654 tweets from February to June 2020. There’s a diverse range of voices from different places on the political spectrum but with a transversal theme of freedom from government interference. Groups seek strength in numbers and seek to ally into movements, leading to the mask becoming a symbol of political identity construction. Anita found very little interaction between the different groups: only 144 messages in total.

Simon Parkin has been working on how we can push back on bad behaviours online while they are linked with good behaviours that we wish to promote. Precision is hard as many of the desirable behaviours are not explicitly recognised as such, and as many behaviours arise as a combination of personal incentives and context. The best way forward is around usability engineering – making the desired behaviours easier.

Bruce Schneier was the final initial speaker, and his topic was covid apps. The initial rush of apps that arrived in March through June have known issues around false positives and false negatives. We’ve also used all sorts of other tools, such as analysis of Google maps to measure lockdown compliance. The third thing is the idea of an immunity passport, saying you’ve had the disease, or a vaccine. That will have the same issues as the fake IDs that Tom talked about. Finally, there’s compliance tracking, where your phone monitors you. The usual countermeasures apply: consent, minimisation, infosec, etc., though the trade-offs might be different for a while. A further bunch of issues concern home working and the larger attack surface that many firms have as a result of unfamiliar tools, less resistance to being tols to do things etc.

The discussion started on fake ID; Tom hasn’t yet done test purchases, and might look at fraudulently obtained documents in the future, as opposed to completely counterfeit ones. Is hackforums helping drug gangs turn paper into coin? This is not clear; more is around cashing out cybercrime rather than street crime. There followed discussion by Anita of how to analyse corpora of tweets, and the implications for policy in real life. Things are made more difficult by the fact that discussions drift off into other platforms we don’t monitor. Another topic was the interaction of fashion: where some people wear masks or not as a political statement, many more buy masks that get across a more targeted statement. Fashion is really powerful, and tends to be overlooked by people in our field. Usability research perhaps focuses too much on the utilitarian economics, and is a bit of a blunt instrument. Another example related to covid is the growing push for monitoring software on employees’ home computers. Unfortunately Uber and Lyft bought a referendum result that enables them to not treat their staff in California as employees, so the regulation of working hours at home will probably fall to the EU. Can we perhaps make some input into what that should look like? Another issue with the pandemic is the effect on information security markets: why should people buy corporate firewalls when their staff are all over the place? And to what extent will some of these changes be permanent, if people work from home more? Another thread of discussion was how the privacy properties of covid apps make it hard for people to make risk-management decisions. The apps appear ineffective because they were designed to do privacy rather than to do public health, in various subtle ways; giving people low-grade warnings which do not require any action appear to be an attempt to raise public awareness, like mask mandates, rather than an effective attempt to get exposed individuals to isolate. Apps that check people into venues have their own issues and appear to be largely security theatre. Security theatre comes into its own where the perceived risk is much greater than the actual risk; covid is the opposite. What can be done in this case? Targeted warnings? Humour? What might happen when fatigue sets in? People will compromise compliance to make their lives bearable. That can be managed to some extent in institutions like universities, but in society it will be harder. We ended up with the suggestion that the next SHB seminar should be in February, which should be the low point; after that we can look forward to things getting better, and hopefully to a meeting in person in Cambridge on June 3-4 2021.

How an Illicit Cybercrime Market Evolves: A Longitudinal Study

Online underground marketplaces are an essential part of the cybercrime economy. They often act as a cash-out market, enabling the trade in illicit goods and services between pseudonymous members. To understand their characteristics, previous research mostly uses vendor ratings, public feedback, sometimes private messages, friend status, and post content. However, most research lacks comprehensive (and important) data about transactions made by the forum members.

Our recent paper (original talk here) published at the Internet Measurement Conference (IMC’20) examines how an online illicit marketplace evolves over time (especially its performance as an infrastructure for trust), including a significant shift through the COVID-19 pandemic. This study draws insights from a novel, rich and powerful dataset containing hundreds of thousands contractual transactions made by members of HackForums — the most popular online cybercrime community. The data includes a two-year historical record of the contract system, originally adopted in June 2018 as an attempt to mitigate scams and frauds occurring between untrusted parties. As well as contractual arrangements, the dataset includes thousands of associated members, threads, posts on the forum, which provide additional context. To study the longitudinal maturation of this marketplace, we split the timespan into three eras: Set-up, Stable, and COVID-19. These eras are defined by two important external milestones: the enforcement of the new forum’s policy in March 2019, and the declaration of the global pandemic in March 2020.

We applied a range of analysis and statistical modelling approaches to outline the maturation of economic and social characteristics of the market since the day it was introduced. We find the market has centralised over time, with a small proportion of ‘power users’ involved in the majority of transactions. In term of trading activities, currency exchange and payments account for the largest proportion of both contracts and users involved, followed by giftcards and accounts/licenses. The other popular products include automated bots, hacking tutorials, remote access tools (RATs), and eWhoring packs. Contracts are settled faster over time, with the completion time dropping from around 70 hours in the early months to less than 10 hours during the COVID-19 Era in June 2020.

We quantitatively estimate a lower bound total trading value of over 6 million USD for public and private transactions. With regards to payment methods preferably used within the market, Bitcoin and PayPal dominate the others at all times in terms of both trading values and number of contracts involved. A subset of new members joining the market face the ‘cold start’ problem, which refers to the difficulties of how to establish and build up a reputation base while initially having no reputation. We find that the majority of these build up their profile by participating in low-level currency exchanges, while some instead establish themselves by offering products and services.

To examine the behaviours of members over time, we use Latent Transition Analysis to discover hidden groups among the forum’s members, including how members move between groups and how they change across the lifetime of the market. In the Set-up Era, we see users gradually shift to the new system with a large number of ‘small scale’ users involved in one-off transactions, and few ‘power-users’. In the Stable Era, we see a shift in the composition and scale of the market when contracts become compulsory, with a growth of ‘business-to-consumer’ trades by ‘power-users’. In the COVID-19 Era, the market further concentrates around already-existing ‘power-users’, who are party to multiple transactions with others.

Overall, the marketplace provides a range of trust capabilities to facilitate trade between pseudonymous parties with the control is becoming further centralised with administrators acting as third-party arbitrators. The platform is clearly being used as a cash-out market, with most trades involving the exchange of currencies. In term of the three eras, the big picture shows two significant rises in the market’s activities in response to two major events that happened at the beginning of Stable and COVID-19 eras. Particularly, we observe a stimulus (rather than transformation) in trading activities during the pandemic: the same kinds of transactions, users, and behaviours, but at increased volumes. By looking at the context of forum posts at that time, we see a period of mass boredom and economic change, when some members are no longer at school while others have become unemployed or are unable to go to work. A need to make money and the availability of time in their hand to do so may be a factor resulting in the increase of trading activities seen at this time.

Some limitations of our dataset include no ground truth verification, in which we have no way to verify if transactions actually proceed as set out in the contractual agreements. Furthermore, the dataset contains a large number of private contracts (around 88%), in which we only can observe minimal information. The dataset is available to academic researchers through the Cambridge Cybercrime Center‘s data-sharing agreements.

Three paper Thursday: COVID-19 and cybercrime

For a slightly different Three Paper Thursday, I’m pulling together some of the work done by our Centre and others around the COVID-19 pandemic and how it, and government responses to it, are reshaping the cybercrime landscape. 

The first thing to note is that there appears to be a nascent academic consensus emerging that the pandemic, or more accurately, lockdowns and social distancing, have indeed substantially changed the topology of crime in contemporary societies, leading to an increase in cybercrime and online fraud. The second is that this large-scale increase in cybercrime appears to be the result of a growth in existing cybercrime phenomena rather than the emergence of qualitatively new exploits, scams, attacks, or crimes. This invites reconsideration not only of our understandings of cybercrime and its relation to space, time, and materiality, but additionally to our understandings of what to do about it.

Continue reading Three paper Thursday: COVID-19 and cybercrime

Three Paper Thursday: Broken Hearts and Empty Wallets

This is a guest post by Cassandra Cross.

Romance fraud (also known as romance scams or sweetheart swindles) affects millions of individuals globally each year. In 2019, the Internet Crime Complaint Centre (IC3) (USA) had over US$475 million reported lost to romance fraud. Similarly, in Australia, victims reported losing over $AUD80 million and British citizens reported over £50 million lost in 2018. Given the known under-reporting of fraud overall, and online fraud more specifically, these figures are likely to only be a minority of actual losses incurred.

Romance fraud occurs when an offender uses the guise of a legitimate relationship to gain a financial advantage from their victim. It differs from a bad relationship, in that from the outset, the offender is using lies and deception to obtain monetary rewards from their partner. Romance fraud capitalises on the fact that a potential victim is looking to establish a relationship and exhibits an express desire to connect with someone. Offenders use this to initiate a connection and start to build strong levels of trust and rapport.

As with all fraud, victims experience a wide range of impacts in the aftermath of victimisation. While many believe these to be only financial, in reality, it extends to a decline in both physical and emotional wellbeing, relationship breakdown, unemployment, homelessness, and in extreme cases, suicide. In the case of romance fraud, there is the additional trauma associated with grieving both the loss of the relationship as well as any funds they have transferred. For many victims, the loss of the relationship can be harder to cope with than the monetary aspect, with victims experiencing large degrees of betrayal and violation at the hands of their offender.

Sadly, there is also a large amount of victim blaming that exists with both romance fraud and fraud in general. Fraud is unique in that victims actively participate in the offence, through the transfer of money, albeit under false pretences. As a result, they are seen to be culpable for what occurs and are often blamed for their own circumstances. The stereotype of fraud victims as greedy, gullible and naïve persists, and presents as a barrier to disclosure as well as inhibiting their ability to report the incident and access any support services.

Given the magnitude of losses and impacts on romance fraud victims, there is an emerging body of scholarship that seeks to better understand the ways in which offenders are able to successfully target victims, the ways in which they are able to perpetrate their offences, and the impacts of victimisation on the individuals themselves. The following three articles each explore different aspects of romance fraud, to gain a more holistic understanding of this crime type.

Continue reading Three Paper Thursday: Broken Hearts and Empty Wallets

A measurement of link rot: 57%

I submitted my PhD on the 31st August 2005 (9 months before Twitter started, almost two years before the first iPhone). The easiest version to find (click here) contains the minor revisions requested by my examiners and some typographical changes to fit it into the Computer Lab’s Technical Report series.

Since it seemed like a good idea at the time, my thesis has an annotated bibliography (so you can read a brief precis of what I referenced, which could assist you in deciding whether to follow it up). I also went to some effort to identify online versions of everything I cited, because it always helpful to just click on a link and immediately see the paper, news article or other material.

The thesis has 153 references, in two cases I provided two URLs, and in three cases I could not provide any URL — though I did note that the three ITU standards documents I cited were available from the ITU bookshop and it was possible to download a small number of standards without charge. That is, the bibliography contained 152 URLs.
Continue reading A measurement of link rot: 57%

Cambridge Cybercrime Centre: COVID briefing papers

The current coronavirus pandemic has significantly disrupted all our societies and, we believe, it has also significantly disrupted cybercrime.

In the Cambridge Cybercrime Centre we collect crime-related datasets of many types and we expect, in due course, to be able to identify, measure and document this disruption. We will not be alone in doing so — a key aim of our centre is to make datasets available to other academic researchers so that they too can identify, measure and document. What’s more, we make this data available in a timely manner — sometimes before we have even looked at it ourselves!

When we have looked at the data and identified what might be changing (or where the criminals are exploiting new opportunities) then we shall of course be taking the traditional academic path of preparing papers, getting them peer reviewed, and then presenting them at conferences or publishing them in academic journals. However, that process is extremely slow — so we have decided to provide a faster route for getting out the message about what we find to be going on.

Our new series of “COVID Briefing Papers” are an ongoing series of short-form, open access reports aimed at academics, policymakers, and practitioners, which aim to provide an accessible summary of our ongoing research into the effects which the coronavirus pandemic (and government responses) are having on cybercrime. We’re hoping, at least for a while, to produce a new briefing paper each week … and you can now read the very first, where Ben Collier explains what has happened to illegal online drug markets… just click here!

Towards greater ecological validity in security usability

When you are a medical doctor, friends and family invariably ask you about their aches and pains. When you are a computer specialist, they ask you to fix their computer. About ten years ago, most of the questions I was getting from friends and family as a security techie had to do with frustration over passwords. I observed that what techies had done to the rest of humanity was not just wrong but fundamentally unethical: asking people to do something impossible and then, if they got hacked, blaming them for not doing it.



So in 2011, years before the Fido Alliance was formed (2013) and Apple announced its smartwatch (2014), I published my detailed design for a clean-slate password replacement I called Pico, an alternative system intended to be easier to use and more secure than passwords. The European Research Council was generous enough to fund my vision with a grant that allowed me to recruit and lead a team of brilliant researchers over a period of five years. We built a number of prototypes, wrote a bunch of papers, offered projects to a number of students and even launched a start-up and thereby learnt a few first-hand lessons about business, venture capital, markets, sales and the difficult process of transitioning from academic research to a profitable commercial product. During all those years we changed our minds a few times about what ought to be done and we came to understand a lot better both the problem space and the mindset of the users.

Continue reading Towards greater ecological validity in security usability

Security and Human Behaviour 2020

I’ll be liveblogging the workshop on security and human behaviour, which is online this year. My liveblogs will appear as followups to this post. This year my program co-chair is Alice Hutchings and we have invited a number of eminent criminologists to join us. Edited to add: here are the videos of the sessions.

How to jam neural networks

Deep neural networks (DNNs) have been a very active field of research for eight years now, and for the last five we’ve seen a steady stream of adversarial examples – inputs that will bamboozle a DNN so that it thinks a 30mph speed limit sign is a 60 instead, and even magic spectacles to make a DNN get the wearer’s gender wrong.

So far, these attacks have targeted the integrity or confidentiality of machine-learning systems. Can we do anything about availability?

Sponge Examples: Energy-Latency Attacks on Neural Networks shows how to find adversarial examples that cause a DNN to burn more energy, take more time, or both. They affect a wide range of DNN applications, from image recognition to natural language processing (NLP). Adversaries might use these examples for all sorts of mischief – from draining mobile phone batteries, though degrading the machine-vision systems on which self-driving cars rely, to jamming cognitive radar.

So far, our most spectacular results are against NLP systems. By feeding them confusing inputs we can slow them down over 100 times. There are already examples in the real world where people pause or stumble when asked hard questions but we now have a dependable method for generating such examples automatically and at scale. We can also neutralize the performance improvements of accelerators for computer vision tasks, and make them operate on their worst case performance.

One implication is that engineers designing real-time systems that use machine learning will have to pay more attention to worst-case behaviour; another is that when custom chips used to accelerate neural network computations use optimisations that increase the gap between worst-case and average-case outcomes, you’d better pay even more attention.