Next month I will be presenting my paper “Hot or Not: Revealing Hidden Services by their Clock Skew” at the 13th ACM Conference on Computer and Communications Security (CCS) held in Alexandria, Virginia.
It is well known that quartz crystals, as used for controlling system clocks of computers, change speed when their temperature is altered. The paper shows how to use this effect to attack anonymity systems. One such attack is to observe timestamps from a PC connected to the Internet and watch how the frequency of the system clock changes.
Absolute clock skew has been previously used to tell whether two apparently different machines are in fact running on the same hardware. My paper adds that because the skew depends on temperature, in principle, a PC can be located by finding out when the day starts and how long it is, or just observing that the pattern is the same as a computer in a known location.
However, the paper is centered around hidden services. This is a feature of Tor which allows servers to be run without giving away the identity of the operator. These can be attacked by repeatedly connecting to the hidden service, causing its CPU load, hence temperature, to increase and so change the clockskew. Then the attacker requests timestamps from all candidate servers and finds the one demonstrating the expected clockskew pattern. I tested this with a private Tor network and it works surprisingly well.
In the graph below, the temperature (orange circles) is modulated by either exercising the hidden service or not. This in turn alters the measured clock skew (blue triangles). The induced load pattern is clear in the clock skew and an attacker could use this to de-anonymise a hidden service. More details can be found in the paper (PDF 1.5M).
I happened upon this effect in a lucky accident, while trying to improve upon the results of the paper “Remote physical device fingerprinting“. A previous paper of mine, “Embedding Covert Channels into TCP/IP” showed how to extract high-precision timestamps from the Linux TCP initial sequence number generator. When I tested this hypothesis it did indeed improve the accuracy of clock skew measurement, to the extent that I noticed an unusual peak at about the time cron caused the hard disk on my test machine to spin-up. Eventually I realised the potential for this effect and ran the necessary further experiments to write the paper.
impressive
Who’d have thunk? You guys certainly are creative! 🙂
So, if the relative use of a CPU can give away the identity of a node, would a plausible countermeasure be to keep the CPU pegged at 100%? Would something as simple as running Folding@Home or SETI@Home on the machine be enough to thwart this?
Just wondering.
This isn’t really something to worry about, right? The attacker has to have physical access to the server. If he does, you’ve got bigger problems than being de-anonymized already.
@James
No, the change in temperature is causing by increasing CPU load, which can be done simply by downloading a file from the hidden service. The clock skew is measured by requesting TCP timestamps, which is a feature enabled by all modern operating systems and seldom blocked by firewalls.
I don’t think there is any need for panic. Primarily the paper is designed to feed into the future design of Tor rather than suggest any short term fixes. There are already known attacks on Tor which will probably work better than this, but the proposed defences to these will not fix the problem I discuss in the paper.
Also, I point out that for clarity the results in the paper are mainly from a private Tor network and running it in reality will be more messy. However, as the performace of the Tor network improves, the attack will be more effective, so is worth bearing in mind for the future.
Oh, thanks for clearing that up. 🙂
As a (very minor) Tor hacker, I’d like to say that this is great work! Well done.
Have you tried it against other services? Like trying to see if you can tell the difference between a real and non-existent user via openssh?
And any source code?
(I apoligise that I’ve only skimmed the paper so far)
Thanks
AGL
@Doug
This wasn’t something I tested, but I expect it will make the attack much harder and possibly infeasible. There might still be some effects apparent. For example, perhaps the hard-disk will have an effect and that can be spun up.
Also, I expect that SETI@Home will pause when Tor is busy. If SETI@Home heats up the CPU more than Tor, then there still will be an effect. In my tests I did note significant difference in the CPU temperature between running Tor and CPUBurn, despite them both keeping the CPU at 100%.
@Adam
No, but that is a very good suggestion. I do remember that OpenSSH gave away whether user did not exist through timing. If they fix the response time, then the temperature attack could work. Hopefully the temperature difference between multiple
crypt()invocations will be different fromsleep().@Adam
Currently it is a bit of a mess, slow, and sparsely documented. I do plan to put it on my site at some point when it is in an acceptable form. Right now I really should be writing my thesis 🙂
This is a very interesting effect!
If the entire sequence relies on timing offsets created by server load then couldn’t the server deliberately include random timing offsets into the displayed timestamp.
@Jason
Random timing offsets would not be a very effective defence, because over time they would average out. Introducing a random skew would help against the fingerprinting attack, but to be complete it would need to be implemented at the timer-interrupt level which is not software controlled.
Neither would be a good defence against modulating the clock skew, since without a good external reference clock, the computer couldn’t tell what the true clock is to base the jitter around. Changes will still be apparent, after removing some noise.
@Adam
I tried this by repeatedly performing keyboard interactive, password and public key authentications, but didn’t see any appreciable difference in CPU usage between users that exist and those that do not. This appears to be because OpenSSH generates a fake user, when the requested one does not exist; for details see
fakepw()in auth.c.The attack might still work in special cases, e.g. where PAM calls a particularly processor intensive module, or I might have missed an avenue in OpenSSH. There could be other packages which are vulnerable too.
This is not a new attack (just the implementation) I was first aware of this when trying to design a random number generator with clock skew (it is not as easy as various reasurch papers would have you belive).
When the paper about the original TCP timestap was posted I mentioned a solution on Bruce Schneire’s blog (plus a couple of other things such as using the change in temprature therfore skew to tell how hevily the machine was being used or side channel),
http://www.schneier.com/blog/archives/2005/03/remote_physical.html
The hardware solution briefly, which is quite simple and fairly cheep to implememnt,
1, identify the CPU (not the backup clock) crystal.
2, Check for the two small value capacitors (around 33pF) that go to ground from the crystal terminals (if they are fitted they often are not).
3, Find which one is on the input (not the driver) side of the crystal (you might need a scope or a data sheet).
4, Replace this with a cap twice the value with a varicap in series (in the ground leg) connect a high value resistor to the junction of the varicap and cap.
5, then supply a suitable very very low frequency signal into the resistor.
6, Make the very slow speed signal random (crypto random would be good) via a D-A or other system in a PIC micro controler etc.
The effect of this is to marginally change the CPU clock frequency which will destroy any timing correlations (if you use the right signal to the varicap).
As an extra thought for those using SSH or other key_press-network program have a look at Matt Blaze’s site (www.crypto.com) and have a look at the JitterBugs paper it shows how the timing of an input signal (ie keypressess) can be modulated to make a side channel that can be reliably picked up on the network. This is without any kind of modification to the computer or it’s software…
Now
@Clive
I am not aware of anyone trying this before. Could you be thinking of “Remote physical device fingerprinting” by Kohno et al? They showed how to fingerprint hosts by absolute clock skew, but this is not what my paper is about.
I show how changes in clock skew are caused by temperature and this can be induced by modulating CPU load. Clock skew can be measured remotely and CPU load can be altered remotely allowing an attacker to tag a machine.
Altering the clock speed randomnly as you suggest would not be effective against the tagging attack, because it would just introduce noise. This would simply be averaged out over time. A better approach would be temperature componsated or oven controlled crystals, provided they could react fast enough.
Very creative, but not particularly useful.
Firstly, this relies on having a server or a list of servers that might be hosting the hidden service – uncertain (see below) verification of if a server is hosting a specific hidden service, but if the hidden service is done properly, you have no idea who could be hosting it 😉
Secondly, you have to (D)DoS the target server to get results – a good firewall or some proper throttling would make it nearly useless, and it is hardly subtle. Also, I imagine multiple CPUs would screw with this.
And, of course, any other system load would contribute – if anything intensive is running, the results would be very unpredictable.
The hidden service operator could just ensure that no one has any reason to suspect that their server is hosting the service, or use a properly configured firewall to prevent attacks like this
@Steven J. Murdoch
Steve,
Below (if it’s emotted correctly) is the last couple of paragraphs of my posting to the Schneier blog a year and a half ago,
Better still use similar technology to modern TCXO’s and use a PIC microcontroler to generate the voltage, also monitor various status lines (like reset etc) and change the voltage each time you reboot the machine, thus giving a different fingerprint each time you turn the laptop etc on.
This “attack” is a form of tempest attack, and the old addage about the information “energy and bandwidth” apply. Interestingly looking at their paper they have missed a couple of things that might provide more information about the computer. Basically the resonant frequency of an Xtal oscilator is decided by the elctrical and physical charecteristics of the circuit. These means that the frequency changes with the applied voltage, temprature, mechanical vibration. So it there is sufficient bandwidth in the time detection method it might well be possible to tell things about the environment the laptop is in and how much it is being used (heavy calculation take the temprature up and drops the powersupply voltage slightly).
Posted by: Clive Robinson at March 10, 2005 05:10 AM
In essence as I understand it from your paper (I willl need a bit more time to read it and go through all the stuff you mention as I printed in BW and the graphs need to be in colour) and from what you have said you are looking at the delta function of the synthetic quatz resonance frequency to temprature and it’s visable effects on the network. The temprature variation being due to the extra load put on the computer by activity of the components.
In the last sentence of the last paragraph of my post I say,
So it there is sufficient bandwidth in the time detection method it might well be possible to tell things about the environment the laptop is in and how much it is being used (heavy calculation take the temprature up and drops the powersupply voltage slightly).
There are a number of issues with the bandwidth, the first and most importantly is that the bandwidth on the transfer of energy to the ambiant suroundings from the CPU or other high energy device is based around the transfer time on the heat sinking mechanism.
A practical example of this is take a bare metal coat hanger and a lighted candle. Hold one corner of the hanger and put the other in the candle flame. Time how long it is before you feel a real temprature difference, it’s usually in the order of seconds.
If you know consider CPU die to CPU package to heatsink to air to Xtal metal case to xtal inside as your thermal path it is fairly obvious that there is going to be a considerable time delay (also by even moderate mechanical changes the thermal path can be broken) .
The bandwidth of this as an information carrying channel is inversly proportional to the time delay.
Therefor any activity that an attacker is going to perform must be sufficiently long for it to fall well within the bandwith.
With regard to TXCO’s long gone are the days when they used heating elerments and thermisters in closed loop feadback (I still have one at home which I show people as a curiosity along with a second world war spy set and it’s Xtals, and old TX valves and 807 based RX antena amp of the sort used in the X stations).
Basicaly the modern version is a small single chip micro like a PIC for example that is capable of producing a voltage to a varicap on the XTAL that is the inverse of the temprature changes (hence my comment about PICs). The whole lot including the AT cut crystal is usually mounted in a small “relay can” with four (or more) contacts.
With regard to supplying a changing signal to a varicap diode that is of very low frequency, providing the frequency changes it makes swamp that of the effects of temprature and are also of an appropriate frequency then the effect it has is to swamp the changes due to temprature from computer load variation (ie bury it in the noise).
Yes you could average the “load signal” out but what method would you use, avaraging is effectivly a low pass filtering function with the cuttoff frequency set by the number of samples and their repetition rate. From the attackers point of view averaging just lowers the available signal bandwidth of the side channel even further.
You could easily make the cutoff point be equivalent to 1/week with fairly minimal effort see stuff on Spread Spectrum to do with Direct Sequence and frequency hopping (also quite a few motherboards these days include spread spectrum clock modulation to help meet EMC requirments).
As I noted at the begining of the last paragraph in my posting to the Schneier blog,
This “attack” is a form of tempest attack, and the old addage about the information “energy and bandwidth” apply.
You can see I am accutly aware of the limitations of the side channel attacks (allmost all practical tempest attacks are side channel attacks).
In fact if you can get sufficient bandwidth, it might be more practical to use the mechanical effects on the XTAL resonant frequency (known as microphonics in the trade) as a low frequency microphone or sizmic sensor 😉
The reason I am acutly aware of this stuff is that I have spent a considerable time one way or another looking into Comms Security.
And more pertanently to the above (which falls into the general class of timing attacks) back in 2000 I was at an EU funded course in InfoSec (IPICS 2K) where Black / Mix nets where discussed and I pointed out the history of Tempest / side channel timing attacks and just how difficult it is to negate their effects to an acceptable level. This gave rise to actually thinking and testing what would be needed to perform a Tempest style attack on these networks (and how to solve many of them 😉
By the way the story usually told about the “first Tempest attack” is quite relevent,
Shortly after WW II Britain had a secure link from Washington to Britain that used an electronic form of the One Time Pad. The Xor function was implemented using a mechanical (post office 600) relay. Unfortunatly it had a bias in it’s hold and release timing. Which enabled you to strip of the encryption, simply by examining the timing on an osciliscope.
Supposedly this is also one of the reasons a Canadian Prof spent so long developing the replacment known as the Rockex which was used by the F&CO for many years.
Therefore it would not be unfair to make the same comment (about Tempest) once made by an NSA employee when talking about DES and differential crypto attacks 😉
Hello! I like this attack, yesterday I implemented clock skew detection in hping3 and I’ll release it in some day. With hping the attack is active, requires sending a packet for second, for 4/5 minutes, but it is very easy to use even for script kiddies 😉
Ciao,
Salvatore
@John
I deal with most of your points in the linked paper.
I would disagree; in fact I used this technique in anger last week with good results. This will hopefully be described in a blog post of its own, later.
“Many hidden servers are also publicly advertised Tor nodes, in order to mask hidden server traffic with other Tor traffic, so this scenario is plausible.”
Also, this attack is orthogonal to other analysis techniques. If one of these produces a list of candidates, the attack presented can narrow down suspects.
This is not necessary; an attacker can be as subtle as it likes, it will just take longer. Over time even slight signals will become apparent. A firewall will not help, since the traffic to the hidden service is encrypted so the firewall will not see the source.
I tried this and there was no problem. Multiple CPU machines still have only one clock crystal.
This was not my experience with “Low-cost Traffic Analysis of Tor”. Noise like this disappears rapidly when you average the results over time.
The first point is unrealistic because the operator must have some motive to set up the hidden service in the first place. The second is a lot more difficult than it sounds. Firstly the operator, would have to block all incoming traffic, which precludes running a Tor node so loses the plausible deniability. Secondly this works for outgoing connections, so web-bugs and Javascript could work as well. An attacker could even snoop in outgoing traffic not destined to him. If all the candidates traffic could be monitored, other attacks will work better, but suppose the attacker could sit at a web proxy or DNS server.
@Steven,
I appriciate that at the moment the attack works.
However I suspect that now it is out in the open as an attack system operators will start to look at the traffic on their machine via the logs etc (and vendors will code the appropriate filters into their IDS/P systems etc if sufficient customers ask for it).
As the attack requires the target machine to be very heavily loaded for a couple of hours (or more) then lightly loaded for an equivalent time with this cycle repeated several times, this behaviour is very likley to give a clear signiture in the system logs (along with several other related indicators if the atack is not skillfuly put together).
As you pointed out in your artical the attacker might have several hundred or more potential targets to attack before localising the network address of the machine. This makes it a clasical time/resource trade off. It is therefore quite likley the attacker will give away their precence to network operators and the TOR ops long before they have succeded.
As I indicated previously there is a very workable solution to the problem, briefly and without getting bogged down in details (and counter attackes),
Firstly you implement delaying tactics on the host to make the attackers time to probable target identification sufficiently long say a week or more. This then moves the advantage over to the system op as they can probably average the system logs more quickly to identify an attack than the attacker can average out the delaying tactics.
Secondly you move the service around various host machines in a more frequent time than target identification takes, again this moves the advantage over to the alert sys ops as they can look for changes in averages across all the machines used.
However the current TOR system does not support the second option, the question is how long before it does (assuming the good will of alternative host system owners).
All in all you have done a very nice job of taking an idea and turning it into a practical attack and then publishing the result, which others have not done. So all credit to you for your work.
Now you (or others) need to do the counter measures as in the old ECM ECCM game. In my experiance offering a possible solution to a problem you have discovered getts a more attentive audiance 😉
Clive, a fascinating dialogue going here which I have enjoyed reading and I’m going to follow in greater depth.
Given the wealth of information and suggestions outlined in your Schneier posts and in the thread here, have you considered putting together a paper of your own? Maybe the time constraints are prohibitive, but it seems to me all these ideas would benefit from coming together in a more formal whitepaper layout, even if the mechanics of a formal publication and review process are not necessarily followed.
Just a thought; and on the possible overlap between such techniques and “tempest” emissions attacks, sadly this is another area where the academic literature is thinner on the ground than it should be too, imho!
Mike
oscillator temperature and age stability is usually of functional concern in oscilloscopes and other instruments. We can probably draw on their solutions to fix this. The difference between expensive instruments and their less expensive brethren is in part the use of oven controlled oscillators. These control the temperature of the oscillator and provide significant improvements in measurement. Some instruments have double ovens…
Also, I imagine if you have some reference data over time on a machine you could detect things like hardware replacement and other potentially interesting events by comparing the age drift of the oscillator.
Mike,
Sorrey for not getting back to you put I ended up in hosiptal due to complications with an earlier visit 🙁
I will send you an EMail to discuss further your proposel.
Clive
With regard to two or more paths to a multihomed host (ie with two or more IP addresses / URLs not neciserilly network cards). Time/frequency skew actually provides a very good way to identify the host. Effectivly you are enumerating the hosts interfaces.
Why is this possible put simply the time stamps etc are (in most OS’s) derived from the same clock source be it hardware or software. This is due simply to trying to minimise the use of resources by the OS.
There are certain security areas where this is of great concern over and above anoymous hosting of services.
First off think of Honey Nets, these are designed to trap the attempts of skilled crackers trying to break in, so that what they do can be anyalised. More often than not these Honey Nets are not made up of a host per IP address, they often use one set of hardware to look like multiple hosts. A Honey Net is only of use if it cannot be detected as such.
Now if you assume you are an intelegent cracker who does not want your latest tricks revealed. Performing what looks like a bad network scan several times (ie script kiddy type behaviour) and harvesting the time stamps and doing comparisons will show that the time skew is possibly the same. Opps play safe you have probably found a Honey Net, time to move on and try somewhere else.
The Honey Net Operator just sees it as one more Skript Kiddy and does not realise that his network has been found by Timestamp Enumeration (it might not even make it past his IDS detectors threshold). The Cracker meanwhile may let their (very few) trusted colleages know their suspicions, meaning that the Crackers the Honey Net Operators are realy after will not knock on their door so no data to analyse (which is not good for the rest of us).
Why do I say “may let” well the best of the crackers these days do not do it for fun they do it for comercial advantage be it either for the write up and subsiquent consulting advantage. Or worse from the ordinary users prespective for snooping Maleware. Eitherway the last thing the cracker needs is for their money making crack to get publisized / rumbeled before they have made good value from it.
As a secondary atack again of value, it is not unknown for Hosting organisations to run more than one customer companies services on a single host but with different IP addresses or URLs.
Now on the assumption you are a cracker for gain, youwill want to get at details on a particular companies website as the data held there is of value to you or your employer.
The target website it’s self may not present any available oportunities to break in. But another companies site on the same host might well do so (actually quite likley for small and startup companies using hosting organisations, why spend money on making a secure site if it a couple of static pages and an emailing script).
So as the cracker you get a time/frequency skew for the site of interest, then scan all the other sites within the hosting organisations domain looking for a match. Even if the Host organisation use the same IP address and different URLs this will be productive as sometimes they move sites from host to host (there are various ways a Cracker can find this information but it’s not relevant to the argument).
As the cracker you can then investigate the other websites on the host of interest, the chances are atleast one will have an exploitable weakness. At this point you are in to the host with the priveladges of the web software. For a poorly implemented host this may be all you need to get to the data you could not otherwise get. If not the chances are that you can escalate your priveladges up to a point where you can. Either way you get what you are looking for no matter how securly the target company produced it’s web site.
Also of note the only way to trully hide a site from time stamp enumeration is to make the timestamps of no use for the attack. That is you lock them to a national standard so that there is no measurable time skew to use as a fingerprint.
This by the way does not mean simply installing NTP software, some of it is poorly writen and the time correction “error signal” will be visable almost like a triangular wave on the time stamps. Again it’s visable and identical on all the IP addresses and URLs on the host due to the use of a common clock source for the network ticks.
To understand why the correction signal is visable imagine
(for arguments sake) the rising slope of the waveform is the time skew of the CPU clock it will carry on rising unless corrected. When the NTP software detects a sufficiently large time error it makes the correction this would be the downward slope, the pitch of the slope is dependent on how hard the NTP software makes the correction. Also the point at which the correction is made is usually not at the minimum detectable time difference but at some greater point as this minimises the NTP softwares use of the hosts resources.
If you look at a NTP software often the writers do make attempts to make the correction slope gradual not a step, this is mainly because a suden step effects other software etc detrimentaly.
Some NTP software is predictive, in that it tryies to apply small corrections prior to them being detectably necesary again this is to minimise the effect on other software, however it usually uses a long time window as the assumption is the time skew is effectivly constant (effectivly an adjustable low pass filter on the previous correction data).
However NTP software is not usually dynamicaly predictive therefore changes in system load etc may take the system quickly to a point where a visable error signal would appear on the interfaces. At which point somebody doing a timestamp enumeration sees a change that they can start corelating against.