I’ve spent the day at the First USENIX Workshop on Offensive Technologies (WOOT07) — an interesting new workshop on attack strategies and technologies. The workshop highlights the tension between the “white” and “black” hats in security research — you can’t design systems to avoid security problems if you don’t understand what they are. USENIX‘s take on such a forum is less far down the questionable ethical spectrum than some other venues, but it certainly presented and talked about both new exploits for new vulnerabilities, and techniques for evading current protections in concrete detail.
I presented, “Exploiting Concurrency Vulnerabilities in System Call Wrappers,” a paper on the topic of compromising system call interposition-based protection systems, such as COTS virus scanners, OpenBSD and NetBSD’s Systrace, the TIS Generic Software Wrappers Toolkit (GSWTK), and CerbNG. The key insight here is that the historic assumption of “atomicity” of system calls is falacious, and that on both uniprocessor and multiprocessing systems, it is trivial to construct a race between system call wrappers and malicious user processes to bypass protections. I demonstrated sample exploit code against the Sysjail policy on Systrace, and IDwrappers on GSWTK, but the paper includes a more extensive discussion including vulnerabilities in sudo‘s Systrace monitor mode. You can read the paper and see the presentation slides here. All affected vendors received at least six months, and in some cases many years advance notice regarding these vulnerabilities.
The moral, for those unwilling to read the paper, is that system call wrappers are a bad idea, unless of course, you’re willing to rewrite the OS to be message-passing. Systems like the TrustedBSD MAC Framework on FreeBSD and Mac OS X Leopard, Linux Security Modules (LSM), Apple’s (and now also NetBSD’s) kauth(9), and other tightly integrated kernel security frameworks offer specific solutions to these concurrency problems. There’s plenty more to be done in that area.
Concurrency issues have been discussed before in computer security, especially relating to races between applications when accessing /tmp, unexpected signal interruption of socket operations, and distributed systems races, but this paper starts to explore the far more sordid area of OS kernel concurrency and security. Given that even notebook computers are multiprocessor these days, emphasizing the importance of correct synchronization and reasoning about high concurrency is critical to thinking about security correctly. As someone with strong interests in both OS parallelism and security, the parallels (no pun intended) seem obvious: in both cases, the details really matter, and it requires thinking about a proverbial Cartesian Evil Genius. Anyone who’s done serious work with concurrent systems knows that they are actively malicious, so a good alignment for the infamous malicious attacker in security research!
Some of the other presentations have included talks about Google’s software fuzzing tool Flayer based on Valgrind, attacks on deployed SIP systems including AT&T’s product, Bluetooth sniffing with BlueSniff, and quantitative analyses of OS fingerprinting techniques. USENIX members will presumably be able to read the full set of papers online immediately; for others, check back in a year or visit the personal web sites of the speakers after you look at the WOOT07 Programme.
What should individual users of OpenBSD do in light of your
findings?
Disable systrace, of course.
“Disable systrace, of course.”
That makes no sense. Just because systrace can be bypassed doesn’t mean it’s worthless. Suppose a way around your antivirus program were demonstrated. Would that mean everyone should uninstall their antivirus software?
I don’t think that’s a proper comparison: anti-virus programs can usually be updated to include more detection methods, heuristics, signatures, etc.; systrace, however, is designed in a way that is outlined to be flawed — unless the underlying OS systrace sits on top of is modified, there’s not much you can do about the issue.
Except, of course, disable systrace, and finally stop treating it like the security tool it’s not.
Although exposing the issue is the best way to come to a soluton, wouldn’t it be grand if the researcher who exposed the problem came up with a solution to the problem…
I gather:
obviously systrace as it is is not longer a security tool. However, it may be useful when trying to inspect not necesarily malicious software (and in order to try and find bugs). “Disabling” it may be overkill. Trusting it as a security tool is the mistake (same as trusting your car-repairer as a security expert) from now on.
As per the diagram in pg 14, cannot systrace “sign” (as in cryptography) the parameters and check memory (the signature) before and after execution of the syscall? Something like that… But I am not a kernel expert. And this will obviously slow things down, but security tools rarely do not.
Pedro.
It’s a matter of priority…
Dear mymyselfandI:
In the paper, I argue that the problem here is not with a specific piece of software (since identical vulnerabilities exist in a broad range of similar such systems), but rather that the system call wrapper approach is fundamentally flawed in the context of current operating system designs.
The implication of your comment is misleading–this paper is about discouraging people from using an approach that doesn’t work, and to instead to select one of several approaches that does work. The paper documents several that have merit, including moving to a true message passing model (offering argument atomicity guarantees) or using an integrated security framework (present on several OS platforms), etc. In fact, if you read my 2003 IEEE DISCEX paper on the TrustedBSD MAC Framework, you’ll see that the design of the MAC Framework is intended to specifically address these exact problems. Tal Garfinkel’s 2003 NDSS paper makes a very similar argument. The solutions are known, and have been for several years, they just need to be adopted.
Thanks,
Robert Watson
Dear Pedro:
If I understand your proposal correctly, the wrapper would sign the arguments as checked before the kernel system call code runs, and then verify the signature later to try to detect modification. Unfortunately, this mechanism itself is also raceable — the attacker can replace the memory after the wrapper signs it, then restore the original value before the wrapper verifies it, in much the same way the attacker can replace arguments after they are checked and change them again before they are audited. For this to be race-free, the actual kernel copyin() would need to perform the signature verification on the argument used in the access–i.e., we’re back to the atomicity issue.
There are many hardening techniques that can be applied, and in several cases are already applied by current systems, but the underlying concern here is that the design fails to take into account concurrency, making these work-arounds that themselves are frequently vulnerable to very similar attacks. What is required is an architecture that addresses concurrency directly.
Thanks,
Robert Watson
With regard to systrace, I’m pretty sure this attack has been known for at least 4 years?
Dear bob:
Yes — I originally reported these bugs to Niels in 2002, and have posted about this general class of vulnerabilities, including those in systrace, on public mailing lists since that time. Among other things, this lead to Niels documenting some aspects of the problem in the systrace man page (although my recent work makes it clear that the caveat regarding clone() is too narrow, as I successfully exploited these vulnerabilities using regular fork()). This general class of vulnerabilities was discussed in both my and Garfinkel’s 2003 papers (see citations and links above).
The purpose of the current paper was to provide a more thorough analysis of the vulnerability class, and concretely explore exploit strategies that might be used by attackers. As it turned out, the concrete exploit strategies were remarkably straight forward. Another objective was to provide more evidence for the corpus regarding why this approach is problematic–widespread use has persisted despite Tal’s excellent paper on the pitfalls of the system call wrapper approach.
Thanks,
Robert Watson
LSM-style implementations have their design problems as well. NOTHING is foolproof (especially with the rise of rootkits)…keep everything in perspective. Layered defense is always the best approach.
LSM Design Issues
http://www.grsecurity.net/lsm.php
http://www.rsbac.org/documentation/why_rsbac_does_not_use_lsm
“On exploiting null ptr derefs, disabling SELinux, and silently fixed Linux vulns”
http://seclists.org/dailydave/2007/q1/0224.html
it seems this class of exploit was foreseen by neils provos in his original systrace paper, but he didn’t apply his solution to the problems for some reason (probably performance).
http://www.citi.umich.edu/u/provos/papers/systrace.pdf
Dear Robert:
Thanks for answering.
Yes, you are right.
I guess a completely different approach (which would require deep kernel modification, *I guess*) would be to timestamp ALL accesses to memory and verifying the timestamps later on…….. This would need a kind of pre & post condition study (in pages 14 and 17, not just the “pre” done by sysjail, systrace in pg 17).
This would not safeguard, but would enable detecting this kind of “parameter forgery”. Probably not ALL accesses would need to be stamped, only those used by system calls (if this makes sense, which I am not certain). So to speak, given a system call
syscall(a,b,c)
Previous step: timestamp the syscall and the space used by a,b,c when the syscall is reached (and mark it as “timestamped”, forcing a new timestamp whenever it changes). This requires a “timestamp table” for syscalls and memory and would require THIS operation to be atomic.
Posterior step: verify timestamps of syscall() and memory at a,b,c. If this is later, forgery has happened.
I think this addresses your issue but is probably overkill. I agree that a better engineering is a better way.
Thanks again,
Pedro
And of course.. disable systrace.. shutdown the machine.. should an attacker know this flaw and break almost 99% of the boxes.. as happends with TRACE method in HTTP.. a well known critical vulnerability in HTTP servers..
stop the web server, shutdown the machine.. the whole world is fucked with this vulnerability!!!
SQL injection is still the most attacked vulnerability, easy , remote, and still the world is not hanging neither broke, and each day appears a new vulnerable application.
Thanks for saving us from ourselfs..
BSD-RIP
Dear Inc_Vega:
đ
Thanks,
Robert Watson
Nice presentation. Although the problem has been fairly well known since systrace was released, and its use for anything other than making sure that ports don’t install files in the wrong place by accident is discouraged, most people still think systrace can stop attacks. In this presentation your examples are perfectly clear and easy to follow. They should be added to the man page as a warning not to rely on systrace as a security mechanism.
Another problem with systrace is that that it’s difficult to know all legitimate code paths a program will take. That might be acceptable when using it on client applications such as a web browser. If you were running your web browser under systrace, and willing to put up with the annoyance of running into new code paths now and then, you would arguably be more secure than someone next to you not running systrace. The reason is because use of systrace is not very common, and most exploits don’t implement the systrace attack. The attacker might not know why his payload failed to run properly. He might not care if he’s out of quantity and not just you. But on the other hand, you might not realize that you were attacked at all. Maybe you just ran into another case of bad systrace policy, turn off automatic enforcement and hit the malicious site again.
NetBSD is very JIHBED.
Dear astraco,
your host is constantly attacking my server. I am sure this is because you just don’t care and not due to malevolent actions from your part. Please don’t shut it down and don’t ever be fucked.
The problem was described and demo provided for Windows
in 2003 http://lists.virus.org/bugtraq-0312/msg00347.html
it seems the blackhats were right all along when making fun of systrace/openbsd
None of my policies mention minherit(), so it’s not allowed and apparently not used by any of my traced application. I see that MAP_INHERIT can be used with mmap() to achieve the same. The claim “only fork() and memcpy()” seems not quite true. Considering that all machines have only single-core CPUs, is systrace really broken in this case?
Maybe I’m missing something but I don’t even see, why it hasn’t been fixed. In fact, systrace has been removed from NetBSD albeit the fix appears to be rather trivial. That is systrace needs to know how and which parameters are passed to each single syscall anyway and I don’t understand what’s so difficult about packing/unpacking the parameters at the same time. Copying costs performance of course but as long as it’s much faster than emulation – few programs spent more time with syscalls than userland code – there is not really a performance issue either. Basically, as far as I grasp this, there is absolutely nothing wrong with systrace as a concept. It’s just broken due to an implementation choice that makes it “rather faster than safe”.
I have another question: Even if systrace is much less useful (at least the way it’s currently implemented), isn’t it still useful to prevent that an (exploited) application can use socket functions or exec() – assuming it would normally use none of these so that you can safely deny permission?
I could imagine that systrace is overkill for this purpose but I’m not aware of an existing alternative. I have always considered it as awkward and a design flaw that Unix with all its filesystem security and idea of privileges, provides absolutely no tool to prevent access to the network except for low-level network functions like raw sockets.
Some firewalls can map rules to users and groups but that’s far too coarse even requiring awkward setgid/uid tricks for effective use in practise.
Dear TheWanAndOnry,
(1) “None of my policies mention minherit(), so itâs not allowed and apparently not used by any of my traced application.”
The use of minherit() is simply an example — there are many, many ways to set up shared memory between two processes in most UNIX systems, from inheritable mmap()’s, explicit shared memory, memory-mapped files, rfork()/clone(), to threading. Shared memory has been a critical part of the UNIX design for 20+ years, and, combined with parallelism/concurrency, is all that is required in order to exploit these races.
(2) “The claim âonly fork() and memcpy()â seems not quite true.”
I can’t find this claim in the blog article, paper, or presentation. Certainly, all my examples make use of other system calls in order to set up shared memory, which is a widely available and heavily used facility; I also manipulate the scheduler process priority to make exploits on UP more successful, which is not strictly necessary but is helpful.
(1) “Considering that all machines have only single-core CPUs, is systrace really broken in this case?”
I think this assumption comes with two false premise:
First, the vulnerability does affect single-core CPUs, as I illustrate in the paper. It just turns out to be even more effective on multi-core.
Second, I would disagree with “all machines have only single-core CPUs”; hyper-threading/SMT, SMP, and multi-core have become widespread, and from commodity server-class boxes shipped 4+ years ago, to the 2-core notebook I got 2+ years ago, to multi-core embedded hardware that’s widespread in networking and telecommunications. The brave new world of parallelism is here in force, and here to stay.
(2) “.. the fix appears rather trivial”
The fix requires significantly rewriting the kernel memory copying code, or, more cleanly, the system call code, and despite suggestions that it would shortly be fixed for the past six+ years since I reported the vulnerability, it still hadn’t been fixed in any of the systems I looked at. Keep in mind also that copying only addresses one half of the problem. Per my comments in the paper about message passing: by moving to true message passing and offering atomic argument access, syntactic races are addressed, but not semantic races, which are inherent to software composition using wrappers.
(3) “Isn’t it still useful…?”
I’ve never said that systrace isn’t useful, just that it is highly vulnerable to a certain class of attacks if used for sandboxing. If your use of systrace does not involve paths, socket addresses, etc, in its policy controls, or if the use isn’t security related, then it presumably not “vulnerable”, although it might be subject to functional race conditions that might well be considered bugs. I have never advocated for the removal of systrace from NetBSD, only that it be properly documented as either a system not appropriate for general-purpose sandboxing as it has been advertised, or that the constraints of sandboxing be properly documented.
thanks for your interest and comments
Robert Watson
Thanks for your answer. My main point is probably, let me have my broken systrace as long as you provide no alternative. “You” is not you but rather NetBSD.
Shared memory: I believe many if not most simpler applications don’t use it or at least can live without it. Despite being simple, they are frequently exploitable and have to handle untrusted input. Consider tools like file(1) or pretty much any kind of “parse and transform” application.
âThe claim âonly fork() and memcpy()â: I might have picked that line from /. or another article. Sorry for attributing it to you.
âConsidering that all machines have only single-core CPUs,”: What I meant was “What if I only use machines with single-core CPUs?” In fact I do and a lot of infrastructure where CPU power is not very important does too.
â.. the fix appears rather trivialâ: That’s certainly exaggerated but I thought of people who are already familiar with the kernel code. I believe the reason nothing happened is mainly that systrace is only maintained by a single person namely Niels Provos and somehow systrace didn’t take-off at all, maybe because it’s complex to configure. Also having to use wrappers is somewhat awkward. People probably just want to apply some restrictions to (application,user) in a central database and the kernel should handle the rest similar to veriexec. There are certainly many usability issue with systrace but then again, if it was reliable, it would still perfect for quite a lot of applications. The question here is also, what’s the alternative? A VM for each application or process would be rather heavy on the CPU and probably just as difficult to configure.
However, I have some idea how the current systrace implementation can still be useful and reliable – not in all – but many real-life relevant cases. An obvious problem are syscalls that use pathnames or socket structures as parameters. In the systrace profile you would simply disallow all of them. The systrace executable would replace the relevant function calls with wrapper functions as in the good ol’ LD_PRELOAD hack. So “open()” would be linked to “systrace_open()” and so on. The actual parameters are send over a unix domain socket to the systrace wrapper (or daemon). The receiving process can identify euid/egid/uid/gid/pid and the pathname of the executable that created the process, so systrace can locate the policy that will rule the request. If it’s granted, systrace calls open() with supplied parameters and passes the file descriptor back over the unix domain socket (an IPC technique known as “file-descriptor passing”). The result code and errno are passed in the same response. The file descriptor number (result code) is of course pointless in this case but matters for stat() and such.
I think this would work for fine-grained access control to the filesystem and sockets.
Hello
I read your paper. It is interesting. I need to know some more about concurrency attack but every where I go I see this paper or link to this paper. do you know any other paper or book and would you please send me link to download them to know more about this subject.
Thank you