From: Mike Perry [mikepery@MIKEPERY.LINUXOS.ORG]
Sent: Thursday, July 15, 1999 1:36 AM
To: BUGTRAQ@SECURITYFOCUS.COM
Subject: Shared memory DoS's

Hello, sorry if it's considered poor form to cross post to both bugtraq and a
development list, but I'm too lazy to fire off two emails.

While fiddling with various IPC mechanisms and reading The Design and
Implementation of 4.4BSD (What a book!), a few things struch me as potentially
dangerous. According to the book, when you request a shared memory segment via
mmap(), the file isn't actually physically in memory until you start to
trigger page faults and cause the vnode-pager to page in the data from the
file.

Then, the following passage from shmctl(2) under Linux caught my eye:
"The user must ensure that a segment is eventually destroyed; otherwise  its  pages
that were faulted in will remain in memory or swap."

So as it turns out that it is in fact possible to create a DoS condition by
requesting a truckload of shared mem, then triggering pagefaults in the entire
shared region.

Now the end result is no different than a simple fork or malloc bomb, but it is
considerably harder to prevent on most systems.

This is mainly because:

  1. The system does not check rlimits for mmap and shmget (FreeBSD)
  2. The system never bothers to offer the ability to set the rlimits for
     virtual memory via shells, login process, or otherwise. (Linux)
  3. b. The system does not actually allocate shared memory until a page
        fault is triggered (this could be argued to be a feature - Linux, *BSD)
     a. The system does not watch to make sure you don't share more memory
        than exists. (Linux, Irix, BSD?)
  4. With System V IPC, shared memory persists even after the process is
     gone. So even though the kernel may kill the process after it exhausts
     all memory from page faults, there still is 0 memory left for the system.
     I suppose with some trickery you might be able to achieve the same results
     by shared mmap()'ing a few large files between pairs of processes. (All)

I've attached a program that will exploit these conditions using either
shmget(), mmap(), or by getting malloc to mmap() (those are in order of
effectivness).

This program should compile on any architecture. SGI Irix is not vulnerable.
Reading The Design and Implementation of 4.4BSD, it sounds as if the BSDs
should all be vulnerable. FreeBSD will mmap as much memory as you tell it.
I haven't tried page faulting the memory, as the system is not mine.
I'd be very interested to hear about OpenBSD...

Also attached is a patch to util-linux-2.9o login.c (and pathnames.h) that
provides a means under Linux (should be pretty portable to other OS's) to set
limits for the address space limit (RLIMIT_AS: the rlimit that controls how
much data you can actually map into your process). The patch is based on an old
program called lshell that set limits by wrapping your shell (I've found that
wrapping the shell in this way caused all sorts of problems with gdb, for some
reason).

sample /etc/limits file:

# Limit the user guest to 5 minutes CPU time and 8 procs, 5Mb address space
guest C5P8V5D2
# 60 min's CPU time, 30 procs, 15Mb data, 50 megs total address space, 5 megs
# stack, 15 megs of RSS.
default C60P30D15V50S5R15

At the very least, I recommend default V<size of physical memory>.
You can use lowercase letters for the next lowest order of magnitude of units.
The comment in the patch explains it in further detail.

Note even in this case, a determined user can probably just login a dozen or
so times and use SysV IPC to steal the system memory. Core wars, anyone? :)

P.S. Util-linux people: I also suspect a small memory leak due to the
strdup(hostname) provided by Ambrose C. Li.

--
Mike Perry
Proud user of both PGP 2.6.3i and GNU Privacy guard.
Considering overthrowing any governments? Count me in!
http://mikepery.linuxos.org/keys.html