ezjail - philosophy

Overview

Computer programs tend to contain programming errors. Chances to avoid them in every program that provides services to the internet are low. What is needed is a way of limiting the impact of programming errors to the system hosting the service.

Computers become faster and their resources in terms of memory, mass storage and internet connectivity can suite several users at once. This led to the development of time sharing computing, most popular in Unix. The next level of abstraction is virtual servers enabling every owner of such an entity to create users and use the power of multitasking as well, without even noticing that the machine is not completely under his control.

Some server software expects its configuration to be always in the same location in file system and makes changing this location difficult to impossible. Some server software requires certain privileges you would not want to grant it, even if you would assume it being error-free. Some users want certain privileges you would not want to grant them, even if you would assume they know what they are doing. What is needed is a way of limiting the impact of permissions you have to grant.

Historical approaches

Several approaches to achieve most of the above have been made. It has shown that the most secure way to do so is to physically separate services to different machines. Of course, this has its costs, is a lot more complex since you have to reproduce and administrate the setup over and over. And, of course, it contradicts the idea of using the grown performance as shown in the second paragraph. Luckily most of the problems were solvable, solutions being condensed in the ezjail script set.

FreeBSD has introduced the concept of jails to eliminate shortcomings seen in so called chroot-environments. chroot changes the file system root for certain programs, leaving them without any means of even expressing the need for access to resources outside their constricted world. Jails take this approach a little further, they represent virtual computers (virtual server), each bound to a single internet address having its file system root somewhere under the real computers (hosting server) file system root. Certain rulesets forbid jailed programs access to specific devices, for example hard drives or kernel memory.

In theory this concept enables you to lock in each (server-)program you run on your computer to its own virtual world limiting the impact it may (deliberately or by mistake) have on other services. In practice each program depends on a set of libraries, tools, configuration directories, system users, other installed software and so on. In short: most servers need at least a small subset of the host servers operating system (OS). Complete virtual servers, of course, need a whole copy of the host servers OS.

Problems in historical approaches

Locking in a single program that way may work out fine, as it "only" doubles the size your operating system needs on hard disc, and hard disc space is becoming cheaper as I write, the FreeBSD OS only needs around 100MB. However, copying all the data does not scale well when you need hundreds of programs being jailed or want to provide dozens of virtual computers. Further, distributing OS software updates to all this instances, which possibly even were created from different versions of the OS source code, becomes an administration nightmare.

Although hard disc space is becoming less of an issue, OSs tend to assume the space rather being occupied by few large files then many small files. So they reserve a limited number of control structures for storing files and directories, called "inodes". Now, copying the whole OS many times consumes many of those valuable inodes on disc, eventually preventing the creation of new virtual servers. Worse: since each library, each executable and each configuration file of the OS exists as several copies, the OS has no way of telling that they're the same and thus has to load each copy to memory, if any of the virtual servers or jailed programs need them.

To complete the picture of the situation I was confronted with when starting the ezjail project: the way to install a copy of the OS to create a virtual server involved (and still involves) installing a copy of the OS source code and invoking several commands to compile, install and configure it. This still left the administrator with the need to start the virtual server by hand and entering a lot of complicated configuration statements to the OSs startup scripts in order to have them run automatically when the server starts up. Jailing programs or virtual servers was rather complicated, achieving any of the goals set at the beginning of this text far away.

Developing a solution

It was very helpful that Unix' file system hierarchy reflects historic optimizations. The Filesystem Hierarchy Standard defines a set of "sharable files", mostly OS binaries, libraries, header files and resources. Those directories are meant to be shared by several computers (via Net-boot, eventually read only from CD-ROM). "unsharable files" on the other hand, contain configuration data, dynamic data like user mails, installed applications etc. While the former directories only need to exist once for all jails and can be provided in a read only fashion, the latter must be provided for each jail and, of course, be writable.

And here a path to the solution shows: most modern OSs know a way of "local mount"/"loopback mount"/"null mount" directories back into the directory hierarchy. That way it is possible to provide all sharable files in a read only fashion to the chroot environment of jailed programs. All unique data a virtual server need is in the "unshared files" which must be copied for each of them. Fortunately, these files are just 2MB in size on the FreeBSD OS. How the sharable files are being provided is subject to implementation. The ezjail project mounts one directory containing all those shareable files and provides soft links from all directories not in the unshared files part of the OS.

Besides having a central point for OS updating all instances at once, this approach can prevent standard root kits to install, since most of the binaries they usually replace are in the unshared files, yet, when someone really wants to install their own version of the OS binaries, one can do so by just removing the soft links and provide self compiled system binaries. System administrators know exactly, which OS version problems in virtual servers are related to, since all virtual servers have the same base system.

Everything left to do is to provide a neat interface, do some sanity checks, wrap it properly around the existing scripts of your OS and find a nice name. In my case: ezjail.