Linux 101: Best practice for security integrity auditing and recovery

by Chad Perrin | Jan 25, 2006 7:25:00 PM

Tags: OPEN SOURCE, Chad Perrin, auditing, security, Linux

Takeaway: Two critical security considerations that are closely related to one another are ignored all too often: integrity auditing and recovery. This document is an overview of good security integrity auditing and recovery practices using a Linux operating system.

Computer and network security involves several important elements. A system administrator must choose secure software, install and configure security systems like firewalls, and keep up with vulnerability patching, among other tasks. Two critical security considerations that are closely related to one another are ignored all too often: integrity auditing and recovery.

Too often, a sysadmin will get all the basic security measures in place, set up a well-secured system, and figure his job is done unless something goes horribly and obviously wrong. It is important, though, to regularly check the systems in your area of responsibility to make sure they haven't been compromised, and to know what to do if they have. Sometimes, a security cracker can penetrate security and gain complete control of your system without you ever knowing about it until it is far too late to prevent the damage that will be done, and sometimes a security breach might be cursorily "cleaned up" so that no visible signs of cracked security remain but, hidden from view, trojans, rootkits, and other deceptive security issues might still exist.

Don't confuse security integrity auditing with security profile auditing. Profile auditing involves testing the intrusion-proofing of your system, checking for avenues of exploiting the system, and testing the ease of achieving malicious goals when cracking the system. Integrity auditing is a purely defensive activity, involving checking to ensure that someone has not already cracked security. Integrity auditing is not the only defensive activity you should be using, by any means: rather, it should be coupled with more well-known security measures such as use of firewalls, software security patching, and access logging.

This is an overview of good security integrity auditing and recovery practices using a Linux operating system.

Linux 101

Check out other articles and downloads in the Linux 101 series. In addition, you can add your own special Linux know-how in the TechRepublic Linux 101 Wiki page.

Integrity auditing

Filesystem

Usually, one of the first things the highly security-conscious Linux administrator does with a new system is set up filesystem integrity auditing. If at all possible, this should be done before the system is connected to the Internet, or even the local network. The most common tool for this, and the industry standard, is Tripwire. The original Tripwire software is known commercially as "Tripwire Change Auditing", and is used to automate and manage the auditing of the filesystem on secured computers to make sure no unauthorized changes are being made.

Tripwire, Inc. contributed code to the open source development community in 2000, however, and there is now an Open Source Tripwire project that provides the same kind of security integrity auditing functionality. Most, if not all, Linux distributions include the Open Source Tripwire software, also known simply as "tripwire", in default installation of the distribution or in additional software that is available. This all conspires to make it very easy to implement filesystem integrity auditing protection.

There are other ways to get the same sort of security integrity auditing for filesystems. While tripwire provides a lot of powerful and flexible functionality for the security conscious system administrator, its easiest and most basic functionality can be duplicated in other ways. One such method in particular involves a clever use of the rsync utility, a tool designed and most commonly used for doing rapid and simple filesystem backups. The rsync utility is, in fact, little more than a means of copying files from point A to point B, but it keeps track of what has been copied so that updating a remote backup when changes have been made is much faster than a complete copy and overwrite would be.

It is this means of keeping track of what has been copied, and thus what changes have been made, that makes it possible to use rsync to do filesystem integrity auditing. Using rsync for this is usually not the ideal solution for a number of reasons, including the fact that it requires a lot of disk space, but if you're using rsync for system backups anyway this operation can be enhanced to provide filesystem integrity auditing instead of, or even in addition to, integrity auditing with tripwire.

In general, filesystem integrity auditing consists of creating a "snapshot" of the filesystem you wish to audit, then later checking it for unauthorized changes. Doing so without a proper tool like tripwire or, at the very least, an efficient copying and difference comparison tool like rsync, can be a very cumbersome task that is prone to occasional errors, but it is possible using utilities like diff, idfile, ls, and md5sum in a shell script to ease the process somewhat.

If you require more comprehensive checking than rsync provides and more efficient and manageable auditing than can be provided with such a shell script, but cannot use tripwire for some reason, consider looking into tools such as Samhain or AIDE. Like tripwire, many Linux distributions make these two tools available in their software management systems.

Rootkit

Another important form of integrity auditing is rootkit checking. Of all the integrity auditing a sysadmin can perform, this is probably the most commonly known, and the easiest to manage. There are commonly used tools for this form of integrity auditing. The most widely used of these is the chkrootkit utility. As with Tripwire, it is often included in a default Linux install, and if it is not installed by default, it is typically very easy to add it to your system configuration by way of your distribution's software management system.

Whereas filesystem auditing just checks for any changes to the filesystem at all, and attempts to make it easy to determine which changes are authorized and which are not, rootkit checking with a tool like chkrootkit checks for rootkit signatures in a manner similar to the way antivirus software works, scanning your filesystem data for bits that are arranged in an order suspiciously like known rootkits. This sort of protection is far more effective than can be achieved with traditional antivirus software. This is because rootkits are necessarily much less common, and because they are used only very deliberately and only once the system's security has already been cracked by other means, making it easier to implement a comprehensive signatures-based auditing solution.

The second most common, and free, rootkit checking software available is called Rootkit Hunter, or rkhunter. This utility largely overlaps the functionality and ease of use of chkrootkit, but there are some differences, and there is no particular reason to avoid using both. The rkhunter tool provides more "user friendly" output than chkrootkit in general, which makes it attractive when one chooses between the two, but for best security it simply makes sense to use both. Either tool can provide the occasional false positive, often in the form of identifying an optical drive as a potentially rootkit infected file, but for the most part they are both very effective and accurate tools within the realm of their intended purpose.

Antivirus

As I have already hinted, antivirus software is in general another form of security integrity auditing. This is not as important a tool for Linux systems as for Windows systems for many reasons, including the fact that it is monumentally difficult to get a virus to actually self-replicate, spread itself, and affect a Linux system. When running a Linux-based mail server or other system that regularly provides files to Windows systems, however, antivirus software can become very important for protecting those Windows systems from infections the Linux system wouldn't even notice.

If you think your Linux system has any reasonable chance of becoming an "immune carrier" for Windows-targeted viruses, you should look into running antivirus software under Linux. The most common and possibly the most effective AV software for Linux is ClamAV. Most distributions of Linux provide ClamAV by way of their software management systems, but it is usually not installed by default because of the relative unimportance in the Linux world of antivirus software.

Other antivirus software is available for Linux as well, including the less permissively licensed F-Prot Antivirus. The F-Prot license does not explicitly allow free distribution, but is generally free for use by way of the vendor's Web site. Some Linux distributions might provide easy installation of F-Prot through their software management systems or (as with Debian) an installation script that can be used to download and install the software from the vendor's FTP site rather than providing it directly.

Secure medium

All integrity auditing security measures are best performed with tools that are stored on, and run from, a known good and strictly secure medium. This usually means copying all the necessary program binaries onto a non-rewritable storage medium such as CD-R. The tools can then be run from the optical media on which they are stored, ensuring that none of the tools you are using have themselves been compromised by rootkits or other problems that might render them blind to the very problems they are designed to detect.

Running chkrootkit or an rsync filesystem integrity auditing operation using utilities stored on the machine that is being checked is better than nothing, but a savvy security cracker who has compromised your system may well have compromised the tools used to audit your system as well so that you will never find evidence of the intrusion. This is far less important for antivirus software, of course, as such software is easily checked by tools like tripwire and chkrootkit to ensure it is still good. The threats against which AV software protect are not typically the sort of thing that can cause a significant system compromise on computers running a Linux operating system.

Security recovery

You've been doing integrity checking faithfully. You discover your system has been compromised. What do you do?

Don't panic

That's the first thing to keep in mind. Lovingly caress your copy of the Hitchhiker's Guide to the Galaxy if you must, but don't panic. You need to be able to think clearly about what you're doing.

Disconnect the network cable

The moment your system has been compromised, you can't trust it on the network. It may be acting as a jumping-off point for compromising more systems, it might be getting searched through for sensitive information, and it may even be serving as part of a file-trading network for kiddie porn. Any attempts to recover while it's still connected to the network may be overcome by the person that cracked system security in the first place, by automatic reinstallers for various bits of malware, and so on. Unplug the network cable immediately, even if the only network cable you have is a phone line plugged into your modem.

Document everything

While you are engaged in the process of recovering your system, you should be making notes about everything you're doing, and everything you find.

Save your data

This is best accomplished by removing hard drives with important data on them from the system and setting them aside. Using backup software may not be good enough. After all, your backup software may have been compromised by the security cracker who gained entry to your system. If you must use backup software to make copies, do so from unwritable media, like a CD-R, so that the backup software won't be compromised while you're trying to back up your data.

Analyze the intrusion

You need to determine how the security cracker got in, what vulnerabilities were exploited, and if possible who did it. Once you have this information, you can use it to better secure your system the second time around and report the illicit activity to the appropriate authorities.

To analyze the intrusion, you might use a tool such as TCT (The Coroner's Tookit). TCT is a collection of utilities for "post-mortem" system analysis in case of compromise, and can even sometimes help recover lost files. It's a good idea to be somewhat familiar with TCT, and have a copy on known good media such as a CD-R, before you ever have to use it. If you decide this incident warrants reporting to security authorities, make sure your gathered data is organized and complete to the best of your ability.

You should inform system or network administrators, both your own and those on other networks that might have been affected by your intrusion situation or attacked in the same manner. You should also contact network administrators whose systems may have been used to launch the attack without the administrators' knowledge. You can, if you deem it necessary or desirable, report the incident to CERT, the Computer Emergency Response Team, which serves as a governmental organization dedicated to monitoring and responding to security cracking activities.

When reporting to CERT or another security incident response team, you should at minimum have information related to the security cracking activities you noticed, the means by which you noticed, the host systems and networks involved in the attack (including your own and other victims as well as the apparent attacker(s)), and any other data you have gathered from log files, network activity analysis, and use of forensic tools like TCT.

Install an OS from scratch

Use a "fresh", cleanly formatted hard drive for this. You should not simply try to recover the old operating system, because once your system has been compromised you cannot trust anything on your computer. Applications, security software, the OS kernel, data and configuration files, logging software, basic system utilities, and even the boot sector may have been modified maliciously. Worse yet, there's no way to be sure what has been modified. Even if you find specific instances of modified files, and can identify exactly how they've been altered, that does not mean that something else hasn't been modified, and all trace of that modification may have been covered up. Make sure you patch and configure your new OS install using what you've learned from analyzing the intrusion.

Restore data carefully

With a Linux system, infected or compromised data files are far less likely to be a later problem than on a system that auto-executes anything it recognizes as executable, as MS Windows often does. You should still be painstakingly careful while restoring data, however, and should prefer older backups over newer backups to minimize the likelihood that your backed up data was compromised by the attack. This is one reason that plain text files are generally preferable over binary file formats, as having a file entirely made up of ASCII text is far easier to be sure isn't infected by malicious code. When in doubt, err on the side of throwing out old files and recreating such data the old-fashioned way.

Double-check

After all of this is done, when you are sure you have dealt with everything else that needs doing, only then you can reconnect to the network with reasonable safety. Double-check before you do so to ensure that there is not a sustained attempt still in progress to crack and control a system with your IP address. Monitor the activity of your recovered system and of servers with which it interacts very closely for a while to ensure that attacks do not resume.

3 comment(s)