Deep System Discovery on your Linux Server With /proc

Live Processes

/proc is a virtual filesystem generated at boot and updated regularly by the kernel. With /proc you can investigate exactly what hardware your Linux kernel sees, what processes it is running, boot options, and also manipulate kernel events.

With Linux, everything is a file – and that includes devices, processes, and system information. /proc allows you to take a look at these 'files'.

/proc (short for "process filesystem") isn't 'real'; it's a virtual filesystem, generated at boot and updated regularly by the kernel. Instead of storing information on disk, when you look at something in /proc, it fetches the information from the kernel to output as a file. This is great both for system communication (utilities can operate in userspace, rather than in kernel space), and for investigating your system's innards.

/proc and processes

If you type ls /proc, you'll see a set of numbered directories. These are your processes: one directory per process ID. Look at your process list with ps -A, and pick one to investigate. (Note that unless you have root/sudo access, you're best off choosing a process that you own.) Here's a sample process from my ps output:

juliet   25175  0.0  0.0  18044  1552 pts/31   Ss   Jul14   0:00 /bin/bash

Look at the proc directory of this process with ls -l /proc/25175/, then use cat or less to look at the file contents. Some files are binary, but they usually have at least some human-readable text. Here are some of the files and directories you'll see; for more info, check the very comprehensive proc manpage. Note that often these files will be links, so ls -l is useful.

• cmdline: the command used to launch this process. Useful if you're debugging a launch problem, or to check which options were passed in.
• environ: the command environment.
• exe: the original executable file.
• fd/: filehandle links to files in use by the process.
• task: hard links to any other processes started by this process, including itself.
• cwd: the process' working directory.
• root: the root path, almost always /, unless the process is being run in a chroot jail.
• maps: memory maps to executables and library files. For each such file, it shows the memory address, permissions (r/w/x, s(hared), and p(rivate - copy on write)), offset into the file, and device and inode.
• stat, status: status information. status has better formatting.

System Discovery

So we've looked at process information, but proc also deals with system information. This is dealt with in the files and directories in /proc that have real names rather than numbers.

You may well already know cpuinfo and meminfo, which tell you about the machine's CPU (including power management and CPU details) and memory (including cache and vmalloc) respectively.

uptime and version give you, unsurprisingly, uptime and version info (this is where the unamecommand gets its information from). /proc/cmdline tells you what options were passed to the kernel at boot time, e.g.:

auto BOOT_IMAGE=Linux ro root=302 hdc=ide-scsi

You can check your LILO or GRUB config to find out which boot image this is. It was mounted read-only (as normal). root=302 means that the root partition is the '3 major, 2 minor' device. Now, take a look at the partitions file, which lists devices by major and minor number:

 
major minor  #blocks  name

   3     0   39082680 hda
   3     1    9767488 hda1
   3     2   14651280 hda2
   3     3   14161297 hda3
   3     4     498015 hda4

So the root partition is hda2 (you can also get this information from df). The final option in/proc/cmdlinetells the kernel to treat my DVD drive (at hdc) as IDE-SCSI.

The acpi/ directory contains ACPI (Advanced Configuration and Power Interface) information. The details of this directory will depend on your hardware, but here's a couple of interesting files:

• acpi/thermal_zone/ may contain information from the internal temperature sensors if that's supported.
• apm looks at APM information if your system has that enabled. apm -v gets its information here, and it's useful for checking your battery status from console.

locks displays the files currently locked by the system. Here's a few sample lines:

 
1: FLOCK  ADVISORY  WRITE 4056 03:02:588739 0 EOF 
2: FLOCK  ADVISORY  WRITE 2747 03:02:596797 0 EOF 
3: POSIX  ADVISORY  READ  2507 03:02:572375 4 4

FLOCK locks result from an flock system call; POSIX locks from a newer lockf system call.ADVISORY locks (unlike MANDATORY ones) allow other processes to read but not to lock the data; and you are also shown whether the lock is for read or write access. Column 5 shows the process ID which owns the lock, and column 6 is the file ID, written as major-device:minor-device:inode. The final columns show the start and end of the locked region: so in the first 4 lines that's the whole file (0 to end of file). This information is most likely to be useful if you're trying to retrieve deleted files by inode number. Or you can check out which file is which with find / -inum INODE_NUM (warning: may take some time!).

/proc/filesystems lists the filesystems available on your system, and marks them with nodev if they're virtual or networked. This is useful if you're trying to connect external or networked disks, to tell you if you'll have to recompile the kernel.

kcore is more memory information, but while it's useful to debuggers like gdb, it's not at all human-parseable. /proc/kmsg deals with kernel messages. net/ provides the raw info for various networking information commands, such as route. It's usually easier to get the data from the relevant commands: technically these files are human-readable but there's largely just a lot of numbers in there! Column headings are provided if you do want to look.

Finding more information

Unfortunately the proc documentation is sometimes a little lacking. Start with the man page; you may also be able to find some information in /usr/src/linux/Documentation/, or at http://www.kernel.org. Alternatively, go take a look through /proc and then Google for whatever file or directory names you find.