Something else you might not have noticed is that the Fedora kernel team has quietly slipped Linux 3.0 into Fedora 15's update channels in disguise. Click that link, scroll down, and you'll see "Rebase to 3.0. Version reports as 2.6.40 for compatibility with older userspace." Although I'm not a fan of calling something what it isn't (2.6.40 doesn't exist on kernel.org), I can understand some of the reasoning behind the choice.

This change makes the Xen installation on Fedora 15 pretty trivial. To get started, update your kernel to the latest if you're not already on Fedora's 2.6.40 kernels:

yum -y upgrade kernel

We need three more packages (quite a few dependencies will roll in with them):

yum -y install xen libvirt python-virtinst

The xen package reels in the hypervisor itself along with libraries and command line tools (like xl and xm). Libvirt gives us easy access to VM management with the virsh command and python-virtinst gives us the handy virt-install command to make OS installations easy.

Once those packages are installed, we need to make some adjustments in your grub configuration. Open /boot/grub/menu.lst in your text editor of choice and add something like this at the bottom:

title Fedora + Xen (2.6.40-4.fc15.x86_64)
        root (hd0,1)
	kernel /boot/xen.gz
        module /boot/vmlinuz-2.6.40-4.fc15.x86_64 ro root=/dev/sda1
        module /boot/initramfs-2.6.40-4.fc15.x86_64.img

Ensure that the root (hd0,1) is applicable to your system (adjust it if it isn't). Also, check the kernel version to ensure it matches your installed kernel and adjust the root= portion to match your root volume. Flip the default line to a value which will boot your new grub entry and ensure the timeout is set to a reasonable number if you need to temporarily switch back to your original grub entry at boot time. (Hey, we all make mistakes.)

I take one extra precaution and change the UPDATEDEFAULT=yes line to no in /etc/sysconfig/kernel. This ensures that future kernel updates don't trample the entry you've just made. Keep in mind that you'll need to manually update your grub configuration when you do kernel upgrades later.

Cross your fingers and reboot. If your system doesn't reboot properly, reboot it again and choose your old kernel from the grub menu. Double-check your configuration for fat-fingering and give it another try. If your system boots and pings but you have no output via a monitor, don't fret. There's a patch for the problem which should appear soon in Linux 3.0. The impatient can snag a kernel source RPM, add the patch file, and build a local kernel (or you can download my local build from when I did it).

Log in and verify that you booted into the dom0:

[root@xenbox ~]# xm dmesg | head -n 5
 __  __            _  _    _   _   ____     __      _ ____  
 \ \/ /___ _ __   | || |  / | / | |___ \   / _| ___/ | ___| 
  \  // _ \ '_ \  | || |_ | | | |__ __) | | |_ / __| |___ \ 
  /  \  __/ | | | |__   _|| |_| |__/ __/ _|  _| (__| |___) |
 /_/\_\___|_| |_|    |_|(_)_(_)_| |_____(_)_|  \___|_|____/

Once you're done with that, make sure libvirtd is running:

/etc/init.d/libvirtd start; chkconfig libvirtd on

Try installing a VM:

virt-install \
  --paravirt \
  --name=testvm \
  --ram=512 \
  --vcpus=4 \
  --file /dev/vmstorage/testvm \
  --graphics vnc,port=5905 --noautoconsole \
  --autostart --noreboot \
  --location=http://mirrors.kernel.org/debian/dists/squeeze/main/installer-amd64/

You should have a VM installation underway pretty quickly and it will be visible via port 5905 on the local host. Enjoy the power and freedom of your brand new type 1 hypervisor.

Xen 4.1 on Fedora 15 with Linux 3.0 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

================================================================================
 Package        Arch           Version                   Repository        Size
================================================================================
Installing:
 kernel         x86_64         2.6.35.13-92.fc14         updates           22 M
Removing:
 kernel         x86_64         2.6.35.11-83.fc14         @updates         104 M
 
Transaction Summary
================================================================================
Install       1 Package(s)
Remove        1 Package(s)

Fedora 15's default behavior is to keep three kernels: the latest one and the two previous versions. However, this behavior may be counter-productive if you compile your own modules, or if you have compatibility issues with subsequent kernel versions.

You can change how yum handles kernel packages with some simple changes to your /etc/yum.conf. The installonly_limit option controls how many old packages are kept:

installonly_limit Number of packages listed in installonlypkgs to keep installed at the same time. Setting to 0 disables this feature. Default is '0'.

I disabled the functionality altogether by setting installonly_limit to 0:

#installonly_limit=3
installonly_limit=0

It's important to keep in mind that you will need to purge these packages from your system yourself now. Kernel packages can occupy a fair amount of disk space, so make a note to go back and clean them up when you no longer need them.

Keep all old kernels when upgrading via yum is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

HOLD IT: The steps below are definitely not meant for those who are new to Linux. Utilizing SystemTap on a production system is a bad idea -- it can chew up significant resources while it runs and it can also cause a running system to kernel panic if you're not careful with the packages you install.

These instructions will work well with Fedora, CentOS and Red Hat Enterprise Linux. Luckily, the SystemTap folks put together some instructions for Debian and Ubuntu as well.

Before you can start working with SystemTap on your RPM-based distribution, you'll need to get some prerequisites together:

yum install gcc systemtap systemtap-runtime systemtap-testsuite kernel-devel
yum --enablerepo=*-debuginfo install kernel-debuginfo kernel-debuginfo-common

WHOA THERE: Ensure that the kernel-devel and kernel-debuginfo* packages that you install via yum match up with your running kernel. If there's a newer kernel available from your yum repo, yum will pull that one. If it's been a while since you updated, you'll either need to upgrade your current kernel to the latest and reboot or you'll need to hunt down the corresponding kernel-devel and kernel-debuginfo* packages from a repository. Installing the wrong package version can lead to kernel panics. Also, bear in mind that the debuginfo packages are quite large: almost 200MB in Red Hat/CentOS and almost 300MB in Fedora.

You can't write the script in just any language. SystemTap uses an odd syntax to get things going:

#! /usr/bin/env stap
probe begin { println("hello world") exit () }

Just run the script with stap:

# stap -v helloworld.stp 
Pass 1: parsed user script and 73 library script(s) using 94380virt/21988res/2628shr kb, in 140usr/30sys/167real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 94776virt/22516res/2692shr kb, in 10usr/0sys/5real ms.
Pass 3: using cached /root/.systemtap/cache/bc/stap_bc368822da380b943d4e845ee15ed047_773.c
Pass 4: using cached /root/.systemtap/cache/bc/stap_bc368822da380b943d4e845ee15ed047_773.ko
Pass 5: starting run.
hello world
Pass 5: run completed in 0usr/20sys/285real ms.

The systemtap-testsuite package gives you a tubload of extremely handy SystemTap scripts. For example:

# cd /usr/share/systemtap/testsuite/systemtap.examples/io/
# stap iotime.stp
15138470 6351 (httpd) access /usr/share/cacti/index.php read: 0 write: 0
15142243 6351 (httpd) access /usr/share/cacti/include/auth.php read: 0 write: 0
15143780 6351 (httpd) access /usr/share/cacti/include/global.php read: 0 write: 0
15144099 6351 (httpd) access /etc/cacti/db.php read: 0 write: 0
15187641 6351 (httpd) access /usr/share/cacti/lib/adodb/adodb.inc.php read: 106486 write: 0
15187664 6351 (httpd) iotime /usr/share/cacti/lib/adodb/adodb.inc.php time: 218
15194965 6351 (httpd) access /usr/share/cacti/lib/adodb/adodb-time.inc.php read: 0 write: 0
15195692 6351 (httpd) access /usr/share/cacti/lib/adodb/adodb-iterator.inc.php read: 0 write: 0
   ... output continues ...

The iotime.stp script dumps out the reads and writes occurring on the system in real time. After starting the script above, I accessed my cacti instance on the server and immediately started seeing some reads as apache began picking up PHP files to parse.

Consider a situation in which you need to decrease interrupts on a Linux machine. This is vital for laptops and systems that need to remain in low power states. Some might suggest powertop for that, but why not give SystemTap a try?

# cd /usr/share/systemtap/testsuite/systemtap.examples/interrupt/
# stap interrupts-by-dev.stp 
        ohci_hcd:usb3 :      1
        ohci_hcd:usb4 :      1
            hda_intel :      1
                 eth0 :      2
                 eth0 :      2
                 eth0 :      2
                 eth0 :      2
                 eth0 :      2
                 eth0 :      2

On this particular system, it's pretty obvious that the ethernet interface is causing a lot of interrupts.

If you want more examples, keep hunting around in the systemtap-testsuite package (remember rpm -ql systemtap-testsuite) or review the giant list of examples on SystemTap's site.

Thanks again to Phil Hopkins at Rackspace for giving us a detailed explanation of system profiling during training.

Tap into your Linux system with SystemTap is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Diagram of two web nodes with a replicated filesystem using DRBD & OCFS2

How does the performance compare to GlusterFS?
On Gluster's best days, the data throughput speeds were quite good, but the latency to retrieve the data was often much too high. Page loads on this site were taking upwards of 3-4 seconds with GlusterFS latency accounting for well over 75% of the delays. For small files, GlusterFS's performance was about 20-25x slower than accessing the disk natively. The performance hit for DRBD and OCFS2 is usually between 1.5-3x for small files and difficult to notice for large file transfers.

Couldn't you keep the data separate and then sync it with rsync?
Everyone knows that rsync can be a resource consuming monster and it seems wasteful to call rsync via a cron job to keep my data in sync. There are some periods of the day where the actual data on the web root rarely changes. There are other times where it changes rapidly and I'd end up with nodes out of sync for a few minutes.

To get the just-in-time synchronization that I want, I'd have to run rsync at least once a minute. If the data isn't changing over a long period, rsync would end up crushing the disk and consuming CPU for no reason. DRBD only syncs data when data changes. Also, all reads with DRBD are done locally. This makes is a highly efficient and effective choice for instant synchronization.

Why OCFS2? Isn't that overkill?
When you use DRBD in dual-primary mode, it's functionally equivalent to having a raw storage device (like a SAN) mounted in two places. If you threw an ext4 filesystem onto a LUN on your SAN and then mounted it on two different servers, you'd be in bad shape very quickly. Non-clustered filesystems like ext3 or ext4 can't handle being mounted in more than one environment.

OCFS2 is built primarily to be mounted in more than one place and it comes with its own distributed locking manager (DLM). The configuration files for OCFS2 are extremely simple and you mount it like any other filesystem. It's been part of the mainline Linux kernel since 2.6.19.

What happens when you lose one of the nodes?
The configuration shown above can operate with just one node in an emergency. When the failed node comes back online, DRBD will resync the block device and you can mount the OCFS2 filesystem as you normally would.

You're using an Oracle product? Really?
You've got me there. I'm not a fan of how they treat the open source community with regards to some of their projects, but the OCFS2 filesystem is robust, free, and it meets my needs.

Where's the how-to?
It's coming soon! Stay tuned.

Keep web servers in sync with DRBD and OCFS2 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The guide: Redundant Cloud Hosting Guide

As always, if you find anything in the guide that needs improvement, I'm all ears.

A simple guide to redundant cloud hosting is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Quite a few distribution maintainers have gone to great lengths to keep Xen domU support in their kernels and it's been an uphill battle. Some kernels, such as Ubuntu's linux-ec2 kernels, have patches from 2.6.18 dragged forward into 2.6.32 and even 2.6.33. It certainly can't be enjoyable to keep dragging those patches forward into new kernel trees.

The paravirt_ops support for Xen guests was added in 2.6.23 and continues to be included and improved in the latest kernel trees. However, there are two significant problems with these new kernels if you're trying to work with legacy environments:

• the console is on hvc0, not tty1
• block devices are now /dev/xvdX rather than /dev/sdX

If you only have a few guests, these changes are generally pretty easy. Switching the console just requires some changes to your inittab or upstart configurations. Changing the block device names requires changes to the guest's Xen configuration file and /etc/fstab within the guest itself.

Considering the amount of environments I work with daily at Rackspace, changing the guest configuration is definitely not an option. I needed a way to keep the console and block devices unchanged so that our customers could have a consistent experience on our infrastructure.

Luckily, Soren Hansen offered to pitch in and a solution became apparent. Through some relatively small patches, the legacy console and block device support was available in the latest 2.6.32 version (2.6.32.12 as of this post's writing).

So far, I've tested x86_64 and i386 versions of 2.6.32.12 with the console and block device patches. It's gone through its paces on Xen 3.0.3, 3.1.2, 3.3.0 and 3.4.2. All revisions of Fedora, CentOS, Ubuntu, Debian, Gentoo and Arch made within the last two years are working well with the new kernels.

Legacy tty1 and block device support for Xen guests with pvops kernels is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

How do I become a better systems administrator?

The best way to become a better systems administrator is to fully understand the theory of what's happening in your server's environment.

What do I mean by that? Learn why things aren't happening as you expected and think about all of the factors that could possibly be involved. Instead of thinking purely about cause and effect, you'll find it much easier and rewarding to consider everything inside and outside your environment before you make any changes.

This still may be a little difficult to fully understand, so he's an example. Let's say you're handling an issue where a customer can't reach a website hosted on their server. When you ask them for more details, they might give you the dreaded reply: "It's not coming up." Start by making a mental list of the problems that are easiest to check:

• Is the web server daemon running?
• If a database server is being used, is it running and accessible?
• Is there a software/hardware firewall blocking port 80?
• Is a script stuck on the server tying up resources?
• Could there be a DNS resolution problem?
• Is the server up?
• Did a switch fail?
• Is the server's hard disk out of space?
• Can the customer reach other websites like Google or Yahoo?
• If SELinux is involved, have the appropriate contexts been set?
• Could the site be a target of a denial of service attack?
• Has the server reached its connection tracking limit?

Of course, this is a relatively short list, but these are all easy to check. If you're thinking about cause and effect, you might only consider the web server daemon and some basic network issues. By considering all of the other factors that may be related, you've ensured that all of the basics are covered before you consider more complex problems.

Most systems administrators have taken an error message and tossed it in en masse into Google before. Occasionally, no results will appear for the search. If you find yourself in this situation, try to understand the individual parts of the error message. Work outward from what you know already. You should know which daemon said it, and you may have an idea of what the application was doing when the error occurred. Take time to consider what the daemon is trying to tell you within the context of what it was doing at the time.

One of the easiest ways to force yourself to be immersed into this way of thinking is to host applications for non-technical people. You'll find that many customers want things done differently, and they're all at different levels of technical aptitude. Some may find it a frustrating experience at first, but you'll think yourself later. It will force you to consider all aspects of how a server operates since you might not always know what's happening within a customer's application.

As always, if you find yourself stumbling, remember to ask your peers and colleagues. Even if they haven't seen the particular issue, they will probably be able to guide you closer to the solution you seek.

A New Year System Administrator Inspiration is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

wget --quiet -O - http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.31.8 | grep -A 4 ^commit | grep -B 1 "^--" | grep -v "^--"

It should give you some output like this:

    Linux 2.6.31.8
    ext4: Fix potential fiemap deadlock (mmap_sem vs. i_data_sem)
    signal: Fix alternate signal stack check
    SCSI: scsi_lib_dma: fix bug with dma maps on nested scsi objects
    SCSI: osd_protocol.h: Add missing #include
    SCSI: megaraid_sas: fix 64 bit sense pointer truncation
    ..

Parse kernel.org changelogs with wget and grep is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

[root@f11 ~]# /etc/init.d/httpd start
Starting httpd: [Fri Aug 14 17:05:24 2009] [crit] (22)Invalid argument: alloc_listener: failed to get a socket for (null)
Syntax error on line 134 of /etc/httpd/conf/httpd.conf:
Listen setup failed
                                                           [FAILED]

The issue is related to three kernel calls that are used in apr-1.3.8-1: accept4(), dup3() and epoll_create1(). Without these calls, apache is unable to start.

Update on August 17, 2009: the Fedora team has pushed apr-1.3.8-2 into the stable repositories for Fedora 11, which eliminates the need for the temporary fix shown below.

Deprecated solution: There is a bug open with the Fedora team, and there is a temporary fix available:

yum --enablerepo=updates-testing update apr

Fedora 11 httpd: alloc_listener: failed to get a socket for (null) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

You can issue the signals like this:

kill -SIGSTOP [pid]
kill -SIGCONT [pid]

Wikipedia has great definitions for SIGSTOP:

When SIGSTOP is sent to a process, the usual behaviour is to pause that process in its current state. The process will only resume execution if it is sent the SIGCONT signal. SIGSTOP and SIGCONT are used for job control in the Unix shell, among other purposes. SIGSTOP cannot be caught or ignored.

and SIGCONT:

When SIGSTOP or SIGTSTP is sent to a process, the usual behaviour is to pause that process in its current state. The process will only resume execution if it is sent the SIGCONT signal. SIGSTOP and SIGCONT are used for job control in the Unix shell, among other purposes.

In short, SIGSTOP tells a process to "hold on" and SIGCONT tells a process to "pick up where you left off". This can work really well for rsync jobs since you can pause the job, clear up some space on the destination device, and then resume the job. The source rsync process just thinks that the destination rsync process is taking a long time to respond.

In the ps output, stopped processes will have a status containing T. Here's an example with crond:

# kill -SIGSTOP `pgrep crond`
# ps aufx | grep crond
root      3499  0.0  0.0 100328  1236 ?        Ts   Jun11   0:01 crond
# kill -SIGCONT `pgrep crond`
# ps aufx | grep crond
root      3499  0.0  0.0 100328  1236 ?        Ss   Jun11   0:01 crond

Two great signals: SIGSTOP and SIGCONT is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

As we know, sometimes servers misbehave due to things put them through, and you can quickly end up in a situation where things are going badly. I'm talking about the type of situation where you're connected via SSH to a server that controls phone lines for five million people and it sits in a tiny building 400 miles away from the nearest human being. We're talking bad. If you issue a plain reboot command, it might not even make it that far. Once SSH stops running, you're going to be out of luck.

If you find yourself in this situation (and I hope you won't!), you have some options to get your way with a misbehaving server remotely. You can force an immediate reboot with the following:

echo 1 > /proc/sys/kernel/sysrq 
echo b > /proc/sysrq-trigger

WHOA THERE! This is pretty much the same as pressing the reset button on the server (if equipped). No daemons will be shut down gracefully, no filesystem sync will occur, and you may get the wrath of a fsck (or worse, a non-booting server) upon reboot. To do things a little more carefully, read on.

These are called magic commands, and they're pretty much synonymous with holding down Alt-SysRq and another key on older keyboards. Dropping 1 into /proc/sys/kernel/sysrq tells the kernel that you want to enable SysRq access (it's usually disabled). The second command is equivalent to pressing Alt-SysRq-b on a QWERTY keyboard.

There's a better way of rebooting a misbehaving server that Wikipedia shows with the mnemonic "Reboot Even If System Utterly Broken":

unRaw      (take control of keyboard back from X),  
 tErminate (send SIGTERM to all processes),
 kIll      (send SIGKILL to all processes), 
  Sync     (flush data to disk),
  Unmount  (remount all filesystems read-only),
reBoot.

I can't vouch for this actually working, but I'm interested to try it. UPDATE: I've been told that doing this series of commands with ReiserFS is a very bad idea.

If you want to shut the machine down entirely (please think about it before using this on a remote system):

echo 1 > /proc/sys/kernel/sysrq 
echo o > /proc/sysrq-trigger

If you want to keep SysRq enabled all the time, you can do that with an entry in your server's sysctl.conf:

kernel.sysrq = 1

Linux: emergency reboot or shutdown with magic commands is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

root@storageserver:~# lspci | grep ATA
00:08.0 IDE interface: nVidia Corporation MCP61 SATA Controller (rev a2)
00:08.1 IDE interface: nVidia Corporation MCP61 SATA Controller (rev a2)
03:00.0 Mass storage controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)

There are two primary drives connected to the onboard controller and four connected to the controller card. One of the primary drives on the onboard controller contains the operating system (Ubuntu, in this case), while the other drive is blank.

When the system booted, the sata_sil24 driver for the add-on card always loaded before the sata_nv drivers for the onboard storage controller:

kernel: [    4.125598] sata_sil24 0000:03:00.0: version 1.1
kernel: [    4.126102] sata_sil24 0000:03:00.0: PCI INT A -> Link[APC6] -> GSI 16 (level, low) -> IRQ 16
kernel: [    4.126161] sata_sil24 0000:03:00.0: setting latency timer to 64
kernel: [    4.129472] scsi0 : sata_sil24
kernel: [    4.129635] scsi1 : sata_sil24
kernel: [    8.293762] sata_nv 0000:00:08.0: version 3.5
kernel: [    8.293779] sata_nv 0000:00:08.0: PCI INT A -> Link[APSI] -> GSI 20 (level, low) -> IRQ 20
kernel: [    8.293829] sata_nv 0000:00:08.0: setting latency timer to 64
kernel: [    8.296764] scsi2 : sata_nv
kernel: [    8.296905] scsi3 : sata_nv
kernel: [    9.285034] sata_nv 0000:00:08.1: PCI INT B -> Link[APSJ] -> GSI 21 (level, low) -> IRQ 21
kernel: [    9.285074] sata_nv 0000:00:08.1: setting latency timer to 64
kernel: [    9.285161] scsi4 : sata_nv
kernel: [    9.286015] scsi5 : sata_nv

After specifying an explicit order in /etc/modules and /etc/modprobe.conf, I wasn't able to see any changes. The sata_sil24 driver still loaded before the onboard sata_nv driver. Luckily, a very wise person on Twitter gave me a strategy that worked just fine.

I added sata_sil24 to the bottom of my /etc/modprobe.d/blacklist file first. Then, in /etc/modules, I listed sata_nv first, followed by sata_sil24. When the system booted, I got the result that I wanted:

kernel: [    3.982909] sata_nv 0000:00:08.0: version 3.5
kernel: [    3.982931] sata_nv 0000:00:08.0: PCI INT A -> Link[APSI] -> GSI 20 (level, low) -> IRQ 20
kernel: [    3.982993] sata_nv 0000:00:08.0: setting latency timer to 64
kernel: [    3.984497] scsi0 : sata_nv
kernel: [    3.986013] scsi1 : sata_nv
kernel: [    4.971755] sata_nv 0000:00:08.1: PCI INT B -> Link[APSJ] -> GSI 21 (level, low) -> IRQ 21
kernel: [    4.971799] sata_nv 0000:00:08.1: setting latency timer to 64
kernel: [    4.973153] scsi2 : sata_nv
kernel: [    4.974031] scsi3 : sata_nv
kernel: [   15.988862] sata_sil24 0000:03:00.0: version 1.1
kernel: [   15.989454] sata_sil24 0000:03:00.0: PCI INT A -> Link[APC6] -> GSI 16 (level, low) -> IRQ 16
kernel: [   15.989511] sata_sil24 0000:03:00.0: setting latency timer to 64
kernel: [   15.990201] scsi6 : sata_sil24
kernel: [   15.991523] scsi7 : sata_sil24

The sata_nv driver is loading first, and Ubuntu boots off of it without an issue. The sata_sil24 driver loads next so that the drives connected to the card show up lower in the boot order.

Many thanks to @Twirrim on Twitter for the suggestion!

Linux: Adjust storage kernel module load order is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

There are a few processes on the server that keep it fairly busy. Two of the processes are vital to the server's operation - if they are stopped, lots of work is required to get them running properly again. I found that a certain java process was being killed by OOM killer regularly, and another perl process was being killed occasionally.

Naturally, my disdain for java made me think that the java process was the source of the issue. The process was configured to use a small amount of RAM, so it was ruled out. The other perl process used even less memory, so it was ruled out as well. When I checked the sysstat data with sar, I found that the server was only using about 2-3GB out of 4GB of physical memory at the time when OOM killer was started. At this point, I was utterly perplexed.

I polled some folks around the office and gathered some ideas. After putting some ideas together, I found that the server was actually caching too much data in the ext3_inode_cache and dentry_cache. These caches hold recently accessed files and directories on the server, and they're purged as the files and directories become stale. Since the operations on the server read and write large amounts of data locally and via NFS, I knew these caches had to be gigantic. If you want to check your own caches, you can use the slabtop command. For those who like things more difficult, you can also cat the contents of /proc/slabinfo and grep for the caches that are important to you.

An immense amount of Googling revealed very little, but I discovered a dirty hack to fix the issue (don't run this yet):

echo 1 > /proc/sys/vm/drop_caches  # free pagecache
     [OR]
echo 2 > /proc/sys/vm/drop_caches  # free dentries and inodes
     [OR]
echo 3 > /proc/sys/vm/drop_caches  # free pagecache, dentries and inodes
sync  # forces the dump to be destructive

There are huge consequences to dumping these caches and running sync. If you are writing data at the time you run these commands, you'll actually be dumping the data out of the filesystem cache before it reaches the disk, which could lead to very bad things.

While discussing the issue with a coworker, he found a different method for correcting the issue that was much safer. You can echo values into /proc/sys/vm/vfs_cache_pressure to tell the kernel what priority it should take when clearing out the inode/dentry caches. LinuxInsight explains the range of values well:

At the default value of vfs_cache_pressure = 100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

In short, values less than 100 won't reduce the caches very much as all. Values over 100 will signal to the kernel that you want to clear out the caches at a higher priority. I found that no matter what value you use, the kernel clears the caches at a slow rate. I've been using a value of 10000 on the server I talked about earlier in the article, and it has kept the caches down to a reasonable level.

Reducing inode and dentry caches to keep OOM killer at bay is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

There's two main kernel variables to know:

vm.dirty_ratio - The highest % of your memory that can be used to hold dirty data. If you set this to a low value, the kernel will flush small writes to the disk more often. Higher values allow the small writes to stack up in memory. They'll go to the disk in bigger chunks.

vm.dirty_background_ratio - The lowest % of your memory where pdflush is told to stop when it is writing dirty data. You'll want to keep this set as low as possible.

These might confuse you. In short, when your memory begins filling with little pieces of data that needs to be written to the disk, it will keep filling until it reaches the dirty_ratio. At that point, pdflush will start up, and it will write data until it reduces the dirty data to the value set by dirty_background_ratio.

Stock 2.6.9 kernels have a dirty_background_ratio of 10% and a dirty_ratio of 40%. Some distributions tweak these defaults to something different, so you may want to review the settings on your system. On a system with heavy disk I/O, you can increase the dirty_ratio and reduce the dirty_background_ratio. A little experimentation may be necessary to find the perfect setting for your server.

If you want to play with the variables, just use your standard echo:

echo 5 > /proc/sys/vm/dirty_background_ratio
echo 60 > /proc/sys/vm/dirty_ratio

Once you've found the right setting, you can set it permanently by adding lines to your /etc/sysctl.conf:

vm.dirty_background_ratio = 5
vm.dirty_ratio = 60

If you have a reliable server with a good RAID card and power supply, you could set the dirty_ratio to 100 and the dirty_background_ratio to 1. This was recommended by a buddy of mine who runs quite a few servers running virtual machines.

Reduce disk I/O for small reads using memory is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First, add the appropriate line to your /etc/modules.conf or /etc/modprobe.conf which corresponds to your new kernel module.

Next, rebuild the initial ram disk after making a backup of the current one:

# cp /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img.bak
# mkinitrd -f initrd-`uname -r`.img `uname -r`

Reboot the server now and make sure the new driver is loaded properly.

Rebuilding the initial ram disk (initrd) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

ip_conntrack: table full, dropping packet

I'd seen this message before, but I headed over to Red Hat's site for more details. It turns out that the server was running iptables, but it was under a very heavy load and also handling a high volume of network connections. Generally, the ip_conntrack_max is set to the total MB of RAM installed multiplied by 16. However, this server had 4GB of RAM, but ip_conntrack_max was set to 65536:

# cat /proc/sys/net/ipv4/ip_conntrack_max
65536

I logged into another server with 1GB of RAM (RHES 5, 32-bit) and another with 2GB of RAM (RHES 4, 64-bit), and both had ip_conntrack_max set to 65536. I'm not sure if this is a known Red Hat issue, or if it's just set to a standard value out of the box.

If you want to check your server's current tracked connections, just run the following:

# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count

If you want to adjust it (as I did), just run the following as root:

# echo 131072 > /proc/sys/net/ipv4/ip_conntrack_max

ip_conntrack: table full, dropping packet is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.