Racker Hacker Words of wisdom from a server administrator

I found a great post on Overstimulate about handling the rotation of rails logs when you use Phusion Passenger. Most of the data for your application should end up in the apache logs, but if your site is highly dynamic, you may end up with a giant production log if you’re not careful.

Toss this into /etc/logrotate.d/yourrailsapplication:

/var/www/yourrailsapp/log/*.log {
  daily
  missingok
  rotate 30
  compress
  delaycompress
  sharedscripts
  postrotate
    touch /var/www/yourrailsapp/tmp/restart.txt
  endscript
}

For a detailed explanation, see the post on Overstimulate.

Occasionally, I’ll end up with a mailbox full of random data, alerts, or other useless things. If you have SSH access to the server, you can always clear out your mail spool, but if you connect to an IMAP server, you can use mutt to do the same thing.

First, use mutt to connect to your server remotely (via IMAP over SSL in this example):

mutt -f imaps://mail.yourdomain.com/

Once you’ve connected and logged in, press SHIFT-D (uppercase d). The status bar of mutt should show:

Delete messages matching:

Type in ~s .* so that the line looks like:

Delete messages matching: ~s .*

When you press enter, mutt will put a D next to all of the messages, which marks them for deletion. Press q to quit, and then y to confirm the deletion. After a brief moment, all of those messages will be deleted and mutt will exit.

The best uses I’ve found for the SIGSTOP and SIGCONT signals are times when a process goes haywire, or when a script spawns too many processes at once.

You can issue the signals like this:

kill -SIGSTOP [pid]
kill -SIGCONT [pid]

Wikipedia has great definitions for SIGSTOP:

When SIGSTOP is sent to a process, the usual behaviour is to pause that process in its current state. The process will only resume execution if it is sent the SIGCONT signal. SIGSTOP and SIGCONT are used for job control in the Unix shell, among other purposes. SIGSTOP cannot be caught or ignored.

and SIGCONT:

When SIGSTOP or SIGTSTP is sent to a process, the usual behaviour is to pause that process in its current state. The process will only resume execution if it is sent the SIGCONT signal. SIGSTOP and SIGCONT are used for job control in the Unix shell, among other purposes.

In short, SIGSTOP tells a process to “hold on” and SIGCONT tells a process to “pick up where you left off”. This can work really well for rsync jobs since you can pause the job, clear up some space on the destination device, and then resume the job. The source rsync process just thinks that the destination rsync process is taking a long time to respond.

In the ps output, stopped processes will have a status containing T. Here’s an example with crond:

# kill -SIGSTOP `pgrep crond`
# ps aufx | grep crond
root      3499  0.0  0.0 100328  1236 ?        Ts   Jun11   0:01 crond
# kill -SIGCONT `pgrep crond`
# ps aufx | grep crond
root      3499  0.0  0.0 100328  1236 ?        Ss   Jun11   0:01 crond

When I tried to do an automatic upgrade from Wordpress 2.7.1 to 2.8 yesterday, it failed miserably. The files were all put in place, but when I tried to load /wp-admin/upgrade.php, this error popped up:

Fatal error: Call to undefined method wpdb::has_cap() in 
/path/to/wordpress/wp-admin/includes/schema.php on line 22

I was perplexed at the error, so I restored from a backup and began upgrading manually. The manual upgrades have always worked well for me in the past, so I figured this would probably fix the problem. After the upgrade, I went to /wp-admin/upgrade.php and saw:

Fatal error: Call to undefined method wpdb::has_cap() in 
/path/to/wordpress/wp-admin/includes/schema.php on line 22

What the heck is going on? I restored from a backup, tried the manual upgrade again, and it still failed. I took a look at the lines causing the problem in schema.php:

if ( $wpdb->has_cap( 'collation' ) ) {
	if ( ! empty($wpdb->charset) )
		$charset_collate = "DEFAULT CHARACTER SET $wpdb->charset";
	if ( ! empty($wpdb->collate) )
		$charset_collate .= " COLLATE $wpdb->collate";
}

I figured I could comment out the if statement and probably still be safe:

// if ( $wpdb->has_cap( 'collation' ) ) {
	if ( ! empty($wpdb->charset) )
		$charset_collate = "DEFAULT CHARACTER SET $wpdb->charset";
	if ( ! empty($wpdb->collate) )
		$charset_collate .= " COLLATE $wpdb->collate";
// }

Success? I could make it through the upgrade.php part fine at this point, but whenever I tried to add a tag to a post, it wasn’t saving to the database. I caught this error in my apache logs:

[Fri Jun 12 23:45:03 2009] [error] [client 72.183.200.144] WordPress database error Duplicate entry 'debian' for key 'slug' for query INSERT INTO wp_terms (`name`,`slug`,`term_group`) VALUES ('debian','debian','0') made by wp_insert_term, referer: http://rackerhacker.com/wp-admin/post.php?action=edit&post=877

Frustration quickly ensued. I moved my /wp-content/ folder out of the way and replaced it with the standard Wordpress stuff, but that didn’t help. I moved plugins out of the way, one by one, but that didn’t fix it either. Then I spotted a strange file sitting in /wp-content/ called db.php. When I opened it, I found a lot of database setup classes for mysqli.

I renamed it to db.pleasedonteverrunthisphp and I was able to save tags properly. So far, I haven’t found any issues after I made chat change.

Does anyone know where that file might have come from? I don’t remember adding it myself, so I’m wondering if it was ever packaged with a Wordpress plugin or a Wordpress installation. I hope this helps someone else!

There are two main ways to upgrade Fedora 10 (Cambridge) to Fedora 11 (Leonidas):

» What the Fedora developers suggest:

yum -y upgrade
yum -y install preupgrade
yum clean all
preupgrade-cli "Fedora 11 (Leonidas)"

Of course, if you’re doing this on a Fedora desktop, you can use preupgrade (rather than preupgrade-cli) to upgrade with a GUI.

» The method I prefer (and it works properly on Slicehost):

yum -y upgrade
yum clean all
wget http://download.fedora.redhat.com/pub/fedora/linux/releases/11/Fedora/x86_64/os/Packages/fedora-release-11-1.noarch.rpm

At this point, you would normally just start upgrading packages, but the Fedora developers threw us a curveball. Since yum in Fedora 10 doesn’t support metalinks, your upgrades will fail with something like this:

# yum -y upgrade
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. /
removing mirrorlist with no valid mirrors: //var/cache/yum/updates/mirrorlist.txt
Error: Cannot retrieve repository metadata (repomd.xml) for repository: updates. Please verify its path and try again

It’s easily fixed, however. Open up /etc/yum.repos.d/fedora.repo and /etc/yum.repos.d/fedora-updates.repo in your favorite text editor and change the mirrorlist URL’s like so:

Fedora Repository

#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch
mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch

Fedora Updates Repository

#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f$releasever&arch=$basearch

Once you make those changes, finish out the upgrade:

yum -y upgrade

This process will take a little while to complete, but there shouldn’t be any interaction required. Once it’s done, change the mirrorlist lines back to the original values so you can benefit from the speedups provided by the metalink format.