Toss this into /etc/logrotate.d/yourrailsapplication:

/var/www/yourrailsapp/log/*.log {
  daily
  missingok
  rotate 30
  compress
  delaycompress
  sharedscripts
  postrotate
    touch /var/www/yourrailsapp/tmp/restart.txt
  endscript
}

For a detailed explanation, see the post on Overstimulate.

©2009 Racker Hacker. All Rights Reserved.

First, use mutt to connect to your server remotely (via IMAP over SSL in this example):

mutt -f imaps://mail.yourdomain.com/

Once you’ve connected and logged in, press SHIFT-D (uppercase d). The status bar of mutt should show:

Delete messages matching:

Type in ~s .* so that the line looks like:

Delete messages matching: ~s .*

When you press enter, mutt will put a D next to all of the messages, which marks them for deletion. Press q to quit, and then y to confirm the deletion. After a brief moment, all of those messages will be deleted and mutt will exit.

©2009 Racker Hacker. All Rights Reserved.

You can issue the signals like this:

kill -SIGSTOP [pid]
kill -SIGCONT [pid]

Wikipedia has great definitions for SIGSTOP:

When SIGSTOP is sent to a process, the usual behaviour is to pause that process in its current state. The process will only resume execution if it is sent the SIGCONT signal. SIGSTOP and SIGCONT are used for job control in the Unix shell, among other purposes. SIGSTOP cannot be caught or ignored.

and SIGCONT:

When SIGSTOP or SIGTSTP is sent to a process, the usual behaviour is to pause that process in its current state. The process will only resume execution if it is sent the SIGCONT signal. SIGSTOP and SIGCONT are used for job control in the Unix shell, among other purposes.

In short, SIGSTOP tells a process to “hold on” and SIGCONT tells a process to “pick up where you left off”. This can work really well for rsync jobs since you can pause the job, clear up some space on the destination device, and then resume the job. The source rsync process just thinks that the destination rsync process is taking a long time to respond.

In the ps output, stopped processes will have a status containing T. Here’s an example with crond:

# kill -SIGSTOP `pgrep crond`
# ps aufx | grep crond
root      3499  0.0  0.0 100328  1236 ?        Ts   Jun11   0:01 crond
# kill -SIGCONT `pgrep crond`
# ps aufx | grep crond
root      3499  0.0  0.0 100328  1236 ?        Ss   Jun11   0:01 crond

©2009 Racker Hacker. All Rights Reserved.

Fatal error: Call to undefined method wpdb::has_cap() in 
/path/to/wordpress/wp-admin/includes/schema.php on line 22

I was perplexed at the error, so I restored from a backup and began upgrading manually. The manual upgrades have always worked well for me in the past, so I figured this would probably fix the problem. After the upgrade, I went to /wp-admin/upgrade.php and saw:

Fatal error: Call to undefined method wpdb::has_cap() in 
/path/to/wordpress/wp-admin/includes/schema.php on line 22

What the heck is going on? I restored from a backup, tried the manual upgrade again, and it still failed. I took a look at the lines causing the problem in schema.php:

if ( $wpdb->has_cap( 'collation' ) ) {
	if ( ! empty($wpdb->charset) )
		$charset_collate = "DEFAULT CHARACTER SET $wpdb->charset";
	if ( ! empty($wpdb->collate) )
		$charset_collate .= " COLLATE $wpdb->collate";
}

I figured I could comment out the if statement and probably still be safe:

// if ( $wpdb->has_cap( 'collation' ) ) {
	if ( ! empty($wpdb->charset) )
		$charset_collate = "DEFAULT CHARACTER SET $wpdb->charset";
	if ( ! empty($wpdb->collate) )
		$charset_collate .= " COLLATE $wpdb->collate";
// }

Success? I could make it through the upgrade.php part fine at this point, but whenever I tried to add a tag to a post, it wasn’t saving to the database. I caught this error in my apache logs:

[Fri Jun 12 23:45:03 2009] [error] [client 72.183.200.144] WordPress database error Duplicate entry 'debian' for key 'slug' for query INSERT INTO wp_terms (`name`,`slug`,`term_group`) VALUES ('debian','debian','0') made by wp_insert_term, referer: http://rackerhacker.com/wp-admin/post.php?action=edit&post=877

Frustration quickly ensued. I moved my /wp-content/ folder out of the way and replaced it with the standard Wordpress stuff, but that didn’t help. I moved plugins out of the way, one by one, but that didn’t fix it either. Then I spotted a strange file sitting in /wp-content/ called db.php. When I opened it, I found a lot of database setup classes for mysqli.

I renamed it to db.pleasedonteverrunthisphp and I was able to save tags properly. So far, I haven’t found any issues after I made chat change.

Does anyone know where that file might have come from? I don’t remember adding it myself, so I’m wondering if it was ever packaged with a Wordpress plugin or a Wordpress installation. I hope this helps someone else!

©2009 Racker Hacker. All Rights Reserved.

» What the Fedora developers suggest:

yum -y upgrade
yum -y install preupgrade
yum clean all
preupgrade-cli "Fedora 11 (Leonidas)"

Of course, if you’re doing this on a Fedora desktop, you can use preupgrade (rather than preupgrade-cli) to upgrade with a GUI.

» The method I prefer (and it works properly on Slicehost):

yum -y upgrade
yum clean all
wget http://download.fedora.redhat.com/pub/fedora/linux/releases/11/Fedora/x86_64/os/Packages/fedora-release-11-1.noarch.rpm

At this point, you would normally just start upgrading packages, but the Fedora developers threw us a curveball. Since yum in Fedora 10 doesn’t support metalinks, your upgrades will fail with something like this:

# yum -y upgrade
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. /
removing mirrorlist with no valid mirrors: //var/cache/yum/updates/mirrorlist.txt
Error: Cannot retrieve repository metadata (repomd.xml) for repository: updates. Please verify its path and try again

It’s easily fixed, however. Open up /etc/yum.repos.d/fedora.repo and /etc/yum.repos.d/fedora-updates.repo in your favorite text editor and change the mirrorlist URL’s like so:

Fedora Repository

#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch
mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch

Fedora Updates Repository

#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f$releasever&arch=$basearch

Once you make those changes, finish out the upgrade:

yum -y upgrade

This process will take a little while to complete, but there shouldn’t be any interaction required. Once it’s done, change the mirrorlist lines back to the original values so you can benefit from the speedups provided by the metalink format.

©2009 Racker Hacker. All Rights Reserved.

Create a simple SOCKS proxy with SSH by using the -D flag:

ssh -D 2400 username@some.host.com

That command will open up a SOCKS proxy on your workstation on port 2400. If you configure your application to use the local SOCKS proxy, any traffic using the proxy will be sent through an encrypted SSH connection to your remote server and out to the internet. Inbound traffic through the proxy is encrypted through the same connection.

You can pair that with autossh to keep your proxy connected at all times:

autossh -f -M 20000 -D 2400 username@some.host.com -N

©2009 Racker Hacker. All Rights Reserved.

mysql> pager md5sum -
PAGER set to 'md5sum -'
mysql> select * from test;
a09bc56ac9aa0cbcc659c3d566c2c7e4  -
4096 rows in set (0.00 sec)

It’s a quick way to determine if you have two tables that are properly in sync. Although there are better ways to compare tables in replicated environments, this method can get it done pretty quickly.

©2009 Racker Hacker. All Rights Reserved.

kernel: sdb : READ CAPACITY failed.
kernel: sdb : status=0, message=00, host=4, driver=00 
kernel: sdb : sense not available. 
kernel: sdb: Write Protect is off
kernel: sdb: Mode Sense: 00 00 00 00
kernel: sdb: asking for cache data failed
kernel: sdb: assuming drive cache: write through
kernel:  sdb:<6>sd 1:0:0:0: SCSI error: return code = 0x00040000
kernel: end_request: I/O error, dev sdb, sector 0
kernel: Buffer I/O error on device sdb, logical block 0
kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
kernel: end_request: I/O error, dev sdb, sector 0
kernel: Buffer I/O error on device sdb, logical block 0
kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
kernel: end_request: I/O error, dev sdb, sector 0

The errors show that the SCSI bus is having issues bringing the new drive online, and it won’t be seen by the OS until the SCSI controller is pleased. You can force the controller to re-scan the drives attached to it, and this should correct the problem:

cd /sys/class/scsi_host/hostX
echo "- - - " > scan

Replace the X with the proper controller number of your SCSI controller. If you’re not sure which controller is which, try running:

# cat /sys/class/scsi_host/host0/proc_name
sata_nv

Credit for this find goes to Tony Dolan

©2009 Racker Hacker. All Rights Reserved.

Edit: Using logger (as suggested by David and Jerry below) will give you a more reliable way to send the data to a syslog server:

cat some.log | logger -t UsefulLabel -h yoursyslogserver.com -p 514

You’ll also be able to set a label for the text before it’s piped into the syslog server, which would be handy if you’re sorting or parsing the data later on.

Also, you can send your data in the raw using netcat:

cat some.log | nc -w 1 -u yoursyslogserver.com 514

©2009 Racker Hacker. All Rights Reserved.

If you don’t want to downgrade to PHPMyAdmin 2.x, just add the following line to the top of your config.inc.php file:

$cfg['InitialSlidersState'] = 'open';

This will cause the indexes to be displayed when you click Structure for a certain table. By default, they are hidden.

Sidenote: Some of you might be thinking: “Hey, you’re a DBA, you should know MySQL queries without needing PHPMyAdmin.” You’re right. I do know how to get the job done without PHPMyAdmin, but I enjoy the way PHPMyAdmin allows me to visualize my table structures. Also, it’s a handy way to present data to others very quickly.

©2009 Racker Hacker. All Rights Reserved.

However, I stumbled upon some documentation that will allow you to VNC to a VMWare Server VM’s main screen. It’s equivalent to having a network KVM connected to the VM so you can have out-of-band management. With VMWare server 2.x, you can enable it by following these steps:

Step 1. Create a new VM in VMWare Server, but don’t start the VM.

Step 2. SSH to the server and find your VM’s .vmx file. Normally, you can find the file in a location like /var/lib/vmware/[vmname]/[vmname].vmx.

Step 3. Add the following lines to the end of the .vmx file:

RemoteDisplay.vnc.enabled = "TRUE"
RemoteDisplay.vnc.password = "vncpassword"
RemoteDisplay.vnc.port = "5900"

Step 4. Change the VNC port and password to values that suit your environment and then start the VM.

DUH! Don’t set two VM’s to use the same vnc port, but that should go without saying.

©2009 Racker Hacker. All Rights Reserved.

First, pull a list of RPM package by name only:

rpm -qa --queryformat='%{NAME}\n' | sort > server.txt

Once you’ve done that on both servers, just use diff to compare the two files:

diff serverold.txt servernew.txt

©2009 Racker Hacker. All Rights Reserved.

Once I switched the TV back over to the Mini, I saw that gnome-keyring had popped up and it was asking for my password. I entered it, and the Mini joined the wireless network. Each time I rebooted, I had to go through this procedure (which is annoying to do with a HTPC that is across the room). I found a pretty fancy solution, but it looked a little complicated for my setup.

Here’s how I did it in a simpler way in Ubuntu 8.10:

• Click Applications > Accessories > Passwords and Encryption Keys
• Click Edit > Preferences
• Click your keyring name (usually default)
• Click Change Unlock Password
• Enter your current password in the top box, but leave the bottom two boxes blank
• Click OK
• Click Use unsafe storage when you are prompted
• Click Close

If you reboot your machine, it should not ask for a password for your keyring any longer. This allowed my system to log into my wireless network automatically.

WHOA THERE: Since the only password being stored on the device is my WPA2 password, I’m not concerned about the security of the keyring. If you’re doing this on a laptop or desktop that other people use, I would highly recommend not following these steps. All of your passwords and keys will be stored unencrypted.

©2009 Racker Hacker. All Rights Reserved.

I started a new customer service blog with a different approach. The blog will cover different situations where companies often make errors while handling customer service issues.

If you have suggestions for a post, please let me know!

©2009 Racker Hacker. All Rights Reserved.

I’ve been a bit slow on new posts lately for two main reasons. First, I’ve been working hard with my new position at Slicehost, (I’m still with Rackspace). Also, I’ve thrown out the old code for Boxcheck.com and I’m re-writing it from scratch.

Most of you probably know that the first version of Boxcheck threw out some odd errors for some tests and caused confusion for those that depended on the results. The code is being overhauled to handle errors and special cases much more efficiently. I’m implementing multiple technologies on the back-end to make the interface simpler and easier to use. There may also be an API that folks can script against – but I haven’t gotten that far yet.

In short – thanks for all of your e-mails! Boxcheck isn’t going away at all. It’s going to be back soon! If you have any questions or suggestions, feel free to give me a shout.

©2009 Racker Hacker. All Rights Reserved.

sudo vim /etc/apt/sources.list     [change 'etch' -> 'lenny']
sudo aptitude update
sudo aptitude install apt dpkg aptitude
sudo aptitude full-upgrade

©2009 Racker Hacker. All Rights Reserved.

@redhaterrata – Red Hat Errata Updates
@ubuntusecurity – Ubuntu Security Notifications
@debiansecurity – Debian Security Advisories

As always, you can let me know if you have any suggestions for improvements, or other bots you’d like to see.

©2009 Racker Hacker. All Rights Reserved.

While it won’t include all situations, it will give ideas to those who are just starting down the path of MySQL optimization. If you find any corrections or if you have suggestions for improvements, please let me know!

UPDATE: I’ve received a lot of positive feedback via twitter and e-mail about the flow chart idea. I’ll try to work on some more charts soon and get them onto the site. Let me know if there are specific topics that you’re interested in.

©2009 Racker Hacker. All Rights Reserved.

As we know, sometimes servers misbehave due to things put them through, and you can quickly end up in a situation where things are going badly. I’m talking about the type of situation where you’re connected via SSH to a server that controls phone lines for five million people and it sits in a tiny building 400 miles away from the nearest human being. We’re talking bad. If you issue a plain reboot command, it might not even make it that far. Once SSH stops running, you’re going to be out of luck.

If you find yourself in this situation (and I hope you won’t!), you have some options to get your way with a misbehaving server remotely. You can force an immediate reboot with the following:

echo 1 > /proc/sys/kernel/sysrq 
echo b > /proc/sysrq-trigger

WHOA THERE! This is pretty much the same as pressing the reset button on the server (if equipped). No daemons will be shut down gracefully, no filesystem sync will occur, and you may get the wrath of a fsck (or worse, a non-booting server) upon reboot. To do things a little more carefully, read on.

These are called magic commands, and they’re pretty much synonymous with holding down Alt-SysRq and another key on older keyboards. Dropping 1 into /proc/sys/kernel/sysrq tells the kernel that you want to enable SysRq access (it’s usually disabled). The second command is equivalent to pressing Alt-SysRq-b on a QWERTY keyboard.

There’s a better way of rebooting a misbehaving server that Wikipedia shows with the mnemonic “Reboot Even If System Utterly Broken”:

unRaw      (take control of keyboard back from X),  
 tErminate (send SIGTERM to all processes),
 kIll      (send SIGKILL to all processes), 
  Sync     (flush data to disk),
  Unmount  (remount all filesystems read-only),
reBoot.

I can’t vouch for this actually working, but I’m interested to try it. UPDATE: I’ve been told that doing this series of commands with ReiserFS is a very bad idea.

If you want to shut the machine down entirely (please think about it before using this on a remote system):

echo 1 > /proc/sys/kernel/sysrq 
echo o > /proc/sysrq-trigger

If you want to keep SysRq enabled all the time, you can do that with an entry in your server’s sysctl.conf:

kernel.sysrq = 1

©2009 Racker Hacker. All Rights Reserved.

root@storageserver:~# lspci | grep ATA
00:08.0 IDE interface: nVidia Corporation MCP61 SATA Controller (rev a2)
00:08.1 IDE interface: nVidia Corporation MCP61 SATA Controller (rev a2)
03:00.0 Mass storage controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)

There are two primary drives connected to the onboard controller and four connected to the controller card. One of the primary drives on the onboard controller contains the operating system (Ubuntu, in this case), while the other drive is blank.

When the system booted, the sata_sil24 driver for the add-on card always loaded before the sata_nv drivers for the onboard storage controller:

kernel: [    4.125598] sata_sil24 0000:03:00.0: version 1.1
kernel: [    4.126102] sata_sil24 0000:03:00.0: PCI INT A -> Link[APC6] -> GSI 16 (level, low) -> IRQ 16
kernel: [    4.126161] sata_sil24 0000:03:00.0: setting latency timer to 64
kernel: [    4.129472] scsi0 : sata_sil24
kernel: [    4.129635] scsi1 : sata_sil24
kernel: [    8.293762] sata_nv 0000:00:08.0: version 3.5
kernel: [    8.293779] sata_nv 0000:00:08.0: PCI INT A -> Link[APSI] -> GSI 20 (level, low) -> IRQ 20
kernel: [    8.293829] sata_nv 0000:00:08.0: setting latency timer to 64
kernel: [    8.296764] scsi2 : sata_nv
kernel: [    8.296905] scsi3 : sata_nv
kernel: [    9.285034] sata_nv 0000:00:08.1: PCI INT B -> Link[APSJ] -> GSI 21 (level, low) -> IRQ 21
kernel: [    9.285074] sata_nv 0000:00:08.1: setting latency timer to 64
kernel: [    9.285161] scsi4 : sata_nv
kernel: [    9.286015] scsi5 : sata_nv

After specifying an explicit order in /etc/modules and /etc/modprobe.conf, I wasn’t able to see any changes. The sata_sil24 driver still loaded before the onboard sata_nv driver. Luckily, a very wise person on Twitter gave me a strategy that worked just fine.

I added sata_sil24 to the bottom of my /etc/modprobe.d/blacklist file first. Then, in /etc/modules, I listed sata_nv first, followed by sata_sil24. When the system booted, I got the result that I wanted:

kernel: [    3.982909] sata_nv 0000:00:08.0: version 3.5
kernel: [    3.982931] sata_nv 0000:00:08.0: PCI INT A -> Link[APSI] -> GSI 20 (level, low) -> IRQ 20
kernel: [    3.982993] sata_nv 0000:00:08.0: setting latency timer to 64
kernel: [    3.984497] scsi0 : sata_nv
kernel: [    3.986013] scsi1 : sata_nv
kernel: [    4.971755] sata_nv 0000:00:08.1: PCI INT B -> Link[APSJ] -> GSI 21 (level, low) -> IRQ 21
kernel: [    4.971799] sata_nv 0000:00:08.1: setting latency timer to 64
kernel: [    4.973153] scsi2 : sata_nv
kernel: [    4.974031] scsi3 : sata_nv
kernel: [   15.988862] sata_sil24 0000:03:00.0: version 1.1
kernel: [   15.989454] sata_sil24 0000:03:00.0: PCI INT A -> Link[APC6] -> GSI 16 (level, low) -> IRQ 16
kernel: [   15.989511] sata_sil24 0000:03:00.0: setting latency timer to 64
kernel: [   15.990201] scsi6 : sata_sil24
kernel: [   15.991523] scsi7 : sata_sil24

The sata_nv driver is loading first, and Ubuntu boots off of it without an issue. The sata_sil24 driver loads next so that the drives connected to the card show up lower in the boot order.

Many thanks to @Twirrim on Twitter for the suggestion!

©2009 Racker Hacker. All Rights Reserved.

If you find yourself needing to write a Rails application without a database, just do the following:

For Rails 1.0 and up:
config/environment.rb:

config.frameworks -= [ :active_record ]

test/test_helper.rb

class Test::Unit::TestCase
  self.use_transactional_fixtures = false
  self.use_instantiated_fixtures  = false
  def load_fixtures
  end
end

For Rails 2.1 and up: Comment out both of the lines that begin with ActiveRecord::Base in config/initializers/new_rails_defaults.rb:

if defined?(ActiveRecord)
  # Include Active Record class name as root for JSON serialized output.
  # ActiveRecord::Base.include_root_in_json = true
 
  # Store the full class name (including module namespace) in STI type column.
  # ActiveRecord::Base.store_full_sti_class = true
end

For more details, review the full article on rubyonrails.org.

©2009 Racker Hacker. All Rights Reserved.

I’ve received a lot of questions lately about Slicehost and how the service works. If you want to know more, check out the “Getting started with Slicehost” screencast that I’ve created. It goes over the following steps:

• Signing up for a Slicehost account
• Provisioning a new slice
• Features and functions of the Slicemanager
• Setting up a LAMP stack on your new slice

The video is 53.2MB in size, and the total running time is 11 minutes and 39 seconds. It’s in Quicktime format (H264 video with AAC audio), but it should play through VLC if you don’t have Quicktime installed. It’s being served through Cloud Files via the Limelight CDN.

Download the screencast here (53.2MB). (Right click and save to your computer for viewing later.)

The screencast was recorded with ScreenFlow and the post processing was done in iMovie.

©2009 Racker Hacker. All Rights Reserved.

To do this, simply bring up a CPAN shell:

perl -MCPAN -e shell

Run these two commands in the CPAN shell:

o conf prerequisites_policy follow
o conf commit

Now, exit the CPAN shell, start the CPAN shell, and try to install a module that you need. All dependencies will be automatically confirmed, downloaded and installed.

The first line sets your dependency policy to follow rather than ask (the default). The second line tells CPAN to write the changes to your user’s CPAN configuration file to make them permanent.

A big thanks goes out to Lee Hambley for the fix.

WARNING: There are some occasions where you would not want to install dependencies from CPAN. Examples of these situations are when your operating system’s package manager (yum, up2date, apt-get, aptitude, etc) has installed Perl modules in an alternative location or when you have manually installed modules in a non-standard way. I’m a Red Hat guy, and these problems rarely arise on Red Hat/Fedora systems, but your mileage may vary.

©2009 Racker Hacker. All Rights Reserved.

Most of the edits aren’t supported by Wordpress, so I ventured into the source code for my plugins as well as my theme and adjusted the links to point to the mirrored files on the Cloud Files service.

©2009 Racker Hacker. All Rights Reserved.

Call to undefined method wpdb::has_cap() in schema.php on line 22

Even though I followed the instructions on the Wordpress site, and I disabled all of my plugins, the error kept appearing. The only parts of the site that remained unchanged after the upgrade were the wp-config.php page and the wp-content directory. I merged the changes in wp-config.php, but the error was still present.

In frustration, I renamed the wp-content directory to a temporary name and uploaded the wp-content provided from the Wordpress tarball. The upgrade completed without a problem! I deleted the new wp-content directory, put the old wp-content directory back in place, and Wordpress sprang to life!

It’s certainly possible that a Wordpress plugin blocked the upgrade process even though it was disabled in the database.

©2009 Racker Hacker. All Rights Reserved.