There has been quite a bit of interest in GlusterFS as of late and I've been inundated with questions from coworkers, other system administrators and developers. Most folks want to know about its reliability and performance in demanding production environments. I'll try to do my best to cover the big points in this post.

First off, here's now I'm using it in production: I have two web nodes that keep content in sync for various web sites. They each run a GlusterFS server instance and they also mount their GlusterFS share. I'm using the replicate translator to keep both web nodes in sync with client side replication.

Here are my impressions after a month:

I/O speed is often tied heavily to network throughput
This one may seem obvious, but it's not always true in all environments. If you deal with a lot of small files like I do, a 40mbit/sec link between the Xen guests is plenty. Adding extra throughput didn't add any performance to my servers. However, if you wrangle large files on your servers regularly, you may want to consider higher throughput links between your servers. I was able to push just under 900mbit/sec by using dd to create a large file within a GlusterFS mount.

Network and I/O latency are big factors for small file performance
If you have a busy network and the latency creeps up from time to time, you'll find that your small file performance will drop significantly (especially with the replicate translator). Without getting too nerdy (you're welcome to read the technical document on replication), replication is an intensive process. When a file is accessed, the client goes around to each server node to ensure that it not only has a copy of the file being read, but that it has the correct copy. If a server didn't save a copy of a file (due to disk failure or the server being offline when the file was written), it has to be synced across the network from one of the good nodes.

When you write files on replicated servers, the client has to roll through the same process first. Once that's done, it has to lock the file, write to the change log, then do the write operation, drop the change log entries, and then unlock the file. All of those operations must be done on all of the servers. High latency networks will wreak havoc on this process and cause it to take longer than it should.

It's quite obvious that if you have a fast, low-latency network between your servers, slow disks can still be a problem. If the client is waiting on the server nodes' disks to write data, the read and write performance will suffer. I've tested this in environments with fast networks and very busy RAID arrays. Even if the network was very underutilized, slow disks could cut performance drastically.

Monitoring GlusterFS isn't easy
When the client has communication problems with the server nodes, some weird things can happen. I've seen situations where the client loses connections to the servers (see the next section on reliability) and the client mount simply hangs. In other situations, the client has been knocked offline entirely and the process is missing from the process tree by the time I logged in. Your monitoring will need to ensure that the mount is active and is responding in a timely fashion.

There's a handy script which allows you to monitor GlusterFS mounts via nagios that Ian Rogers put together. Also, you can get some historical data with acrollet's munin-glusterfs plugin.

GlusterFS 3.x is pretty reliable
When I first started working with GlusterFS, I was using a version from the 2.x tree. The Fedora package maintainer hadn't updated the package in quite some time, but I figured it should work well enough for my needs. I found that the small file performance was lacking and the nodes often had communication issues when many files were being accessed or written simultaneously. This improved when I built my own RPMs of 3.0.4 (and later 3.0.5) and began using those instead.

I did some failure testing by hard cycling the server and client nodes and found some interesting results. First off, abruptly pulling clients had no effects on the other clients or the server nodes. The connection eventually timed out and the servers logged the timeout as expected.

Abruptly pulling servers led to some mixed results. In the 2.x branch, I saw client hangs and timeouts when I abruptly removed a server. This appears to be mostly corrected in the 3.x branch. If you're using replicate, it's important to keep in mind that the first server volume listed in your client's volume file is the one that will be coordinating the file and directory locking. Should that one fall offline quickly, you'll see a hiccup in performance for a brief moment and the next server will be used for coordinating the locking. When your original server comes back up, the locking coordination will shift back.

Conclusion
I'm really impressed with how much GlusterFS can do with the simplicity of how it operates. Sure, you can get better performance and more features (sometimes) from something like Lustre or GFS2, but the amount of work required to stand up that kind of cluster isn't trivial. GlusterFS really only requires that your kernel have FUSE support (it's been in mainline kernels since 2.6.14).

There are some things that GlusterFS really needs in order to succeed:

• Documentation - The current documentation is often out of date and confusing. I've even found instances where the documentation contradicts itself. While there are some good technical documents about the design of some translators, they really ought to do some more work there.
• Statistics gathering - It's very difficult to find out what GlusterFS is doing and where it can be optimized. Profiling your environment to find your bottlenecks is nearly impossible with the 2.x and 3.x branches. It doesn't make it easier when some of the performance translators actually decrease performance.
• Community involvement - This ties back into the documentation part a little, but it would be nice to see more participation from Gluster employees on IRC and via the mailing lists. They're a little better with mailing list responses than other companies I've seen, but there is still room for improvement.

If you're considering GlusterFS for your servers but you still have more questions, feel free to leave a comment or find me on Freenode (I'm 'rackerhacker').

One month with GlusterFS in production is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

However, when you load up quite a few plugins or a heavy theme, the performance will dip due to the increased number of stylesheets, javascript files, and images. You can compress and combine the stylesheets and javascript to decrease load times, but this may not get the performance to a level you like.

I was in this situation and I found a great solution: the W3 Total Cache plugin and the MaxCDN service.

To get started, visit MaxCDN's site and set up an account. Their current promotion gives you 1TB of CDN bandwidth for one year for $10 (regularly $99). Once you sign up, do the following:

• Click Manage Zones
• Click Create pull zone

At this point, you'll see a list of form fields to complete:

• Enter an alias for the pull zone name
• The origin server URL is the URL that's normally used to access your site (i.e. rackerhacker.com)
• The custom CDN domain is the URL you want to use for your CDN (i.e. cdn.rackerhacker.com)
• The label can be anything you'd like to use to remember which zone is which
• Enabling compression is generally a good idea

Once you save the zone, MaxCDN will give you a new domain name. You'll want to create a CNAME record that points from your CDN URL (for me, that's cdn.rackerhacker.com) to the really long URL that MaxCDN provides.

STOP HERE: Ensure that all of your DNS servers are replying with the CNAME record before you continue with the W3 Total Cache installation and CDN setup. If you proceed without waiting for that, some of your blog's visitors will get errors when they try to load content via your CDN domain.

You're ready for W3 Total Cache now. Install the plugin within your WordPress installation and activate it. Hop into the settings for the plugin and make these adjustments:

• Enable Page Caching and set it to Disk (enhanced)
• Enable Minify and set it to Disk
• Enable Database Caching and set it to Disk
• Leave the CDN disabled for now, but flip the CDN Type to Origin Pull (Mirror)
• Press Save changes

Click CDN Settings at the top of the page and configure the CDN:

• Enter your CDN domain (for me, it's cdn.rackerhacker.com) in the top form field
• Leave the other options as they are by default and click Save changes

W3 Total Cache should prompt you to clear out your page cache, and that would be recommended at this step. If you fully reload your blog's main page in your browser (may require you to hold SHIFT while you click reload/refresh) and check the page source, you should see your CDN URL appear for some of the javascript or CSS files.

You may discover that some CSS files, stylesheets, or images aren't being loaded via the CDN automatically. Luckily, that's an easy fix. Under the Minify Settings section of the W3 Total Cache plugin settings, scroll to the very bottom. Add in your javascript or CSS files via the form fields at the bottom and the plugin should handle the minifying (is that even a word?) and the CDN URL rewriting for you.

Further reading:

• MaxCDN WordPress Integration Overview
• W3 Total Cache plugin page at wordpress.org
• W3 Total Cache main site

WordPress + W3 Total Cache + MaxCDN How-To is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Parameter 1 to update_comment_type_cache() expected to be a reference, 
value given in wp-includes/plugin.php on line 166

The problem wasn't in a plugin, actually. It was within my theme's (R755-light) functions.php:

function update_comment_type_cache(&$queried_posts) {

The temporary fix is to remove the & from that line so it looks like this:

function update_comment_type_cache($queried_posts) {

After clearing out the WP Super Cache, the page was loading properly again. It turns out that the function actually calculates how many comments are available for a given post, so that functionality is working properly right now. A few theme authors are already releasing new versions to fix this bug, but my theme's author has not.

The credit for the fix goes to someone in the WordPress forums.

WordPress and PHP 5.3.x: update_comment_type_cache() expected to be a reference is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The automatic upgrade via the admin interface actually worked just fine for me. Of course, I backed up my database and files first, just to be sure.

Upgraded to WordPress 2.9 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Also, I've dropped the Adsimple theme I was using, and I'm now using a slightly modified Dojo theme. It's a little easier on the eyes, but it's still lightweight enough to be fast on mobile broadband connections.

Let me know what you think!

Upgraded to WordPress 2.8.6 with some theme changes is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Then I stumbled upon the WordPress documentation that shows you how to require SSL with no plugins at all. If you're using WordPress 2.6+, you can use these super-simple instructions:

Require encryption just for the /wp-admin/ login, but leave the rest of the administrative area on HTTP:

# Add this line to wp-config.php
define('FORCE_SSL_LOGIN', true);

To encrypt the login and the entire administrative area:

# Add this line to wp-config.php
define('FORCE_SSL_ADMIN', true);

Of course, for this to work, you'll need virtual hosts on ports 80 and 443 for your blog. Also, you'll need an SSL certificate for your blog. You can snag one from a reputable provider or make your own.

Requiring SSL encryption for WordPress administration is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Upgraded WordPress to 2.8.2 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Fatal error: Call to undefined method wpdb::has_cap() in 
/path/to/wordpress/wp-admin/includes/schema.php on line 22

I was perplexed at the error, so I restored from a backup and began upgrading manually. The manual upgrades have always worked well for me in the past, so I figured this would probably fix the problem. After the upgrade, I went to /wp-admin/upgrade.php and saw:

Fatal error: Call to undefined method wpdb::has_cap() in 
/path/to/wordpress/wp-admin/includes/schema.php on line 22

What the heck is going on? I restored from a backup, tried the manual upgrade again, and it still failed. I took a look at the lines causing the problem in schema.php:

if ( $wpdb->has_cap( 'collation' ) ) {
	if ( ! empty($wpdb->charset) )
		$charset_collate = "DEFAULT CHARACTER SET $wpdb->charset";
	if ( ! empty($wpdb->collate) )
		$charset_collate .= " COLLATE $wpdb->collate";
}

I figured I could comment out the if statement and probably still be safe:

// if ( $wpdb->has_cap( 'collation' ) ) {
	if ( ! empty($wpdb->charset) )
		$charset_collate = "DEFAULT CHARACTER SET $wpdb->charset";
	if ( ! empty($wpdb->collate) )
		$charset_collate .= " COLLATE $wpdb->collate";
// }

Success? I could make it through the upgrade.php part fine at this point, but whenever I tried to add a tag to a post, it wasn't saving to the database. I caught this error in my apache logs:

[Fri Jun 12 23:45:03 2009] [error] [client 72.183.200.144] WordPress database error Duplicate entry 'debian' for key 'slug' for query INSERT INTO wp_terms (`name`,`slug`,`term_group`) VALUES ('debian','debian','0') made by wp_insert_term, referer: http://rackerhacker.com/wp-admin/post.php?action=edit&post=877

Frustration quickly ensued. I moved my /wp-content/ folder out of the way and replaced it with the standard WordPress stuff, but that didn't help. I moved plugins out of the way, one by one, but that didn't fix it either. Then I spotted a strange file sitting in /wp-content/ called db.php. When I opened it, I found a lot of database setup classes for mysqli.

I renamed it to db.pleasedonteverrunthisphp and I was able to save tags properly. So far, I haven't found any issues after I made chat change.

Does anyone know where that file might have come from? I don't remember adding it myself, so I'm wondering if it was ever packaged with a WordPress plugin or a WordPress installation. I hope this helps someone else!

Ugly upgrade path from WordPress 2.7.1 to 2.8 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Most of the edits aren't supported by WordPress, so I ventured into the source code for my plugins as well as my theme and adjusted the links to point to the mirrored files on the Cloud Files service.

Moved static content to Cloud Files is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Call to undefined method wpdb::has_cap() in schema.php on line 22

Even though I followed the instructions on the WordPress site, and I disabled all of my plugins, the error kept appearing. The only parts of the site that remained unchanged after the upgrade were the wp-config.php page and the wp-content directory. I merged the changes in wp-config.php, but the error was still present.

In frustration, I renamed the wp-content directory to a temporary name and uploaded the wp-content provided from the WordPress tarball. The upgrade completed without a problem! I deleted the new wp-content directory, put the old wp-content directory back in place, and WordPress sprang to life!

It's certainly possible that a WordPress plugin blocked the upgrade process even though it was disabled in the database.

Upgraded to WordPress 2.7 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Upgraded to WordPress 2.6.5 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Another improvement to the site is that it is now hosted on Mosso. This should allow the site to handle a lot more traffic and be more redundant than the original configuration. Eventually, I'll move this blog to Mosso® as well.

What is PleskHacker? It's part of an effort to provide more detailed documentation of Plesk for system administrators. PleskHacker has documentation covering one third of the Plesk database so far. It includes definitions for each column and table, but it also shows relationships between tables so you can create your own JOIN queries with ease.

Working hard on PleskHacker is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Leave a comment and let me know what you think of the changes. Have a great weekend!

New look is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Slide on over to the WordPress site and download it now.

Upgraded to WordPress 2.6.2 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Upgraded to WordPress 2.6.1 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First post using the WordPress iPhone application is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

WordPress 2.6 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

WordPress upgraded to 2.3.3 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

This blog runs on a server with dual Xeon Woodcrest CPU's, 64-bit CentOS 4.5 and a 100mbit network connection. Here's the first test with WP-Cache turned off:

$ http_load -parallel 10 -seconds 30 urltocheck.txt
346 fetches, 10 max parallel, 1.78616e+07 bytes, in 30 seconds
51623.2 mean bytes/connection
11.5333 fetches/sec, 595387 bytes/sec
msecs/connect: 15.1661 mean, 16.97 max, 14.922 min
msecs/first-response: 445.984 mean, 2328.82 max, 189.62 min
HTTP response codes:
code 200 -- 346

346 fetches in 30 seconds is not a very exciting performance number for me. That's just over 10 fetches per second, and on a busy day, I sometimes reach that number. Also, while this test ran, the server's CPU usage was extremely high and over 80% of all four cores were in use. The iowait was about 20% across the board.

I decided to turn on WP-Cache and give it another go with the same test:

$ http_load -parallel 10 -seconds 30 urltocheck.txt
3482 fetches, 10 max parallel, 1.79671e+08 bytes, in 30 seconds
51600 mean bytes/connection
116.067 fetches/sec, 5.98904e+06 bytes/sec
msecs/connect: 15.2259 mean, 18.257 max, 14.891 min
msecs/first-response: 20.7297 mean, 69.39 max, 18.861 min
HTTP response codes:
code 200 -- 3482

Wow, that's a 10-fold improvement, and I can handle over 100 requests per second with 10 parallel requests. Also, the iowait dropped to 5%, and overall CPU usage remained under 8%.

I kicked it up to 20 parallel connections and tried again:

$ http_load -parallel 20 -seconds 30 urltocheck.txt
5817 fetches, 20 max parallel, 3.02176e+08 bytes, in 30 seconds
51947 mean bytes/connection
193.9 fetches/sec, 1.00725e+07 bytes/sec
msecs/connect: 17.9175 mean, 30.831 max, 14.911 min
msecs/first-response: 24.5352 mean, 97.475 max, 18.978 min
HTTP response codes:
code 200 -- 5817

Almost 194 connections served per second! Also, the CPU usage was only at about 14% during the duration of the test.

I decided to tempt fate and see if I could blow the roof off the test with 50 parallel connections:

$ http_load -parallel 50 -seconds 30 urltocheck.txt
5794 fetches, 50 max parallel, 2.99718e+08 bytes, in 30 seconds
51729 mean bytes/connection
193.133 fetches/sec, 9.99059e+06 bytes/sec
msecs/connect: 43.286 mean, 63.878 max, 14.942 min
msecs/first-response: 68.967 mean, 202.854 max, 20.014 min
HTTP response codes:
code 200 -- 5794

The performance suffered a bit, but the server was still pumping out almost 200 connections per second, and I'm okay with that. Well, unless anyone has a spare Cisco 11501 laying around that I could have. And, of course, one additional server.

Just as a sidenote, I installed Zend Optimizer v3.3 on the server and performance actually dropped by 1%-3% for each test. I found that a bit surprising.

I used http_load to perform the benchmarks after I found it on Caleb's blog.

Why you should use caching for WordPress blogs is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.