You can't talk about cloud hosting without defining it first. When I think of "cloud", these are the things that come to mind:

• quickly add/remove resources with little or no lead time
• hosting platforms that allow for quick provisioning of highly available systems
• self-service adjustment of tangible and intangible resources that normally require human intervention

That list may seem a bit vague at first, but try to let it sink in just a bit. Hosting applications in a "cloud" shouldn't mean that you must have a virtual instance running on Xen, KVM or VMWare, and it shouldn't mean that you must have an account with Rackspace Cloud, Amazon EC2, or Microsoft Azure. It means that your hosting operations are highly automated and you can rapidly allocate and deallocate resources for the requirements of your current projects.

Consider this: a customer of a traditional dedicated hosting provider decides to take their applications and host them on one VPS at a leading commercial provider. That provider allows the customer to spin up new VM's in a matter of minutes and re-image the VM's whenever they like. Is that cloud hosting? I'd say yes -- even if it's one single virtual instance. That customer has moved from a hosting system with manual interventions and extended lead times to a system where they have instant control over their resources.

It's not possible to talk about what cloud is without talking about what it isn't.

• Cloud is not infinitely scalable. If any provider ever claims that their solution is "infinitely scalable", you should be skeptical. Regardless of the provider, everyone eventually runs out of datacenter space, servers, network bandwidth, or power. (If you know of a provider that is infinitely scalable, please let me know as I'd love to see their facilities and review their supply chain.)
• Cloud isn't right for everybody. Some applications have demands that cloud hosting might not be able to meet (yet). If an application depends on proprietary hardware that is difficult to virtualize or rapidly allocate, cloud hosting is probably not the answer for that particular application.
• Cloud doesn't mean VPS. VPS doesn't mean cloud. As I said before, having a virtual private server environment is not a pre-requisite for cloud hosting. Also, not all VPS solutions fit my definition of cloud as they don't allow for rapid deployments and resource adjustments.

It's important to remember that cloud hosting is a marketing term. As for the technology of cloud, it's what you make of it. You should be looking to reduce costs, solidify availability and increase performance every day. If the ideals of cloud hosting help you do that, it might be the right option for you.

©2010 Racker Hacker. All Rights Reserved.

The guide: Redundant Cloud Hosting Guide

As always, if you find anything in the guide that needs improvement, I'm all ears.

©2010 Racker Hacker. All Rights Reserved.

Quite a few different sites argue that the default GlusterFS performance translator configuration from glusterfs-volgen doesn't allow for good performance. You can find other sites which say you should stick with the defaults that come from the script. I decided to run some simple tests to see which was true in my environment.

Here's the testbed:

• GlusterFS 3.0.5 running on RHEL 5.4 Xen guests with ext3 filesystems
• one GlusterFS client and two GlusterFS servers are running in separate Xen guests
• cluster/replicate translator is being used to keep the servers in sync
• the instances are served by a gigabit network

It's about time for some pretty graphs, isn't it?

The test run on the left used default stock client and server volume files as they come from glusterfs-volgen. The test run on the right used a client volume file with no performance translators (the server volume file was untouched). Between each test run, the GlusterFS mount was unmounted and remounted. I repeated this process four times (for a total of five runs) and averaged the data.

You'll have to forgive the color mismatches and the lack of labeling on the legend (that's KB/sec transferred) as I'm far from an Excel expert.

The graphs show that running without any translators at all will drastically hinder read caching in GlusterFS -- exactly as I expected. Without any translators, the performance is very even across the board. Since my instances had 256MB of RAM each, their iocache translator was limited to about 51MB of cache. That's reflected in the graph on the left -- look for the vertical red/blue divider between the 32MB and 64MB file sizes. I'll be playing around with that value soon to see how it can improve performance for large and small files.

Keep in mind that this test was very unscientific and your results may vary depending on your configuration. While I hope to have more detailed benchmarks soon, this should help some of the folks who have been asking for something basic and easy to understand.

©2010 Racker Hacker. All Rights Reserved.

There has been quite a bit of interest in GlusterFS as of late and I've been inundated with questions from coworkers, other system administrators and developers. Most folks want to know about its reliability and performance in demanding production environments. I'll try to do my best to cover the big points in this post.

First off, here's now I'm using it in production: I have two web nodes that keep content in sync for various web sites. They each run a GlusterFS server instance and they also mount their GlusterFS share. I'm using the replicate translator to keep both web nodes in sync with client side replication.

Here are my impressions after a month:

I/O speed is often tied heavily to network throughput
This one may seem obvious, but it's not always true in all environments. If you deal with a lot of small files like I do, a 40mbit/sec link between the Xen guests is plenty. Adding extra throughput didn't add any performance to my servers. However, if you wrangle large files on your servers regularly, you may want to consider higher throughput links between your servers. I was able to push just under 900mbit/sec by using dd to create a large file within a GlusterFS mount.

Network and I/O latency are big factors for small file performance
If you have a busy network and the latency creeps up from time to time, you'll find that your small file performance will drop significantly (especially with the replicate translator). Without getting too nerdy (you're welcome to read the technical document on replication), replication is an intensive process. When a file is accessed, the client goes around to each server node to ensure that it not only has a copy of the file being read, but that it has the correct copy. If a server didn't save a copy of a file (due to disk failure or the server being offline when the file was written), it has to be synced across the network from one of the good nodes.

When you write files on replicated servers, the client has to roll through the same process first. Once that's done, it has to lock the file, write to the change log, then do the write operation, drop the change log entries, and then unlock the file. All of those operations must be done on all of the servers. High latency networks will wreak havoc on this process and cause it to take longer than it should.

It's quite obvious that if you have a fast, low-latency network between your servers, slow disks can still be a problem. If the client is waiting on the server nodes' disks to write data, the read and write performance will suffer. I've tested this in environments with fast networks and very busy RAID arrays. Even if the network was very underutilized, slow disks could cut performance drastically.

Monitoring GlusterFS isn't easy
When the client has communication problems with the server nodes, some weird things can happen. I've seen situations where the client loses connections to the servers (see the next section on reliability) and the client mount simply hangs. In other situations, the client has been knocked offline entirely and the process is missing from the process tree by the time I logged in. Your monitoring will need to ensure that the mount is active and is responding in a timely fashion.

There's a handy script which allows you to monitor GlusterFS mounts via nagios that Ian Rogers put together. Also, you can get some historical data with acrollet's munin-glusterfs plugin.

GlusterFS 3.x is pretty reliable
When I first started working with GlusterFS, I was using a version from the 2.x tree. The Fedora package maintainer hadn't updated the package in quite some time, but I figured it should work well enough for my needs. I found that the small file performance was lacking and the nodes often had communication issues when many files were being accessed or written simultaneously. This improved when I built my own RPMs of 3.0.4 (and later 3.0.5) and began using those instead.

I did some failure testing by hard cycling the server and client nodes and found some interesting results. First off, abruptly pulling clients had no effects on the other clients or the server nodes. The connection eventually timed out and the servers logged the timeout as expected.

Abruptly pulling servers led to some mixed results. In the 2.x branch, I saw client hangs and timeouts when I abruptly removed a server. This appears to be mostly corrected in the 3.x branch. If you're using replicate, it's important to keep in mind that the first server volume listed in your client's volume file is the one that will be coordinating the file and directory locking. Should that one fall offline quickly, you'll see a hiccup in performance for a brief moment and the next server will be used for coordinating the locking. When your original server comes back up, the locking coordination will shift back.

Conclusion
I'm really impressed with how much GlusterFS can do with the simplicity of how it operates. Sure, you can get better performance and more features (sometimes) from something like Lustre or GFS2, but the amount of work required to stand up that kind of cluster isn't trivial. GlusterFS really only requires that your kernel have FUSE support (it's been in mainline kernels since 2.6.14).

There are some things that GlusterFS really needs in order to succeed:

• Documentation - The current documentation is often out of date and confusing. I've even found instances where the documentation contradicts itself. While there are some good technical documents about the design of some translators, they really ought to do some more work there.
• Statistics gathering - It's very difficult to find out what GlusterFS is doing and where it can be optimized. Profiling your environment to find your bottlenecks is nearly impossible with the 2.x and 3.x branches. It doesn't make it easier when some of the performance translators actually decrease performance.
• Community involvement - This ties back into the documentation part a little, but it would be nice to see more participation from Gluster employees on IRC and via the mailing lists. They're a little better with mailing list responses than other companies I've seen, but there is still room for improvement.

If you're considering GlusterFS for your servers but you still have more questions, feel free to leave a comment or find me on Freenode (I'm 'rackerhacker').

©2010 Racker Hacker. All Rights Reserved.

To make an iptables rule with a comment, simply add on the following arguments to the rule:

-m comment --comment "limit ssh access"

Depending on your distribution, you may need to load the ipt_comment or xt_comment modules into your running kernel first.

A full iptables rule to limit ssh access would look something like this:

iptables -A INPUT -j DROP -p tcp --dport 22 -m comment --comment "limit ssh access"

©2010 Racker Hacker. All Rights Reserved.

Evan Michael was born earlier this month and although he's had a bit of a rough start, he's proving that he's a strong fellow. I'll try to pick up where I left off with my posts when we have him settled.

©2010 Racker Hacker. All Rights Reserved.

Typical configuration for a
proxy-type load balancer

• client makes a request
• load balancer receives the request
• load balancer sends request to a web node
• the web server sends content back to the load balancer
• the load balancer responds to the client

If you're not familiar with load balancing, here's an analogy. Consider a fast food restaurant. When you walk up to the counter and place an order, you're asking the person at the counter (the load balancer) for a hamburger. The person at the counter is going to submit your order, and then a group of people (web nodes) are going to work on it. Once your hamburger (web request) is ready, your order will be given to the person at the counter and then back to you.

This style of organization can become a problem as your web nodes begin to scale. It requires you to ensure that your load balancers can keep up with the requests and sustain higher transfer rates that come from having more web nodes serving a greater number of requests. Imagine the fast food restaurant where you have one person taking the orders but you have 30 people working on the food. The person at the counter may be able to take orders very quickly, but they may not be able to keep up with the orders coming out of the kitchen.

LVS allows for application servers
to respond to clients directly

• client makes a request
• load balancer receives the request
• load balancer sends request to a web node
• the web server sends the response directly to the client

The key difference is that the load balancer sends the unaltered request to the web server and the web server responds directly to the client. Here's the fast food analogy again. If you ask the person at the counter (the load balancer) for a hamburger, that person is going to take your order and give it to the kitchen staff (the web nodes) to work on it. This time around, the person at the counter is going to advise the kitchen staff that the order needs to go directly to you once it's complete. When your hamburger is ready, a member of the kitchen staff will walk to the counter and give it directly to you.

In the fast food analogy, what are the benefits? As the number of orders and kitchen staff increases, the job of the person at the counter doesn't drastically increase in difficulty. While that person will have to handle more orders and keep tabs on which of the kitchen staff is working on the least amount of orders, they don't have to worry about returning food to customers. Also, the kitchen staff doesn't need to waste time handing orders to the person at the counter. Instead, they can pass these orders directly to the customer that ordered them.

In the world of servers, this is a large benefit. Since the web servers' responses no longer pass through the load balancer, they can spend more time on what they do best -- balancing traffic. This allows for smaller, lower-powered load balancing servers from the beginning. It also allows for increases in web nodes without big changes for the load balancers.

There are three main implementations of LVS to consider:

LVS-DR: Direct Routing
The load balancer receives the request and sends the packet directly to a waiting real server to process. LVS-DR has the best performance, but all of your servers must be on the same network subnet and they have to be able to share the same router (with no other routing devices in between them).

LVS-TUN: Tunneling
This is very similar to the direct routing approach, but the packets are encapsulated and sent directly to the real servers once the load balancer receives them. This removes the restriction that all of the devices must be on the same network. Thanks to encapsulation, you can use this method to load balance between multiple datacenters.

LVS-NAT: Network Address Translation
Using NAT for LVS yields the least performance and scaling of all of the implementation options. In this configuration, the incoming requests are rewritten so that they will be transported correctly in a NAT environment. This puts a bigger burden on the load balancer as it must rewrite the requests quickly while still keeping up with how much work is being done by each web server.

©2010 Racker Hacker. All Rights Reserved.

Well, it hasn't returned, but you can build your own replacement with ruby, a jabber account, and a few gems. While it won't do everything that the original jabber bot did, you can still track tweets mentioning certain terms very quickly.

Here's how to get started:

First, install the tweetstream and xmpp4r-simple gems:

gem install tweetstream xmpp4r-simple

Next, you'll need a jabber account. You'll probably want to make one for the exclusive use of your jabber bot. I chose to make up a quick account at ChatMask for mine.

The last step is to drop a copy of this script on your server:

#!/usr/bin/ruby
require 'rubygems'
require 'tweetstream/client'
require 'tweetstream/hash'
require 'tweetstream/status'
require 'tweetstream/user'
require 'tweetstream/daemon'
require 'xmpp4r-simple'
 
jabber = Jabber::Simple.new('jabberbot@yourjabberserver.com','jabberpassword')
 
tweets = TweetStream::Client.new(twitterusername,twitterpassword)
 
tweets.track('celtics','lakers','finals','nba') do |status, client|
  imtext = "#{status.user.screen_name}: #{status.text} \r\n" 
  imtext += "[http://twitter.com/#{status.user.screen_name}/status/#{status.id}]"
  jabber.deliver("yourjabberusername@yourjabberserver.com",imtext)
end
 
jabber.disconnect

You'll want to be sure to fill in the following:

• your jabber bot's username and password
• the username and password for the twitter account that will monitor the stream
• the search terms you want to track
• the destination jabber account where the messages should be sent

Ensure that your jabber account has authorized the jabber bot's account so that you'll actually receive the messages. Also, Twitter is very strict with their streaming API tracking terms. It's a good idea to review their entire Streaming API documentation to ensure that you're not going to end up having your account temporarily or permanently blacklisted.

Once everything is ready to go, you can just run the script within GNU screen or via nohup. There's still a bit more error checking to do around jabber reconnections, but the script has run non-stop for well over two weeks at a time without a failure.

©2010 Racker Hacker. All Rights Reserved.

The man page offers a very simple description of its features:

Write lines consisting of the sequentially corresponding lines from each FILE, separated by TABs, to standard output.

Here's an example of how it works. Let's say you want to parse some software raid output that looks like this:

# mdadm --brief --verbose --detail /dev/md0
ARRAY /dev/md0 level=raid1 num-devices=2 metadata=00.90 UUID=7bea4601:d5a02f5c:2da69848:3184a367
   devices=/dev/sda1,/dev/sdb1

It would be handy if we had both on one line as that would make it easier to parse with a script. Of course, you can do this with utilities like awk and tr, but paste makes it so much easier:

# mdadm --brief --verbose --detail /dev/md0 | paste - -
ARRAY /dev/md0 level=raid1 num-devices=2 metadata=00.90 UUID=7bea4601:d5a02f5c:2da69848:3184a367	   devices=/dev/sda1,/dev/sdb1

By default, paste uses tabs to separate the lines, but you can use the -d argument to specify any delimiter you like:

# mdadm --brief --verbose --detail /dev/md0 | paste -d"*" - -
ARRAY /dev/md0 level=raid1 num-devices=2 metadata=00.90 UUID=7bea4601:d5a02f5c:2da69848:3184a367*   devices=/dev/sda1,/dev/sdb1

©2010 Racker Hacker. All Rights Reserved.

If you want to go active-active, or if you want multiple nodes active at the same time, you need to use a clustered filesystem like GFS2, OCFS2 or Lustre. These are certainly good options to consider but they're not trivial to implement. They usually rely on additional systems and scripts to provide reliable fencing and STONITH capabilities.

What about the rest of us who want multiple active VM's with simple replicated storage that doesn't require any additional elaborate systems? This is where GlusterFS really shines. GlusterFS can ride on top of whichever filesystem you prefer, and that's a huge win for those who want a simple solution. However, that means that it has to use fuse, and that will limit your performance.

Let's get this thing started!

Consider a situation where you want to run a WordPress blog on two VM's with load balancers out front. You'll probably want to use GlusterFS's replicated volume mode (RAID 1-ish) so that the same files are on both nodes all of the time. To get started, build two small Slicehost slices or Rackspace Cloud Servers. I'll be using Fedora 13 in this example, but the instructions for other distributions should be very similar.

First things first -- be sure to set a new root password and update all of the packages on the system. This should go without saying, but it's important to remember. We can clear out the default iptables ruleset since we will make a customized set later:

# iptables -F
# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:        [  OK  ]

GlusterFS communicates over the network, so we will want to ensure that traffic only moves over the private network between the instances. We will need to add the private IP's and a special hostname for each instance to /etc/hosts on both instances. I'll call mine gluster1 and gluster2:

10.xx.xx.xx gluster1
10.xx.xx.xx gluster2

You're now ready to install the required packages on both instances:

yum install glusterfs-client glusterfs-server glusterfs-common glusterfs-devel

Make the directories for the GlusterFS volumes on each instance:

mkdir -p /export/store1

We're ready to make the configuration files for our storage volumes. Since we want the same files on each instance, we will use the --raid 1 option. This only needs to be run on the first node:

# glusterfs-volgen --name store1 --raid 1 gluster1:/export/store1 gluster2:/export/store1
Generating server volfiles.. for server 'gluster2'
Generating server volfiles.. for server 'gluster1'
Generating client volfiles.. for transport 'tcp'

Once that's done, you'll have four new files:

• booster.fstab - you won't need this file
• gluster1-store1-export.vol - server-side configuration file for the first instance
• gluster2-store1-export.vol - server-side configuration file for the second instance
• store1-tcp.vol - client side configuration file for GlusterFS clients

Copy the gluster1-store1-export.vol file to /etc/glusterfs/glusterfsd.vol on your first instance. Then, copy gluster2-store1-export.vol to /etc/glusterfs/glusterfsd.vol on your second instance. The store1-tcp.vol should be copied to /etc/glusterfs/glusterfs.vol on both instances.

At this point, you're ready to start the GlusterFS servers on each instance:

/etc/init.d/glusterfsd start

You can now mount the GlusterFS volume on both instances:

mkdir -p /mnt/glusterfs
glusterfs /mnt/glusterfs/

You should now be able to see the new GlusterFS volume in both instances:

# df -h /mnt/glusterfs
Filesystem            Size  Used Avail Use% Mounted on
/etc/glusterfs/glusterfs.vol
                      9.4G  831M  8.1G  10% /mnt/glusterfs

As a test, you can create a file on your first instance and verify that your second instance can read the data:

[root@gluster1 ~]# echo "We're testing GlusterFS" > /mnt/glusterfs/test.txt
.....
[root@gluster2 ~]# cat /mnt/glusterfs/test.txt
We're testing GlusterFS

If you remove that file on your second instance, it should disappear from your first instance as well.

Obviously, this is a very simple and basic implementation of GlusterFS. You can increase performance by making dedicated VM's just for serving data and you can adjust the default performance options when you mount a GlusterFS volume. Limiting access to the GlusterFS servers is also a good idea.

If you want to read more, I'd recommend reading the GlusterFS Technical FAQ and the GlusterFS User Guide.

©2010 Racker Hacker. All Rights Reserved.

You may not have realized it, but I covered the most important part of selling a technical product in the first paragraph without even mentioning the word "sell". That was intentional. As a technical person, you have an innate ability to interact with customers without needing to actively sell them the product.

Whenever I meet a customer at a conference, trade show, or some other relatively informal event, I try to keep a few things in mind. I'll share them with you:

Learn why your customers are seeking out your product and what they really need
It's pretty obvious that this step requires more listening than talking. While the customer is explaining what they need but haven't found, try to keep a running tally in your brain of what technologies are important to them so that you can rank your suggestions for them. Don't think about which product will work best for them yet -- just keep keep their general requirements in mind.

This is also a good opportunity to relate to what they've told you. If there's a certain solution that ended up working really well or one that failed miserably, and you're familiar with one of those solutions, tell them briefly about your experiences. This will re-affirm how the customer feels about that solution and it also shows them that you've been in their shoes before. They'll also appreciate that you've been listening to their concerns and looking for ways to relate to their unique situation.

Make thoughtful production suggestions and discuss implementation
Some folks might say this is where the selling starts, but if you're doing it correctly, you've been selling your product and your company the whole time. This is where things can get tricky. Most technical people I've met will try to avoid being pushy when suggesting a product for a customer to use, and that's a good idea.

You need to do three things: pick the right product (or group of products), explain what needs it meets, and briefly cover some example implementations. As a technical person, this is where you really shine. Interpreting the customer's needs and turning it into a mini technical sales pitch is a piece of cake when you know the product well and you've implemented it before.

It's great to give a customer multiple options, but it's a bad idea to overwhelm them. If you find that you're talking a bit too much, there's no harm in offering to talk about details later during a formal meeting. You can say things like these:

• "this product will meet all your needs, but if you want to save a little money, you can use this other product like this."
• "if you combine these two products, you can meet these needs and save some time, but you can just use one and set it up like this..."
• "then later on, if you need to expand, you can start using this product by..."

Think about the customer's future growth
Even if you have products that meet your customer's needs, they're going to be concerned about what's going to happen down the road. What happens when they scale to a level that they can't even comprehend right now? I don't think any customer would expect you to cover all the bases, but try to think of some basic future-proofing for the customer. Even if it might involve a product that your company doesn't sell, just mention it.

Of course, there are some things that you shouldn't do:

• Don't overpromise or push hard about a future product.
• Don't feel obligated to know the answer to every question.
• Don't use words like "infinite", "forever", or "perfect".
• Don't talk about cost constantly.
• Don't force a customer to choose a product, take product literature, or take your contact information.
• Don't make assumptions about the customer's technical level, needs, or purchasing power.
• Don't let it bother you if the customer isn't interested in your product -- it's not personal.

And that's about it. If you follow those three tips and avoid the things you shouldn't do, you'll get the confidence you need to engage the customer and create the beginnings of a relationship with them.

©2010 Racker Hacker. All Rights Reserved.

I formulated a pretty simple script using John Nunemaker's twitter gem that will get it done manually for any scripts you have that read from or update Twitter:

#!/usr/bin/ruby
require 'rubygems'
require 'twitter'
 
# These credentials are specific to your *application* and not your *user*
# Get these credentials from Twitter directly: http://twitter.com/apps
application_token = '[this should be the shorter one]'
application_secret = '[this should be the longer one]'
 
oauth = Twitter::OAuth.new(application_token,application_secret)
 
request_token = oauth.request_token.token
request_secret = oauth.request_token.secret
puts "Request token => #{request_token}"
puts "Request secret => #{request_secret}"
puts "Authentication URL => #{oauth.request_token.authorize_url}"
 
print "Provide the PIN that Twitter gave you here: "
pin = gets.chomp
 
oauth.authorize_from_request(request_token,request_secret,pin)
access_token = oauth.access_token.token
access_secret = oauth.access_token.secret
puts "Access token => #{oauth.access_token.token}"
puts "Access secret => #{oauth.access_token.secret}"
 
oauth.authorize_from_access(access_token, access_secret)
twitter = Twitter::Base.new(oauth)
puts twitter.friends_timeline(:count => 1)

When you run the script, it will give you a request token, request secret and a URL to visit. When you access the URL, you'll be given a PIN. Type the PIN into the prompt and you'll get your access token and secret. This is what you can use to continue authenticating with Twitter, so be sure to save the access token and secret.

From then on, you should be able to login with a script like this:

#!/usr/bin/ruby
require 'rubygems'
require 'twitter'
 
application_token = '[this should be the shorter one]'
application_secret = '[this should be the longer one]'
 
oauth = Twitter::OAuth.new(application_token,application_secret)
 
oauth.authorize_from_access(access_token, access_secret)
twitter = Twitter::Base.new(oauth)
puts twitter.friends_timeline(:count => 1)

I hope this helps!

©2010 Racker Hacker. All Rights Reserved.

Quite a few distribution maintainers have gone to great lengths to keep Xen domU support in their kernels and it's been an uphill battle. Some kernels, such as Ubuntu's linux-ec2 kernels, have patches from 2.6.18 dragged forward into 2.6.32 and even 2.6.33. It certainly can't be enjoyable to keep dragging those patches forward into new kernel trees.

The paravirt_ops support for Xen guests was added in 2.6.23 and continues to be included and improved in the latest kernel trees. However, there are two significant problems with these new kernels if you're trying to work with legacy environments:

• the console is on hvc0, not tty1
• block devices are now /dev/xvdX rather than /dev/sdX

If you only have a few guests, these changes are generally pretty easy. Switching the console just requires some changes to your inittab or upstart configurations. Changing the block device names requires changes to the guest's Xen configuration file and /etc/fstab within the guest itself.

Considering the amount of environments I work with daily at Rackspace, changing the guest configuration is definitely not an option. I needed a way to keep the console and block devices unchanged so that our customers could have a consistent experience on our infrastructure.

Luckily, Soren Hansen offered to pitch in and a solution became apparent. Through some relatively small patches, the legacy console and block device support was available in the latest 2.6.32 version (2.6.32.12 as of this post's writing).

So far, I've tested x86_64 and i386 versions of 2.6.32.12 with the console and block device patches. It's gone through its paces on Xen 3.0.3, 3.1.2, 3.3.0 and 3.4.2. All revisions of Fedora, CentOS, Ubuntu, Debian, Gentoo and Arch made within the last two years are working well with the new kernels.

©2010 Racker Hacker. All Rights Reserved.

Thanks for the e-mails and support!

©2010 Racker Hacker. All Rights Reserved.

Ian Pratt and Keir Fraser started off the day with an overview of Xen's current roadmap. They talked about their largest products and where they're going.

Open source Xen development is going strong with the recent release of Xen 4.0 and there are some great features in the works for Xen 4.1. There should be a new credit scheduler called "credit2" in that release. Also, the Xen Cloud Platform project is improving and the feature set is growing. The Open vSwitch was recently integrated with XCP.

The Xen Client Initiative caught my attention quickly as it allows a user to run a very thin hypervisor on a client machine, such as a laptop or desktop computer, and then run multiple operating systems on top of that hypervisor. This would reduce the need for products like VMWare Fusion, VirtualBox or Parallels Desktop.

Still not impressed? Watch Ian Pratt do a quick demonstration of XCI:

www.youtube.com/watch?v=b2-K1dbzZPk

Tom Woller from AMD talked about some of the new hardware enhancements that allow Xen to virtualize devices more efficiently. While much of the low-level hardware discussion was a little over my head (I'm terrible with hardware), the improvements make sense and should improve the relationship with Xen.

Yuvraj Agarwal from the University of California San Diego talked about SleepServers. He's done some impressive work with Xen to reduce the energy usage of PC's left on when their users are no longer in the office.

Jonathan Ludlum covered the roadmap for Xen Cloud Platform in greater detail. This is definitely going to be a product to watch as it has a tight integration between various products, including XAPI, Open vSwitch, and the Xen hypervisor itself. It currently runs a CentOS-based userland and Jonathan said there are no plans to change it.

Later in the afternoon, Konrad Wilk from Oracle talked about the current status of the Xen kernels. The PVOps kernels are used with Xen 4.0 for the dom0 (and soon for the 3.4.x branch as well). The domU support has been present in PVOps code in the upstream kernels for some time. Many distribution vendors are making one kernel for bare metal and virtualized instances by simply adding PVOps support when they build their kernels. This eliminates the need for the traditional "kernel-xen" kernels that needed to be loaded for virtualized instances to work properly. Jeremy Fitzhardinge helped Konrad answer some questions from the audience and their presentation was one of the most informative ones of the day.

George Dunlap talked about the new credit2 scheduler that is due to be released with Xen 4.1 later this year. He found that the current scheduler negatively affects VM's running applications that have low CPU requirements but are affected greatly by higher latency. He tested this with a VM that was playing back an audio stream. When the rest of the VM's on the server used a lot of CPU time, the audio skipped many times. The credit2 scheduler allows for these low-CPU, latency-sensitive applications to keep running as expected without interruptions. I'm looking forward to testing this out later this year.

The last presentation of the day was from Stefano Stabellini. He covered the work being done to simplify how applications interact with Xen. The new library, libxenlight, strives to be a common layer between client applications and the Xen hypervisor.

The day wrapped up with a great party at Dave and Buster's. We had the opportunity to meet many people from Citrix as well as other people working in the virtualization space. Overall, the first day was very informative and I'm eager to hear some of the presentations scheduled for the second day.

©2010 Racker Hacker. All Rights Reserved.

Understand how iptables operates
Before you can begin using iptables, you need to fully understand how it matches packets with chains and rules. There is a terrific diagram in Wikipedia that will make it easier to understand. It's imperative to remember that iptables rules are read top-down until a matching rule is found. If no matching rule is found, the default policy of the chain will be applied (more on that in a moment).

Don't set the default policy to DROP
All iptables chains have a default policy setting. If a packet doesn't match any of the rules in a relevant chain, it will match the default policy and will be handled accordingly. I've seen quite a few users set their default policy to DROP, and this can bring about some unintended consequences.

Consider a situation where your INPUT chain contains quite a few rules allowing traffic, and you've set the default policy to DROP. Later on, another administrator logs into the server and flushes the rules (which isn't a good practice, either). I've met quite a few good systems administrators who are unaware of the default policy for iptables chains. Your server will be completely inaccessible immediately. All of the packets will be dropped since they match the default policy in the chain.

Instead of using the default policy, I normally recommend making an explicit DROP/REJECT rule at the bottom of your chain that matches everything. You can leave your default policy set to ACCEPT and this should reduce the chance of blocking all access to the server.

Don't blindly flush iptables rules
Before running iptables -F, always check each chain's default policy. If the INPUT chain is set to DROP, you'll need to set it to ACCEPT if you want to access the server after the rules are flushed. Also, consider the security implications of your network when you clear the rules. Your services will be completely exposed and any masquerading or NAT rules will be removed.

Remember localhost
Lots of applications require access to the lo interface. Ensure that you set up your rules carefully so that the lo interface is not disturbed.

Split complicated rule groups into separate chains
Even if you're the only systems administrator for your particular network, it's important to keep your iptables rules manageable. If you have a certain subset of rules that may be a little complicated, consider breaking them out into their own chain. You can just add in a jump to that chain from your default set of chains.

Use REJECT until you know your rules are working properly
When you're writing iptables rules, you'll probably be testing them pretty often. One way to speed up that process is to use the REJECT target rather than DROP. You'll get an immediate rejection of your traffic (a TCP reset) instead of wondering if your packet is being dropped or if it's making it to your server at all. Once you're done with your testing, you can flip the rules from REJECT to DROP if you prefer.

For those folks working towards their RHCE, this is a huge help during the test. When you're nervous and in a hurry, the immediate packet rejection is a welcomed sight.

Be stringent with your rules
Try to make your rules as specific as possible for your needs. For example, I like to allow ICMP pings on my servers so that I can run network tests against them. I could easily toss a rule into my INPUT chain that looks like this:

iptables -A INPUT -p icmp -m icmp -j ACCEPT

However, I don't want to simply allow all ICMP traffic. There have been some ICMP flaws from time to time and I'd rather keep as low of a profile as possible. There are many types of ICMP control messages, but I only want to allow echo requests:

iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

This will allow echo requests (standard ICMP pings), but it won't explicitly allow any other ICMP traffic to pass through the firewall.

Use comments for obscure rules
If you have rules to cover edge cases that other administrators might not understand, consider using iptables comments by adding the following arguments to your rules:

-m comment --comment "limit ssh access"

The comments will appear in the iptables output if you list the current rules. They will also appear in your saved iptables rules.

Always save your rules
Most distributions offer some way to save your iptables rules so that they persist through reboots. Red Hat-based distributions offer /etc/init.d/iptables save, but Debian and Ubuntu require some manual labor. An errant reboot would easily take out your unsaved rules, so save them often.

©2010 Racker Hacker. All Rights Reserved.

# color => new RH6.0 bootup
# verbose => old-style bootup
# anything else => new style bootup without ANSI colors or positioning
BOOTUP=color
# column to start "[  OK  ]" label in 
RES_COL=60
# terminal sequence to move to that column. You could change this
# to something like "tput hpa ${RES_COL}" if your terminal supports it
MOVE_TO_COL="echo -en \\033[${RES_COL}G"
# terminal sequence to set color to a 'success' color (currently: green)
SETCOLOR_SUCCESS="echo -en \\033[0;32m"
# terminal sequence to set color to a 'failure' color (currently: red)
SETCOLOR_FAILURE="echo -en \\033[0;31m"
# terminal sequence to set color to a 'warning' color (currently: yellow)
SETCOLOR_WARNING="echo -en \\033[0;33m"
# terminal sequence to reset to the default color.
SETCOLOR_NORMAL="echo -en \\033[0;39m"
# default kernel loglevel on boot (syslog will reset this)
LOGLEVEL=3
# Set to anything other than 'no' to allow hotkey interactive startup...
PROMPT=yes
# Set to 'yes' to allow probing for devices with swap signatures
AUTOSWAP=no
# What ttys should gettys be started on?
ACTIVE_CONSOLES=/dev/tty[1-6]

The very last line controls the number of tty's that are kept alive on your system. If you need more tty's, simply increase the 6 to a higher number. If you only want one terminal (which is usually what I want in Xen), just make this adjustment:

# What ttys should gettys be started on?
ACTIVE_CONSOLES=/dev/tty1

A normal telinit q doesn't seem to adjust the terminals on the fly as it did before upstart was involved. I'm not sure if this is a bug or an intended feature. Either way, a reboot solves the problem and you should see the changes afterwards.

©2010 Racker Hacker. All Rights Reserved.

After the post went up, I received as much feedback from people outside of Rackspace as I received from Rackers.  The negative feedback I received was centered around the assertion that the post's content was "fluffed" to make the Rackspace experience seem better than it actually is.  That couldn't be further from the truth.

If you want to make comments on the post, or if you want to know more about working at Rackspace, let me know.  Although I'm not in sales and I'm not in recruiting, I always enjoy talking to people about using Rackspace's services or working for Rackspace.

Here's a link to the post: Rackspace Talent - Why I'm a Racker

©2010 Racker Hacker. All Rights Reserved.

In short, iperf can be installed on two machines within your network. You'll run one as a server, and one as a client. On the server side, simply run:

iperf -s

On the client side, run:

iperf -c [server_ip]

The client side will try to shove TCP packets through the network interface as quickly as possible for a period of 10 seconds by default. Once that's complete, you'll see a report on the server and client that will look like this:

$ iperf -c 192.168.10.10
------------------------------------------------------------
Client connecting to 192.168.10.10, TCP port 5001
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.30 port 53345 connected with 192.168.10.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  37.9 MBytes  31.8 Mbits/sec

The previous test was run over an 802.11n network between a wired and wireless device. The typical downlink for an 802.11n network is about 40Mbit/s, so it's obvious that my home network could use an adjustment.

You can also run bidirectional tests from the client either at the same time (-d flag) or one after the other (-r flag). The server side will keep running until you stop it, so you can leave it running and run tests from multiple locations over time. You can daemonize the server end if that makes things easier.

For the full list of options, refer to iperf's man page.

©2010 Racker Hacker. All Rights Reserved.

Anytime you use kill on a process, you're actually sending the process a signal (in almost all situations - I'll get into that soon). Standard C applications have a header file that contains the steps that the process should follow if it receives a particular signal. You can get an entire list of the available signals on your system by checking the man page for kill.

Consider a command like this:

kill 2563

This would send a signal called SIGTERM to the process. Once the process receives the notice, a few different things can happen:

• the process may stop immediately
• the process may stop after a short delay after cleaning up resources
• the process may keep running indefinitely

The application can determine what it wants to do once a SIGTERM is received. While most applications will clean up their resources and stop, some may not. An application may be configured to do something completely different when a SIGTERM is received. Also, if the application is in a bad state, such as waiting for disk I/O, it may not be able to act on the signal that was sent.

Most system administrators will usually resort to the more abrupt signal when an application doesn't respond to a SIGTERM:

kill -9 2563

The -9 tells the kill command that you want to send signal #9, which is called SIGKILL. With a name like that, it's obvious that this signal carries a little more weight.

Although SIGKILL is defined in the same signal header file as SIGTERM, it cannot be ignored by the process. In fact, the process isn't even made aware of the SIGKILL signal since the signal goes straight to the kernel init. At that point, init will stop the process. The process never gets the opportunity to catch the signal and act on it.

However, the kernel may not be able to successfully kill the process in some situations. If the process is waiting for network or disk I/O, the kernel won't be able to stop it. Zombie processes and processes caught in an uninterruptible sleep cannot be stopped by the kernel, either. A reboot is required to clear those processes from the system.

©2010 Racker Hacker. All Rights Reserved.

www.youtube.com/watch?v=QC4KxSbWKEQ

I participated in a podcast for the Rackspace Cloud with Robert Collazo last week. We covered some important topics including network security and convenient deployment tools.

©2010 Racker Hacker. All Rights Reserved.

I kept having an issue where the shift key would become stuck regardless of the settings I set for the client or server. The halfDuplexCapsLock configuration option had no effect. After installing xkeycaps, I found that both shift keys were getting stuck if I brought the mouse back and forth between Mac and Fedora twice.

I decided to run a test. I started the client with the debug argument and moved the mouse to my Fedora box. At that point, I pressed the letter 'a' and saw:

DEBUG1: CXWindowsKeyState.cpp,195:   032 (00000000) up
DEBUG1: CXWindowsKeyState.cpp,195:   03e (00000000) up
DEBUG1: CXWindowsKeyState.cpp,195:   026 (00000000) down
DEBUG1: CXWindowsKeyState.cpp,195:   032 (00000000) down
DEBUG1: CXWindowsKeyState.cpp,195:   03e (00000000) down
DEBUG1: CXWindowsKeyState.cpp,195:   026 (00000000) up

I brought the mouse back to the Mac and then back to Fedora. I pressed 'a' again and saw:

DEBUG1: CXWindowsKeyState.cpp,195:   026 (00000000) down
DEBUG1: CXWindowsKeyState.cpp,195:   026 (00000000) up
DEBUG1: CXWindowsKeyState.cpp,195:   026 (00000000) down
DEBUG1: CXWindowsKeyState.cpp,195:   026 (00000000) up

After dumping the keyboard layout with xmodmap I found the keys that corresponded with the key numbers:

• 032 - Left shift
• 03e - Right shift
• 026 - a

If I tapped the left shift, I could clear the key press, but I couldn't clear the right shift key (it was stuck down according to Fedora's X server). When I hooked up a physical keyboard and mouse, I was able to use them normally without any keybinding problems.

The root cause: When synergy started in /etc/gdm/PreSession/Default after the gdm login, the keyboard layout wasn't set up properly. The X server was setting up the keyboard layout later in the startup process and this confusion caused the shift keys to get stuck. Fedora 12 uses evdev to probe for keyboards during X's startup and eventually settles on a default layout if none are explicitly defined.

The fix: I added the synergy startup to the GNOME startup items and it works flawlessly.

©2010 Racker Hacker. All Rights Reserved.

Many cloud providers allow you access to a private network environment where you can exchange data with other instances or other services offered by the provider. The convenience of this access comes with a price: other instances can access your instance on the private network just as easily as they could on the public interface.

Here are some security tips for your private interfaces:

Disable the private interface
This one is pretty simple. If you have only one instance or server, and you don't need to communicate privately with any other instances, just disable the interface. Remember to configure your networking scripts to leave the interface disabled after reboots.

Use packet filtering
The actual mechanism will vary based on your operating system, but filtering packets is the one of the simplest ways to secure your private interface. You can take some different approaches with them, but I find the easiest method is to allow access from your other instances and reject all other traffic.

For additional security, you can limit access based on ports as well as source IP addresses. This could prevent an attacker from having easy access to your other instances if they're able to break into one of them.

Configure your daemons to listen on the appropriate interfaces
If there are services that don't need to be listening on the private network, don't allow them to listen on your private interface. For example, MySQL might need to listen on the private interface so the web server can talk to it, but apache won't need to listen on the private interface. This reduces the profile of your instance on the private network and makes it a less likely target for attack.

Use hosts.allow and hosts.deny
Many new systems administrators forget about how handy tcpwrappers can be for limiting access. If your firewall is down in error, host.allow and hosts.deny could be an extra layer of protection. It's important to ensure that the daemons you are attempting to control are build with tcpwrappers support. Daemons like sshd support it, but apache and MySQL do not.

Encrypt all traffic on the private network
Just because it's called a "private" network doesn't mean that your traffic can traverse the network privately. You should always err on the side of caution and encrypt all traffic traversing the private network. You can use ssh tunnels, stunnel, or the built-in SSL features found in most daemons.

This also brings up an important point: you should know how your provider's private network works. Are there safeguards to prevent sniffing? Could someone else possibly ARP spoof your instance's private IP addresses? Is your private network's subnet shared among many customers?

With all of that said, it's also very important to have proper change control policies so that administrators working after you are fully aware of the security measures in place and why they are important. This will ensure that all of the administrators on your instances will understand the security of the system and they should be able to make sensible adjustments later for future functionality.

©2010 Racker Hacker. All Rights Reserved.

Almost every system administrator has found themselves in a situation where they're confronted with a server which has a problem. However, if you're not the primary administrator for the server, you may not always know what has changed recently or you may not be aware of changes in the server's environment. In these situations, if the fix isn't obvious, try going through these steps:

Localize the problem to a specific daemon or service
In the case of a problem where a website isn't loading properly, is it a problem with the web server itself? Could something other than the actual web server daemon be having an issue?

As an example, consider a ruby on rails application which runs through apache's mod_proxy_balancer and queries data from MySQL. If any of those individual puzzle pieces were not functioning correctly, you'd get a different result. A downed MySQL instance could make the application throw errors or appear to be unresponsive. If the mongrel cluster had failed, apache might be returning internal server errors. Your browser might return a connection refused if apache was down. These are all relatively easy to determine.

What if you are unable to determine which daemon is causing the problem?

If it's broken, break it a little more
Let's say that you've reviewed the process list and all of the appropriate daemons appear to be running. However, the website is still not loading properly. What do you do? Bring down a service and try again. Did something change? Did a new error appear? If not, bring that daemon back up and try taking down one of the other ones.

I've also had some good results by making small adjustments in the web server's configuration file. If you have a virtual host that isn't returning the correct data, try commenting it out temporarily. For rewrite rules, try removing them temporarily or strip them down to a more basic form. Test again, and then begin adding lines back incrementally. As much as a single period or quotation mark can derail a perfectly good set of rewrite rules.

In short - try to think outside the box when you're troubleshooting a difficult issue on an unfamiliar system. Always remember to back up your configurations before making changes and ensure your daemons will start properly if you bring them down.

©2010 Racker Hacker. All Rights Reserved.

ERROR 1206 (HY000): The total number of locks exceeds the lock table size

MySQL is trying to tell you that it doesn't have enough room to store all of the row locks that it would need to execute your query. The only way to fix it for sure is to adjust innodb_buffer_pool_size and restart MySQL. By default, this is set to only 8MB, which is too small for anyone who is using InnoDB to do anything.

If you need a temporary workaround, reduce the amount of rows you're manipulating in one query. For example, if you need to delete a million rows from a table, try to delete the records in chunks of 50,000 or 100,000 rows. If you're inserting many rows, try to insert portions of the data at a single time.

Further reading:

• MySQL Bug #15667 - The total number of locks exceeds the lock table size
• MySQL Error 1206 » Mike R's Blog

©2010 Racker Hacker. All Rights Reserved.