First things first, getting a job at Rackspace isn't easy. We don't intentionally make the process difficult. It's just that the work we do is unique and demanding.

We work in a fast-paced, extremely dynamic team-centric environment. While some people in the company work in extremely small teams or sometimes all by themselves, that's pretty few and far between. We look for people who can survive and flourish in this atmosphere and we look for people who can do it all while working as a team. Even with all of this hustle and bustle, we still remember why we're doing it: the pursuit of Fanatical Support for our customers.

Another thing to keep in mind is that there's no true secret for making it through the application process. There's no magic combination of skills or "silver bullet" that will scoot you through. Every candidate is reviewed individually for each position. There have been several times at the end of an interview where we've gotten together and said: "Wow, this candidate is solid, but they're just not right for this position. Let's find the right spot and see if there's a spot open." We look for the right candidate for the right position at the right time.

One of the best ways to get ahead in the screening or interview process is to do a little homework about Rackspace and the products we offer. Much of this is covered in a post I wrote in 2011. You'll go into the interviews with more confidence and it will be much more obvious that you're really interested in the position.

Don't be discouraged if the process takes a little longer than you expected. When I was hired in 2006, I went through two phone pre-screens and then three back-to-back interviews in person. Things have changed a little since then and I've heard of some candidates receiving two to three pre-screens via telephone and then one or two interviews in person. The additional screening and interviews may be due to Rackers trying to find the right fit for a particular applicant. As I said previously, we look for the right fit for each applicant. We may consider you for a different position than you applied for if we feel like your skill set or personality fits that role better.

A very common question is what to wear to a Rackspace interview. It's confusing to know exactly what's expected since we have Rackers in the building wearing everything from suits to flip-flops. This is where you really have to go with your gut. Interviewing for a customer-facing sales position while wearing a hoodie and shorts is probably going to bring a suboptimal result. Keep in mind that there's really nothing negative about overdressing (but keep your tuxedo in the closet, seriously). I wore a shirt and tie for my interviews in 2006 but my tie got caught in the car door and was shredded. After a lot of cursing, I took off the tie and decided to wing it with my dress shirt. Nobody ever said a word about it.

Remember to be flexible during the interviews. You might be asked to draw a solution on a whiteboard or think through a really complicated situation. Roll with it and keep your confidence up. When you don't know something, admit it, but then talk about how you'd research an answer.

There's one last thing to keep in mind and it's really critical. If you're ever asked about how you would solve a problem or how you solved a problem in the past, don't divulge any information which is confidential or proprietary to your current company. Just tell the interviewers that you've solved the solution in the past but you'll need to keep things vague to maintain confidentiality. We will definitely understand and we will encourage you to maintain that confidentiality.

Leave your comments if you have any! I'll be glad to answer any questions you have.

Getting a Technical Job at Rackspace is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Why do we need a JSON API for MySQL?
The real need sprang from a situation I was facing daily at Rackspace. We have a lot of production and pre-production environments which are in flux but we need a way to query data from various MySQL servers for multiple purposes. Some folks need data in ruby or python scripts while others need to drag in data with .NET and Java. Wrestling with the various adapters and all of the user privileges on disparate database servers behind different firewalls on different networks was less than enjoyable.

That's where this bridge comes in.

The bridge essentially gives anyone the ability to talk to multiple database servers across different environments by talking to a single endpoint with easily configurable security and encryption. As long as the remote user can make an HTTP POST and parse some JSON, they can query data from multiple MySQL endpoints.

How does it work?
It all starts with a simple HTTP POST. I've become a big fan of the Python requests module. If you're using it, this is all you need to submit a query:

import requests
payload = {'sql': 'SELECT * FROM some_tables WHERE some_column=some_value'}
url = "http://localhost:5000/my_environment/my_database"
r = requests.post(url, data=payload)
print r.text

The bridge takes your query and feeds it into the corresponding MySQL server. When the results come back, they're converted to JSON and returned via the same HTTP connection.

What technology does it use?
Flask does the heavy lifting for the HTTP requests and Facebook's Tornado database class wraps the MySQLdb module in something a little more user friendly. Other than those modules, PyYAML and requests are the only other modules not provided by the standard Python libraries.

Is it fast?
Yes. I haven't done any detailed benchmarks on it yet, but the overhead is quite low even with a lot of concurrency. The biggest slowdowns come from network latency between you and the bridge or between the bridge and the database server. Keep in mind that gigantic result sets will take a longer time to transfer across the network and get transformed into JSON.

I found a bug. I have an idea for an improvement. You're terrible at Python.
All feedback (and every pull request) is welcome. I'm still getting the hang of Python (hey, I've only been writing in it seriously for a few weeks!) and I'm always eager to learn a new or better way to accomplish something. Feel free to create an issue in GitHub or submit a pull request with a patch.

mysql-json-bridge: a simple JSON API for MySQL is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Photo credit: Will Clayton

As I mentioned already, the blog started out just as a place to stuff my notes from the things I learned at work. I figured that it would be nice to store it in a searchable format but it would also be great if I could link other people to certain posts if they needed more information to fix a problem. It was a way to retain knowledge but yet give it back to the people around me who needed it.

The blog has hit 456 posts (this one is #457) and it's gone from a few page views per day to just over 20,000 per day. Here are the top five most accessed posts (since I've been keeping stats):

1. Syncing an iPhone with a new Mac without hassles
2. ip_conntrack: table full, dropping packet
3. Delete a single iptables rule
4. Increase MySQL connection limit
5. MySQL Error 1040: too many connections

I'd like to send out a big thanks to the people who read this blog, add comments (or complaints!), and suggest new topics. You are the reason why I take the time to keep this blog going.

Five years of rackerhacker.com is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First things first: there are going to be many part of this post which probably aren't as specific as you'd like. A lot of that is due to the NDA that all Red Hat examinees agree to when they take an exam. We aren't allowed to talk about what was on the exam or our experiences during the exam. If we do, penalties range from smaller things like losing certifications all the way up to serious stuff like legal action. It goes without saying that I want to protect the security of the exams, I don't want to lose my certifications, and I don't want to hire a lawyer. Please try to keep this in mind if you yearn for more specifics than I'm able to give.

Red Hat Certified Engineer
The RHCSA and RHCE exams are the first step on the path to the RHCA. You can't take any of the RHCA prerequisite exams without it. These exams cover a really broad spectrum of material including apache configuration, NFS, iptables and mail services. The two links above will take you to the exam objectives for each exam.

I've always recommended the RHCE exam for Linux administrators who are trying to sharpen their skills and get to the next level whether they use Red Hat or not. The exam covers a lot of good material that makes a solid foundation for any Linux user without throwing in too many Red Hat-specific knowledge.

The exam (like all Red Hat exams) is fully practical. There are no multiple choice questions or essays. You'll have to meet all of the objectives by logging into a local Red Hat system and making the system do what it needs to do.

Quick tips for the RHCSA/RHCE exams:

• Keep your eye on the clock. Time can really get away from you if you get stuck in the weeds on a problem that should be relatively straightforward.
• Leave time at the end to check your work. When you set up a lot of services, it's inevitable that you might configure a service for one problem that breaks the functionality required by a problem you completed already.
• Always reboot before you leave. We all forget to use chkconfig when we're in a hurry.
• Practice, practice, practice. There's not one objective on this exam that you can't test in a VM on your own.

Red Hat Enterprise System Monitoring and Performance Tuning
Our group at Rackspace started off with EX442 and it was a very difficult way to start off the RHCA track. Take a look at the objectives and you'll see that much of the exam is related to tweaking system performance and then monitoring that performance with graphs and raw data. You'll have to turn a lot of knobs on the kernel and you'll need to know where to store these configurations so they'll be persistent.

In addition, the objective regarding TCP buffers and related settings is a real challenge. You'll have to wrestle with some math that appears to be relatively simple, but can get confusing quickly. Some of the settings can't really be checked to know if your setting is correct. The objectives mention tuning disk scheduling -- you don't really have the time or tools to know if your setting is ideal.

Quick tips for EX442:

• Use the documentation available to you. Install the kernel-doc package while you practice and during the exam.
• Be careful with your math. You have a Linux machine in front of you! Don't forget about bc.
• Watch your units. Know the difference between a kilobyte (KB) and a kibibyte (KiB).
• Make comments in files where you adjust kernel configurations. It will help you keep track of which question the kernel adjustment is meant to satisfy.

Red Hat Enterprise Storage Management
I'm surprised to say this now, but I actually enjoyed EX436. I've always used other clustering tools like heartbeat and pacemaker, but I've never had the need to use the Red Hat Cluster Suite. Although RHCS definitely has a lot of quirks and rough edges, it's pretty solid once you get familiar with the GUI and command line tools.

You get the opportunity to mess around with some pretty useful technology like iSCSI, GFS, and clustered LVM. These are things that you're probably already using or will be using soon in a large server environment. The web interface for RHCS is quite peculiar and you may find yourself wanting to put your fist through the screen when you're staring down the endless animated GIFs when the cluster is syncing its configuration. Do your best to be patient because you certainly don't want to short circuit the cluster sync.

Quick tips for EX436:

• Be patient. You'll feel like the RHCS web interface is mocking you when you're pressed for time.
• Watch the clock. It's extremely easy to burn a lot of time on this exam if you get stuck on a particular problem.
• Double check your entries in the web interface. Make sure you're doing things in the right order and that you've set up the prerequisites before adding services to the cluster. If you get it wrong, you could put your cluster into a weird state.
• Use man pages. If you don't mess with GFS a lot, the man pages will save you in a pinch.

Red Hat Enterprise Deployment and Systems Management
If there's one exam where time management is critical, it's EX401. Importing data into the Satellite Server takes quite a bit of time and there's almost nothing you can do to speed it up. It probably goes without saying, but as with most long-running tasks, you'll want to run it in screen. The last thing you'd ever want is to abort the import due to an errant click or CTRL-C (I did it while practicing -- it's aggravating).

There are other test objectives which you can either complete or partially complete while you wait for the import to finish.

Also, take the time to really dig into the Satellite Server web interface while your practicing for the exam. Knowing where to find the most common configuration items will really save some time when you're in the exam. You can sometimes get pretty bogged down in the interface so don't forget to use multiple tabs to keep your work organized.

I felt like this exam was the easiest out of the bunch since you could go back and test every single question with good time management. Did I mention how important time management was on this exam already? If I forgot to mention it earlier, be sure to focus on time management for this test.

Quick tips for EX401:

• Time management will make or break you on this test. Keep an eye on the clock and make sure you've done absolutely every piece of the exam that you can while you wait for the server to do its work.
• Scour the web interface. Keep a mental map in your mind where the big chunks of configuration items are.
• Go back and test everything. If you manage your time well, you should have enough time to verify each and every objective on this exam.

Red Hat Enterprise Directory Services and Authentication
At first, EX423 looks pretty straightforward. Red Hat's authentication configuration tools make LDAP authentication setup pretty easy. However, this exam comes with a lot of curveballs.

The GUI interface for the Directory Services component is a little frustrating to use. I found that the GUI stopped responding to keyboard input occasionally unless I clicked on another window and came back. If you misconfigure the SSL certificates in the interface, your LDAP server is down for the count. If you don't input the correct data into the setup scripts at the beginning, you might not notice it until much later when it's either too difficult to dig yourself out of the hole or it's too late to start over with a clean configuration.

I didn't feel pressed for time on this exam too much and that was pretty refreshing after taking the EX401 test. It's extremely critical to watch what you type and click on this exam. Some mistakes can be quickly corrected while others may require you to blow away the LDAP server configuration and re-provision the whole thing.

Quick tips for EX423:

• Always watch what you're typing. A simple mistake can lead to confusion or bigger issues down the road.
• Don't ignore the LDIF objectives. As you practice, you'll find that manipulating LDIF files is a little more involved than you expected.
• Practice starting over. Throw out your Directory Services configuration and get the experience of what it's like to start over and get back in the game.

Red Hat Enterprise Security: Network Services
There's no sugar coating it -- EX333 is a beast. It's a six hour exam broken into two three-hour chunks. It covers a ton of material and I refer to it as "the RHCE on steroids." You might argue that I thought it was hard since it was the last test and I was ready to be finished, but I really think this exam is a tough one.

Practicing for the Kerberos and DNS objectives was the hardest for me. I just couldn't understand Kerberos, no matter how hard I tried. The realization that I would really have to learn it soon set in. I dug into the Kerberos design documentation on MIT's site, read the summaries on Wikipedia, and scoured the documentation available in the Kerberos RPM packages. Once I understood why Kerberos is set up the way it is and why the security measures are present, everything began to come together. I was able to remember the steps not because I was memorizing them, but because I understood how Kerberos worked.

When you're working through the DNS objectives, keep an eye out for punctuation. I blew through a good 20 minutes in what seemed like the blink of an eye when I forgot a period in my TSIG key configuration while studying. Make sure you use the resources available to you, like system-config-bind and sample configs in /usr/share/doc/bind*/examples/. Get to know commands like dig really well.

If you're overwhelmed by OpenSSL's command line syntax, check out the /etc/pki/tls/misc/CA script. There are some handy comments at the top of the script that explain how to use it. You can also pluck OpenSSL commands right out of the script if you need to run them yourself.

• Don't just memorize. Do some research to understand how everything fits together.
• Manage your time. DNS and Kerberos have lots of small nuances that can become time sinks when done incorrectly.
• Use the available documentation and tools. Try practicing without study materials so that you're forced to use the docs and tools available within the server.

Ranking the exams
A couple of folks on Twitter asked me to rank the exams from most difficult to least difficult. Keep in mind that these are a little subjective since I was more familiar with some objectives than others for certain tests.

• EX333 - Enterprise Security: Network Services: a tubload of material and a very long exam
• EX442 - System Monitoring and Performance Tuning: very difficult to check your work, lots of calculations
• EX423 - Directory Services and Authentication: not a lot of material to cover, but tons of curveballs
• EX436 - Storage Management: the web interface made things much easier, lots of documentation available
• EX401 - Deployment and Systems Management: every objective can be tested, I build RPM's already

Looking back at the long road to becoming a Red Hat Certified Architect is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Also, I don't really hate Kerberos. It's a bit of an inside joke with my coworkers who are studying for some of the RHCA exams at Rackspace. The additional security provided by Kerberos is quite good but the setup involves a lot of small steps. If you miss one of the steps or if you get something done out of order, you may have to scrap the whole setup and start over unless you can make sense of the errors in the log files. A lot of my dislikes for Kerberos comes from the number of steps required in the setup process and the difficulty in tracking down issues when they crop up.

To complete this guide, you'll need the following:

• two CentOS, Red Hat Enterprise Linux or Scientific Linux 5 servers or VM's
• some patience

Here's how I plan to name my servers:

• kdc.example.com - the Kerberos KDC server at 192.168.250.2
• client.example.com - the Kerberos client at 192.168.250.3

CRITICAL STEP: Before getting started, ensure that both systems have their hostnames properly set and both systems have the hostnames and IP addresses of both systems in /etc/hosts. Your server and client must be able to know the IP and hostname of the other system as well as themselves.

First off, we will need NIS working to serve up the user information for our client. Install the NIS server components on the KDC server:

[root@kdc ~]# yum install ypserv

Set the NIS domain and set a static port for ypserv to make it easier to firewall off. Edit /etc/sysconfig/network on the KDC server:

NISDOMAINNAME=EXAMPLE.COM
YPSERV_ARGS="-p 808"

Manually set the NIS domain on the KDC server and add it to /etc/yp.conf:

[root@kdc ~]# nisdomain EXAMPLE.COM
[root@kdc ~]# echo "domain EXAMPLE.COM server kdc.example.com" >> /etc/yp.conf

Adjust /var/yp/securenets on the KDC server for additional security:

[root@kdc ~]# echo "255.0.0.0 127.0.0.0" >> /var/yp/securenets
[root@kdc ~]# echo "255.255.255.0 192.168.250.0" >> /var/yp/securenets

Start the NIS server and generate the NIS maps:

[root@kdc ~]# /etc/init.d/ypserv start; chkconfig ypserv on
[root@kdc ~]# make -C /var/yp

I usually like to prepare my iptables rules ahead of time so I ensure that it doesn't derail me later on. Paste this into the KDC's terminal:

iptables -N SERVICES
iptables -I INPUT -j SERVICES
iptables -A SERVICES -p tcp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p udp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p tcp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p udp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p tcp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 464 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p tcp --dport 749 -j ACCEPT -m comment --comment "kerberos"
/etc/init.d/iptables save

We need our time in sync for Kerberos to work properly. Install NTP on both nodes, start it, and ensure it comes up at boot time:

[root@kdc ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start
[root@client ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start

Now we're ready to set up Kerberos. Start by installing some packages on the KDC:

[root@kdc ~]# yum install krb5-server krb5-workstation

We will need to make some edits to /etc/krb5.conf on the KDC to set up our KDC realm. Ensure that the default_realm is set:

default_realm = EXAMPLE.COM

The [realms] section should look like this:

[realms]
EXAMPLE.COM = {
	kdc = 192.168.250.2:88
	admin_server = 192.168.250.2:749
}

The [domain_realm] section should look like this:

[domain_realm]
kdc.example.com = EXAMPLE.COM
client.example.com = EXAMPLE.COM

Add validate = true within the pam { } block of the [appdefaults] section:

[appdefaults]
 pam = {
   validate = true

Adjust /var/kerberos/krb5kdc/kdc.conf on the KDC:

[realms]
EXAMPLE.COM = {
	master_key_type = des-hmac-sha1
	default_principal_flags = +preauth
}

There's one last configuration file to edit on the KDC! Ensure that /var/kerberos/krb5kdc/kadm5.acl looks like this:

*/admin@EXAMPLE.COM	    *

We're now ready to make a KDC database to hold our sensitive Kerberos data. Create the database and set a good password which you can remember. This command also stashes your password on the KDC so you don't have to enter it each time you start the KDC:

kdb5_util create -r EXAMPLE.COM -s

On the KDC, create a principal for the admin user as well as user1 (which we'll create shortly). Also, export the admin details to the kadmind key tab. You'll get some extra output after each one of these commands but I've snipped it to reduce the length of the post.

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc root/admin
kadmin.local:  addprinc user1
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/admin
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/changepw
kadmin.local:  exit

Let's start the Kerberos KDC and kadmin daemons:

[root@kdc ~]# /etc/init.d/krb5kdc start; /etc/init.d/kadmin start
[root@kdc ~]# chkconfig krb5kdc on; chkconfig kadmin on

Now that the administration work is done, let's create a principal for our KDC server and stick it in it's keytab:

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc -randkey host/kdc.example.com
kadmin.local:  ktadd host/kdc.example.com

Transfer your /etc/krb5.conf from the KDC server to the client. Hop onto the client server, install the Kerberos client package and add some host principals:

[root@client ~]# yum install krb5-workstation
[root@client ~]# kadmin.local
kadmin.local:  addpinc --randkey host/client.example.com
kadmin.local:  ktadd host/kdc.example.com

There aren't any daemons on the client side, so the configuration is pretty much wrapped up there for Kerberos. However, we now need to tell both servers to use Kerberos for auth and your client servers needs to use NIS to get user data.

• On the KDC:
  • run authconfig-tui
  • choose Use Kerberos from the second column
  • press Next
  • don't edit the configuration (authconfig got the data from /etc/krb.conf)
  • press OK
• On the client:
  • run authconfig-tui
  • choose Use NIS and Use Kerberos
  • press Next
  • enter your NIS domain (EXAMPLE.COM) and NIS server (kdc.example.com or 192.168.250.2)
  • press Next
  • don't edit the Kerberos configuration (authconfig got the data from /etc/krb.conf)
  • press OK

Got NIS problems? If the NIS connection stalls on the client, ensure that you have the iptables rules present on the KDC that we added near the beginning of this guide. Also, if you forgot to add both hosts to both servers' /etc/hosts, go do that now.

Let's make our test user on the KDC. Don't add this user to the client -- we'll get the user information via NIS and authenticate via Kerberos shortly. We'll also rebuild our NIS maps after adding the user:

[root@kdc ~]# useradd user1
[root@kdc ~]# passwd user1
[root@kdc ~]# make -C /var/yp/

On the client, see if you can get the password hash for the user1 account via NIS:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:$1$sUlSTlCv$riK5El3z8N4y.mi5Fe3Q60:500:500::/home/user1:/bin/bash

You can see why NIS isn't a good way to authenticate users. Someone could easily pull the hash for any account and brute force the hash on their own server. Go back to the KDC and lock out the user account:

[root@kdc ~]# usermod -p '!!' user1

Go back to the client and try to pull the password hash now:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:!!:500:500::/home/user1:/bin/bash

On the plus side, the user's password hash is now gone. On the negative side, you've just prevented this user from logging in locally or via NIS. Don't worry, the user can log in via Kerberos now. Let's prepare a home directory on the client for the user:

[root@client ~]# mkdir /home/user1
[root@client ~]# cp -av /etc/skel/.bash* /home/user1/
[root@client ~]# chown -R user1:user1 /home/user1/

Note: In a real-world scenario, you'd probably want to export this user's home directory via NFS so they didn't get a different home directory on every server.

While you're still on the client, try to log into the client via the user. Use the password that you used when you created the user1 principal on the KDC.

[root@client ~]# ssh user1@localhost
user1@localhost's password:
[user1@client ~]$ whoami
user1

List your Kerberos tickets and you should see one for your user principal:

[user1@client ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500_fCKPnZ
Default principal: user1@EXAMPLE.COM
 
Valid starting     Expires            Service principal
02/05/12 14:18:53  02/06/12 00:18:53  krbtgt/EXAMPLE.COM@EXAMPLE.COM
	renew until 02/05/12 14:18:53

Your KDC should have a couple of lines in its /var/log/krb5kdc.log showing the authentication:

Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): AS_REQ (12 etypes {18 17 16 23 1 3 2 11 10 15 12 13}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=16 ses=16}, user1@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=18 ses=18}, user1@EXAMPLE.COM for host/client.example.com@EXAMPLE.COM

The first line shows that the client asked for a Authentication Server Request (AS_REQ) and the second line shows that the client then asked for a Ticket Granting Server Request (TGS_REQ). In layman's terms, the client first asked for a ticket-granting ticket (TGT) so it could authenticate to other services. When it actually tried to log in via ssh it asked for a ticket (and received it).

YOU JUST CONFIGURED KERBEROS!

From here, the sky's the limit. Another popular implementation of Kerberos is encrypted NFSv4. You can even go crazy and use Kerberos with apache.

Let me know if you have any questions about this post or if you spot any errors. With this many steps, there's bound to be a typo or two in this guide. Keep in mind that there are some obvious spots for network-level and service-level security improvements. This guide was intended to give you the basics and it doesn't cover all of the security implications involved with a Kerberos implementation.

The Kerberos-hater's guide to installing Kerberos is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Thanks to @pinojo for snapping the photo.

You might see me on CNN Money soon is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The feeling I had at last year's summit is that Xen was on the verge of losing traction in the market. Very few distributions still had Xen support going forward and much of the discussion was around the lack of dom0 support in upstream Linux kernels. Distribution vendors were hesitant to drag patches forward into modern kernels and this made it much more difficult to get Xen working for many people.

This year was quite different. The number of attendees was up, the venue was much better, and there was an obvious buzz of energy in the room. As many of the presenters noted, this excitement stemmed from the upstream dom0 support in Linux 3.0. This inclusion is a huge win and it helps to drive Xen forward since the developers don't have to worry about dragging patches forward. They can focus on improving performance, adding features, and tightening security.

Many of the discussions this year focused on security and performance. Ian Pratt discussed Xen's ability to view memory pages of virtual machines via an API to detect malware running inside the instance. Memory pages could be identified and marked as not executable or applications could be triggered when a VM attempts to touch a particular memory page. Also, the whole VM could be frozen if needed.

There's also a big push to bring code out of the dom0 and push it into utility VM's. Driver domains could manage the network or I/O infrastructure and this would further reduce the amount of privileged code actively running in dom0. There is already very little code required for the Xen hypervisor itself (much much less than the Linux kernel -- I'm looking at you, KVM) and this reduces the attack surface for potential compromises of the hypervisor. Some projects even aim to restart driver domains multiple times per minute to ensure that any malicious code injected into those virtual machines can't exist for long periods.

Pradeep Vincent from Amazon talked about how Amazon uses Xen and the pain points they have with its current architecture. Much of his discussion was around scaling problems (and we see many of the same issues at Rackspace). Higher performance could easily be gained by multi-threaded operations in dom0 when attaching block devices and creating virtual network interfaces. He also saw some areas for performance gains in the pvops I/O code.

Quite a few of the talks centered on the ARM architecture and what Xen is able to do on those systems after Samsung published their port in 2008. HVM is on the way for ARM and it might even show up in Xen 4.2. Some demos of Xen on mobile phones from Samsung were amazing. They showed how an attacker could compromise the web browser on the phone with a keylogger, but that application was running in a VM. Once the user switched back to the phone's main menu, the keylogger couldn't access the keystrokes any longer. After that, a simple close of the browser killed the VM and destroyed the malicious code.

Xen 4.2 should be available in early 2012 and the feature list is staggering. Improvements to libxenlight, pvops performance (even in HVM), and guest memory sharing should be available with the new release. Nested virtualization (run a hypervisor inside a hypervisor) is also coming in Xen 4.2 and I'm sure Xzibit will be a huge fan. This should streamline hypervisor testing, allow for embedded hypervisor options and extend the capabilities of client hypervisors. Remus should be available in 4.2 as well, but it might be marked as experimental. OVMF will be added as a BIOS option for UEFI (along with the standard SeaBIOS) and this should allow for Mac OS X guests. UEFI allows Windows to boot faster since it switches to PV mode sooner and it allows for simpler platform certification for software vendors.

Mike McClurg's presentation on XCP was pretty important to me since Rackspace is a big consumer of XenServer. If you're not familiar with XCP, it's basically open-source XenServer which runs on bleeding edge (and sometimes unstable) components. XCP 1.5 and XenServer 6 should be available in November with Xen 4.1 and Linux 2.6.32. GPU passthrough, up to 1TB RAM, and disaster recovery will be available. Another goal for the XCP team is to work closely with OpenStack via Project Olympus. Mike's vision is to have XCP become the configuration of choice for open source clouds. Project Kronos was also extremely interesting. It's essentially XCP's XenAPI stack running on Debian and Ubuntu. You'd be able to install either OS on a physical server and run XCP's services on it for a fully OSS hypervisor.

Konrad Wilk gave an update on Linux pvops and it appears there is a shift to get Xen working well on a desktop. This includes 3D graphics support, S3/hibernate capabilities and various bug fixes. There's also a push to get PV functionality into HVM and get HVM functionality into PV. Driver/device domains were discussed again in Patrick Kolp's talk and he had plenty of graphs showing performance changes when regularly restarting device domains. The performance dips were almost negligible with 10 second restarts and the security gains were significant.

There were several other great presentations on other topics like GlusterFS, OpenStack Nova, and Linpicker (from the NSA!). If these types of things interests you, keep your eyes peeled for Xen Summit 2012 next year. The weather in the bay area is well worth the trip.

Xen Summit 2011: My Takeways is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

What do I mean by "do your homework?" Here are some bullet points to get you on your way:

Know what the company does.
This one is critical and it should be easy. However, make sure to do thorough research first. For example, if you interviewed at a company like Apple, becoming familiar with their hardware lineup should be a no-brainer. That's their bread and butter. On the other hand, remember that Apple isn't solely a hardware company; they write lots of software, provide online productivity services, and they distribute music, movies, and other entertainment media.

While you're doing this research, try to discover what makes the company unique. Sure, Apple sells laptops and desktops (just like a lot of other companies), but what makes their particular products unique? Is there something unique about the way they provide their services? Have they cornered a certain market segment by providing a combination of products and services to that group of consumers? Answering these simple questions may help you tip the scales in the interview process.

Try one or more of the company's products.
The feasibility of trying a company's product before an interview could be debatable. For example, if you wanted to interview at Cray, you probably don't need to drop $2M USD on your own XE6 before walking in the door. For companies where the barrier to entry for purchasing a product is much lower, such as cloud computing companies, there's no excuse to not try things out first. Amazon has a free tier and a Rackspace Cloud Server could cost you as little as $2.50 per week.

It's concerning when I talk to an applicant about a job working with Rackspace's Cloud Servers and they haven't tried out any cloud products from any provider. How can I take a candidate's interest seriously when they haven't shown interest in any portion of my group's market segment?

Know what the company's competitors do.
It's often more impressive to an interviewer to know what a company's competitors are doing and how it compares to what that company is doing in the market. For example, if you can walk into an interview and say "I like the way your company makes these widgets, but Company X is able to make them more lightweight, and I value that more than the added customer service your company offers." This shows the interviewer that you're familiar with various products in the segment and you've used them enough to understand what makes them different.

Some of you might be thinking: "Why would I say something like that to the interviewer? They'll think I'm being too negative about their product." That's always possible, but you can guard against it by wording everything carefully. Make sure you have a solid reason for the way you feel that is based on something substantial (usability, price, features, etc). I've had candidates talk for five to ten minutes about why one of our product is inferior to one of our competitors' products and I was very impressed.

One quick gotcha: your interviewer might turn your comments back on you and ask you how you would improve one of the inferior products (I do this regularly). Make sure that you're prepared for that question and consider offering up a suggestion before the question is presented to you.

Can't get the information you need? Ask!
When you reach the end of the interview and the interviewer asks if you have questions, be sure to ask any questions about topics you had trouble researching. Going back to the Cray example, compare what you know about an XE6 to servers you've used before. You could mention a problem you had with the density of your previous configurations and ask how they overcame that hurdle at Cray. If it's a proprietary trade secret, you might not get an answer, but they'll know that you did some serious research ahead of time. If they can share the answer, they might still be impressed, and you might end up learning something you didn't know prior to the interview.

Conclusion
In summary, doing your homework and learning about the company shows the interviewers that you not only have what it takes to do the work, but that the work interests you as well. I've interviewed folks in the past who lacked on technical ability but had plenty of desire and drive. More often than not, those people are now Rackers.

Do your homework before a technical interview is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Taking on the job of a technical manager presents its own unique challenges. It's easy for a technical manager to lose focus and get down in the weeds of daily work. It's also very difficult to let go of the reins and resign to the fact that the direct involvement in technical work will have to be reduced.

These problems resonate with me as I've recently taken on another technical management role at Rackspace. My previous experience involved managing a team of technicians at various skill levels who were working on customer environments made up of dedicated servers and network equipment. The current position has quite a few differences. I'm now managing a small group of highly technical and extremely dedicated Linux engineers and we're responsible for maintaining the systems and networks which run the Cloud Servers product.

One of my goals of this blog is to help others learn things much more easily than I have. Here are some things I've had to learn the hard way while working as a technical manager:

Get out of the mindset of an individual contributor
When you're a system administrator on a team (or by yourself), you're often judged on your personal job performance. Team interaction is important for some companies (especially at Rackspace), but not for others. Breaking the mindset of being an individual contributor was extremely difficult for me to do.

Managers are judged on the success of the team as a whole. Encouraging your team members to succeed and playing an active role in their personal and professional development is key. Each time you find yourself buried in the weeds of a problem rather than facilitating your team's work on the problem is when your performance as a manager will drop. If you do it more often, you may find that your team members aren't getting the support they need.

Don't be afraid of your team becoming smarter than you
One of the biggest things I've heard from my team is: "Aren't you worried about losing your technical skills when you're a manager?" My answer: "Of course." Anyone who has technical abilities will always be afraid of watching those abilities wane over time. However, as your team becomes stronger, you should be able to continue learning not through your own work, but through theirs. When your team members see that you're still interested in learning and you're now able to learn from them, they'll become more energized about their own work.

If you find yourself thinking negatively about a potential job candidate because they're smarter than you, step back and think for a moment. Put your own ego aside and consider what's best for you, your team, and your company. Your goal is to build a strong and successful team, not to pad your own ego. If your managers are judging you (as a technical manager) on your technical ability, then you need to solve that problem first.

Inspire instead of direct
Every manager faces the challenge of working with team members who disagree with a particular company policy or with the direction of their particular infrastructure. Keep in mind that your team members are probably not intending to be insubordinate and they might have something useful to contribute.

When you find yourself locking horns with your team members, inspire a discussion about the problem. Break out the disagreement onto a whiteboard and let the team make suggestions for improvements. Even if the entire discussion leads back to the fact that the original problem is inevitable, fostering that feedback loop is critical. You'll learn more about your team while they find ways to express their opinions and feel empowered.

The really tough part is when your team comes up with an alternative plan and you find yourself presenting to your leadership team. Always remember to take it seriously and know that you may need to refine the plan many times over before you find something acceptable for your team and the business.

De-stress by staying on task
If you're anything like me, you need some way to keep tabs on action items coming from meetings, e-mails, phone calls, and walk-ups. I've heard great things about applications like OmniFocus and Things, but I settled on 2Do. I really enjoy a strong to-do list which allows me to set priorities, due dates, and write extended notes about a particular task.

The best way to tackle a wall of tasks is to keep them organized into a concise list. Even if it's a small task, get it into your list so it's on your radar and you won't forget it. Work through the simple tasks and the high priority ones first but watch out for tasks with due dates.

Conclusion
All of these processes get easier over time and although your job will surely have challenges and pitfalls, the enjoyment will greatly increase. I feel privileged to lead a team of talented people who work on a complex and ever-expanding product.

Also, I'd like to mention that I'm not an expert on management! There are probably much better ways to do much of this than I've outlined in this post. Be sure to share your ideas in the comments section below.

How to survive as a technical manager is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The day started out with Jared Smith's "State of Fedora" address. The audio has already been posted on the wiki, but the speech was very positive overall. He talked about some of the struggles that have happened in the past and how they'll probably happen again in some form or another. It was pretty inspirational and you could obviously tell that people in the room were energized by it.

After the address, all of the talks were pitched in BarCamp format. It was a very efficient and entertaining way to create a schedule for the conference. Everyone had 15-20 seconds to present their talk and then they had to rush outside to post their topic on the wall. We all had the opportunity to go outside and vote for the talks that sounded interesting. Once the votes were tallied, the schedule was set and the conference was fully underway.

The first talk for me was about Marek Goldmann's BoxGrinder. (Note: If you Google for BoxGrinder, make sure that you enter it as a single word. You'll get some wild unrelated results if you use two words.) In short, BoxGrinder gives you the ability to have a kickstart-ish method for automatically building images for virtual machine environments. It's completely plugin-based, so you can have different platform and delivery plugins depending on where your VM needs to be deployed. For example, you could deploy a VM with BoxGrinder that is in a format for VMWare (platform) and is delivered to the target server via SFTP (delivery). The public cloud plugins are only compatible with Amazon's products, but I'm eager to change that during one of the upcoming hackfests.

The Sheepdog talk started up right after lunch and although it was interesting, I think it left most people with quite a few questions when it was over. However, I think people are generally apprehensive when anyone tries to do anything innovative with storage. Losing data due to a bug is a big concern and many of the questions went deeper into data safety than performance and functionality.

Next up was Dave Malcolm's talk about the different implementations of python. This was definitely an eye-opening talk for my coworker and I. Dave covered CPython, Jython, PyPy and various other implementations and compared their advantages and disadvantages. I'm still pretty new to Python (I'm clutching on to ruby, PHP and perl still), but this talk really had me thinking about which implementations are best for a particular environment or task. It was quite a bit of fun to learn about some of the deep underpinnings of Python and how they differ depending on the specific implementation.

Jeff Darcy's talk about CloudFS was very intriguing. I've been a fan of GlusterFS recently, but I eventually moved away due to a lack of enterprise features and degrading performance. Jeff is working to add in encryption and authentication without rewriting the filesystem itself. There are quite a few tricky problems involved in the encryption portion due to partial writes and general security during the handshake process. CloudFS could potentially be a network filesystem which could be shared by multiple tenants with their own individual namespaces and segregated UID's. This could be a big win for providers as they could offer up large amounts of storage in an organized fashion without too many management headaches.

We wrapped up the day of talks with Chris Lalancette's presentation about Deltacloud. In short, it's a bag of daemons that allow you to manage multiple public or private clouds. Everything from image management to provisioning are included in the project. Questions were raised about whether another application was needed since vendor-specific libraries are abundant and libcloud offers many of the same features in a simpler package.

Tonight's social event was FUDPub at ASU's Memoral Union building. The food and drinks were excellent (thanks to Rackspace!) and it was a great opportunity to relax and talk with other Fedora users and developers. We had the opportunity to meet people from around the world while playing round after round of bowling and billiards. The discussions were extremely valuable, but as I said before, it was quite tiring.

I've compiled the FUDCon photos I've taken into a Flickr photo set.

That's the end of today's summary. I'll try to keep this going tomorrow as well. Thanks for reading!

FUDCon 2011: Day One is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Diagram: OpenVPN to Rackspace Cloud Servers and Slicehost

The diagram at the right gives an idea of what this guide will allow you to accomplish. Consider a situation where you want to talk to the MySQL installation on db1 directly without requiring extra ssh tunnels or MySQL over SSL via the public network. If you tunnel into one of your instances, you can utilize the private network to talk between your instances very easily.

There's one important thing to keep in mind here: even though you'll be utilizing the private network between your tunnel endpoint and your other instances, your traffic will still traverse the public network. That means that the instance with your tunnel endpoint will still get billed for the traffic flowing through your tunnel.

You'll only need the openvpn package on the server side:

yum -y install openvpn

Throw down this simple configuration file into /etc/openvpn/server.conf:

port 1194
proto tcp
dev tun
persist-key
persist-tun
 
server 10.66.66.0 255.255.255.0
ifconfig-pool-persist ipp.txt
 
#push "route 10.0.0.0 255.0.0.0"
push "route 10.176.0.0 255.248.0.0"
keepalive 10 120
 
ca      /etc/openvpn/my_certificate_authority.pem
cert    /home/major/vpn_server_cert.pem
key     /home/major/vpn_server_key.pem
dh      /etc/openvpn/easy-rsa/2.0/keys/dh1024.pem
 
status log/openvpn-status.log
verb 3

Here's a bit of explanation for some things you may want to configure:

• push - These are the routes that will be sent over the VPN that are pushed to the clients. If you don't use any IP addresses in the 10.0.0.0/8 network block in your office, you can probably use the commented out line above. However, you may want to be more specific with the routes if you happen to use any 10.0.0.0/8 space in your office.
• server - These are the IP addresses that the VPN server will assign and NAT out through the private interface. I've used a /24 above, but you may want to adjust the netmask if you have a lot of users making tunnels to your VPN endpoint.
• ca, cert, key - You will need to create a certificate authority as well as a certificate/key pair for your VPN endpoint. I already use SimpleAuthority on my Mac to manage some other CA's and certificates, but you can use openvpn's easy-rsa scripts if you wish. They are already included with the openvpn installation.

Build your Diffie-Hellman parameters file:

cd /etc/openvpn/easy-rsa/2.0/ && ./build-dh

Tell iptables that you want to NAT your VPN endpoint traffic out to all 10.x.x.x IP addresses on the private network:

iptables -t nat -A POSTROUTING -s 10.0.0.0/8 -o eth1 -j MASQUERADE

The last step on the server side is to ensure that the kernel will forward packets from the VPN endpoint out through the private interface. Ensure that your /etc/sysctl.conf looks like this:

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

Adjusting your sysctl.conf ensures that forwarding is enabled at boot time, but you'll need to enable it on your VPN endpoint right now:

echo 1 > /proc/sys/net/ipv4/ip_forward

Start the openvpn server:

/etc/init.d/openvpn start

If all is well, you should see openvpn listening on port 1194:

[root@lb2 ~]# netstat -ntlp | grep openvpn
tcp        0      0 0.0.0.0:1194      0.0.0.0:*         LISTEN      2020/openvpn

You'll need to configure a client to talk to your VPN now. This involves three steps: creating a new certificate/key pair for the client (same procedure as making your server certificates), signing the client's certificate with your CA certificate (same one that you used above to sign your server certificates), and then configuring your client application to access the VPN.

There are many openvpn clients out there to choose from.

If you're using a Linux desktop, you may want to consider using the built-in VPN functionality in NetworkManager. For Mac users, I'd highly recommend using Viscosity ($9), but there's also tunnelblick (free).

Accessing Rackspace Cloud Servers and Slicehost slices privately via OpenVPN is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

My main purpose for the blog at the beginning was to create a place where I could write quick articles about problems I found and how to fix them. Most of the people around me were using their own handy systems to store notes (Stickies on the Mac, Tomboy notes on Linux, or just simple text files), but they weren't able to share them easily. I wanted a way to write up a solution and instantly share it with someone. I also wanted that person to be able to pass along the fix to someone else if they wanted.

Needless to say, it took off from there.

It's important to note that I couldn't have done this by myself. I've learned some efficient strategies for managing large systems and troubleshooting complex issues from my peers, my managers, and colleagues outside of Rackspace. There have been many triumphs and there have been quite a few failures.

The failures have taught me the most. I've made some pretty large mistakes and here are a few:

• inserted data into a MySQL slave in an active replication pair
• run a fsck on an online ext3 partition
• marked a failed drive online in a hardware RAID array
• mangled Plesk installations in ways that you can't comprehend
• typed 'reboot' into a terminal and pressed enter, only to realize I was in the wrong terminal
• ran UPDATE statements without a WHERE clause in MySQL (well, I only did this one twice)

Even after all that, people occasionally tell me that I'm very good at what I do. I don't know if that's true or not, but I'm glad some people think so! Many of those folks end up asking me this question:

How do I learn how to be a successful Linux systems administrator?

My answer is this: Be humble. Always be thirsty for knowledge. Don't be afraid to make mistakes. Love what you do and the people you serve.

400th post: Looking back at where this blog started is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Here are the steps for an upgrade from Fedora 13 to Fedora 14 via yum:

yum -y upgrade
wget http://mirror.rackspace.com/fedora/releases/14/Fedora/x86_64/os/Packages/fedora-release-14-1.noarch.rpm
rpm -Uvh fedora-release-14-1.noarch.rpm
yum -y install yum
yum -y upgrade

If you happen to be upgrading a 32-bit instance on Slicehost, simply replace x86_64 with i386 in the url shown above.

Upgrading Fedora 13 to Fedora 14 on Slicehost and Rackspace Cloud Servers is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The guide: Redundant Cloud Hosting Guide

As always, if you find anything in the guide that needs improvement, I'm all ears.

A simple guide to redundant cloud hosting is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you want to go active-active, or if you want multiple nodes active at the same time, you need to use a clustered filesystem like GFS2, OCFS2 or Lustre. These are certainly good options to consider but they're not trivial to implement. They usually rely on additional systems and scripts to provide reliable fencing and STONITH capabilities.

What about the rest of us who want multiple active VM's with simple replicated storage that doesn't require any additional elaborate systems? This is where GlusterFS really shines. GlusterFS can ride on top of whichever filesystem you prefer, and that's a huge win for those who want a simple solution. However, that means that it has to use fuse, and that will limit your performance.

Let's get this thing started!

Consider a situation where you want to run a WordPress blog on two VM's with load balancers out front. You'll probably want to use GlusterFS's replicated volume mode (RAID 1-ish) so that the same files are on both nodes all of the time. To get started, build two small Slicehost slices or Rackspace Cloud Servers. I'll be using Fedora 13 in this example, but the instructions for other distributions should be very similar.

First things first -- be sure to set a new root password and update all of the packages on the system. This should go without saying, but it's important to remember. We can clear out the default iptables ruleset since we will make a customized set later:

# iptables -F
# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:        [  OK  ]

GlusterFS communicates over the network, so we will want to ensure that traffic only moves over the private network between the instances. We will need to add the private IP's and a special hostname for each instance to /etc/hosts on both instances. I'll call mine gluster1 and gluster2:

10.xx.xx.xx gluster1
10.xx.xx.xx gluster2

You're now ready to install the required packages on both instances:

yum install glusterfs-client glusterfs-server glusterfs-common glusterfs-devel

Make the directories for the GlusterFS volumes on each instance:

mkdir -p /export/store1

We're ready to make the configuration files for our storage volumes. Since we want the same files on each instance, we will use the --raid 1 option. This only needs to be run on the first node:

# glusterfs-volgen --name store1 --raid 1 gluster1:/export/store1 gluster2:/export/store1
Generating server volfiles.. for server 'gluster2'
Generating server volfiles.. for server 'gluster1'
Generating client volfiles.. for transport 'tcp'

Once that's done, you'll have four new files:

• booster.fstab - you won't need this file
• gluster1-store1-export.vol - server-side configuration file for the first instance
• gluster2-store1-export.vol - server-side configuration file for the second instance
• store1-tcp.vol - client side configuration file for GlusterFS clients

Copy the gluster1-store1-export.vol file to /etc/glusterfs/glusterfsd.vol on your first instance. Then, copy gluster2-store1-export.vol to /etc/glusterfs/glusterfsd.vol on your second instance. The store1-tcp.vol should be copied to /etc/glusterfs/glusterfs.vol on both instances.

At this point, you're ready to start the GlusterFS servers on each instance:

/etc/init.d/glusterfsd start

You can now mount the GlusterFS volume on both instances:

mkdir -p /mnt/glusterfs
glusterfs /mnt/glusterfs/

You should now be able to see the new GlusterFS volume in both instances:

# df -h /mnt/glusterfs
Filesystem            Size  Used Avail Use% Mounted on
/etc/glusterfs/glusterfs.vol
                      9.4G  831M  8.1G  10% /mnt/glusterfs

As a test, you can create a file on your first instance and verify that your second instance can read the data:

[root@gluster1 ~]# echo "We're testing GlusterFS" > /mnt/glusterfs/test.txt
.....
[root@gluster2 ~]# cat /mnt/glusterfs/test.txt
We're testing GlusterFS

If you remove that file on your second instance, it should disappear from your first instance as well.

Obviously, this is a very simple and basic implementation of GlusterFS. You can increase performance by making dedicated VM's just for serving data and you can adjust the default performance options when you mount a GlusterFS volume. Limiting access to the GlusterFS servers is also a good idea.

If you want to read more, I'd recommend reading the GlusterFS Technical FAQ and the GlusterFS User Guide.

GlusterFS on the cheap with Rackspace's Cloud Servers or Slicehost is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

You may not have realized it, but I covered the most important part of selling a technical product in the first paragraph without even mentioning the word "sell". That was intentional. As a technical person, you have an innate ability to interact with customers without needing to actively sell them the product.

Whenever I meet a customer at a conference, trade show, or some other relatively informal event, I try to keep a few things in mind. I'll share them with you:

Learn why your customers are seeking out your product and what they really need
It's pretty obvious that this step requires more listening than talking. While the customer is explaining what they need but haven't found, try to keep a running tally in your brain of what technologies are important to them so that you can rank your suggestions for them. Don't think about which product will work best for them yet -- just keep keep their general requirements in mind.

This is also a good opportunity to relate to what they've told you. If there's a certain solution that ended up working really well or one that failed miserably, and you're familiar with one of those solutions, tell them briefly about your experiences. This will re-affirm how the customer feels about that solution and it also shows them that you've been in their shoes before. They'll also appreciate that you've been listening to their concerns and looking for ways to relate to their unique situation.

Make thoughtful production suggestions and discuss implementation
Some folks might say this is where the selling starts, but if you're doing it correctly, you've been selling your product and your company the whole time. This is where things can get tricky. Most technical people I've met will try to avoid being pushy when suggesting a product for a customer to use, and that's a good idea.

You need to do three things: pick the right product (or group of products), explain what needs it meets, and briefly cover some example implementations. As a technical person, this is where you really shine. Interpreting the customer's needs and turning it into a mini technical sales pitch is a piece of cake when you know the product well and you've implemented it before.

It's great to give a customer multiple options, but it's a bad idea to overwhelm them. If you find that you're talking a bit too much, there's no harm in offering to talk about details later during a formal meeting. You can say things like these:

• "this product will meet all your needs, but if you want to save a little money, you can use this other product like this."
• "if you combine these two products, you can meet these needs and save some time, but you can just use one and set it up like this..."
• "then later on, if you need to expand, you can start using this product by..."

Think about the customer's future growth
Even if you have products that meet your customer's needs, they're going to be concerned about what's going to happen down the road. What happens when they scale to a level that they can't even comprehend right now? I don't think any customer would expect you to cover all the bases, but try to think of some basic future-proofing for the customer. Even if it might involve a product that your company doesn't sell, just mention it.

Of course, there are some things that you shouldn't do:

• Don't overpromise or push hard about a future product.
• Don't feel obligated to know the answer to every question.
• Don't use words like "infinite", "forever", or "perfect".
• Don't talk about cost constantly.
• Don't force a customer to choose a product, take product literature, or take your contact information.
• Don't make assumptions about the customer's technical level, needs, or purchasing power.
• Don't let it bother you if the customer isn't interested in your product -- it's not personal.

And that's about it. If you follow those three tips and avoid the things you shouldn't do, you'll get the confidence you need to engage the customer and create the beginnings of a relationship with them.

How to sell: a guide for technical people is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

After the post went up, I received as much feedback from people outside of Rackspace as I received from Rackers.  The negative feedback I received was centered around the assertion that the post's content was "fluffed" to make the Rackspace experience seem better than it actually is.  That couldn't be further from the truth.

If you want to make comments on the post, or if you want to know more about working at Rackspace, let me know.  Although I'm not in sales and I'm not in recruiting, I always enjoy talking to people about using Rackspace's services or working for Rackspace.

Here's a link to the post: Rackspace Talent - Why I'm a Racker

Why I'm a Racker is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

It's moments like these that make me proud to be a Racker! In just a year and a half that I've been there, I've seen a lot of great changes and this one is the best. Congratulations to RAX!

Congratulations to RAX is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

How it started
The initial recruiter that I spoke with was a very friendly fellow that didn't seem too technical. He didn't get into the job description much, but he was interested mostly in whether I wanted to relocate and what type of job I enjoy most. We ran through a few cursory technical questions and he tried to find out what my skill level was in certain areas. When it was all said and done, he said I'd be contacted from someone else at Google within a few weeks.

Two weeks later, I received some e-mails, went through [redacted] phone screens (with some pretty intelligent people), and learned more about the position. The folks from Google that I spoke with ranged from friendly and chatty to very direct and somewhat terse. Overall, I got the idea that they weren't interested in running a quiz, but they wanted to know how deep my knowledge and understanding was with regards to critical topics relating to the position. I know this sounds vague, but it's about as much as I can tell you.

The middle
I received a few more e-mails after the phone screens and my recruiter wanted to bring me out to California. Travel arrangements were made, I flew out to San Jose, and then drove the short drive to Mountain View. The city and the surrounding areas were a little different than I was used to. Most of the buildings and structures look as if they were built between 1960 and 1980 and they had a peculiar architecture. I stayed in the Hotel Avante (which was quite comfortable) and made the short drive to the Googleplex in the morning.

This was about the point where I slapped myself and said "Holy crap, I'm interviewing at GOOGLE!"

When I arrived, I went into the wrong buildings twice until I found the right one, but some Google employees finally pointed me in the right direction. I met with my recruiter, who was actually pretty entertaining, and he gave me a run down of how the day would go. I spent the morning interviewing, and then I joined a Google employee for lunch. He answered many of my questions about the cost of living, job benefits, and how he liked Google. When that was over, I went back to interviewing and was escorted out of the building at the end of the day.

Towards the end
I spoke with my recruiter a few more times after the interview for some basic paperwork-related issues, and he worked hard to keep me in the loop on my application status. There wasn't much of a concern job-wise as I work for one of the best companies in my industry already. However, I was getting ready to move to a new home, so I let my recruiter know that I was in a bit of a time crunch.

You'll probably want to know what happened next, but there's not really anything that I'm allowed to say about it! What I can tell you is that I'm still with the best company in my industry, and I'm still enjoying it each day.

So I know what you're probably thinking...

Why did you stay at Rackspace?
It's easy to answer this question: I learn something new every day at Rackspace. Sometimes it's something technical, and sometimes it's something related to managing people or designing technology. The people that I share this learning opportunity with make it all worthwhile. I've never worked for a company where my managers cared so much about my personal and technical development. Also, I've never worked at a company where, as a manager, I'm encouraged to care for my own technicians' personal and technical development.

If you have any more questions about why I love working at Rackspace, please let me know. I'll be happy to fill you in.

Why I interviewed at Google and stayed at Rackspace is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.