Here's an example of what httpry's output looks like on a busy site like icanhazip.com:

2012-03-13 23:29:39 186.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	186.x.x.x < -	-	-	HTTP/1.1	200	OK
2012-03-13 23:29:39 187.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.0	-	-
2012-03-13 23:29:39 192.x.x.x	187.x.x.x < -	-	-	HTTP/1.0	200	OK
2012-03-13 23:29:39 188.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	188.x.x.x < -	-	-	HTTP/1.1	200	OK
2012-03-13 23:29:39 189.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	189.x.x.x < -	-	-	HTTP/1.1	200	OK

You can watch the requests come in and the responses go out in real time. It even allows for BPF-style packet filters which allow you to narrow down the source and/or destination IP addresses and ports you want to watch. You can run it as a foreground process or as a daemon depending on your needs.

It's now available as a RPM package for Fedora 15, 16, 17 (and rawhide) as well as EPEL 6 (for RHEL/CentOS/SL 6).

New Fedora and EPEL package: httpry is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First things first: there are going to be many part of this post which probably aren't as specific as you'd like. A lot of that is due to the NDA that all Red Hat examinees agree to when they take an exam. We aren't allowed to talk about what was on the exam or our experiences during the exam. If we do, penalties range from smaller things like losing certifications all the way up to serious stuff like legal action. It goes without saying that I want to protect the security of the exams, I don't want to lose my certifications, and I don't want to hire a lawyer. Please try to keep this in mind if you yearn for more specifics than I'm able to give.

Red Hat Certified Engineer
The RHCSA and RHCE exams are the first step on the path to the RHCA. You can't take any of the RHCA prerequisite exams without it. These exams cover a really broad spectrum of material including apache configuration, NFS, iptables and mail services. The two links above will take you to the exam objectives for each exam.

I've always recommended the RHCE exam for Linux administrators who are trying to sharpen their skills and get to the next level whether they use Red Hat or not. The exam covers a lot of good material that makes a solid foundation for any Linux user without throwing in too many Red Hat-specific knowledge.

The exam (like all Red Hat exams) is fully practical. There are no multiple choice questions or essays. You'll have to meet all of the objectives by logging into a local Red Hat system and making the system do what it needs to do.

Quick tips for the RHCSA/RHCE exams:

• Keep your eye on the clock. Time can really get away from you if you get stuck in the weeds on a problem that should be relatively straightforward.
• Leave time at the end to check your work. When you set up a lot of services, it's inevitable that you might configure a service for one problem that breaks the functionality required by a problem you completed already.
• Always reboot before you leave. We all forget to use chkconfig when we're in a hurry.
• Practice, practice, practice. There's not one objective on this exam that you can't test in a VM on your own.

Red Hat Enterprise System Monitoring and Performance Tuning
Our group at Rackspace started off with EX442 and it was a very difficult way to start off the RHCA track. Take a look at the objectives and you'll see that much of the exam is related to tweaking system performance and then monitoring that performance with graphs and raw data. You'll have to turn a lot of knobs on the kernel and you'll need to know where to store these configurations so they'll be persistent.

In addition, the objective regarding TCP buffers and related settings is a real challenge. You'll have to wrestle with some math that appears to be relatively simple, but can get confusing quickly. Some of the settings can't really be checked to know if your setting is correct. The objectives mention tuning disk scheduling -- you don't really have the time or tools to know if your setting is ideal.

Quick tips for EX442:

• Use the documentation available to you. Install the kernel-doc package while you practice and during the exam.
• Be careful with your math. You have a Linux machine in front of you! Don't forget about bc.
• Watch your units. Know the difference between a kilobyte (KB) and a kibibyte (KiB).
• Make comments in files where you adjust kernel configurations. It will help you keep track of which question the kernel adjustment is meant to satisfy.

Red Hat Enterprise Storage Management
I'm surprised to say this now, but I actually enjoyed EX436. I've always used other clustering tools like heartbeat and pacemaker, but I've never had the need to use the Red Hat Cluster Suite. Although RHCS definitely has a lot of quirks and rough edges, it's pretty solid once you get familiar with the GUI and command line tools.

You get the opportunity to mess around with some pretty useful technology like iSCSI, GFS, and clustered LVM. These are things that you're probably already using or will be using soon in a large server environment. The web interface for RHCS is quite peculiar and you may find yourself wanting to put your fist through the screen when you're staring down the endless animated GIFs when the cluster is syncing its configuration. Do your best to be patient because you certainly don't want to short circuit the cluster sync.

Quick tips for EX436:

• Be patient. You'll feel like the RHCS web interface is mocking you when you're pressed for time.
• Watch the clock. It's extremely easy to burn a lot of time on this exam if you get stuck on a particular problem.
• Double check your entries in the web interface. Make sure you're doing things in the right order and that you've set up the prerequisites before adding services to the cluster. If you get it wrong, you could put your cluster into a weird state.
• Use man pages. If you don't mess with GFS a lot, the man pages will save you in a pinch.

Red Hat Enterprise Deployment and Systems Management
If there's one exam where time management is critical, it's EX401. Importing data into the Satellite Server takes quite a bit of time and there's almost nothing you can do to speed it up. It probably goes without saying, but as with most long-running tasks, you'll want to run it in screen. The last thing you'd ever want is to abort the import due to an errant click or CTRL-C (I did it while practicing -- it's aggravating).

There are other test objectives which you can either complete or partially complete while you wait for the import to finish.

Also, take the time to really dig into the Satellite Server web interface while your practicing for the exam. Knowing where to find the most common configuration items will really save some time when you're in the exam. You can sometimes get pretty bogged down in the interface so don't forget to use multiple tabs to keep your work organized.

I felt like this exam was the easiest out of the bunch since you could go back and test every single question with good time management. Did I mention how important time management was on this exam already? If I forgot to mention it earlier, be sure to focus on time management for this test.

Quick tips for EX401:

• Time management will make or break you on this test. Keep an eye on the clock and make sure you've done absolutely every piece of the exam that you can while you wait for the server to do its work.
• Scour the web interface. Keep a mental map in your mind where the big chunks of configuration items are.
• Go back and test everything. If you manage your time well, you should have enough time to verify each and every objective on this exam.

Red Hat Enterprise Directory Services and Authentication
At first, EX423 looks pretty straightforward. Red Hat's authentication configuration tools make LDAP authentication setup pretty easy. However, this exam comes with a lot of curveballs.

The GUI interface for the Directory Services component is a little frustrating to use. I found that the GUI stopped responding to keyboard input occasionally unless I clicked on another window and came back. If you misconfigure the SSL certificates in the interface, your LDAP server is down for the count. If you don't input the correct data into the setup scripts at the beginning, you might not notice it until much later when it's either too difficult to dig yourself out of the hole or it's too late to start over with a clean configuration.

I didn't feel pressed for time on this exam too much and that was pretty refreshing after taking the EX401 test. It's extremely critical to watch what you type and click on this exam. Some mistakes can be quickly corrected while others may require you to blow away the LDAP server configuration and re-provision the whole thing.

Quick tips for EX423:

• Always watch what you're typing. A simple mistake can lead to confusion or bigger issues down the road.
• Don't ignore the LDIF objectives. As you practice, you'll find that manipulating LDIF files is a little more involved than you expected.
• Practice starting over. Throw out your Directory Services configuration and get the experience of what it's like to start over and get back in the game.

Red Hat Enterprise Security: Network Services
There's no sugar coating it -- EX333 is a beast. It's a six hour exam broken into two three-hour chunks. It covers a ton of material and I refer to it as "the RHCE on steroids." You might argue that I thought it was hard since it was the last test and I was ready to be finished, but I really think this exam is a tough one.

Practicing for the Kerberos and DNS objectives was the hardest for me. I just couldn't understand Kerberos, no matter how hard I tried. The realization that I would really have to learn it soon set in. I dug into the Kerberos design documentation on MIT's site, read the summaries on Wikipedia, and scoured the documentation available in the Kerberos RPM packages. Once I understood why Kerberos is set up the way it is and why the security measures are present, everything began to come together. I was able to remember the steps not because I was memorizing them, but because I understood how Kerberos worked.

When you're working through the DNS objectives, keep an eye out for punctuation. I blew through a good 20 minutes in what seemed like the blink of an eye when I forgot a period in my TSIG key configuration while studying. Make sure you use the resources available to you, like system-config-bind and sample configs in /usr/share/doc/bind*/examples/. Get to know commands like dig really well.

If you're overwhelmed by OpenSSL's command line syntax, check out the /etc/pki/tls/misc/CA script. There are some handy comments at the top of the script that explain how to use it. You can also pluck OpenSSL commands right out of the script if you need to run them yourself.

• Don't just memorize. Do some research to understand how everything fits together.
• Manage your time. DNS and Kerberos have lots of small nuances that can become time sinks when done incorrectly.
• Use the available documentation and tools. Try practicing without study materials so that you're forced to use the docs and tools available within the server.

Ranking the exams
A couple of folks on Twitter asked me to rank the exams from most difficult to least difficult. Keep in mind that these are a little subjective since I was more familiar with some objectives than others for certain tests.

• EX333 - Enterprise Security: Network Services: a tubload of material and a very long exam
• EX442 - System Monitoring and Performance Tuning: very difficult to check your work, lots of calculations
• EX423 - Directory Services and Authentication: not a lot of material to cover, but tons of curveballs
• EX436 - Storage Management: the web interface made things much easier, lots of documentation available
• EX401 - Deployment and Systems Management: every objective can be tested, I build RPM's already

Looking back at the long road to becoming a Red Hat Certified Architect is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

However, what if the RPM isn't installed already? How do you figure out which one to install?

Fedora has well over 20,000 packages in the standard repositories without adding additional repositories like RPM Fusion. Narrowing that list down to find the package you want can be daunting, but you can use yum to help.

Consider this: you're following a guide online and the author says you need to run deallocvt:

# deallocvt
-bash: deallocvt: command not found

Perhaps it's in a package with deallocvt in the name:

# yum search deallocvt
Warning: No matches found for: deallocvt
No Matches found

This is where yum's whatprovides (provides works in recent yum versions) command works really well:

# yum whatprovides */deallocvt
kbd-1.15-11.fc14.x86_64 : Tools for configuring the console
Repo        : fedora
Matched from:
Filename    : /usr/bin/deallocvt

From there, you can install the kbd RPM package via yum and you'll be on your way.

Author's note: Regular readers will probably think this is pretty basic, but I often find people who don't know this functionality exists in yum.

UPDATE: I forgot to include another handy command in this article (thanks to Jason Gill for reminding me). If you have file on your system already, but you need to know which RPM package it came from, you can do this very quickly:

# rpm -qf /usr/bin/free
procps-3.2.8-14.fc14.x86_64

Locate RPM packages which contain a certain file is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Here are the steps for an upgrade from Fedora 13 to Fedora 14 via yum:

yum -y upgrade
wget http://mirror.rackspace.com/fedora/releases/14/Fedora/x86_64/os/Packages/fedora-release-14-1.noarch.rpm
rpm -Uvh fedora-release-14-1.noarch.rpm
yum -y install yum
yum -y upgrade

If you happen to be upgrading a 32-bit instance on Slicehost, simply replace x86_64 with i386 in the url shown above.

Upgrading Fedora 13 to Fedora 14 on Slicehost and Rackspace Cloud Servers is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

To reduce the total data downloaded, I'd recommend installing the yum-presto package first. It downloads delta RPM's and builds them on the fly, which allows you to upgrade packages without having to download the entire RPM's.

yum install yum-presto

Now, upgrade your current system to the latest packages and clean up yum's metadata:

yum upgrade
yum clean all

Get the latest fedora-release package and install it (replace x86_64 with x86 if you're using a 32-bit system):

wget ftp://download.fedora.redhat.com/pub/fedora/linux/releases/12/Fedora/x86_64/os/Packages/fedora-release-*.noarch.rpm
rpm -Uvh fedora-release-*.rpm

Now, upgrade your system to Fedora 12:

yum upgrade

For detailed documentation on the entire process, refer to Fedora using yum on the FedoraProject Wiki.

Upgrading Fedora 11 to 12 using yum is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

» What the Fedora developers suggest:

yum -y upgrade
yum -y install preupgrade
yum clean all
preupgrade-cli "Fedora 11 (Leonidas)"

Of course, if you're doing this on a Fedora desktop, you can use preupgrade (rather than preupgrade-cli) to upgrade with a GUI.

» The method I prefer (and it works properly on Slicehost):

yum -y upgrade
yum clean all
wget http://download.fedora.redhat.com/pub/fedora/linux/releases/11/Fedora/x86_64/os/Packages/fedora-release-11-1.noarch.rpm
rpm -Uvh fedora-release-11-1.noarch.rpm

At this point, you would normally just start upgrading packages, but the Fedora developers threw us a curveball. Since yum in Fedora 10 doesn't support metalinks, your upgrades will fail with something like this:

# yum -y upgrade
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. /
removing mirrorlist with no valid mirrors: //var/cache/yum/updates/mirrorlist.txt
Error: Cannot retrieve repository metadata (repomd.xml) for repository: updates. Please verify its path and try again

It's easily fixed, however. Open up /etc/yum.repos.d/fedora.repo and /etc/yum.repos.d/fedora-updates.repo in your favorite text editor and change the mirrorlist URL's like so:

Fedora Repository

#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch
mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch

Fedora Updates Repository

#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f$releasever&arch=$basearch

Once you make those changes, finish out the upgrade:

yum -y upgrade

This process will take a little while to complete, but there shouldn't be any interaction required. Once it's done, change the mirrorlist lines back to the original values so you can benefit from the speedups provided by the metalink format.

Upgrading from Fedora 10 (Cambridge) to Fedora 11 (Leonidas) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First, pull a list of RPM package by name only:

rpm -qa --queryformat='%{NAME}\n' | sort > server.txt

Once you've done that on both servers, just use diff to compare the two files:

diff serverold.txt servernew.txt

Compare the RPM packages installed on two different servers is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you have an RPM-based linux distribution, you can use a spec file that is available on SourceForge. You can follow the instructions on the project's page, or you can follow these abbreviated instructions here:

Install some prerequisites:

// RHEL 4
up2date -i rpm-build wget ttmkfdir
// RHEL 5
yum install rpm-build wget ttmkfdir

Install cabextract.

Build the RPM:

wget -O /usr/src/redhat/SPECS/msttcorefonts-2.0-1.spec http://corefonts.sourceforge.net/msttcorefonts-2.0-1.spec
rpmbuild -bb msttcorefonts-2.0-1.spec
rpm -Uvh /usr/src/redhat/SPECS/msttcorefonts-2.0-1.spec

Test it to be sure that they're installed:

xlsfonts | grep ^-microsoft
rpm -ql msttcorefonts

Installing Microsoft's TrueType fonts on Linux servers is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The errors:

rpmdb: Lock table is out of available locker entries
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm

Make a backup of /var/lib/rpm in case you break something:

tar cvzf rpmdb-backup.tar.gz /var/lib/rpm

Remove the Berkeley databases that rpm uses:

rm /var/lib/rpm/__db.00*

Make rpm rebuild the databases from scratch (may take a short while):

rpm --rebuilddb

Now, check rpm to make sure everything is okay:

rpm -qa | sort

Why does this happen?
When rpm accesses the Berkeley database files, it makes temporary locker entries within the tables while it searches for data. If you control-c your rpm processes often, this issue will occur much sooner because the locks are never cleared.

rpmdb: Lock table is out of available locker entries is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.