If you haven't tried dstat (I hadn't until I saw Kristóf's post), this is a great one to try. You can keep a running tally on various server metrics including load average, network transfer, and disk operations.

Here is some sample output:

----total-cpu-usage---- ---paging-- ---load-avg--- ------memory-usage----- -net/total- ---procs--- --io/total- ---system-- ----tcp-sockets----
usr sys idl wai hiq siq|  in   out | 1m   5m  15m | used  buff  cach  free| recv  send|run blk new| read  writ| int   csw |lis act syn tim clo
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1314B  180B|  0   0   0|   0     0 |  70    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1779B 1004B|  0   0   0|   0     0 |  84    78 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M| 904B  362B|1.0   0 1.0|   0     0 |  75    86 | 13   9   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  386M|2203B 1559B|  0   0   0|   0     0 | 180   127 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  386M| 260B  130B|  0   0   0|   0     0 |  53    66 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|  52B  114B|  0   0   0|   0     0 |  54    77 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|2271B  872B|  0   0   0|   0     0 |  94    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|  52B  130B|  0   0   0|   0     0 |  54    74 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1126B 1254B|  0   0   0|   0  24.0 |  80    87 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1030B  130B|  0   0   0|   0     0 |  88    82 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 578B  114B|  0   0   0|   0     0 |  53    64 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1597B  890B|  0   0   0|   0     0 |  85    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 552B  114B|  0   0   0|   0     0 |  63    77 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1624B 1254B|  0   0   0|   0     0 |  81    75 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 478B  114B|  0   0   0|   0     0 |  67    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 418B  114B|  0   0   0|   0     0 |  59    74 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1265B  874B|  0   0   0|   0     0 |  82    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 758B  114B|  0   0   0|   0     0 |  60    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1236B 1255B|  0   0   0|   0  4.00 |  93    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|  52B  130B|  0   0   0|   0     0 |  71    70 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 214B  114B|  0   0   0|   0     0 |  55    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M|1201B  890B|  0   0   0|   0     0 |  80    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 108B  114B|  0   0   0|   0     0 |  53    66 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M|1344B 1254B|  0   0   0|   0  10.0 | 119    85 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 172B  130B|  0   0   0|   0  8.00 |  80    82 | 13   7   0   0   5

Learn more about dstat on Dag Wieërs' site.

Lesser-known but extremely handy Linux tools is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

My mother was always the one who told me (and her students) that everyone has a story. She said that writing could be therapeutic in ways you probably won't consider until you've written something that someone else enjoys. Just as software developers exist to write software for their users, writers exist to write stories for their readers. There's nothing that says technical people can't become excellent writers who inspire others to learn and share their knowledge with others.

The goal of this post is to encourage technical people to enjoy writing, write efficiently and feel comfortable doing it. I'll roll through some of the most common responses I've received about why technical people don't blog about what they know.

I don't think I'm really an expert on anything. I'm not an authority on any topic I can think of.

I'm leading off with this response because it's the most critical to refute. If you don't take away anything else from this post, let it be this: you don't need to be an expert on a topic to write about it.

You can find examples of this by rolling through some of the posts on my blog. I'd consider myself to be an expert on one, maybe two topics, but I've written over 450 posts in the span of just over five years. I certainly didn't write all of those about the one or two topics I know best.

Write about what you know and don't be afraid to do a little research to become an authority on something. A great example of this was my post, entitled "Kerberos for haters." I had almost no expertise in Kerberos. In fact, I couldn't even configure it properly for my RHCA exam! However, I did a ton of research and began to understand how most of the pieces fit together. Many other people were just as confused and I decided to pack all of the knowledge I had about Kerberos into a blog post. Positive and negative feedback rolled in and it was obvious that my post taught some readers, inspired some others and angered a few.

What a great way to lead into the next response:

What if I say something that isn't correct? I'll look like an idiot in front of the whole internet!

Been there, done that. Every writer makes errors and comes up with bad assumptions at least once. Readers will call you out on your mistakes (some do it delicately while others don't) and it's your duty to correct your post or correct the reader. I've written posts with errors, and I've gotten a little lazy on my fact-checking from time to time. As my middle school journalism teacher always reminded me, the most important part of a mistake is what you do to clean it up and learn from it.

In short: you'll make mistakes. As long as you've done your due diligence to minimize them and respond to them promptly, your readers should forgive you.

Speaking of errors:

I'm great at a command prompt but my spelling and grammar are awful. I write terribly.

This is easily fixed. If you're one of those folks who live the do-it-yourself type of lifestyle, pick up a copy of The Elements of Style by Strunk & White. There are free PDF versions online or you can borrow one from your nearest journalist. No matter the situation you're in, this book has details about where punctuation should and shouldn't be, how to structure sentences and paragraphs, and how to properly cite your sources (really vital for research posts).

Hauling around a copy of an ultra-dry reference book may not be your thing. If that's the case, find someone you know who has a knack for writing. You can usually find helpful folks in marketing or corporate communications in most big companies who will take your post and return it covered in red ink ready for corrections (thanks, Garrett!). I've even spotted some folks on Fiverr who will do this for as low as $5.

I'll wrap up with the second most common response:

I don't know who I'm writing for? What if I write about something simple and the really technical folks think I'm a noob? What if I write something crazy complex and it goes over most people's heads?

I've done both of these. Most Linux system administrators worth their salt know how to add and remove iptables rules, and they'd consider it to be pretty trivial work. Would it surprise you to know that out of over 450 posts, my post about deleting a single iptables rule is in the top five most accessed posts per month? I receive just over 11 percent of my monthly hits to this post. People are either learning from it or they can't remember how to delete the rule and they want to use the post as a quick reference. Either way, the post is valuable to many people even if I think it's the simplest topic possible.

On the flip side, I went nuts and wrote up a complete how-to for a redundant cloud hosting configuration complete with LVS, glusterfs, MySQL on DRBD, memcached, haproxy and ldirectord. I thought it would be valuable knowledge to a few folks but that it might sail over the heads of most of my readers. Again, I was wrong. The post is constantly in the top 10 most visited posts on the blog and I've probably received more feedback via comments, email and IRC about that post than any other. Once again, a post I thought would be mostly useless turned into a real conversation starter.

Let's conclude and wrap up. Keep these things in mind if you feel discouraged about writing:

• Write about what interests you whether you're an expert on it or not
• Don't be afraid to fail
• Be responsive to your readers
• Even if you think nobody will read your post, write it
• Always ensure your voice shines through in your writing — this is what makes it special and appealing

Why technical people should blog (but don't) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Getting yourself ready for any type of examination is usually a stressful experience that involves procrastination and some late nights leading up to the test. Every time I take one, I always say to myself, “I’m really going to get ahead of this next time and study early. This last minute stuff is terrible.” But I always forget all of this as the next exam rolls around.

Quick note: As you read through the remainder of the post, you may wonder why some of it is a bit vague. Every Red Hat test taker is under a NDA to prevent disclosure of test information that may reduce the security of the exam itself. Penalties start with losing credit for the exams previously taken and they can escalate up to legal action. I hope you’ll understand why I’m not able to go into details about certain portions of the Red Hat examinations.

I’ve taken seven Red Hat exams already: two for the RHCE and five for the RHCA. These tests certainly aren’t easy, but there are some good guidelines and tips you can use to make your studying efforts less stressful and more productive. Without further ado, here are my recommendations for prospective Red Hat examinees:

Build a flexible study environment

This is critical. You’ll need some spare servers or some available virtual machines to practice the objectives on each exam. However, don’t feel like you need to spend the money on a Red Hat subscription to get your studying done. Most of the test objectives on the majority of exams can be completed with very similar Linux distributions, like Scientific Linux or CentOS. Look for a version of the distribution that is closest to what you’ll be tested on at exam time. Your study environment should meet some basic criteria:

• You should be able to quickly build and tear down servers or virtual machines
• Keep the latency to your environment low to avoid getting frustrated
• Use applications like VirtualBox, VMWare Fusion/Workstation to practice on your own computer
• Consider using VMs from cloud providers if you’re under a time crunch

Some exams may require some bare-metal access to the server itself (especially EX442), so keep that in mind when you’re looking for a good practice environment. You may need some specific network or storage setups for some exams (as with EX436). If you’re not sure what you need, be sure to ask your instructor or someone else you know who has taken the exam already.

Prioritize doing over reading

The Red Hat exams are all hands-on, practical exams. You won’t find any essays or multiple-choice questions in these exams. Although the materials from Red Hat are full of good information, reading this information can only get you so far. You need to practice setting up the services on your own to be fully prepared for the test. If you’re not pressed for time, reading through the book can give you some details about the lab sequences, which you might miss by solely reading through labs themselves.

Research the why, not the what, to remember

This is especially important for the RHCA exam track. You may find that there is a ton of material to cover for the exam and that it’s difficult to remember each command to bring a certain service online or to repair a problem. Instead of thinking through the problem as “first, I do this, then I do this”, try to understand why each step is important in the first place.

Here’s a good example. I’ll be the first one to admit that Kerberos drives me crazy. I’ve even written posts about it. The commands seemed really archaic, the daemons didn’t make sense, and the lack of readline support in the Kerberos tools made me want to throw my computer out the window (come on, MIT!). I put my class materials aside, went to Google in a browser, and started researching Kerberos.

I read some of MIT’s documentation, ventured over to Wikipedia, and poked at some of the documentation within the Kerberos RPM packages. After a while, I began to realize how it all fit together. “Okay,” I thought to myself, “I need principals in a keytab to do these things, but I need to have a database for the admin stuff first.” Suddenly, the order of things in my head wasn’t just memorized any longer. The process of operations seemed to make logical sense because I fully understood how the pieces of a Kerberos infrastructure fit together.

If you start to get discouraged, take a break and learn more about why you’re doing what you’re doing. Once it becomes second nature, working through the problems on the exam becomes much easier.

Lean on your available resources

Don’t forget that there are other knowledgeable folks available to talk to when you get bogged down. Lean on other RHCE’s, RHCA’s, or experienced Linux users to get the answers or explanations you need. If you already have a Red Hat certification, head over to the Red Hat Certification Forums and meet up with other examinees that are discussing test preparation.

Also, you’ll find some knowledgeable (but sometimes snarky or quirky) people on IRC who are eager to point you in the right direction. Try the #rhel, #centos, or #fedora channels if you’re struggling through the configuration of a certain service. Many Linux users may roll their eyes about it, but Twitter is also a pretty good way to reach out to people who have a lot of Linux experience.

Summary

Remember to lean on the knowledge of others, get hands-on with the test objectives and do your research when you’re frustrated. The exams from Red Hat are generally difficult and cover a lot of material, but with the right amount of preparation and determination you can pass the exams and get the certifications you want.

Preparing for Red Hat Exams is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Photo credit: Will Clayton

As I mentioned already, the blog started out just as a place to stuff my notes from the things I learned at work. I figured that it would be nice to store it in a searchable format but it would also be great if I could link other people to certain posts if they needed more information to fix a problem. It was a way to retain knowledge but yet give it back to the people around me who needed it.

The blog has hit 456 posts (this one is #457) and it's gone from a few page views per day to just over 20,000 per day. Here are the top five most accessed posts (since I've been keeping stats):

1. Syncing an iPhone with a new Mac without hassles
2. ip_conntrack: table full, dropping packet
3. Delete a single iptables rule
4. Increase MySQL connection limit
5. MySQL Error 1040: too many connections

I'd like to send out a big thanks to the people who read this blog, add comments (or complaints!), and suggest new topics. You are the reason why I take the time to keep this blog going.

Five years of rackerhacker.com is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First things first: there are going to be many part of this post which probably aren't as specific as you'd like. A lot of that is due to the NDA that all Red Hat examinees agree to when they take an exam. We aren't allowed to talk about what was on the exam or our experiences during the exam. If we do, penalties range from smaller things like losing certifications all the way up to serious stuff like legal action. It goes without saying that I want to protect the security of the exams, I don't want to lose my certifications, and I don't want to hire a lawyer. Please try to keep this in mind if you yearn for more specifics than I'm able to give.

Red Hat Certified Engineer
The RHCSA and RHCE exams are the first step on the path to the RHCA. You can't take any of the RHCA prerequisite exams without it. These exams cover a really broad spectrum of material including apache configuration, NFS, iptables and mail services. The two links above will take you to the exam objectives for each exam.

I've always recommended the RHCE exam for Linux administrators who are trying to sharpen their skills and get to the next level whether they use Red Hat or not. The exam covers a lot of good material that makes a solid foundation for any Linux user without throwing in too many Red Hat-specific knowledge.

The exam (like all Red Hat exams) is fully practical. There are no multiple choice questions or essays. You'll have to meet all of the objectives by logging into a local Red Hat system and making the system do what it needs to do.

Quick tips for the RHCSA/RHCE exams:

• Keep your eye on the clock. Time can really get away from you if you get stuck in the weeds on a problem that should be relatively straightforward.
• Leave time at the end to check your work. When you set up a lot of services, it's inevitable that you might configure a service for one problem that breaks the functionality required by a problem you completed already.
• Always reboot before you leave. We all forget to use chkconfig when we're in a hurry.
• Practice, practice, practice. There's not one objective on this exam that you can't test in a VM on your own.

Red Hat Enterprise System Monitoring and Performance Tuning
Our group at Rackspace started off with EX442 and it was a very difficult way to start off the RHCA track. Take a look at the objectives and you'll see that much of the exam is related to tweaking system performance and then monitoring that performance with graphs and raw data. You'll have to turn a lot of knobs on the kernel and you'll need to know where to store these configurations so they'll be persistent.

In addition, the objective regarding TCP buffers and related settings is a real challenge. You'll have to wrestle with some math that appears to be relatively simple, but can get confusing quickly. Some of the settings can't really be checked to know if your setting is correct. The objectives mention tuning disk scheduling -- you don't really have the time or tools to know if your setting is ideal.

Quick tips for EX442:

• Use the documentation available to you. Install the kernel-doc package while you practice and during the exam.
• Be careful with your math. You have a Linux machine in front of you! Don't forget about bc.
• Watch your units. Know the difference between a kilobyte (KB) and a kibibyte (KiB).
• Make comments in files where you adjust kernel configurations. It will help you keep track of which question the kernel adjustment is meant to satisfy.

Red Hat Enterprise Storage Management
I'm surprised to say this now, but I actually enjoyed EX436. I've always used other clustering tools like heartbeat and pacemaker, but I've never had the need to use the Red Hat Cluster Suite. Although RHCS definitely has a lot of quirks and rough edges, it's pretty solid once you get familiar with the GUI and command line tools.

You get the opportunity to mess around with some pretty useful technology like iSCSI, GFS, and clustered LVM. These are things that you're probably already using or will be using soon in a large server environment. The web interface for RHCS is quite peculiar and you may find yourself wanting to put your fist through the screen when you're staring down the endless animated GIFs when the cluster is syncing its configuration. Do your best to be patient because you certainly don't want to short circuit the cluster sync.

Quick tips for EX436:

• Be patient. You'll feel like the RHCS web interface is mocking you when you're pressed for time.
• Watch the clock. It's extremely easy to burn a lot of time on this exam if you get stuck on a particular problem.
• Double check your entries in the web interface. Make sure you're doing things in the right order and that you've set up the prerequisites before adding services to the cluster. If you get it wrong, you could put your cluster into a weird state.
• Use man pages. If you don't mess with GFS a lot, the man pages will save you in a pinch.

Red Hat Enterprise Deployment and Systems Management
If there's one exam where time management is critical, it's EX401. Importing data into the Satellite Server takes quite a bit of time and there's almost nothing you can do to speed it up. It probably goes without saying, but as with most long-running tasks, you'll want to run it in screen. The last thing you'd ever want is to abort the import due to an errant click or CTRL-C (I did it while practicing -- it's aggravating).

There are other test objectives which you can either complete or partially complete while you wait for the import to finish.

Also, take the time to really dig into the Satellite Server web interface while your practicing for the exam. Knowing where to find the most common configuration items will really save some time when you're in the exam. You can sometimes get pretty bogged down in the interface so don't forget to use multiple tabs to keep your work organized.

I felt like this exam was the easiest out of the bunch since you could go back and test every single question with good time management. Did I mention how important time management was on this exam already? If I forgot to mention it earlier, be sure to focus on time management for this test.

Quick tips for EX401:

• Time management will make or break you on this test. Keep an eye on the clock and make sure you've done absolutely every piece of the exam that you can while you wait for the server to do its work.
• Scour the web interface. Keep a mental map in your mind where the big chunks of configuration items are.
• Go back and test everything. If you manage your time well, you should have enough time to verify each and every objective on this exam.

Red Hat Enterprise Directory Services and Authentication
At first, EX423 looks pretty straightforward. Red Hat's authentication configuration tools make LDAP authentication setup pretty easy. However, this exam comes with a lot of curveballs.

The GUI interface for the Directory Services component is a little frustrating to use. I found that the GUI stopped responding to keyboard input occasionally unless I clicked on another window and came back. If you misconfigure the SSL certificates in the interface, your LDAP server is down for the count. If you don't input the correct data into the setup scripts at the beginning, you might not notice it until much later when it's either too difficult to dig yourself out of the hole or it's too late to start over with a clean configuration.

I didn't feel pressed for time on this exam too much and that was pretty refreshing after taking the EX401 test. It's extremely critical to watch what you type and click on this exam. Some mistakes can be quickly corrected while others may require you to blow away the LDAP server configuration and re-provision the whole thing.

Quick tips for EX423:

• Always watch what you're typing. A simple mistake can lead to confusion or bigger issues down the road.
• Don't ignore the LDIF objectives. As you practice, you'll find that manipulating LDIF files is a little more involved than you expected.
• Practice starting over. Throw out your Directory Services configuration and get the experience of what it's like to start over and get back in the game.

Red Hat Enterprise Security: Network Services
There's no sugar coating it -- EX333 is a beast. It's a six hour exam broken into two three-hour chunks. It covers a ton of material and I refer to it as "the RHCE on steroids." You might argue that I thought it was hard since it was the last test and I was ready to be finished, but I really think this exam is a tough one.

Practicing for the Kerberos and DNS objectives was the hardest for me. I just couldn't understand Kerberos, no matter how hard I tried. The realization that I would really have to learn it soon set in. I dug into the Kerberos design documentation on MIT's site, read the summaries on Wikipedia, and scoured the documentation available in the Kerberos RPM packages. Once I understood why Kerberos is set up the way it is and why the security measures are present, everything began to come together. I was able to remember the steps not because I was memorizing them, but because I understood how Kerberos worked.

When you're working through the DNS objectives, keep an eye out for punctuation. I blew through a good 20 minutes in what seemed like the blink of an eye when I forgot a period in my TSIG key configuration while studying. Make sure you use the resources available to you, like system-config-bind and sample configs in /usr/share/doc/bind*/examples/. Get to know commands like dig really well.

If you're overwhelmed by OpenSSL's command line syntax, check out the /etc/pki/tls/misc/CA script. There are some handy comments at the top of the script that explain how to use it. You can also pluck OpenSSL commands right out of the script if you need to run them yourself.

• Don't just memorize. Do some research to understand how everything fits together.
• Manage your time. DNS and Kerberos have lots of small nuances that can become time sinks when done incorrectly.
• Use the available documentation and tools. Try practicing without study materials so that you're forced to use the docs and tools available within the server.

Ranking the exams
A couple of folks on Twitter asked me to rank the exams from most difficult to least difficult. Keep in mind that these are a little subjective since I was more familiar with some objectives than others for certain tests.

• EX333 - Enterprise Security: Network Services: a tubload of material and a very long exam
• EX442 - System Monitoring and Performance Tuning: very difficult to check your work, lots of calculations
• EX423 - Directory Services and Authentication: not a lot of material to cover, but tons of curveballs
• EX436 - Storage Management: the web interface made things much easier, lots of documentation available
• EX401 - Deployment and Systems Management: every objective can be tested, I build RPM's already

Looking back at the long road to becoming a Red Hat Certified Architect is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

git clone git://github.com/rackerhacker/kickstarts.git kickstarts

There are some big issues with Fedora 16 which cause problems for installations within XenServer:

• the installer sets up a console on something other than hvc0
• anaconda won't start without being in serial mode
• anaconda tries to use GPT partitions by default
• grub2 is now standard, but it causes problems for older XenServer versions

My kickstart works around the grub2 problem by throwing down an old-style grub configuration file and creating the proper symlinks. This config will still be updated when you upgrade kernels (at least in Fedora 16). It also sets up a very simple partitioning schema with one root and one swap partition. A DOS partition table is used in lieu of a GPT partition table.

When you start the installation, be sure to review the README.md in the git repository. It has some special instructions for boot options to meet the requirements of Fedora 16 and the kickstart file.

Installing Fedora 16 in XenServer is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

openssl s_client -connect encrypted.google.com:443

You'll see the chain of certificates back to the original certificate authority where Google bought its certificate at the top, a copy of their SSL certificate in plain text in the middle, and a bunch of session-related information at the bottom.

This works really well when a site has one SSL certificate installed per IP address (this used to be a hard requirement). With Server Name Indication (SNI), a web server can have multiple SSL certificates installed on the same IP address. SNI-capable browsers will specify the hostname of the server they're trying to reach during the initial handshake process. This allows the web server to determine the correct SSL certificate to use for the connection.

If you try to connect to rackerhacker.com with s_client, you'll find that you receive the default SSL certificate installed on my server and not the one for this site:

$ openssl s_client -connect rackerhacker.com:443
Certificate chain
 0 s:/C=US/ST=Texas/L=San Antonio/O=MHTX Enterprises/CN=*.mhtx.net
   i:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
 1 s:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
   i:/C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority

Add on the -servername argument and s_client will do the additional SNI negotiation step for you:

$ openssl s_client -connect rackerhacker.com:443 -servername rackerhacker.com
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=rackerhacker.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
 2 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

You may be asking yourself this question:

Why doesn't the web server just use the Host: header that my browser sends already to figure out which SSL certificate to use?

Keep in mind that the SSL negotiation must occur prior to sending the HTTP request through to the remote server. That means that the browser and the server have to do the certificate exchange earlier in the process and the browser wouldn't get the opportunity to specify which site it's trying to reach. SNI fixes that by allowing a Host: header type of exchange during the SSL negotiation process.

Using OpenSSL's s_client command with web servers using Server Name Indication (SNI) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Also, I don't really hate Kerberos. It's a bit of an inside joke with my coworkers who are studying for some of the RHCA exams at Rackspace. The additional security provided by Kerberos is quite good but the setup involves a lot of small steps. If you miss one of the steps or if you get something done out of order, you may have to scrap the whole setup and start over unless you can make sense of the errors in the log files. A lot of my dislikes for Kerberos comes from the number of steps required in the setup process and the difficulty in tracking down issues when they crop up.

To complete this guide, you'll need the following:

• two CentOS, Red Hat Enterprise Linux or Scientific Linux 5 servers or VM's
• some patience

Here's how I plan to name my servers:

• kdc.example.com - the Kerberos KDC server at 192.168.250.2
• client.example.com - the Kerberos client at 192.168.250.3

CRITICAL STEP: Before getting started, ensure that both systems have their hostnames properly set and both systems have the hostnames and IP addresses of both systems in /etc/hosts. Your server and client must be able to know the IP and hostname of the other system as well as themselves.

First off, we will need NIS working to serve up the user information for our client. Install the NIS server components on the KDC server:

[root@kdc ~]# yum install ypserv

Set the NIS domain and set a static port for ypserv to make it easier to firewall off. Edit /etc/sysconfig/network on the KDC server:

NISDOMAINNAME=EXAMPLE.COM
YPSERV_ARGS="-p 808"

Manually set the NIS domain on the KDC server and add it to /etc/yp.conf:

[root@kdc ~]# nisdomain EXAMPLE.COM
[root@kdc ~]# echo "domain EXAMPLE.COM server kdc.example.com" >> /etc/yp.conf

Adjust /var/yp/securenets on the KDC server for additional security:

[root@kdc ~]# echo "255.0.0.0 127.0.0.0" >> /var/yp/securenets
[root@kdc ~]# echo "255.255.255.0 192.168.250.0" >> /var/yp/securenets

Start the NIS server and generate the NIS maps:

[root@kdc ~]# /etc/init.d/ypserv start; chkconfig ypserv on
[root@kdc ~]# make -C /var/yp

I usually like to prepare my iptables rules ahead of time so I ensure that it doesn't derail me later on. Paste this into the KDC's terminal:

iptables -N SERVICES
iptables -I INPUT -j SERVICES
iptables -A SERVICES -p tcp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p udp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p tcp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p udp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p tcp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 464 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p tcp --dport 749 -j ACCEPT -m comment --comment "kerberos"
/etc/init.d/iptables save

We need our time in sync for Kerberos to work properly. Install NTP on both nodes, start it, and ensure it comes up at boot time:

[root@kdc ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start
[root@client ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start

Now we're ready to set up Kerberos. Start by installing some packages on the KDC:

[root@kdc ~]# yum install krb5-server krb5-workstation

We will need to make some edits to /etc/krb5.conf on the KDC to set up our KDC realm. Ensure that the default_realm is set:

default_realm = EXAMPLE.COM

The [realms] section should look like this:

[realms]
EXAMPLE.COM = {
	kdc = 192.168.250.2:88
	admin_server = 192.168.250.2:749
}

The [domain_realm] section should look like this:

[domain_realm]
kdc.example.com = EXAMPLE.COM
client.example.com = EXAMPLE.COM

Add validate = true within the pam { } block of the [appdefaults] section:

[appdefaults]
 pam = {
   validate = true

Adjust /var/kerberos/krb5kdc/kdc.conf on the KDC:

[realms]
EXAMPLE.COM = {
	master_key_type = des-hmac-sha1
	default_principal_flags = +preauth
}

There's one last configuration file to edit on the KDC! Ensure that /var/kerberos/krb5kdc/kadm5.acl looks like this:

*/admin@EXAMPLE.COM	    *

We're now ready to make a KDC database to hold our sensitive Kerberos data. Create the database and set a good password which you can remember. This command also stashes your password on the KDC so you don't have to enter it each time you start the KDC:

kdb5_util create -r EXAMPLE.COM -s

On the KDC, create a principal for the admin user as well as user1 (which we'll create shortly). Also, export the admin details to the kadmind key tab. You'll get some extra output after each one of these commands but I've snipped it to reduce the length of the post.

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc root/admin
kadmin.local:  addprinc user1
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/admin
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/changepw
kadmin.local:  exit

Let's start the Kerberos KDC and kadmin daemons:

[root@kdc ~]# /etc/init.d/krb5kdc start; /etc/init.d/kadmin start
[root@kdc ~]# chkconfig krb5kdc on; chkconfig kadmin on

Now that the administration work is done, let's create a principal for our KDC server and stick it in it's keytab:

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc -randkey host/kdc.example.com
kadmin.local:  ktadd host/kdc.example.com

Transfer your /etc/krb5.conf from the KDC server to the client. Hop onto the client server, install the Kerberos client package and add some host principals:

[root@client ~]# yum install krb5-workstation
[root@client ~]# kadmin.local
kadmin.local:  addpinc --randkey host/client.example.com
kadmin.local:  ktadd host/kdc.example.com

There aren't any daemons on the client side, so the configuration is pretty much wrapped up there for Kerberos. However, we now need to tell both servers to use Kerberos for auth and your client servers needs to use NIS to get user data.

• On the KDC:
  • run authconfig-tui
  • choose Use Kerberos from the second column
  • press Next
  • don't edit the configuration (authconfig got the data from /etc/krb.conf)
  • press OK
• On the client:
  • run authconfig-tui
  • choose Use NIS and Use Kerberos
  • press Next
  • enter your NIS domain (EXAMPLE.COM) and NIS server (kdc.example.com or 192.168.250.2)
  • press Next
  • don't edit the Kerberos configuration (authconfig got the data from /etc/krb.conf)
  • press OK

Got NIS problems? If the NIS connection stalls on the client, ensure that you have the iptables rules present on the KDC that we added near the beginning of this guide. Also, if you forgot to add both hosts to both servers' /etc/hosts, go do that now.

Let's make our test user on the KDC. Don't add this user to the client -- we'll get the user information via NIS and authenticate via Kerberos shortly. We'll also rebuild our NIS maps after adding the user:

[root@kdc ~]# useradd user1
[root@kdc ~]# passwd user1
[root@kdc ~]# make -C /var/yp/

On the client, see if you can get the password hash for the user1 account via NIS:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:$1$sUlSTlCv$riK5El3z8N4y.mi5Fe3Q60:500:500::/home/user1:/bin/bash

You can see why NIS isn't a good way to authenticate users. Someone could easily pull the hash for any account and brute force the hash on their own server. Go back to the KDC and lock out the user account:

[root@kdc ~]# usermod -p '!!' user1

Go back to the client and try to pull the password hash now:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:!!:500:500::/home/user1:/bin/bash

On the plus side, the user's password hash is now gone. On the negative side, you've just prevented this user from logging in locally or via NIS. Don't worry, the user can log in via Kerberos now. Let's prepare a home directory on the client for the user:

[root@client ~]# mkdir /home/user1
[root@client ~]# cp -av /etc/skel/.bash* /home/user1/
[root@client ~]# chown -R user1:user1 /home/user1/

Note: In a real-world scenario, you'd probably want to export this user's home directory via NFS so they didn't get a different home directory on every server.

While you're still on the client, try to log into the client via the user. Use the password that you used when you created the user1 principal on the KDC.

[root@client ~]# ssh user1@localhost
user1@localhost's password:
[user1@client ~]$ whoami
user1

List your Kerberos tickets and you should see one for your user principal:

[user1@client ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500_fCKPnZ
Default principal: user1@EXAMPLE.COM
 
Valid starting     Expires            Service principal
02/05/12 14:18:53  02/06/12 00:18:53  krbtgt/EXAMPLE.COM@EXAMPLE.COM
	renew until 02/05/12 14:18:53

Your KDC should have a couple of lines in its /var/log/krb5kdc.log showing the authentication:

Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): AS_REQ (12 etypes {18 17 16 23 1 3 2 11 10 15 12 13}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=16 ses=16}, user1@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=18 ses=18}, user1@EXAMPLE.COM for host/client.example.com@EXAMPLE.COM

The first line shows that the client asked for a Authentication Server Request (AS_REQ) and the second line shows that the client then asked for a Ticket Granting Server Request (TGS_REQ). In layman's terms, the client first asked for a ticket-granting ticket (TGT) so it could authenticate to other services. When it actually tried to log in via ssh it asked for a ticket (and received it).

YOU JUST CONFIGURED KERBEROS!

From here, the sky's the limit. Another popular implementation of Kerberos is encrypted NFSv4. You can even go crazy and use Kerberos with apache.

Let me know if you have any questions about this post or if you spot any errors. With this many steps, there's bound to be a typo or two in this guide. Keep in mind that there are some obvious spots for network-level and service-level security improvements. This guide was intended to give you the basics and it doesn't cover all of the security implications involved with a Kerberos implementation.

The Kerberos-hater's guide to installing Kerberos is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

However, certain folks may find the need to host a local PyPi repository for their own packages. You may need it to store Python code which you don't plan to release publicly or you may need to add proprietary patches to upstream Python packages. Regardless of the reason to have it, a local PyPi repository is relatively easy to configure.

You'll need to start with a base directory for your PyPi repository. For this example, I chose /var/pypi. The directory structure should look something like this:

/var/pypi/simple/[package_name]/[package_tarball]

For a package like pip, you'd make a structure like this:

/var/pypi/simple/pip/pip-1.0.2.tar.gz

Once you have at least one package stored locally, it's time to configure apache. Here's a snippet from the virtual host I configured:

DocumentRoot /var/pypi/
ServerName pypi.example.com
 
Options +Indexes
 
RewriteEngine On
RewriteRule ^/robots.txt - [L]
RewriteRule ^/icons/.* - [L]
RewriteRule ^/index\..* - [L]
 
RewriteCond /var/pypi/$1 !-f
RewriteCond /var/pypi/$1 !-d
RewriteRule ^/(.*)/?$ http://pypi.python.org/$1 [R,L]

The last set of rewrite directives check to see if the request refers to an existing file or directory under your document root. If it does, your server will reply with a directory listing or with the actual file to download. If the directory or file doesn't exist, apache will send the client a redirection to the main PyPi site.

Reload your apache configuration to bring in your new changes. Let's try to download the pip tarball from our local server in the example I mentioned above:

$ curl -I http://pypi.example.com/simple/pip/
HTTP/1.1 200 OK
 
$ curl -I http://pypi.example.com/simple/pip/pip-1.0.2.tar.gz
HTTP/1.1 200 OK

I've obviously snipped a bit of the response above, but you can see that apache is responding with 200's since it has the directories and files that I was trying to retrieve via curl. Let's try to get something we don't have locally, like kombu:

$ curl -I http://pypi.example.com/simple/kombu/
HTTP/1.1 302 Found
Location: http://pypi.python.org/simple/kombu/

Our local PyPi repository doesn't have kombu so it will refer our Python tools over to the official PyPi repository to get the listing of available package versions for kombu.

Now we need to tell pip to use our local repository. Edit ~/.pip/pip.conf and add:

[global]
index-url = http://pypi.example.com/simple/

If you'd rather use easy_install, edit ~/.pydistutils.cfg and add:

[easy_install]
index_url = http://pypi.example.com/simple/

Once your tools are configured, try installing a package you have locally and try to install one that you know you won't have locally. You can add -v to pip install to watch it retrieve different URL's to get the packages it needs. If you spot any peculiar behavior or unexpected redirections, double-check your mod_rewrite rules in your apache configuration and check the spelling of your directories under your document root.

Create a local PyPi repository using only mod_rewrite is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Later on, I found that one of my servers (which I'd previously secured quite thoroughly) had some rogue processes running that were spawned through httpd. Had I actually been using SELinux in enforcing mode, those processes would have probably never even started.

If you're trying to get started with SELinux but you're not sure how to do it without completely disrupting your server's workflow, these tips should help:

Get some good reporting and monitoring
Two of the most handy SELinux tools are setroubleshoot and setroubleshoot-server. If you're running a server without X, you can use my guide for configuring setroubleshoot-server. You will receive email alerts within seconds of an AVC denial and the emails should contain tips on how to resolve the denial if the original action should be allowed. If the AVC denial caught something you didn't expect, you'll know about the potential security breach almost immediately.

Start out with SELinux in permissive mode
If you're overly concerned about SELinux getting in your way, or if you're enabling SELinux on a server that has been running without SELinux since it was installed, start out with SELinux in permissive mode. To make the change effective immediately, just run:

# setenforce 0
# getenforce
Permissive

Edit /etc/sysconfig/selinux to make it persistent across reboots:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=permissive

Adjust booleans before adding your own custom modules
There are a lot of booleans you can toggle to get the functionality you need without adding your own custom SELinux modules with audit2allow. If you wanted to see all of the applicable booleans for httpd, just use getsebool:

# getsebool -a | grep httpd
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_network_connect --> on
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> on
httpd_can_sendmail --> on
... and so on ...

Toggling booleans is easy with togglesebool:

# togglesebool httpd_can_network_memcache
httpd_can_network_memcache: active

Now httpd can talk to memcache. You can also use setsebool if you want to be specific about your setting (this is good for scripts):

# setsebool httpd_can_network_memcache on

Tracking your history of AVC denials
All of your AVC denals are logged by auditd in /var/log/audit/audit.log but it's not the easiest file to read and parse. That's where aureport comes in:

# aureport --avc | tail -n 5
45. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 4 fifo_file getattr system_u:object_r:postfix_public_t:s0 denied 1061
46. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 2 fifo_file write system_u:object_r:postfix_public_t:s0 denied 1062
47. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 2 fifo_file open system_u:object_r:postfix_public_t:s0 denied 1062
48. 01/24/2012 14:01:58 sendmail unconfined_u:system_r:httpd_t:s0 160 process setrlimit unconfined_u:system_r:httpd_t:s0 denied 1123
49. 01/24/2012 14:01:58 postdrop unconfined_u:system_r:httpd_t:s0 4 dir search system_u:object_r:postfix_public_t:s0 denied 1124

Summary
There's no need to be scared of or be annoyed by SELinux in your server environment. While it takes some getting used to (and what new software doesn't?), you'll have an extra layer of security and access restrictions which should let you sleep a little better at night.

Getting started with SELinux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Get informed about what the US government is trying to accomplish with SOPA and PIPA. Get involved and do what you can to prevent it from moving any further.

• WordPress > Help Stop SOPA/PIPA
• Reddit - SOPA
• Stop American Censorship
• Wikipedia - Stop Online Piracy Act
• EFF - Take Action
• Why Rackspace opposes the Stop Online Piracy Act
• Mozilla - Protect The Internet

You may need a little humor after all of that reading. Head on over to Know Your Meme (warning: NSFW language in certain areas) for a laugh.

Fight SOPA & PIPA is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

echo "user@example.com" > /root/.forward

With SELinux, you'll end up getting an AVC denial each time your mail server tries to read the contents of the .forward file:

type=AVC msg=audit(1325543823.787:7416): avc:  denied  { open } for  pid=9850 
  comm="local" name=".forward" dev=md0 ino=17694734 
  scontext=system_u:system_r:postfix_local_t:s0 
  tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file

The reason is that your .forward file doesn't have the right SELinux contexts. You can set the correct contest quickly with restorecon:

# ls -Z /root/.forward 
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 /root/.forward
# restorecon -v /root/.forward
restorecon reset /root/.forward context unconfined_u:object_r:admin_home_t:s0->system_u:object_r:mail_forward_t:s0
# ls -Z /root/.forward 
-rw-r--r--. root root system_u:object_r:mail_home_t:s0 /root/.forward

Try to send another e-mail to root and you should see the mail server forward the e-mail properly without any additional AVC denials.

SELinux and .forward files is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Upgrading Fedora via yum is not the recommended method. Your first choice for upgrading Fedora should be to use preupgrade. Seriously.

This begs the question: When should you use another method to upgrade Fedora? What other methods are there?

You have a few other methods to get the upgrade done:

• Toss in a CD or DVD: You can upgrade via the anaconda installer provided on the CD, DVD or netinstall media. My experiences with this method for Fedora (as well as CentOS, Scientific Linux, and Red Hat) haven't been too positive, but your results may vary.
• Download the newer release's fedora-release RPM, install it with rpm, and yum upgrade: This is the really old way of doing things. Don't try this (read the next bullet).
• Use yum's distro-sync functionality: If you can't go the preupgrade route, I'd recommend giving this a try. However, leave plenty of time to fix small glitches after it's done (and after your first reboot).

Personal anecdote time (Keep scrolling for the meat and potatoes)
I have a dedicated server at Joe's Datacenter (love those folks) with IPMI and KVM-over-LAN access. The preupgrade method won't work for me because my /boot partition is on a software RAID volume. There's a rat's nest of a Bugzilla ticket over on Red Hat's site about this problem. I'm really only left with a live upgrade using yum.

Live yum upgrade process
Before even beginning the upgrade, I double-checked that I'd applied all of the available updates for my server. Once that was done, I realized I was one kernel revision behind and I rebooted to ensure I was in the latest Fedora 15 kernel.

A good practice here is to run package-cleanup --orphans (it's in the yum-utils package) to find any packages which don't exist on any Fedora mirrors. In my case, I had two old kernels and a JungleDisk package. I removed the two old kernels (probably wasn't necessary) and left JungleDisk alone (it worked fine after the upgrade). If you have any external repositories, such as Livna or RPMForge, you may want to disable those until the upgrade is done. Should the initial upgrade checks bomb out, try adding as few repositories back in as possible to see if it clears up the problem.

Once you make it this far, just follow the instructions available in Fedora's documentation: Upgrading Fedora using yum. I set SELinux to permissive mode during the upgrade just in case it caused problems.

I'd recommend skipping the grub2-install portion since your original grub installation will still be present after the upgrade. If your server has EFI (not BIOS), don't use grub2 yet. Keep an eye on the previously mentioned documentation page to see if the problems get ironed out between grub2 and EFI.

Before you reboot, be sure to get a list of your active processes and daemons. After your reboot, some old SysVinit scripts will be converted into Systemd service scripts. They might not start automatically and you might need to enable and/or start some services.

New to Systemd? This will be an extremely handy resource: SysVinit to Systemd Cheatsheet.

I haven't seen too many issues after cleaning up some daemons that didn't start properly. There is a problem between asterisk and SELinux that I haven't nailed down yet but it's not a showstopper.

Good luck during your upgrades. Keep in mind that Fedora 15 could be EOL'd as early as May or June 20102 when Fedora 17 is released.

Live upgrade Fedora 15 to Fedora 16 using yum is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

For example, all onboard ethernet adapters are labeled as emX (em1, em2...) and all PCI ethernet adapters are labeled as pXpX (p[slot]p[port], like p7p1 for port 1 on slot 7). Ethernet devices within Xen virtual machines aren't adjusted.

This may make sense to people who swap out the chassis on servers regularly and they don't want to mess with hard-coding MAC addresses in network configuration files. Also, it should give users predictable names even if a running system's drives are inserted into a newer hardware revision of the same server.

However, I don't like this on my personal dedicated servers and I prefer to revert back to the old way of doing things. Getting back to eth0 is pretty simple and it only requires a few configuration files to be edited followed by a reboot.

First, add biosdevname=0 to your grub.conf on the kernel line:

title Fedora (2.6.40.4-5.fc15.x86_64)
	root (hd0,0)
	kernel /boot/vmlinuz-2.6.40.4-5.fc15.x86_64 ro root=/dev/md0 SYSFONT=latarcyrheb-sun16 KEYTABLE=us biosdevname=0 quiet LANG=en_US.UTF-8
	initrd /boot/initramfs-2.6.40.4-5.fc15.x86_64.img

Open /etc/udev/rules.d/70-persistent-net.rules in your favorite text editor (create it if it doesn't exist) and add in the following:

# Be sure to put your MAC addresses in the fields below
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:10", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:11", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Be sure to rename your ifcfg-* files in /etc/sysconfig/network-scripts/ to match the device names you've assigned. Just for good measure, I add in the MAC address in /etc/sysconfig/network-scripts/ifcfg-ethX:

...
HWADDR=00:11:22:33:44:10
...

Reboot the server and you should be back to eth0 and eth1 after a reboot.

Getting back to using eth0 in Fedora 15 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The purpose behind setroubleshoot is to let users know when access has been denied, help them resolve it if necessary, and to reduce overall frustration while working through tight security restrictions in the default SELinux policies. The GUI frontend for setroubleshoot is great for users who run Linux desktops or those who run servers with a display attached. Don't worry, you can configure setroubleshoot on remote servers to send alerts elsewhere when a GUI alert isn't an option.

Install a few packages to get started:

yum install setroubleshoot{-server,-plugins,-doc}

Open /etc/setroubleshoot/setroubleshoot.conf in your favorite text editor and adjust the [email] section to fit your server:

recipients_filepath = /var/lib/setroubleshoot/email_alert_recipients
smtp_port = 25
smtp_host = localhost
from_address = selinux@myserver.com
subject = [MyServer] SELinux AVC Alert

You could probably see it coming, but you need to put the e-mail addresses for your recipients into /var/lib/setroubleshoot/email_alert_recipients:

echo "selinux@mycompany.com" >> /var/lib/setroubleshoot/email_alert_recipients

You'll notice that setroubleshoot doesn't have an init script and it doesn't exist in systemd in Fedora 15. It runs through the dbus-daemon and a quick bounce of the messagebus via its init script brings in the necessary components to run setroubleshoot:

service messagebus restart

A really easy (and safe) test is to ask sshd to bind to a non-standard port. Simply define an additional port on in your /etc/ssh/sshd_config like this:

Port 22
Port 222

When you restart sshd, it will bind to port 22 with success, but it won't be allowed to bind to port 222 (since that's blocked by SELinux as a non-standard port for the ssh_port_t port type). DON'T WORRY! Your sshd server will still be listening on port 22. If you wait a moment, you'll get an e-mail (perhaps two) that not only notify you of the denial, but they make suggestions for how to fix it:

SELinux is preventing /usr/sbin/sshd from name_bind access on the tcp_socket port 222.
 
*****  Plugin bind_ports (99.5 confidence) suggests  *************************
 
If you want to allow /usr/sbin/sshd to bind to network port 222
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 222
   where PORT_TYPE is one of the following: ...

For this particular example, the quick fix would be to run:

semanage port -a -t ssh_port_t -p tcp 222

Receive e-mail reports for SELinux AVC denials is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

[Thu Sep 08 03:23:40 2011] [error] [client 11.22.33.44] PHP Warning:  
Unknown: Failed to write session data (memcached). Please verify that 
the current setting of session.save_path is correct (127.0.0.1:11211) 
in Unknown on line 0

I ran through my usual list of checks:

• netstat showed memcached bound to the correct ports/interfaces
• memcached was running and I could reach it via telnet
• memcached-tool could connect and pull stats from memcached
• double-checked my php.ini
• tested memcached connectivity via a PHP and ruby script -- they worked

Even after all that, I still couldn't figure out what was wrong. I ran strace on memcached while I ran a curl against the page which creates a session and I found something significant -- memcached wasn't seeing any connections whatsoever at that time. A quick check of the lo interface with tcpdump showed the same result. Just before I threw a chair, I remembered one thing:

SELinux.

A quick check for AVC denials showed the problem:

# aureport --avc | tail -n 1
4021. 09/08/2011 03:23:38 httpd system_u:system_r:httpd_t:s0 42 tcp_socket name_connect system_u:object_r:memcache_port_t:s0 denied 31536

I'm far from being a guru on SELinux, so I leaned on audit2allow for help:

# grep memcache /var/log/audit/audit.log | audit2allow
 
#============= httpd_t ==============
#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_relay, httpd_can_network_memcache, httpd_can_network_connect
 
allow httpd_t memcache_port_t:tcp_socket name_connect;

The boolean we're looking for is httpd_can_network_memcache. Flipping the boolean can be done in a snap:

# setsebool -P httpd_can_network_memcache 1
# getsebool httpd_can_network_memcache
httpd_can_network_memcache --> on

After adjusting the boolean, apache was able to make connections to memcached without a hitch. My page which created sessions loaded quickly and I could see data being stored in memcached. If you want to check the status of all of the apache-related SELinux booleans, just use getsebool:

# getsebool -a | grep httpd | grep off$
allow_httpd_anon_write --> off
allow_httpd_mod_auth_ntlm_winbind --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_sys_script_anon_write --> off
httpd_can_check_spam --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_read_user_content --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_tmp_exec --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off

If you're interested in SELinux, a good way to get your feet wet is to head over to the CentOS Wiki and review their SELinux Howtos

Getting apache, PHP, and memcached working with SELinux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you're not technical, you might be asking yourself: "I need to e-mail technical people and they need to take what I say seriously? How do I do it?" It's not impossible, but the rest of this blog post should help.

Brevity is key

There are some people who thrive on receiving e-mail, sending e-mail, and talking about e-mail that they've sent or received. Most nerds don't feel this way.

You need to get your point across concisely and succinctly so that your e-mail is seen as less of a distraction. Avoid adding a lot of context where it isn't needed and try to summarize business needs and processes unless details are absolutely critical. If you need to send your e-mail to multiple recipients and some of those recipients need additional details, provide an abstract at the beginning of the e-mail.

Learn the ways of TL;DR

I've heard quite a few conversations like these around the office:

Nerd 1: "Did you get that e-mail from [name here]?"
Nerd 2: "The six page one with four PDF files attached?"
Nerd 1: "Yeah. That one."
Nerd 2: "TL;DR dude, seriously. Did you read it?"
Nerd 1: "Nah. I might read it later."

If someone's ever mentioned "TL;DR" (too long; didn't read) when your e-mail was mentioned, don't fret. It's a quick fix. Just add a quick summary to the top of your e-mail prefaced with "TL;DR". Provide a really brief summary (bulleted lists are a plus) of your e-mail in the section and then start your e-mail right afterwards. Here's an example:

TL;DR
  * next software release deploys Monday
  * two bugs remaining to fix
  * we will get started at 8AM Saturday, yeaaaaah

Missed the joke? Head over to Wikipedia.

If one of the summary points interests a recipient, they'll scan your e-mail for the pertinent sections. Some recipients may only need to see what's in the summary and they won't bother reading the remainder. Either way, the effectiveness of your e-mail increases by leaps and bounds.

Plain text

Users of mutt prefer plain text e-mails

Here are a few more tips under this category:

• Don't use Outlook stationery.
• Never send e-mails with an image as the e-mail itself.
• No Comic Sans at any time. Period.
• Avoid graphical e-mail signatures (more on that in a moment).

E-mail signatures

Brevity can definitely be applied to e-mail signatures, too. How many times have you seen e-mails that end like this:

Frank Frankelton MCSE, RHCSA, RHCE, CCNA, RHCA, LPIC-3, Ph.D., M.D., Esq., CMDBA
Systems Adminstrator Extraordinaire, Database Administrator, All-around great guy
Office: 210-555-1212
Mobile: 210-555-1213
Other Mobile: 210-555-1214
Fax: 210-555-1215
VOIP: 210-555-1216
AIM: frankeltonia
Twitter: @frankyfrank
Jabber: frankfurter@frankeltonisinthehouse.com
Big Company, Inc

You might think that nobody would ever send out e-mails with a signature like the one above, but I've seen some that are actually worse. Keep the signature short and only put in the information that people really need to know. Generally, your name and title or department is sufficient for e-mail signatures (unless your local/federal laws require otherwise). Always preface it with a double dash "--" on a line by itself to signify that the remainder of the e-mail is the signature.

Summary

Keep it simple, keep it brief, and keep it relevant. While the suggestions above might not apply to every business or every person, following the suggestions will increase the effectiveness of your e-mails and ensure that your voice is heard on the other end.

I'm really interested to hear your comments. Are there some suggestions you have that I missed in the post? Did I make some suggestions which didn't make sense or don't apply to you? Let me know!

How to write e-mails to nerds (that they will actually read) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Here are the list of winners:

• Grand Prize ($50 at ThinkGeek): Dan Udey
• Runners-Up ($25 at ThinkGeek): Joe Wright, Susan Price, and Giovanni Tirloni

Dan's comment rang true with me since much of a sysadmin's job involves responding to crises regardless of how much planning you put forth:

Keep a cool head. Focus. Work methodically. Figure out what to do and get it done, and people will remember you as the person who performs under pressure. Once you can do that, you're a sysadmin.

Joe touched on a critical point about system administration:

Tell the truth. If you break something, 'fess up and fix it. If you don't know how to do something, admit it and learn how to do the task. Create your own culture of honesty on the job; others will respect and follow your example.

Susan offered some inspiration for system administrators stuck in frustrating situations:

I know, I know - dumb users, RTFM. Believe me, I've been there. In fact - one of your strategies should be to establish a trusted community where you can VENT about these issues, and get support for yourself. Ask for answers when you don't know them. Restock on the compassion and patience.

Giovanni talked about the basics and what every system administrator should know to get started in a career. We probably take this for granted, but this is critical to keep in mind:

If you are starting in the system administration area, don't praise yourself only because you (blindly?) fixed an issue or helped that friend with his/her server. Ask yourself: Why what I did fixed the issue? Why that was happening in the first place? And more importantly, how to avoid it for all eternity? You won't but it doesn't hurt to aim high.

Even though it isn't a runner-up, Paul's comment certainly deserves an honorable mention. His comment is actually a true story (with a slight amount of embellishment, of course) and it serves as a reminder that system administrators and developers must stand up for their beliefs even if it goes against the beliefs of their superiors. If your managers don't value the feedback, it might be a sign that a career change is in order.

Once again, a big thanks goes out to everyone who submitted a comment. I'll reach out to the winners today and get the gift certificates sent out to them.

Contest winners from the "Inspire a sysadmin" contest is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The feeling I had at last year's summit is that Xen was on the verge of losing traction in the market. Very few distributions still had Xen support going forward and much of the discussion was around the lack of dom0 support in upstream Linux kernels. Distribution vendors were hesitant to drag patches forward into modern kernels and this made it much more difficult to get Xen working for many people.

This year was quite different. The number of attendees was up, the venue was much better, and there was an obvious buzz of energy in the room. As many of the presenters noted, this excitement stemmed from the upstream dom0 support in Linux 3.0. This inclusion is a huge win and it helps to drive Xen forward since the developers don't have to worry about dragging patches forward. They can focus on improving performance, adding features, and tightening security.

Many of the discussions this year focused on security and performance. Ian Pratt discussed Xen's ability to view memory pages of virtual machines via an API to detect malware running inside the instance. Memory pages could be identified and marked as not executable or applications could be triggered when a VM attempts to touch a particular memory page. Also, the whole VM could be frozen if needed.

There's also a big push to bring code out of the dom0 and push it into utility VM's. Driver domains could manage the network or I/O infrastructure and this would further reduce the amount of privileged code actively running in dom0. There is already very little code required for the Xen hypervisor itself (much much less than the Linux kernel -- I'm looking at you, KVM) and this reduces the attack surface for potential compromises of the hypervisor. Some projects even aim to restart driver domains multiple times per minute to ensure that any malicious code injected into those virtual machines can't exist for long periods.

Pradeep Vincent from Amazon talked about how Amazon uses Xen and the pain points they have with its current architecture. Much of his discussion was around scaling problems (and we see many of the same issues at Rackspace). Higher performance could easily be gained by multi-threaded operations in dom0 when attaching block devices and creating virtual network interfaces. He also saw some areas for performance gains in the pvops I/O code.

Quite a few of the talks centered on the ARM architecture and what Xen is able to do on those systems after Samsung published their port in 2008. HVM is on the way for ARM and it might even show up in Xen 4.2. Some demos of Xen on mobile phones from Samsung were amazing. They showed how an attacker could compromise the web browser on the phone with a keylogger, but that application was running in a VM. Once the user switched back to the phone's main menu, the keylogger couldn't access the keystrokes any longer. After that, a simple close of the browser killed the VM and destroyed the malicious code.

Xen 4.2 should be available in early 2012 and the feature list is staggering. Improvements to libxenlight, pvops performance (even in HVM), and guest memory sharing should be available with the new release. Nested virtualization (run a hypervisor inside a hypervisor) is also coming in Xen 4.2 and I'm sure Xzibit will be a huge fan. This should streamline hypervisor testing, allow for embedded hypervisor options and extend the capabilities of client hypervisors. Remus should be available in 4.2 as well, but it might be marked as experimental. OVMF will be added as a BIOS option for UEFI (along with the standard SeaBIOS) and this should allow for Mac OS X guests. UEFI allows Windows to boot faster since it switches to PV mode sooner and it allows for simpler platform certification for software vendors.

Mike McClurg's presentation on XCP was pretty important to me since Rackspace is a big consumer of XenServer. If you're not familiar with XCP, it's basically open-source XenServer which runs on bleeding edge (and sometimes unstable) components. XCP 1.5 and XenServer 6 should be available in November with Xen 4.1 and Linux 2.6.32. GPU passthrough, up to 1TB RAM, and disaster recovery will be available. Another goal for the XCP team is to work closely with OpenStack via Project Olympus. Mike's vision is to have XCP become the configuration of choice for open source clouds. Project Kronos was also extremely interesting. It's essentially XCP's XenAPI stack running on Debian and Ubuntu. You'd be able to install either OS on a physical server and run XCP's services on it for a fully OSS hypervisor.

Konrad Wilk gave an update on Linux pvops and it appears there is a shift to get Xen working well on a desktop. This includes 3D graphics support, S3/hibernate capabilities and various bug fixes. There's also a push to get PV functionality into HVM and get HVM functionality into PV. Driver/device domains were discussed again in Patrick Kolp's talk and he had plenty of graphs showing performance changes when regularly restarting device domains. The performance dips were almost negligible with 10 second restarts and the security gains were significant.

There were several other great presentations on other topics like GlusterFS, OpenStack Nova, and Linpicker (from the NSA!). If these types of things interests you, keep your eyes peeled for Xen Summit 2012 next year. The weather in the bay area is well worth the trip.

Xen Summit 2011: My Takeways is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Something else you might not have noticed is that the Fedora kernel team has quietly slipped Linux 3.0 into Fedora 15's update channels in disguise. Click that link, scroll down, and you'll see "Rebase to 3.0. Version reports as 2.6.40 for compatibility with older userspace." Although I'm not a fan of calling something what it isn't (2.6.40 doesn't exist on kernel.org), I can understand some of the reasoning behind the choice.

This change makes the Xen installation on Fedora 15 pretty trivial. To get started, update your kernel to the latest if you're not already on Fedora's 2.6.40 kernels:

yum -y upgrade kernel

We need three more packages (quite a few dependencies will roll in with them):

yum -y install xen libvirt python-virtinst

The xen package reels in the hypervisor itself along with libraries and command line tools (like xl and xm). Libvirt gives us easy access to VM management with the virsh command and python-virtinst gives us the handy virt-install command to make OS installations easy.

Once those packages are installed, we need to make some adjustments in your grub configuration. Open /boot/grub/menu.lst in your text editor of choice and add something like this at the bottom:

title Fedora + Xen (2.6.40-4.fc15.x86_64)
        root (hd0,1)
	kernel /boot/xen.gz
        module /boot/vmlinuz-2.6.40-4.fc15.x86_64 ro root=/dev/sda1
        module /boot/initramfs-2.6.40-4.fc15.x86_64.img

Ensure that the root (hd0,1) is applicable to your system (adjust it if it isn't). Also, check the kernel version to ensure it matches your installed kernel and adjust the root= portion to match your root volume. Flip the default line to a value which will boot your new grub entry and ensure the timeout is set to a reasonable number if you need to temporarily switch back to your original grub entry at boot time. (Hey, we all make mistakes.)

I take one extra precaution and change the UPDATEDEFAULT=yes line to no in /etc/sysconfig/kernel. This ensures that future kernel updates don't trample the entry you've just made. Keep in mind that you'll need to manually update your grub configuration when you do kernel upgrades later.

Cross your fingers and reboot. If your system doesn't reboot properly, reboot it again and choose your old kernel from the grub menu. Double-check your configuration for fat-fingering and give it another try. If your system boots and pings but you have no output via a monitor, don't fret. There's a patch for the problem which should appear soon in Linux 3.0. The impatient can snag a kernel source RPM, add the patch file, and build a local kernel (or you can download my local build from when I did it).

Log in and verify that you booted into the dom0:

[root@xenbox ~]# xm dmesg | head -n 5
 __  __            _  _    _   _   ____     __      _ ____  
 \ \/ /___ _ __   | || |  / | / | |___ \   / _| ___/ | ___| 
  \  // _ \ '_ \  | || |_ | | | |__ __) | | |_ / __| |___ \ 
  /  \  __/ | | | |__   _|| |_| |__/ __/ _|  _| (__| |___) |
 /_/\_\___|_| |_|    |_|(_)_(_)_| |_____(_)_|  \___|_|____/

Once you're done with that, make sure libvirtd is running:

/etc/init.d/libvirtd start; chkconfig libvirtd on

Try installing a VM:

virt-install \
  --paravirt \
  --name=testvm \
  --ram=512 \
  --vcpus=4 \
  --file /dev/vmstorage/testvm \
  --graphics vnc,port=5905 --noautoconsole \
  --autostart --noreboot \
  --location=http://mirrors.kernel.org/debian/dists/squeeze/main/installer-amd64/

You should have a VM installation underway pretty quickly and it will be visible via port 5905 on the local host. Enjoy the power and freedom of your brand new type 1 hypervisor.

Xen 4.1 on Fedora 15 with Linux 3.0 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Stress can come from a variety of sources. Some of the obvious ones involve dealing with outages or tight deadlines, but there are some that aren’t so obvious, such as maintaining the customers’ trust and interpersonal issues.

There’s one thing you must remember: stress doesn’t have to rule your life. I’ve learned (and sometimes stumbled upon) some good techniques to prevent many of the negative effects of stressful situations at work and they’re definitely worth a try.

Know what you’re up against
It’s hard to battle a source of stress if you don’t know why it’s bothering you. Take the problem you’re facing and break it down into pieces. There are going to be some things you can and can’t change. Put the things you can’t change aside and focus on the things you’re able to change. As you tackle the list of things you can change, you might find ways to work around the things you can’t.

Interpersonal issues are easy
Stress that comes from dealing with coworkers may seem insurmountable at times. However, this type of stress is easily fixed and it normally stems from insufficient communication or conflicting goals. There’s an informal policy I’ve had on most of my teams called “Take it to the Racker” and it’s been quite successful. The basic idea is that if you have problems with another Racker, whether it’s something personal or work-related, take the grievance to them directly (in private, of course) and find common ground.

More often than not, this process leads to a good work relationship. It also improves communication drastically in the short term and it generally lasts if the people involved keep up the communication over time. I’ve seen Rackers who are so upset that they refuse to sit next to each other and after this process, they’re eating lunch together and working on the same projects.

Don’t fight your battles alone
Your best resources for fighting stress are all around you. Lean on your manager or your coworkers for help. Remember what your mother always told you: a trouble shared is a trouble halved. Your coworker might have a solution to a particular problem which frees up an hour for you each day and allows you to work on other projects. Your manager might not know that a particular task doesn’t fit your strengths and they might be able to provide you with another project that plays to your strengths.

Is it possible to reduce your stress level to zero at work? I don’t think so. However, you should always have a goal to reduce it when it makes sense.

As always, I’m interested to hear your comments. Which stress-reduction strategies work best for you? What is the source of most of your stress?

Success with stress is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

================================================================================
 Package        Arch           Version                   Repository        Size
================================================================================
Installing:
 kernel         x86_64         2.6.35.13-92.fc14         updates           22 M
Removing:
 kernel         x86_64         2.6.35.11-83.fc14         @updates         104 M
 
Transaction Summary
================================================================================
Install       1 Package(s)
Remove        1 Package(s)

Fedora 15's default behavior is to keep three kernels: the latest one and the two previous versions. However, this behavior may be counter-productive if you compile your own modules, or if you have compatibility issues with subsequent kernel versions.

You can change how yum handles kernel packages with some simple changes to your /etc/yum.conf. The installonly_limit option controls how many old packages are kept:

installonly_limit Number of packages listed in installonlypkgs to keep installed at the same time. Setting to 0 disables this feature. Default is '0'.

I disabled the functionality altogether by setting installonly_limit to 0:

#installonly_limit=3
installonly_limit=0

It's important to keep in mind that you will need to purge these packages from your system yourself now. Kernel packages can occupy a fair amount of disk space, so make a note to go back and clean them up when you no longer need them.

Keep all old kernels when upgrading via yum is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

What do I mean by "do your homework?" Here are some bullet points to get you on your way:

Know what the company does.
This one is critical and it should be easy. However, make sure to do thorough research first. For example, if you interviewed at a company like Apple, becoming familiar with their hardware lineup should be a no-brainer. That's their bread and butter. On the other hand, remember that Apple isn't solely a hardware company; they write lots of software, provide online productivity services, and they distribute music, movies, and other entertainment media.

While you're doing this research, try to discover what makes the company unique. Sure, Apple sells laptops and desktops (just like a lot of other companies), but what makes their particular products unique? Is there something unique about the way they provide their services? Have they cornered a certain market segment by providing a combination of products and services to that group of consumers? Answering these simple questions may help you tip the scales in the interview process.

Try one or more of the company's products.
The feasibility of trying a company's product before an interview could be debatable. For example, if you wanted to interview at Cray, you probably don't need to drop $2M USD on your own XE6 before walking in the door. For companies where the barrier to entry for purchasing a product is much lower, such as cloud computing companies, there's no excuse to not try things out first. Amazon has a free tier and a Rackspace Cloud Server could cost you as little as $2.50 per week.

It's concerning when I talk to an applicant about a job working with Rackspace's Cloud Servers and they haven't tried out any cloud products from any provider. How can I take a candidate's interest seriously when they haven't shown interest in any portion of my group's market segment?

Know what the company's competitors do.
It's often more impressive to an interviewer to know what a company's competitors are doing and how it compares to what that company is doing in the market. For example, if you can walk into an interview and say "I like the way your company makes these widgets, but Company X is able to make them more lightweight, and I value that more than the added customer service your company offers." This shows the interviewer that you're familiar with various products in the segment and you've used them enough to understand what makes them different.

Some of you might be thinking: "Why would I say something like that to the interviewer? They'll think I'm being too negative about their product." That's always possible, but you can guard against it by wording everything carefully. Make sure you have a solid reason for the way you feel that is based on something substantial (usability, price, features, etc). I've had candidates talk for five to ten minutes about why one of our product is inferior to one of our competitors' products and I was very impressed.

One quick gotcha: your interviewer might turn your comments back on you and ask you how you would improve one of the inferior products (I do this regularly). Make sure that you're prepared for that question and consider offering up a suggestion before the question is presented to you.

Can't get the information you need? Ask!
When you reach the end of the interview and the interviewer asks if you have questions, be sure to ask any questions about topics you had trouble researching. Going back to the Cray example, compare what you know about an XE6 to servers you've used before. You could mention a problem you had with the density of your previous configurations and ask how they overcame that hurdle at Cray. If it's a proprietary trade secret, you might not get an answer, but they'll know that you did some serious research ahead of time. If they can share the answer, they might still be impressed, and you might end up learning something you didn't know prior to the interview.

Conclusion
In summary, doing your homework and learning about the company shows the interviewers that you not only have what it takes to do the work, but that the work interests you as well. I've interviewed folks in the past who lacked on technical ability but had plenty of desire and drive. More often than not, those people are now Rackers.

Do your homework before a technical interview is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

• your server could be used to gain access other servers
• data could be stolen from your server's databases or storage devices
• an attacker could capture data from your server's local network
• denial of service attacks could be launched using your server as an active participant

The best ways to limit your server's attack surface are pretty obvious: limit network access, keep your OS packages up to date, and regularly audit any code which is accessible externally or internally. As we all know, your server can still become compromised even with all of these preventative measures in place.

Here are some tips which will allow you to rapidly detect a compromise on your servers:

Abnormal network usage patterns and atypical bandwidth consumption
Most sites will have a fairly normal traffic pattern which repeats itself daily. If your traffic graph suddenly has a plateau or spikes drastically during different parts of the day, that could signify that there is something worth reviewing. Also, if your site normally consumes about 2TB of traffic per month and you're at the 1.5TB mark on the fifth day of the month, you might want to examine the server more closely.

On the flip side, look for dips in network traffic as well. This may mean that a compromise is interfering with the operation of a particular daemon, or there may be a rogue daemon listening on a trusted port during certain periods.

Many compromises consist of simple scripts which scan for other servers to infect or participate in large denial of service attacks. The scans may show up as a large amount of packets, but the denial of service attacks will usually consume a large amount of bandwidth. Keeping tabs on network traffic is easily done with open source software like munin, cacti, or MRTG.

Unusual open ports
If you run a web server on port 80, but netstat -ntlp shows something listening on various ports over 1024, those processes are worth reviewing. Use commands like lsof to probe the system for the files and network ports held open by the processes. You can also check within /proc/[pid] to find the directory where the processes were originally launched.

Watch out for processes started within directories like /dev/shm, /tmp or any directories in which your daemons have write access. You might see that some processes were started in a user's home directory. If that's the case, it might be a good time to reset that user's password or clear out their ssh key. Review the output from last authentication logs to see if there are account logins from peculiar locations. If you know the user lives in the US, but there are logins from various other countries over a short period, you've got a serious problem.

I've used applications like chkrootkit and rkhunter in the past, but I still prefer a keen eye and netstat on most occasions.

Command output is unusual
I've seen compromises in the past where the attacker actually took the time to replace integral applications like ps, top and lsof to hide the evidence of the ongoing compromise. However, a quick peek in /proc revealed that there was a lot more going on.

If you suspect a compromise like this one, you may want to use the functionality provided by rpm to verify the integrity of the packages currently installed. You can quickly hunt for changed files by running rpm -Va | grep ^..5.

Keeping tabs on changing files can be a challenge, but applications like tripwire and good ol' logwatch can save you in a pinch.

Summary
We can all agree that the best way to prevent a compromise is to take precautions before putting anything into production. In real life, something will always be forgotten, so detection is a must. It's critical to keep in mind that monitoring a server means more than keeping track on uptime. Keeping tabs on performance anomalies will allow you to find the compromise sooner and that keeps the damage done to a minimum.

Strategies for detecting a compromised Linux server is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

In this guide, I'll be using two Fedora 14 instances in the Rackspace Cloud with separate public and private networks. The instances are called server1 and server2 to make things easier to follow.

NOTE: All of the instructions below should be done on both servers unless otherwise specified.

yum -y install drbd-utils
dd if=/dev/zero of=/drbd-loop.img bs=1M count=1000

Put this loop file initialization init script in /etc/init.d/loop-for-drbd and finish setting it up:

chmod a+x /etc/init.d/loop-for-drbd
chkconfig loop-for-drbd on
/etc/init.d/loop-for-drbd start

Place this DRBD resource file in /etc/drbd.d/r0.res. Be sure to adjust the server names and IP addresses for your servers.

resource r0 {
	meta-disk internal;
	device /dev/drbd0;
	disk /dev/loop7;
 
	syncer { rate 1000M; }
        net { 
                allow-two-primaries; 
                after-sb-0pri discard-zero-changes;
                after-sb-1pri discard-secondary;
                after-sb-2pri disconnect;
        }
	startup { become-primary-on both; }
 
	on server1 { address 10.181.76.0:7789; }
	on server2 { address 10.181.76.1:7789; }
}

The net section is telling DRBD to do the following:

• allow-two-primaries - Generally, DRBD has a primary and a secondary node. In this case, we will allow both nodes to have the filesystem mounted at the same time. Do this only with a clustered filesystem. If you do this with a non-clustered filesystem like ext2/ext3/ext4 or reiserfs, you will have data corruption. Seriously!
• after-sb-0pri discard-zero-changes - DRBD detected a split-brain scenario, but none of the nodes think they're a primary. DRBD will take the newest modifications and apply them to the node that didn't have any changes.
• after-sb-1pri discard-secondary - DRBD detected a split-brain scenario, but one node is the primary and the other is the secondary. In this case, DRBD will decide that the secondary node is the victim and it will sync data from the primary to the secondary automatically.
• after-sb-2pri disconnect - DRBD detected a split-brain scenario, but it can't figure out which node has the right data. It tries to protect the consistency of both nodes by disconnecting the DRBD volume entirely. You'll have to tell DRBD which node has the valid data in order to reconnect the volume. Use extreme caution if you find yourself in this scenario.

If you'd like to read about DRBD split-brain behavior in more detail, review the documentation.

I generally turn off the usage reporting functionality in DRBD within /etc/drbd.d/global_common.conf:

global {
	usage-count no;
}

Now we can create the volume and start DRBD:

drbdadm create-md r0
/etc/init.d/drbd start && chkconfig drbd on

You may see some errors thrown about having two primaries but neither are up to date. That can be fixed by running the following command on the primary node only:

drbdsetup /dev/drbd0 primary -o

If you run cat /proc/drbd on the secondary node, you should see the DRBD sync running:

version: 8.3.8 (api:88/proto:86-94)
srcversion: 299AFE04D7AFD98B3CA0AF9
 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:210272 dw:210272 dr:0 al:0 bm:12 lo:1 pe:2682 ua:0 ap:0 ep:1 wo:b oos:813660  
        [===>................] sync'ed: 20.8% (813660/1023932)K queue_delay: 0.0 ms
        finish: 0:01:30 speed: 8,976 (6,368) want: 1024,000 K/sec

Before you go any further, wait for the DRBD sync to fully finish. When it completes, it should look like this:

version: 8.3.8 (api:88/proto:86-94)
srcversion: 299AFE04D7AFD98B3CA0AF9
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
    ns:0 nr:1023932 dw:1023932 dr:0 al:0 bm:63 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

Now, on the secondary node only make it a primary node as well:

drbdadm primary r0

You should see this on the secondary node if you've done everything properly:

version: 8.3.8 (api:88/proto:86-94)
srcversion: 299AFE04D7AFD98B3CA0AF9 
 0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
    ns:1122 nr:1119 dw:2241 dr:4550 al:2 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

We're now ready to move on to configuring OCFS2. Only one package is needed:

yum -y install ocfs2-tools

Ensure that you have your servers and their private IP addresses in /etc/hosts before proceeding. Create the /etc/ocfs2 directory and place the following configuration in /etc/ocfs2/cluster.conf (adjust the server names and IP addresses):

cluster:
	node_count = 2          
	name = web
 
node:
	ip_port = 7777
	ip_address = 10.181.76.0
	number = 1
	name = server1
	cluster = web
 
node:
	ip_port = 7777
	ip_address = 10.181.76.1
	number = 2
	name = server2
	cluster = web

Now it's time to configure OCFS2. Run service ocfs2 configure and follow the prompts. Use the defaults for all of the responses except for two questions:

• Answer "y" to "Load O2CB driver on boot"
• Answer "web" to "Cluster to start on boot"

Start OCFS2 and enable it at boot up:

chkconfig o2cb on && chkconfig ocfs2 on
/etc/init.d/o2cb start && /etc/init.d/ocfs2 start

Create an OCFS2 partition on the primary node only:

mkfs.ocfs2 -L "web" /dev/drbd0

Mount the volumes and configure them to automatically mount at boot time. You might be wondering why I do the mounting within /etc/rc.local. I chose to go that route since mounting via fstab was often unreliable for me due to the incorrect ordering of events at boot time. Using rc.local allows the mounts to work properly upon every reboot.

mkdir /mnt/storage
echo "/dev/drbd0  /mnt/storage  ocfs2  noauto,noatime  0 0" >> /etc/fstab
mount /dev/drbd0
echo "mount /dev/drbd0" >> /etc/rc.local

At this point, you should be all done. If you want to test OCFS2, copy a file into your /mnt/storage mount on one node and check that it appears on the other node. If you remove it, it should be gone instantly on both nodes. This is a great opportunity to test reboots of both machines to ensure that everything comes up properly at boot time.

Dual-primary DRBD with OCFS2 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.