First things first, getting a job at Rackspace isn't easy. We don't intentionally make the process difficult. It's just that the work we do is unique and demanding.

We work in a fast-paced, extremely dynamic team-centric environment. While some people in the company work in extremely small teams or sometimes all by themselves, that's pretty few and far between. We look for people who can survive and flourish in this atmosphere and we look for people who can do it all while working as a team. Even with all of this hustle and bustle, we still remember why we're doing it: the pursuit of Fanatical Support for our customers.

Another thing to keep in mind is that there's no true secret for making it through the application process. There's no magic combination of skills or "silver bullet" that will scoot you through. Every candidate is reviewed individually for each position. There have been several times at the end of an interview where we've gotten together and said: "Wow, this candidate is solid, but they're just not right for this position. Let's find the right spot and see if there's a spot open." We look for the right candidate for the right position at the right time.

One of the best ways to get ahead in the screening or interview process is to do a little homework about Rackspace and the products we offer. Much of this is covered in a post I wrote in 2011. You'll go into the interviews with more confidence and it will be much more obvious that you're really interested in the position.

Don't be discouraged if the process takes a little longer than you expected. When I was hired in 2006, I went through two phone pre-screens and then three back-to-back interviews in person. Things have changed a little since then and I've heard of some candidates receiving two to three pre-screens via telephone and then one or two interviews in person. The additional screening and interviews may be due to Rackers trying to find the right fit for a particular applicant. As I said previously, we look for the right fit for each applicant. We may consider you for a different position than you applied for if we feel like your skill set or personality fits that role better.

A very common question is what to wear to a Rackspace interview. It's confusing to know exactly what's expected since we have Rackers in the building wearing everything from suits to flip-flops. This is where you really have to go with your gut. Interviewing for a customer-facing sales position while wearing a hoodie and shorts is probably going to bring a suboptimal result. Keep in mind that there's really nothing negative about overdressing (but keep your tuxedo in the closet, seriously). I wore a shirt and tie for my interviews in 2006 but my tie got caught in the car door and was shredded. After a lot of cursing, I took off the tie and decided to wing it with my dress shirt. Nobody ever said a word about it.

Remember to be flexible during the interviews. You might be asked to draw a solution on a whiteboard or think through a really complicated situation. Roll with it and keep your confidence up. When you don't know something, admit it, but then talk about how you'd research an answer.

There's one last thing to keep in mind and it's really critical. If you're ever asked about how you would solve a problem or how you solved a problem in the past, don't divulge any information which is confidential or proprietary to your current company. Just tell the interviewers that you've solved the solution in the past but you'll need to keep things vague to maintain confidentiality. We will definitely understand and we will encourage you to maintain that confidentiality.

Leave your comments if you have any! I'll be glad to answer any questions you have.

Getting a Technical Job at Rackspace is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

My mother was always the one who told me (and her students) that everyone has a story. She said that writing could be therapeutic in ways you probably won't consider until you've written something that someone else enjoys. Just as software developers exist to write software for their users, writers exist to write stories for their readers. There's nothing that says technical people can't become excellent writers who inspire others to learn and share their knowledge with others.

The goal of this post is to encourage technical people to enjoy writing, write efficiently and feel comfortable doing it. I'll roll through some of the most common responses I've received about why technical people don't blog about what they know.

I don't think I'm really an expert on anything. I'm not an authority on any topic I can think of.

I'm leading off with this response because it's the most critical to refute. If you don't take away anything else from this post, let it be this: you don't need to be an expert on a topic to write about it.

You can find examples of this by rolling through some of the posts on my blog. I'd consider myself to be an expert on one, maybe two topics, but I've written over 450 posts in the span of just over five years. I certainly didn't write all of those about the one or two topics I know best.

Write about what you know and don't be afraid to do a little research to become an authority on something. A great example of this was my post, entitled "Kerberos for haters." I had almost no expertise in Kerberos. In fact, I couldn't even configure it properly for my RHCA exam! However, I did a ton of research and began to understand how most of the pieces fit together. Many other people were just as confused and I decided to pack all of the knowledge I had about Kerberos into a blog post. Positive and negative feedback rolled in and it was obvious that my post taught some readers, inspired some others and angered a few.

What a great way to lead into the next response:

What if I say something that isn't correct? I'll look like an idiot in front of the whole internet!

Been there, done that. Every writer makes errors and comes up with bad assumptions at least once. Readers will call you out on your mistakes (some do it delicately while others don't) and it's your duty to correct your post or correct the reader. I've written posts with errors, and I've gotten a little lazy on my fact-checking from time to time. As my middle school journalism teacher always reminded me, the most important part of a mistake is what you do to clean it up and learn from it.

In short: you'll make mistakes. As long as you've done your due diligence to minimize them and respond to them promptly, your readers should forgive you.

Speaking of errors:

I'm great at a command prompt but my spelling and grammar are awful. I write terribly.

This is easily fixed. If you're one of those folks who live the do-it-yourself type of lifestyle, pick up a copy of The Elements of Style by Strunk & White. There are free PDF versions online or you can borrow one from your nearest journalist. No matter the situation you're in, this book has details about where punctuation should and shouldn't be, how to structure sentences and paragraphs, and how to properly cite your sources (really vital for research posts).

Hauling around a copy of an ultra-dry reference book may not be your thing. If that's the case, find someone you know who has a knack for writing. You can usually find helpful folks in marketing or corporate communications in most big companies who will take your post and return it covered in red ink ready for corrections (thanks, Garrett!). I've even spotted some folks on Fiverr who will do this for as low as $5.

I'll wrap up with the second most common response:

I don't know who I'm writing for? What if I write about something simple and the really technical folks think I'm a noob? What if I write something crazy complex and it goes over most people's heads?

I've done both of these. Most Linux system administrators worth their salt know how to add and remove iptables rules, and they'd consider it to be pretty trivial work. Would it surprise you to know that out of over 450 posts, my post about deleting a single iptables rule is in the top five most accessed posts per month? I receive just over 11 percent of my monthly hits to this post. People are either learning from it or they can't remember how to delete the rule and they want to use the post as a quick reference. Either way, the post is valuable to many people even if I think it's the simplest topic possible.

On the flip side, I went nuts and wrote up a complete how-to for a redundant cloud hosting configuration complete with LVS, glusterfs, MySQL on DRBD, memcached, haproxy and ldirectord. I thought it would be valuable knowledge to a few folks but that it might sail over the heads of most of my readers. Again, I was wrong. The post is constantly in the top 10 most visited posts on the blog and I've probably received more feedback via comments, email and IRC about that post than any other. Once again, a post I thought would be mostly useless turned into a real conversation starter.

Let's conclude and wrap up. Keep these things in mind if you feel discouraged about writing:

• Write about what interests you whether you're an expert on it or not
• Don't be afraid to fail
• Be responsive to your readers
• Even if you think nobody will read your post, write it
• Always ensure your voice shines through in your writing — this is what makes it special and appealing

Why technical people should blog (but don't) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Getting yourself ready for any type of examination is usually a stressful experience that involves procrastination and some late nights leading up to the test. Every time I take one, I always say to myself, “I’m really going to get ahead of this next time and study early. This last minute stuff is terrible.” But I always forget all of this as the next exam rolls around.

Quick note: As you read through the remainder of the post, you may wonder why some of it is a bit vague. Every Red Hat test taker is under a NDA to prevent disclosure of test information that may reduce the security of the exam itself. Penalties start with losing credit for the exams previously taken and they can escalate up to legal action. I hope you’ll understand why I’m not able to go into details about certain portions of the Red Hat examinations.

I’ve taken seven Red Hat exams already: two for the RHCE and five for the RHCA. These tests certainly aren’t easy, but there are some good guidelines and tips you can use to make your studying efforts less stressful and more productive. Without further ado, here are my recommendations for prospective Red Hat examinees:

Build a flexible study environment

This is critical. You’ll need some spare servers or some available virtual machines to practice the objectives on each exam. However, don’t feel like you need to spend the money on a Red Hat subscription to get your studying done. Most of the test objectives on the majority of exams can be completed with very similar Linux distributions, like Scientific Linux or CentOS. Look for a version of the distribution that is closest to what you’ll be tested on at exam time. Your study environment should meet some basic criteria:

• You should be able to quickly build and tear down servers or virtual machines
• Keep the latency to your environment low to avoid getting frustrated
• Use applications like VirtualBox, VMWare Fusion/Workstation to practice on your own computer
• Consider using VMs from cloud providers if you’re under a time crunch

Some exams may require some bare-metal access to the server itself (especially EX442), so keep that in mind when you’re looking for a good practice environment. You may need some specific network or storage setups for some exams (as with EX436). If you’re not sure what you need, be sure to ask your instructor or someone else you know who has taken the exam already.

Prioritize doing over reading

The Red Hat exams are all hands-on, practical exams. You won’t find any essays or multiple-choice questions in these exams. Although the materials from Red Hat are full of good information, reading this information can only get you so far. You need to practice setting up the services on your own to be fully prepared for the test. If you’re not pressed for time, reading through the book can give you some details about the lab sequences, which you might miss by solely reading through labs themselves.

Research the why, not the what, to remember

This is especially important for the RHCA exam track. You may find that there is a ton of material to cover for the exam and that it’s difficult to remember each command to bring a certain service online or to repair a problem. Instead of thinking through the problem as “first, I do this, then I do this”, try to understand why each step is important in the first place.

Here’s a good example. I’ll be the first one to admit that Kerberos drives me crazy. I’ve even written posts about it. The commands seemed really archaic, the daemons didn’t make sense, and the lack of readline support in the Kerberos tools made me want to throw my computer out the window (come on, MIT!). I put my class materials aside, went to Google in a browser, and started researching Kerberos.

I read some of MIT’s documentation, ventured over to Wikipedia, and poked at some of the documentation within the Kerberos RPM packages. After a while, I began to realize how it all fit together. “Okay,” I thought to myself, “I need principals in a keytab to do these things, but I need to have a database for the admin stuff first.” Suddenly, the order of things in my head wasn’t just memorized any longer. The process of operations seemed to make logical sense because I fully understood how the pieces of a Kerberos infrastructure fit together.

If you start to get discouraged, take a break and learn more about why you’re doing what you’re doing. Once it becomes second nature, working through the problems on the exam becomes much easier.

Lean on your available resources

Don’t forget that there are other knowledgeable folks available to talk to when you get bogged down. Lean on other RHCE’s, RHCA’s, or experienced Linux users to get the answers or explanations you need. If you already have a Red Hat certification, head over to the Red Hat Certification Forums and meet up with other examinees that are discussing test preparation.

Also, you’ll find some knowledgeable (but sometimes snarky or quirky) people on IRC who are eager to point you in the right direction. Try the #rhel, #centos, or #fedora channels if you’re struggling through the configuration of a certain service. Many Linux users may roll their eyes about it, but Twitter is also a pretty good way to reach out to people who have a lot of Linux experience.

Summary

Remember to lean on the knowledge of others, get hands-on with the test objectives and do your research when you’re frustrated. The exams from Red Hat are generally difficult and cover a lot of material, but with the right amount of preparation and determination you can pass the exams and get the certifications you want.

Preparing for Red Hat Exams is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First things first: there are going to be many part of this post which probably aren't as specific as you'd like. A lot of that is due to the NDA that all Red Hat examinees agree to when they take an exam. We aren't allowed to talk about what was on the exam or our experiences during the exam. If we do, penalties range from smaller things like losing certifications all the way up to serious stuff like legal action. It goes without saying that I want to protect the security of the exams, I don't want to lose my certifications, and I don't want to hire a lawyer. Please try to keep this in mind if you yearn for more specifics than I'm able to give.

Red Hat Certified Engineer
The RHCSA and RHCE exams are the first step on the path to the RHCA. You can't take any of the RHCA prerequisite exams without it. These exams cover a really broad spectrum of material including apache configuration, NFS, iptables and mail services. The two links above will take you to the exam objectives for each exam.

I've always recommended the RHCE exam for Linux administrators who are trying to sharpen their skills and get to the next level whether they use Red Hat or not. The exam covers a lot of good material that makes a solid foundation for any Linux user without throwing in too many Red Hat-specific knowledge.

The exam (like all Red Hat exams) is fully practical. There are no multiple choice questions or essays. You'll have to meet all of the objectives by logging into a local Red Hat system and making the system do what it needs to do.

Quick tips for the RHCSA/RHCE exams:

• Keep your eye on the clock. Time can really get away from you if you get stuck in the weeds on a problem that should be relatively straightforward.
• Leave time at the end to check your work. When you set up a lot of services, it's inevitable that you might configure a service for one problem that breaks the functionality required by a problem you completed already.
• Always reboot before you leave. We all forget to use chkconfig when we're in a hurry.
• Practice, practice, practice. There's not one objective on this exam that you can't test in a VM on your own.

Red Hat Enterprise System Monitoring and Performance Tuning
Our group at Rackspace started off with EX442 and it was a very difficult way to start off the RHCA track. Take a look at the objectives and you'll see that much of the exam is related to tweaking system performance and then monitoring that performance with graphs and raw data. You'll have to turn a lot of knobs on the kernel and you'll need to know where to store these configurations so they'll be persistent.

In addition, the objective regarding TCP buffers and related settings is a real challenge. You'll have to wrestle with some math that appears to be relatively simple, but can get confusing quickly. Some of the settings can't really be checked to know if your setting is correct. The objectives mention tuning disk scheduling -- you don't really have the time or tools to know if your setting is ideal.

Quick tips for EX442:

• Use the documentation available to you. Install the kernel-doc package while you practice and during the exam.
• Be careful with your math. You have a Linux machine in front of you! Don't forget about bc.
• Watch your units. Know the difference between a kilobyte (KB) and a kibibyte (KiB).
• Make comments in files where you adjust kernel configurations. It will help you keep track of which question the kernel adjustment is meant to satisfy.

Red Hat Enterprise Storage Management
I'm surprised to say this now, but I actually enjoyed EX436. I've always used other clustering tools like heartbeat and pacemaker, but I've never had the need to use the Red Hat Cluster Suite. Although RHCS definitely has a lot of quirks and rough edges, it's pretty solid once you get familiar with the GUI and command line tools.

You get the opportunity to mess around with some pretty useful technology like iSCSI, GFS, and clustered LVM. These are things that you're probably already using or will be using soon in a large server environment. The web interface for RHCS is quite peculiar and you may find yourself wanting to put your fist through the screen when you're staring down the endless animated GIFs when the cluster is syncing its configuration. Do your best to be patient because you certainly don't want to short circuit the cluster sync.

Quick tips for EX436:

• Be patient. You'll feel like the RHCS web interface is mocking you when you're pressed for time.
• Watch the clock. It's extremely easy to burn a lot of time on this exam if you get stuck on a particular problem.
• Double check your entries in the web interface. Make sure you're doing things in the right order and that you've set up the prerequisites before adding services to the cluster. If you get it wrong, you could put your cluster into a weird state.
• Use man pages. If you don't mess with GFS a lot, the man pages will save you in a pinch.

Red Hat Enterprise Deployment and Systems Management
If there's one exam where time management is critical, it's EX401. Importing data into the Satellite Server takes quite a bit of time and there's almost nothing you can do to speed it up. It probably goes without saying, but as with most long-running tasks, you'll want to run it in screen. The last thing you'd ever want is to abort the import due to an errant click or CTRL-C (I did it while practicing -- it's aggravating).

There are other test objectives which you can either complete or partially complete while you wait for the import to finish.

Also, take the time to really dig into the Satellite Server web interface while your practicing for the exam. Knowing where to find the most common configuration items will really save some time when you're in the exam. You can sometimes get pretty bogged down in the interface so don't forget to use multiple tabs to keep your work organized.

I felt like this exam was the easiest out of the bunch since you could go back and test every single question with good time management. Did I mention how important time management was on this exam already? If I forgot to mention it earlier, be sure to focus on time management for this test.

Quick tips for EX401:

• Time management will make or break you on this test. Keep an eye on the clock and make sure you've done absolutely every piece of the exam that you can while you wait for the server to do its work.
• Scour the web interface. Keep a mental map in your mind where the big chunks of configuration items are.
• Go back and test everything. If you manage your time well, you should have enough time to verify each and every objective on this exam.

Red Hat Enterprise Directory Services and Authentication
At first, EX423 looks pretty straightforward. Red Hat's authentication configuration tools make LDAP authentication setup pretty easy. However, this exam comes with a lot of curveballs.

The GUI interface for the Directory Services component is a little frustrating to use. I found that the GUI stopped responding to keyboard input occasionally unless I clicked on another window and came back. If you misconfigure the SSL certificates in the interface, your LDAP server is down for the count. If you don't input the correct data into the setup scripts at the beginning, you might not notice it until much later when it's either too difficult to dig yourself out of the hole or it's too late to start over with a clean configuration.

I didn't feel pressed for time on this exam too much and that was pretty refreshing after taking the EX401 test. It's extremely critical to watch what you type and click on this exam. Some mistakes can be quickly corrected while others may require you to blow away the LDAP server configuration and re-provision the whole thing.

Quick tips for EX423:

• Always watch what you're typing. A simple mistake can lead to confusion or bigger issues down the road.
• Don't ignore the LDIF objectives. As you practice, you'll find that manipulating LDIF files is a little more involved than you expected.
• Practice starting over. Throw out your Directory Services configuration and get the experience of what it's like to start over and get back in the game.

Red Hat Enterprise Security: Network Services
There's no sugar coating it -- EX333 is a beast. It's a six hour exam broken into two three-hour chunks. It covers a ton of material and I refer to it as "the RHCE on steroids." You might argue that I thought it was hard since it was the last test and I was ready to be finished, but I really think this exam is a tough one.

Practicing for the Kerberos and DNS objectives was the hardest for me. I just couldn't understand Kerberos, no matter how hard I tried. The realization that I would really have to learn it soon set in. I dug into the Kerberos design documentation on MIT's site, read the summaries on Wikipedia, and scoured the documentation available in the Kerberos RPM packages. Once I understood why Kerberos is set up the way it is and why the security measures are present, everything began to come together. I was able to remember the steps not because I was memorizing them, but because I understood how Kerberos worked.

When you're working through the DNS objectives, keep an eye out for punctuation. I blew through a good 20 minutes in what seemed like the blink of an eye when I forgot a period in my TSIG key configuration while studying. Make sure you use the resources available to you, like system-config-bind and sample configs in /usr/share/doc/bind*/examples/. Get to know commands like dig really well.

If you're overwhelmed by OpenSSL's command line syntax, check out the /etc/pki/tls/misc/CA script. There are some handy comments at the top of the script that explain how to use it. You can also pluck OpenSSL commands right out of the script if you need to run them yourself.

• Don't just memorize. Do some research to understand how everything fits together.
• Manage your time. DNS and Kerberos have lots of small nuances that can become time sinks when done incorrectly.
• Use the available documentation and tools. Try practicing without study materials so that you're forced to use the docs and tools available within the server.

Ranking the exams
A couple of folks on Twitter asked me to rank the exams from most difficult to least difficult. Keep in mind that these are a little subjective since I was more familiar with some objectives than others for certain tests.

• EX333 - Enterprise Security: Network Services: a tubload of material and a very long exam
• EX442 - System Monitoring and Performance Tuning: very difficult to check your work, lots of calculations
• EX423 - Directory Services and Authentication: not a lot of material to cover, but tons of curveballs
• EX436 - Storage Management: the web interface made things much easier, lots of documentation available
• EX401 - Deployment and Systems Management: every objective can be tested, I build RPM's already

Looking back at the long road to becoming a Red Hat Certified Architect is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you're not technical, you might be asking yourself: "I need to e-mail technical people and they need to take what I say seriously? How do I do it?" It's not impossible, but the rest of this blog post should help.

Brevity is key

There are some people who thrive on receiving e-mail, sending e-mail, and talking about e-mail that they've sent or received. Most nerds don't feel this way.

You need to get your point across concisely and succinctly so that your e-mail is seen as less of a distraction. Avoid adding a lot of context where it isn't needed and try to summarize business needs and processes unless details are absolutely critical. If you need to send your e-mail to multiple recipients and some of those recipients need additional details, provide an abstract at the beginning of the e-mail.

Learn the ways of TL;DR

I've heard quite a few conversations like these around the office:

Nerd 1: "Did you get that e-mail from [name here]?"
Nerd 2: "The six page one with four PDF files attached?"
Nerd 1: "Yeah. That one."
Nerd 2: "TL;DR dude, seriously. Did you read it?"
Nerd 1: "Nah. I might read it later."

If someone's ever mentioned "TL;DR" (too long; didn't read) when your e-mail was mentioned, don't fret. It's a quick fix. Just add a quick summary to the top of your e-mail prefaced with "TL;DR". Provide a really brief summary (bulleted lists are a plus) of your e-mail in the section and then start your e-mail right afterwards. Here's an example:

TL;DR
  * next software release deploys Monday
  * two bugs remaining to fix
  * we will get started at 8AM Saturday, yeaaaaah

Missed the joke? Head over to Wikipedia.

If one of the summary points interests a recipient, they'll scan your e-mail for the pertinent sections. Some recipients may only need to see what's in the summary and they won't bother reading the remainder. Either way, the effectiveness of your e-mail increases by leaps and bounds.

Plain text

Users of mutt prefer plain text e-mails

Here are a few more tips under this category:

• Don't use Outlook stationery.
• Never send e-mails with an image as the e-mail itself.
• No Comic Sans at any time. Period.
• Avoid graphical e-mail signatures (more on that in a moment).

E-mail signatures

Brevity can definitely be applied to e-mail signatures, too. How many times have you seen e-mails that end like this:

Frank Frankelton MCSE, RHCSA, RHCE, CCNA, RHCA, LPIC-3, Ph.D., M.D., Esq., CMDBA
Systems Adminstrator Extraordinaire, Database Administrator, All-around great guy
Office: 210-555-1212
Mobile: 210-555-1213
Other Mobile: 210-555-1214
Fax: 210-555-1215
VOIP: 210-555-1216
AIM: frankeltonia
Twitter: @frankyfrank
Jabber: frankfurter@frankeltonisinthehouse.com
Big Company, Inc

You might think that nobody would ever send out e-mails with a signature like the one above, but I've seen some that are actually worse. Keep the signature short and only put in the information that people really need to know. Generally, your name and title or department is sufficient for e-mail signatures (unless your local/federal laws require otherwise). Always preface it with a double dash "--" on a line by itself to signify that the remainder of the e-mail is the signature.

Summary

Keep it simple, keep it brief, and keep it relevant. While the suggestions above might not apply to every business or every person, following the suggestions will increase the effectiveness of your e-mails and ensure that your voice is heard on the other end.

I'm really interested to hear your comments. Are there some suggestions you have that I missed in the post? Did I make some suggestions which didn't make sense or don't apply to you? Let me know!

How to write e-mails to nerds (that they will actually read) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Here are the list of winners:

• Grand Prize ($50 at ThinkGeek): Dan Udey
• Runners-Up ($25 at ThinkGeek): Joe Wright, Susan Price, and Giovanni Tirloni

Dan's comment rang true with me since much of a sysadmin's job involves responding to crises regardless of how much planning you put forth:

Keep a cool head. Focus. Work methodically. Figure out what to do and get it done, and people will remember you as the person who performs under pressure. Once you can do that, you're a sysadmin.

Joe touched on a critical point about system administration:

Tell the truth. If you break something, 'fess up and fix it. If you don't know how to do something, admit it and learn how to do the task. Create your own culture of honesty on the job; others will respect and follow your example.

Susan offered some inspiration for system administrators stuck in frustrating situations:

I know, I know - dumb users, RTFM. Believe me, I've been there. In fact - one of your strategies should be to establish a trusted community where you can VENT about these issues, and get support for yourself. Ask for answers when you don't know them. Restock on the compassion and patience.

Giovanni talked about the basics and what every system administrator should know to get started in a career. We probably take this for granted, but this is critical to keep in mind:

If you are starting in the system administration area, don't praise yourself only because you (blindly?) fixed an issue or helped that friend with his/her server. Ask yourself: Why what I did fixed the issue? Why that was happening in the first place? And more importantly, how to avoid it for all eternity? You won't but it doesn't hurt to aim high.

Even though it isn't a runner-up, Paul's comment certainly deserves an honorable mention. His comment is actually a true story (with a slight amount of embellishment, of course) and it serves as a reminder that system administrators and developers must stand up for their beliefs even if it goes against the beliefs of their superiors. If your managers don't value the feedback, it might be a sign that a career change is in order.

Once again, a big thanks goes out to everyone who submitted a comment. I'll reach out to the winners today and get the gift certificates sent out to them.

Contest winners from the "Inspire a sysadmin" contest is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Stress can come from a variety of sources. Some of the obvious ones involve dealing with outages or tight deadlines, but there are some that aren’t so obvious, such as maintaining the customers’ trust and interpersonal issues.

There’s one thing you must remember: stress doesn’t have to rule your life. I’ve learned (and sometimes stumbled upon) some good techniques to prevent many of the negative effects of stressful situations at work and they’re definitely worth a try.

Know what you’re up against
It’s hard to battle a source of stress if you don’t know why it’s bothering you. Take the problem you’re facing and break it down into pieces. There are going to be some things you can and can’t change. Put the things you can’t change aside and focus on the things you’re able to change. As you tackle the list of things you can change, you might find ways to work around the things you can’t.

Interpersonal issues are easy
Stress that comes from dealing with coworkers may seem insurmountable at times. However, this type of stress is easily fixed and it normally stems from insufficient communication or conflicting goals. There’s an informal policy I’ve had on most of my teams called “Take it to the Racker” and it’s been quite successful. The basic idea is that if you have problems with another Racker, whether it’s something personal or work-related, take the grievance to them directly (in private, of course) and find common ground.

More often than not, this process leads to a good work relationship. It also improves communication drastically in the short term and it generally lasts if the people involved keep up the communication over time. I’ve seen Rackers who are so upset that they refuse to sit next to each other and after this process, they’re eating lunch together and working on the same projects.

Don’t fight your battles alone
Your best resources for fighting stress are all around you. Lean on your manager or your coworkers for help. Remember what your mother always told you: a trouble shared is a trouble halved. Your coworker might have a solution to a particular problem which frees up an hour for you each day and allows you to work on other projects. Your manager might not know that a particular task doesn’t fit your strengths and they might be able to provide you with another project that plays to your strengths.

Is it possible to reduce your stress level to zero at work? I don’t think so. However, you should always have a goal to reduce it when it makes sense.

As always, I’m interested to hear your comments. Which stress-reduction strategies work best for you? What is the source of most of your stress?

Success with stress is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

What do I mean by "do your homework?" Here are some bullet points to get you on your way:

Know what the company does.
This one is critical and it should be easy. However, make sure to do thorough research first. For example, if you interviewed at a company like Apple, becoming familiar with their hardware lineup should be a no-brainer. That's their bread and butter. On the other hand, remember that Apple isn't solely a hardware company; they write lots of software, provide online productivity services, and they distribute music, movies, and other entertainment media.

While you're doing this research, try to discover what makes the company unique. Sure, Apple sells laptops and desktops (just like a lot of other companies), but what makes their particular products unique? Is there something unique about the way they provide their services? Have they cornered a certain market segment by providing a combination of products and services to that group of consumers? Answering these simple questions may help you tip the scales in the interview process.

Try one or more of the company's products.
The feasibility of trying a company's product before an interview could be debatable. For example, if you wanted to interview at Cray, you probably don't need to drop $2M USD on your own XE6 before walking in the door. For companies where the barrier to entry for purchasing a product is much lower, such as cloud computing companies, there's no excuse to not try things out first. Amazon has a free tier and a Rackspace Cloud Server could cost you as little as $2.50 per week.

It's concerning when I talk to an applicant about a job working with Rackspace's Cloud Servers and they haven't tried out any cloud products from any provider. How can I take a candidate's interest seriously when they haven't shown interest in any portion of my group's market segment?

Know what the company's competitors do.
It's often more impressive to an interviewer to know what a company's competitors are doing and how it compares to what that company is doing in the market. For example, if you can walk into an interview and say "I like the way your company makes these widgets, but Company X is able to make them more lightweight, and I value that more than the added customer service your company offers." This shows the interviewer that you're familiar with various products in the segment and you've used them enough to understand what makes them different.

Some of you might be thinking: "Why would I say something like that to the interviewer? They'll think I'm being too negative about their product." That's always possible, but you can guard against it by wording everything carefully. Make sure you have a solid reason for the way you feel that is based on something substantial (usability, price, features, etc). I've had candidates talk for five to ten minutes about why one of our product is inferior to one of our competitors' products and I was very impressed.

One quick gotcha: your interviewer might turn your comments back on you and ask you how you would improve one of the inferior products (I do this regularly). Make sure that you're prepared for that question and consider offering up a suggestion before the question is presented to you.

Can't get the information you need? Ask!
When you reach the end of the interview and the interviewer asks if you have questions, be sure to ask any questions about topics you had trouble researching. Going back to the Cray example, compare what you know about an XE6 to servers you've used before. You could mention a problem you had with the density of your previous configurations and ask how they overcame that hurdle at Cray. If it's a proprietary trade secret, you might not get an answer, but they'll know that you did some serious research ahead of time. If they can share the answer, they might still be impressed, and you might end up learning something you didn't know prior to the interview.

Conclusion
In summary, doing your homework and learning about the company shows the interviewers that you not only have what it takes to do the work, but that the work interests you as well. I've interviewed folks in the past who lacked on technical ability but had plenty of desire and drive. More often than not, those people are now Rackers.

Do your homework before a technical interview is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Taking on the job of a technical manager presents its own unique challenges. It's easy for a technical manager to lose focus and get down in the weeds of daily work. It's also very difficult to let go of the reins and resign to the fact that the direct involvement in technical work will have to be reduced.

These problems resonate with me as I've recently taken on another technical management role at Rackspace. My previous experience involved managing a team of technicians at various skill levels who were working on customer environments made up of dedicated servers and network equipment. The current position has quite a few differences. I'm now managing a small group of highly technical and extremely dedicated Linux engineers and we're responsible for maintaining the systems and networks which run the Cloud Servers product.

One of my goals of this blog is to help others learn things much more easily than I have. Here are some things I've had to learn the hard way while working as a technical manager:

Get out of the mindset of an individual contributor
When you're a system administrator on a team (or by yourself), you're often judged on your personal job performance. Team interaction is important for some companies (especially at Rackspace), but not for others. Breaking the mindset of being an individual contributor was extremely difficult for me to do.

Managers are judged on the success of the team as a whole. Encouraging your team members to succeed and playing an active role in their personal and professional development is key. Each time you find yourself buried in the weeds of a problem rather than facilitating your team's work on the problem is when your performance as a manager will drop. If you do it more often, you may find that your team members aren't getting the support they need.

Don't be afraid of your team becoming smarter than you
One of the biggest things I've heard from my team is: "Aren't you worried about losing your technical skills when you're a manager?" My answer: "Of course." Anyone who has technical abilities will always be afraid of watching those abilities wane over time. However, as your team becomes stronger, you should be able to continue learning not through your own work, but through theirs. When your team members see that you're still interested in learning and you're now able to learn from them, they'll become more energized about their own work.

If you find yourself thinking negatively about a potential job candidate because they're smarter than you, step back and think for a moment. Put your own ego aside and consider what's best for you, your team, and your company. Your goal is to build a strong and successful team, not to pad your own ego. If your managers are judging you (as a technical manager) on your technical ability, then you need to solve that problem first.

Inspire instead of direct
Every manager faces the challenge of working with team members who disagree with a particular company policy or with the direction of their particular infrastructure. Keep in mind that your team members are probably not intending to be insubordinate and they might have something useful to contribute.

When you find yourself locking horns with your team members, inspire a discussion about the problem. Break out the disagreement onto a whiteboard and let the team make suggestions for improvements. Even if the entire discussion leads back to the fact that the original problem is inevitable, fostering that feedback loop is critical. You'll learn more about your team while they find ways to express their opinions and feel empowered.

The really tough part is when your team comes up with an alternative plan and you find yourself presenting to your leadership team. Always remember to take it seriously and know that you may need to refine the plan many times over before you find something acceptable for your team and the business.

De-stress by staying on task
If you're anything like me, you need some way to keep tabs on action items coming from meetings, e-mails, phone calls, and walk-ups. I've heard great things about applications like OmniFocus and Things, but I settled on 2Do. I really enjoy a strong to-do list which allows me to set priorities, due dates, and write extended notes about a particular task.

The best way to tackle a wall of tasks is to keep them organized into a concise list. Even if it's a small task, get it into your list so it's on your radar and you won't forget it. Work through the simple tasks and the high priority ones first but watch out for tasks with due dates.

Conclusion
All of these processes get easier over time and although your job will surely have challenges and pitfalls, the enjoyment will greatly increase. I feel privileged to lead a team of talented people who work on a complex and ever-expanding product.

Also, I'd like to mention that I'm not an expert on management! There are probably much better ways to do much of this than I've outlined in this post. Be sure to share your ideas in the comments section below.

How to survive as a technical manager is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

• your server could be used to gain access other servers
• data could be stolen from your server's databases or storage devices
• an attacker could capture data from your server's local network
• denial of service attacks could be launched using your server as an active participant

The best ways to limit your server's attack surface are pretty obvious: limit network access, keep your OS packages up to date, and regularly audit any code which is accessible externally or internally. As we all know, your server can still become compromised even with all of these preventative measures in place.

Here are some tips which will allow you to rapidly detect a compromise on your servers:

Abnormal network usage patterns and atypical bandwidth consumption
Most sites will have a fairly normal traffic pattern which repeats itself daily. If your traffic graph suddenly has a plateau or spikes drastically during different parts of the day, that could signify that there is something worth reviewing. Also, if your site normally consumes about 2TB of traffic per month and you're at the 1.5TB mark on the fifth day of the month, you might want to examine the server more closely.

On the flip side, look for dips in network traffic as well. This may mean that a compromise is interfering with the operation of a particular daemon, or there may be a rogue daemon listening on a trusted port during certain periods.

Many compromises consist of simple scripts which scan for other servers to infect or participate in large denial of service attacks. The scans may show up as a large amount of packets, but the denial of service attacks will usually consume a large amount of bandwidth. Keeping tabs on network traffic is easily done with open source software like munin, cacti, or MRTG.

Unusual open ports
If you run a web server on port 80, but netstat -ntlp shows something listening on various ports over 1024, those processes are worth reviewing. Use commands like lsof to probe the system for the files and network ports held open by the processes. You can also check within /proc/[pid] to find the directory where the processes were originally launched.

Watch out for processes started within directories like /dev/shm, /tmp or any directories in which your daemons have write access. You might see that some processes were started in a user's home directory. If that's the case, it might be a good time to reset that user's password or clear out their ssh key. Review the output from last authentication logs to see if there are account logins from peculiar locations. If you know the user lives in the US, but there are logins from various other countries over a short period, you've got a serious problem.

I've used applications like chkrootkit and rkhunter in the past, but I still prefer a keen eye and netstat on most occasions.

Command output is unusual
I've seen compromises in the past where the attacker actually took the time to replace integral applications like ps, top and lsof to hide the evidence of the ongoing compromise. However, a quick peek in /proc revealed that there was a lot more going on.

If you suspect a compromise like this one, you may want to use the functionality provided by rpm to verify the integrity of the packages currently installed. You can quickly hunt for changed files by running rpm -Va | grep ^..5.

Keeping tabs on changing files can be a challenge, but applications like tripwire and good ol' logwatch can save you in a pinch.

Summary
We can all agree that the best way to prevent a compromise is to take precautions before putting anything into production. In real life, something will always be forgotten, so detection is a must. It's critical to keep in mind that monitoring a server means more than keeping track on uptime. Keeping tabs on performance anomalies will allow you to find the compromise sooner and that keeps the damage done to a minimum.

Strategies for detecting a compromised Linux server is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

For my environments, I follow a strategy like this: I have some backups immediately accessible, others that are accessible very quickly (but not instantly), and others that are offsite and may take a bit more time to access.

Immediately accessible backups
One of the easiest way to have an immediately accessible backup is to have multiple machines online running the same versions of code or databases in a high availability group. If you have a node which fails, the remaining nodes should be able to handle the requests immediately. You may not consider this to be a backup under the traditional definition of what a backup should be, but it's functionally similar.

Backups that are accessible quickly
This second level of backups should be stored very close to your environment or within the environment itself. If you have multiple database and web server nodes, you could consider storing your web backups on the database servers and vice versa. For those who run very sensitive applications, this may violate the provisions of different certifications and regulations. A server dedicated to holding backups may be a viable alternative for additional security.

Offsite backup storage
These are the backups that need to be geographically distant from your main environment. Also, you should always consider storing these backups on more than one medium with more than one company.

For example, if your hosting providers offers a storage service, it's fine to store one set of your backups there, but consider storing them with a competitor as well. If you store your backups with your hosting provider in multiple places, you could be caught be a provider issue and lose access to your backups entirely. Hosting with multiple providers will allow you to access at least one copy of your backups even if there are billing or technical issues with a particular provider.

Another thing to keep in mind with offsite backup storage is how long it will take to transfer the backups to your hosting environment in case of an emergency. If your hosting environment is in Texas, but your backups are stored in Australia, you're going to have a longer wait when you transfer your data back.

A specific example
My environments are all in Dallas, Texas and I have a highly available environment with multiple instances. My second layer of backups are stored within the environment as well as in Rackspace's Cloud Files in Dallas. My third layer of backups are stored with Amazon S3 via Jungle Disk and at my home on a RAID array.

While I hope you never need to access your backups under duress, these tips should help to reduce your stress if you need to restore data in a hurry.

Strategies for storing backups is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Many of the topics covered on the EX442 exam are really handy for Linux system administrators. The advanced filesystem configuration, software RAID optimization, and process scheduling knowledge can provide some tangible performance gains on just about any Linux server.

I especially enjoyed learning about SystemTap. I'd heard about it before, but I never really understood what it did and how much power it can give a system administrator on a server. You can sample a wide variety of system events quickly with relative ease and you're not bound to the hardware dependencies of OProfile.

Bear in mind that you'll need a valid RHCE to get started with the RHCA exams. That's another certification that I'd definitely recommend as it helps you learn a wide variety of administration topics that can be applied to almost any Linux distribution available today.

One RHCA exam down, five to go is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Without a doubt, I'm most thankful for my family. They're my support base and I couldn't get through my days without them.

I wish I could thank my coworkers, mentors, and close friends individually, but this blog post would get rather lengthy. To everyone who has had taken the time to teach me something: I thank you. You've inspired me to continue learning and inspire others to do the same.

Happy Thanksgiving to you and your families. Please travel safely and take some time to relax.

Happy Thanksgiving is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

My main purpose for the blog at the beginning was to create a place where I could write quick articles about problems I found and how to fix them. Most of the people around me were using their own handy systems to store notes (Stickies on the Mac, Tomboy notes on Linux, or just simple text files), but they weren't able to share them easily. I wanted a way to write up a solution and instantly share it with someone. I also wanted that person to be able to pass along the fix to someone else if they wanted.

Needless to say, it took off from there.

It's important to note that I couldn't have done this by myself. I've learned some efficient strategies for managing large systems and troubleshooting complex issues from my peers, my managers, and colleagues outside of Rackspace. There have been many triumphs and there have been quite a few failures.

The failures have taught me the most. I've made some pretty large mistakes and here are a few:

• inserted data into a MySQL slave in an active replication pair
• run a fsck on an online ext3 partition
• marked a failed drive online in a hardware RAID array
• mangled Plesk installations in ways that you can't comprehend
• typed 'reboot' into a terminal and pressed enter, only to realize I was in the wrong terminal
• ran UPDATE statements without a WHERE clause in MySQL (well, I only did this one twice)

Even after all that, people occasionally tell me that I'm very good at what I do. I don't know if that's true or not, but I'm glad some people think so! Many of those folks end up asking me this question:

How do I learn how to be a successful Linux systems administrator?

My answer is this: Be humble. Always be thirsty for knowledge. Don't be afraid to make mistakes. Love what you do and the people you serve.

400th post: Looking back at where this blog started is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Typical configuration for a
proxy-type load balancer

• client makes a request
• load balancer receives the request
• load balancer sends request to a web node
• the web server sends content back to the load balancer
• the load balancer responds to the client

If you're not familiar with load balancing, here's an analogy. Consider a fast food restaurant. When you walk up to the counter and place an order, you're asking the person at the counter (the load balancer) for a hamburger. The person at the counter is going to submit your order, and then a group of people (web nodes) are going to work on it. Once your hamburger (web request) is ready, your order will be given to the person at the counter and then back to you.

This style of organization can become a problem as your web nodes begin to scale. It requires you to ensure that your load balancers can keep up with the requests and sustain higher transfer rates that come from having more web nodes serving a greater number of requests. Imagine the fast food restaurant where you have one person taking the orders but you have 30 people working on the food. The person at the counter may be able to take orders very quickly, but they may not be able to keep up with the orders coming out of the kitchen.

LVS allows for application servers
to respond to clients directly

• client makes a request
• load balancer receives the request
• load balancer sends request to a web node
• the web server sends the response directly to the client

The key difference is that the load balancer sends the unaltered request to the web server and the web server responds directly to the client. Here's the fast food analogy again. If you ask the person at the counter (the load balancer) for a hamburger, that person is going to take your order and give it to the kitchen staff (the web nodes) to work on it. This time around, the person at the counter is going to advise the kitchen staff that the order needs to go directly to you once it's complete. When your hamburger is ready, a member of the kitchen staff will walk to the counter and give it directly to you.

In the fast food analogy, what are the benefits? As the number of orders and kitchen staff increases, the job of the person at the counter doesn't drastically increase in difficulty. While that person will have to handle more orders and keep tabs on which of the kitchen staff is working on the least amount of orders, they don't have to worry about returning food to customers. Also, the kitchen staff doesn't need to waste time handing orders to the person at the counter. Instead, they can pass these orders directly to the customer that ordered them.

In the world of servers, this is a large benefit. Since the web servers' responses no longer pass through the load balancer, they can spend more time on what they do best -- balancing traffic. This allows for smaller, lower-powered load balancing servers from the beginning. It also allows for increases in web nodes without big changes for the load balancers.

There are three main implementations of LVS to consider:

LVS-DR: Direct Routing
The load balancer receives the request and sends the packet directly to a waiting real server to process. LVS-DR has the best performance, but all of your servers must be on the same network subnet and they have to be able to share the same router (with no other routing devices in between them).

LVS-TUN: Tunneling
This is very similar to the direct routing approach, but the packets are encapsulated and sent directly to the real servers once the load balancer receives them. This removes the restriction that all of the devices must be on the same network. Thanks to encapsulation, you can use this method to load balance between multiple datacenters.

LVS-NAT: Network Address Translation
Using NAT for LVS yields the least performance and scaling of all of the implementation options. In this configuration, the incoming requests are rewritten so that they will be transported correctly in a NAT environment. This puts a bigger burden on the load balancer as it must rewrite the requests quickly while still keeping up with how much work is being done by each web server.

A modern implementation and explanation of Linux Virtual Server (LVS) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

How do I become a better systems administrator?

The best way to become a better systems administrator is to fully understand the theory of what's happening in your server's environment.

What do I mean by that? Learn why things aren't happening as you expected and think about all of the factors that could possibly be involved. Instead of thinking purely about cause and effect, you'll find it much easier and rewarding to consider everything inside and outside your environment before you make any changes.

This still may be a little difficult to fully understand, so he's an example. Let's say you're handling an issue where a customer can't reach a website hosted on their server. When you ask them for more details, they might give you the dreaded reply: "It's not coming up." Start by making a mental list of the problems that are easiest to check:

• Is the web server daemon running?
• If a database server is being used, is it running and accessible?
• Is there a software/hardware firewall blocking port 80?
• Is a script stuck on the server tying up resources?
• Could there be a DNS resolution problem?
• Is the server up?
• Did a switch fail?
• Is the server's hard disk out of space?
• Can the customer reach other websites like Google or Yahoo?
• If SELinux is involved, have the appropriate contexts been set?
• Could the site be a target of a denial of service attack?
• Has the server reached its connection tracking limit?

Of course, this is a relatively short list, but these are all easy to check. If you're thinking about cause and effect, you might only consider the web server daemon and some basic network issues. By considering all of the other factors that may be related, you've ensured that all of the basics are covered before you consider more complex problems.

Most systems administrators have taken an error message and tossed it in en masse into Google before. Occasionally, no results will appear for the search. If you find yourself in this situation, try to understand the individual parts of the error message. Work outward from what you know already. You should know which daemon said it, and you may have an idea of what the application was doing when the error occurred. Take time to consider what the daemon is trying to tell you within the context of what it was doing at the time.

One of the easiest ways to force yourself to be immersed into this way of thinking is to host applications for non-technical people. You'll find that many customers want things done differently, and they're all at different levels of technical aptitude. Some may find it a frustrating experience at first, but you'll think yourself later. It will force you to consider all aspects of how a server operates since you might not always know what's happening within a customer's application.

As always, if you find yourself stumbling, remember to ask your peers and colleagues. Even if they haven't seen the particular issue, they will probably be able to guide you closer to the solution you seek.

A New Year System Administrator Inspiration is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Download It
Upgrade It

Upgraded to WordPress 2.5.1 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

No one has every really asked me "hey, if I want to fire my technical guy and get a new one, how do I do it?" So, how can I answer this question with any authority? Simple. I used to run my own company doing technical work for homes and businesses, I was hired and fired by people (much more hiring than firing), and I've learned a lot from being "the tech guy". Also, from working at Rackspace, and a previous job, I've seen many situations in which a company lets their technical person go without any plan in place. You never realize how valuable your IT staff are until they're not in the office, and your e-mail server falls apart.

Firing your technician

I'll start with how to fire your current technical person. It should go without saying, but be sure that you're firing this person for a substantial and legal reason. If you're firing this person for something trivial or petty, stop right here and re-evaluate. But, if this is the kind of person that ignores your phone calls, takes down services to increase job security, or prints pornography on the office laserjet, then it's time for them to go.

First, create a plan. How much does this technical person know about the company that could be detrimental if they were fired abruptly? You'll need to consider things like their access to your buildings, computers, corporate credit cards, cars, and colocated/dedicated servers. Take an inventory of the access that they have, and also how they access these items. For example, if they have multiple user accounts on your computer network, then make sure that all of those accounts are accounted for. If you have secret passwords with any of your service providers, be sure those are documented as well.

If you don't know some of these items, but your technical person does, you might want to get this information from them in a careful manner. I'd recommend against going in and saying something like "we need to inventory all of our user accounts before you're canned". You need to find a plausible excuse for you to have a list of this information, and it needs to be something that the technical person won't argue with. Some good ones I've heard are PCI, SOX or SAS/70 compliance. Let your employee know that compliance with these standards requires that you keep all of the access to all of these services in a safe place.

By the time you reach this stage, you really should have a new technician in mind. Interview them after work, or at night time so that the current technical person doesn't become suspicious. I'll talk more about how to find a new technical person a little later.

At this point, if you can trust your technical person, you should have a proper list of their entry points into your infrastructure. It's more likely that you don't trust this person at this point since you're firing them after all. Some might argue with me here, I'd recommend bringing in some other technical person that you undoubtedly trust. The reason for this is that your technical person may have given you a partial list, or they may have left "backdoors" so that they can access your infrastructure after they leave. A trusted tech can review your company for any possible issues and can give you a heads up if they find any red flags. Of course, if your current technical person has set up traps to know when someone logs in, then you may have blown your cover entirely. I would certainly hope that your situation wouldn't end this badly.

Now that you have a complete list of everything to which your former tech had access, you have an idea of what will be involved in changing everything over. Most people like to fire employees on Fridays to reduce the chance of violence or uncomfortable moments, so here's my recommendation. If anything financial needs to be taken care of, get it done late on Thursday or early on Friday. Then, on Friday, set a time with your current technical person to have a short meeting. Coordinate this time with your trusted technical person so they can begin changing passwords on accounts which the current tech has access to. Change the most sensitive passwords first, like the passwords on database servers. Also, change the root passwords as a high priority, but make sure you eventually change them all since you can be bitten by sudo or SSH keys.

When your current technical person exits the meeting, you're covered. If the meeting goes well, and the current technical person is amicable, then you're going to be covered since their access is revoked. If the meeting goes badly, your still covered in case they try to do something nasty. Their access to your network and corporate infrastructure should be eliminated or minimized before they can do anything destructive.

Hiring your technician

Luckily, hiring a new technical person is a bit easier than firing one. However, if you do a bad job on the hiring, you'll be referring to the beginning of this article fairly soon.

The best way to find a new technical person is via recommendations from another person. They've probably had interactions with the tech, and they can give you an idea of their technical prowess and social skills (yes, these are important). If you can't find any techs through recommendations, you can always check big job sites like Monster, LinkedIn, or Dice. Whichever route you choose, be sure to meet the technician in person. Don't hire someone based on the initials after their name, their previous job experience, or how they sound over the phone. Your technical person is like a central pillar in your organization, and this needs to be a responsible, sensible, and practical person.

Once you've found one or more technicians that you'd like to hire, you need to test them just a little. I'd recommend contacting them late in the evening (8-10PM) or early/late on the weekend. See how receptive and cordial they are at these times, because when something explodes later, you'll probably be calling them after business hours. You don't want to pick up the phone at 4AM on Saturday when your Exchange server dies only to hear your tech tell you that he'll be in on Monday to fix it for you, and that it can wait until then. When you talk to the technician on the phone, ask them to do something that forces them to go use the computer. For example, send them an e-mail for something reasonable that they need to respond to. Or, tell them that you discovered some neat product or service, and you want to know if they could start working at your company and maintain that product or service. If they respond quickly and they don't give you the vibe that you've just inconvenienced them horribly, then that's a good sign that you've found a worthy technician.

It's up to you when you bring them on at your company. Some people might want to hold off until the current technician is out of the way, but some might want to bring the technician in a little early to help with the cleanup of the last technician. Either way is good in my opinion. However, I would recommend against having both of them employed at your company simultaneously. If your old technician is upset about something, that could rub off on the new guy, and you may be returning to the top part of this article sooner rather than later.

Also, don't expect the new technician to be knee-deep in your problems immediately. They will need some time to figure out your network, review your vital services, and get an idea how everything works together. If you have the giant list your previous tech made, be sure to furnish it to the new technician so they have an idea of where to go to fix a certain problem.

I certainly hope this article helps! If you have any questions, drop me a comment and I'll be happy to give additional recommendations.

Small Companies: How to hire and fire a technical person is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.