Typical configuration for a
proxy-type load balancer

• client makes a request
• load balancer receives the request
• load balancer sends request to a web node
• the web server sends content back to the load balancer
• the load balancer responds to the client

If you're not familiar with load balancing, here's an analogy. Consider a fast food restaurant. When you walk up to the counter and place an order, you're asking the person at the counter (the load balancer) for a hamburger. The person at the counter is going to submit your order, and then a group of people (web nodes) are going to work on it. Once your hamburger (web request) is ready, your order will be given to the person at the counter and then back to you.

This style of organization can become a problem as your web nodes begin to scale. It requires you to ensure that your load balancers can keep up with the requests and sustain higher transfer rates that come from having more web nodes serving a greater number of requests. Imagine the fast food restaurant where you have one person taking the orders but you have 30 people working on the food. The person at the counter may be able to take orders very quickly, but they may not be able to keep up with the orders coming out of the kitchen.

LVS allows for application servers
to respond to clients directly

• client makes a request
• load balancer receives the request
• load balancer sends request to a web node
• the web server sends the response directly to the client

The key difference is that the load balancer sends the unaltered request to the web server and the web server responds directly to the client. Here's the fast food analogy again. If you ask the person at the counter (the load balancer) for a hamburger, that person is going to take your order and give it to the kitchen staff (the web nodes) to work on it. This time around, the person at the counter is going to advise the kitchen staff that the order needs to go directly to you once it's complete. When your hamburger is ready, a member of the kitchen staff will walk to the counter and give it directly to you.

In the fast food analogy, what are the benefits? As the number of orders and kitchen staff increases, the job of the person at the counter doesn't drastically increase in difficulty. While that person will have to handle more orders and keep tabs on which of the kitchen staff is working on the least amount of orders, they don't have to worry about returning food to customers. Also, the kitchen staff doesn't need to waste time handing orders to the person at the counter. Instead, they can pass these orders directly to the customer that ordered them.

In the world of servers, this is a large benefit. Since the web servers' responses no longer pass through the load balancer, they can spend more time on what they do best -- balancing traffic. This allows for smaller, lower-powered load balancing servers from the beginning. It also allows for increases in web nodes without big changes for the load balancers.

There are three main implementations of LVS to consider:

LVS-DR: Direct Routing
The load balancer receives the request and sends the packet directly to a waiting real server to process. LVS-DR has the best performance, but all of your servers must be on the same network subnet and they have to be able to share the same router (with no other routing devices in between them).

LVS-TUN: Tunneling
This is very similar to the direct routing approach, but the packets are encapsulated and sent directly to the real servers once the load balancer receives them. This removes the restriction that all of the devices must be on the same network. Thanks to encapsulation, you can use this method to load balance between multiple datacenters.

LVS-NAT: Network Address Translation
Using NAT for LVS yields the least performance and scaling of all of the implementation options. In this configuration, the incoming requests are rewritten so that they will be transported correctly in a NAT environment. This puts a bigger burden on the load balancer as it must rewrite the requests quickly while still keeping up with how much work is being done by each web server.

©2010 Racker Hacker. All Rights Reserved.

How do I become a better systems administrator?

The best way to become a better systems administrator is to fully understand the theory of what's happening in your server's environment.

What do I mean by that? Learn why things aren't happening as you expected and think about all of the factors that could possibly be involved. Instead of thinking purely about cause and effect, you'll find it much easier and rewarding to consider everything inside and outside your environment before you make any changes.

This still may be a little difficult to fully understand, so he's an example. Let's say you're handling an issue where a customer can't reach a website hosted on their server. When you ask them for more details, they might give you the dreaded reply: "It's not coming up." Start by making a mental list of the problems that are easiest to check:

• Is the web server daemon running?
• If a database server is being used, is it running and accessible?
• Is there a software/hardware firewall blocking port 80?
• Is a script stuck on the server tying up resources?
• Could there be a DNS resolution problem?
• Is the server up?
• Did a switch fail?
• Is the server's hard disk out of space?
• Can the customer reach other websites like Google or Yahoo?
• If SELinux is involved, have the appropriate contexts been set?
• Could the site be a target of a denial of service attack?
• Has the server reached its connection tracking limit?

Of course, this is a relatively short list, but these are all easy to check. If you're thinking about cause and effect, you might only consider the web server daemon and some basic network issues. By considering all of the other factors that may be related, you've ensured that all of the basics are covered before you consider more complex problems.

Most systems administrators have taken an error message and tossed it in en masse into Google before. Occasionally, no results will appear for the search. If you find yourself in this situation, try to understand the individual parts of the error message. Work outward from what you know already. You should know which daemon said it, and you may have an idea of what the application was doing when the error occurred. Take time to consider what the daemon is trying to tell you within the context of what it was doing at the time.

One of the easiest ways to force yourself to be immersed into this way of thinking is to host applications for non-technical people. You'll find that many customers want things done differently, and they're all at different levels of technical aptitude. Some may find it a frustrating experience at first, but you'll think yourself later. It will force you to consider all aspects of how a server operates since you might not always know what's happening within a customer's application.

As always, if you find yourself stumbling, remember to ask your peers and colleagues. Even if they haven't seen the particular issue, they will probably be able to guide you closer to the solution you seek.

©2010 Racker Hacker. All Rights Reserved.

Download It
Upgrade It

©2010 Racker Hacker. All Rights Reserved.

No one has every really asked me "hey, if I want to fire my technical guy and get a new one, how do I do it?" So, how can I answer this question with any authority? Simple. I used to run my own company doing technical work for homes and businesses, I was hired and fired by people (much more hiring than firing), and I've learned a lot from being "the tech guy". Also, from working at Rackspace, and a previous job, I've seen many situations in which a company lets their technical person go without any plan in place. You never realize how valuable your IT staff are until they're not in the office, and your e-mail server falls apart.

Firing your technician

I'll start with how to fire your current technical person. It should go without saying, but be sure that you're firing this person for a substantial and legal reason. If you're firing this person for something trivial or petty, stop right here and re-evaluate. But, if this is the kind of person that ignores your phone calls, takes down services to increase job security, or prints pornography on the office laserjet, then it's time for them to go.

First, create a plan. How much does this technical person know about the company that could be detrimental if they were fired abruptly? You'll need to consider things like their access to your buildings, computers, corporate credit cards, cars, and colocated/dedicated servers. Take an inventory of the access that they have, and also how they access these items. For example, if they have multiple user accounts on your computer network, then make sure that all of those accounts are accounted for. If you have secret passwords with any of your service providers, be sure those are documented as well.

If you don't know some of these items, but your technical person does, you might want to get this information from them in a careful manner. I'd recommend against going in and saying something like "we need to inventory all of our user accounts before you're canned". You need to find a plausible excuse for you to have a list of this information, and it needs to be something that the technical person won't argue with. Some good ones I've heard are PCI, SOX or SAS/70 compliance. Let your employee know that compliance with these standards requires that you keep all of the access to all of these services in a safe place.

By the time you reach this stage, you really should have a new technician in mind. Interview them after work, or at night time so that the current technical person doesn't become suspicious. I'll talk more about how to find a new technical person a little later.

At this point, if you can trust your technical person, you should have a proper list of their entry points into your infrastructure. It's more likely that you don't trust this person at this point since you're firing them after all. Some might argue with me here, I'd recommend bringing in some other technical person that you undoubtedly trust. The reason for this is that your technical person may have given you a partial list, or they may have left "backdoors" so that they can access your infrastructure after they leave. A trusted tech can review your company for any possible issues and can give you a heads up if they find any red flags. Of course, if your current technical person has set up traps to know when someone logs in, then you may have blown your cover entirely. I would certainly hope that your situation wouldn't end this badly.

Now that you have a complete list of everything to which your former tech had access, you have an idea of what will be involved in changing everything over. Most people like to fire employees on Fridays to reduce the chance of violence or uncomfortable moments, so here's my recommendation. If anything financial needs to be taken care of, get it done late on Thursday or early on Friday. Then, on Friday, set a time with your current technical person to have a short meeting. Coordinate this time with your trusted technical person so they can begin changing passwords on accounts which the current tech has access to. Change the most sensitive passwords first, like the passwords on database servers. Also, change the root passwords as a high priority, but make sure you eventually change them all since you can be bitten by sudo or SSH keys.

When your current technical person exits the meeting, you're covered. If the meeting goes well, and the current technical person is amicable, then you're going to be covered since their access is revoked. If the meeting goes badly, your still covered in case they try to do something nasty. Their access to your network and corporate infrastructure should be eliminated or minimized before they can do anything destructive.

Hiring your technician

Luckily, hiring a new technical person is a bit easier than firing one. However, if you do a bad job on the hiring, you'll be referring to the beginning of this article fairly soon.

The best way to find a new technical person is via recommendations from another person. They've probably had interactions with the tech, and they can give you an idea of their technical prowess and social skills (yes, these are important). If you can't find any techs through recommendations, you can always check big job sites like Monster, LinkedIn, or Dice. Whichever route you choose, be sure to meet the technician in person. Don't hire someone based on the initials after their name, their previous job experience, or how they sound over the phone. Your technical person is like a central pillar in your organization, and this needs to be a responsible, sensible, and practical person.

Once you've found one or more technicians that you'd like to hire, you need to test them just a little. I'd recommend contacting them late in the evening (8-10PM) or early/late on the weekend. See how receptive and cordial they are at these times, because when something explodes later, you'll probably be calling them after business hours. You don't want to pick up the phone at 4AM on Saturday when your Exchange server dies only to hear your tech tell you that he'll be in on Monday to fix it for you, and that it can wait until then. When you talk to the technician on the phone, ask them to do something that forces them to go use the computer. For example, send them an e-mail for something reasonable that they need to respond to. Or, tell them that you discovered some neat product or service, and you want to know if they could start working at your company and maintain that product or service. If they respond quickly and they don't give you the vibe that you've just inconvenienced them horribly, then that's a good sign that you've found a worthy technician.

It's up to you when you bring them on at your company. Some people might want to hold off until the current technician is out of the way, but some might want to bring the technician in a little early to help with the cleanup of the last technician. Either way is good in my opinion. However, I would recommend against having both of them employed at your company simultaneously. If your old technician is upset about something, that could rub off on the new guy, and you may be returning to the top part of this article sooner rather than later.

Also, don't expect the new technician to be knee-deep in your problems immediately. They will need some time to figure out your network, review your vital services, and get an idea how everything works together. If you have the giant list your previous tech made, be sure to furnish it to the new technician so they have an idea of where to go to fix a certain problem.

I certainly hope this article helps! If you have any questions, drop me a comment and I'll be happy to give additional recommendations.

©2010 Racker Hacker. All Rights Reserved.