UPDATE: THE STAKES ARE RAISED! Check the end of this post for details.
Today is my birthday and I'm doing things in reverse -- you are getting gifts today. I'm giving away four $25 gift certificates to ThinkGeek today (yep, that's $100 out of my pocket) but you'll have to do something to earn them.
I'm looking for words of wisdom and guidance from the readers of my blog for system administrators who are just getting started. I talk to brand new sysadmins and college graduates regularly and they're all hungry for what the seasoned folks in the industry know. They're not specifically on the hunt for hard facts and how-to's; they're looking for guidance on how to gain experience, reduce errors and learn efficiently.
Let's get to the important stuff: How does this contest work?
- Write a comment. Put an inspirational story, anecdote, or random words of wisdom for system administrators who are new to the industry in a comment on this post. Although it doesn't have to be extraordinarily lengthy, try to write more than just a sentence or two.
- Give me a way to contact you. Add something to your comment so I can contact you if you're the winner.
- Do it soon. The contest ends at 11:59PM CDT tonight.
I'll be the judge of the comments and I'm going to choose the winners based on the content of the comment. The more inspirational and profound your comment is, the better chance you have of winning. Any comment written in LOLCats caption style will lose points immediately. 
One last thing: This contest isn't affiliated with my employer or ThinkGeek. I'm doing this on my own. However, I'm a big fan of both my employer and ThinkGeek, but that's irrelevant right now.
UPDATE: The folks at ThinkGeek decided to not only pay for one of the gift certificates, but they're going to double it. There's now a $50 certificate for the best entry and three more $25 certificates for second, third and fourth best entries. Thanks again to ThinkGeek for offering this up!
UPDATE: The winners have been announced!
Happy birthday! Is that inspiring enough?
But seriously, happy birthday.
Thanks, Baron.
Happy birthday! And never forget about man man
I wish that when I start as a Sysadmin someone would have told me to not be afraid to make mistakes as long as you learn from them. To say "I don't know, but I'll find out" when you don't have an answer to a problem. Most importantly never stop learning your job.
Happy birthday!
And to aspiring sysadmins: You can NEVER have a strong enough disaster recovery plan. Backup, test your backup, then test it again. Unless your website is http://www.zombo.com/, you should be ready to recover from almost any disaster—fire, earthquake, connectivity problem, etc. Plain for the fail whale, and you're half of the way towards success in sysadmin work! As Netflix says: the best way to avoid failure is to fail constantly.
Congrats.
Well. The first thing I always learn newbies here is the golden support rule: "Never assume anything".
In dutch it even sounds better: "Neem niets aan van je klant, ... behalve zijn geld."
Being a system administration is all about understanding how things work, building new things and fixing broken things. You can only build new things and fix broken things if you know how things work and that's what I think is lacking the most: curiosity.
If you are starting in the system administration area, don't praise yourself only because you (blindly?) fixed an issue or helped that friend with his/her server. Ask yourself: Why what I did fixed the issue? Why that was happening in the first place? And more importantly, how to avoid it for all eternity? You won't but it doesn't hurt to aim high.
Decent system administrators are rare because what makes them decent is a mix of technical and human skills. Sometimes you'll need to step out of the technical area and understand the business, what your client is trying to do and ask yourself: Geez, I don't know anything about accounting but is there a better way to do this? When you do that, your client/boss/customer/friend you see that you are really interested in helping.. and system administration is all about that: using your technical skills to do good.
BTW, happy birthday
Don't be afraid to break software - I have learned more by breaking things in SCCM and having to fix it than I would have without having it break. Also if you are adding drivers to a PXE image in sccm make sure they are the PXE drivers and not normal windows drivers.
In 1998, my fledgling web design company went through a growth spurt. We had acquired about 25 clients - mostly small and medium businesses - and utilized the services of a "buddy of mine" for hosting services. I could work in Photoshop, write HTML by hand, and write Director Lingo code in my sleep, but I couldn't (and wouldn't) approach a Linux command line.
Through a series of events I found myself outgrowing this friend's level of service and took the leap to move all my client's sites and email to a dedicated server. This led to a falling out of the previous friendship, and a mandate of "You've got one week to get you crap off my box."
Scouring a few web sites, I chose a contract system administrator to build the box and do the migration. That's when my personal hell began.
My contractor talked the talk, but couldn't deliver. Not only did he already have a day contract for 60 hours per week as a PERL programmer, but he didn't have the chops to do the most basic setup of BIND, Apache, and Sendmail. His excuses were peppered with tales of his new baby cyring all night and complaints of his dialup internet connection. Meanwhile the clock was ticking over my eviction.
This was quite possibly the most painful time in the growth of my small company. Having spent my money on a contractor who couldn't deliver, I was forced to do the unthinkable. I had to do it myself. And thank God for that!
My eviction came swiftly, which brought downtime to my clients. For an entire week, I dug deep into online tutorials for Apache, BIND and Sendmail. And every day, I took time out to phone or visit each client and give them a status update. Even though their sites and email were down for a full 5 business days, Incredibly, I didn't lose a single customer from it. In today's world this would have been completely unacceptable. But given the kind of clients I had back then and email being mostly a novelty to them, they offered me grace and encouragement.
I never wanted to be a System Administrator. But countless times over the past 13 years, I have realized that without that horribly steep learning curve in the very beginning, I'd still be a slave to "some guy" who charges me by the hour. Even to this day, the music I listened to during that time of failure makes me nervous to hear on the radio. It was singly the most painful time in my life, realizing that my family's well being rested on whether I could figure out how to set up /etc/mail/virtusertable.
13 years later I'm no guru, but I can handle 99% of the day to day needs of a system administrator. It took some time and pain in the beginning, but now I play nicely with Sendmail M4 files, BIND zones, and Apache and mySQL, even adminning a high-traffic, high-availability cluster of servers for several of my company's clients.
If your business is resting on your mastery of a certain bit of knowledge and the clock is bearing down on you, the best thing you can think about is how great you'll feel once you've kicked that task's ass and tamed it. And although it might be painful, particularly when you're a breadwinner for those who depend on you, it's worth the pain and that knowledge that was so painfully hard to acquire. Especially if you build on that knowledge over time.
That sinking feeling you get in your gut when you realize a server is down will never go away. But having the skill to bring that bitch back online is priceless. So stick with it. Keep learning. And try not to piss anyone off in the process, because you might need to ask them a question later!
(^^ spoken like the old fart that I am ^^)
Happy Birthday!
In 1998 I changed careers and jumped headfirst into Unix administration. If I could go back in time and give myself advice, here are some things I'd say.
1. Programming ability will take you far. Ability to automate is a quality that makes you stand out; many sysadmins simply can't/won't do it. Knowledge of Perl and ksh got me my first Unix admin job, and I've spent most of my career doing automation and tool development.
2. Never allow people (or yourself) to do development on production systems.
3. Learn some sort of version control system. Use it as much as possible.
4. Never stop learning your profession. Never stop reading.
5. Treat others well. Your users are your customers. They don't know your job, but they do learn quickly (and tell others) whether you are pleasant or a jerk.
6. Tell the truth. If you break something, 'fess up and fix it. If you don't know how to do something, admit it and learn how to do the task. Create your own culture of honesty on the job; others will respect and follow your example.
7. Take care of yourself. No one else will.
8. Have fun. You're getting paid to do your hobby: it doesn't get any better than that!
My first tech job was doing phone support for a little ISP in 1998. As I worked there, and got more familiar with FreeBSD, I got more of the sys admin duties. There was one full-time sys admin, and he wound up being fired because he chewed out a customer on the phone (why he was on the phone with a customer, I have no idea). His last task was to school me on adminning BIND. He sat down, showed me around a zone file, and then said: "And the most important thing is, don't forget to update the serial number. Okay, bye." He got up, walked out, and that was the last I saw of him.
Major,
That is an awesome thing you are doing!
I agree with most of these comments, especially the ones about making mistakes.. we are human and it happens. The important thing is to learn from it.
Joe Wright has hit very close to home for me with comment 8 Have fun. You're getting paid to do your hobby: it doesn't get any better than that! -- seriously!! I feel like I get paid to play with other peoples money all day, its an amazing feeling!
I don't have a lot of advice for new sysadmins (no one ever gave me any when I was starting out), but one piece of advice I do give out has saved me repeatedly.
You will screw up. You will delete important files, you will delete backups, you will overwrite documents, you will reboot the wrong servers, you will forget to add a WHERE clause to your DELETE FROM statement. Things will go wrong.
When they do, allow yourself the luxury of panic. Panic, freak out, and scream 'oh crap oh crap oh crap' – but do it in your head, and do it for five or ten seconds. Once you're done, get back to work and fix your mistakes.
When you realize that something has gone horribly wrong there's an immediate adrenaline spike, which can be a shock to your system. This is your opportunity to freak out. Once that's done, you've got adrenaline and you've got your skills as a system administrator, and unfortunately, only one of these will help you fix your problems.
Keep a cool head. Focus. Work methodically. Figure out what to do and get it done, and people will remember you as the person who performs under pressure. Once you can do that, you're a sysadmin.
System Administrators are silent professionals. We're the Lee Atwaters of the tech world. It's a behind the scenes type of gig and you shouldn't expect to be a rock star. An Admin who receives frequent praise probably sucks. A good Admin is so organized that things rarely break and when they do, they break in very predictable ways so that remediation looks easy.
I exclude certain Admin positions such as those who work in support queues where they don't really manage things fully and customers have root access. I consider them more as tech support than System Adminstrators. I had my start at Rackspace and know the queue life well but it's a very different role.
I think the best piece of advice I can give to anyone is to be organized. Documentation, config management, revision control, and veering from the standard in only the most extreme cases are the most important. You may have a very deep understanding of the Linux scheduler or the TCP/IP stack but if things are changing constantly in production without documentation, it's likely to cause problems on the business side which is all you're paid to do. Just support the business people so they can make more money and hope you can get a slice.
Way back in the halcyon days of 2006 I reconnected with a guy I had met in college. He said "come work for me, we'll write PHP and shit". And lo, we did, and it was good (if not the code, then at least the job).
But all was not well in coding land, for the grinch was our boss and when not stockpiling lima beans for the coming apocalypse he had some sales techniques that caused friction with his dev team.
"I sold a new feature; we need it done yesterday!" said the Boss.
"You can't keep changing requirements and making things up just because the potential customer said they needed that..." said our Hero.
The Boss shook his head slowly, as if trying to digest this radical concept. Could it be that this guy, this punk kid, might have a point?
"Look, I know you're a perfectionist," said the Boss man. "But you need to realize that in the real world we ship software first and do cursory testing later. In fact, we might never do that testing. You should be writing code that works perfectly the first time like the classic ASP I originally designed our systems around."
Our Hero stared into the distance, pondering his incipient insanity. "Well then, I guess I'm going to scatter my resumé to the winds. Surely it will take root in more fertile ground!"
Our Boss laughed maniacally, confident that no one would want to hire an ambitious and hardworking young college graduate who had everything to offer a company that treated him well. Unfortunately for him, our Hero landed a job at a small company nearby. It wasn't much, really just a few servers run in a shed down by the river, but he rapidly rose through the ranks of the company to lead his own team of homeless people.
*audible stage whispering*
I'm sorry, I've been informed that while having the appearance of homeless people, these were actually highly trained and well-educated Linux sysadmins. Your narrator regrets this error.
From a lowly level 2 technician (barely trusted to check the crontab), our Hero swiftly moved to level 3, team lead, and then into the cloud division. From there he made his way into the VM support team and has continued to rapidly expand his base of knowledge. Yes, the past 4.5 years have been professionally eventful for our Hero, but I want to share with you some words of wisdom I heard from him one fateful day in 2006. I believe they are the secret to all his success:
"Gentoo is the best server distribution."
For you see, even those who are successful can (and will continue to) say and do really stupid things! Happy birthday Major!
*BEGIN HEMMINGWAY*
For sale: Instruction manual. Never opened.
*END HEMMINGWAY*
Happy Birthday, Major!
My best advice for beginning sysadmins is to cultivate a customer-centric mindset in themselves. It will serve them better than any single factor in their careers.
SysAdmin personality types are often:
Ruthlessly efficient
Smart, and quick-thinking
Problem solvers
About 10 beats ahead of everyone else
You geniuses with your valuable skills can sometimes be light on:
Compassion
Patience
Listening skills
I know, I know - dumb users, RTFM. Believe me, I've been there. In fact - one of your strategies should be to establish a trusted community where you can VENT about these issues, and get support for yourself. Ask for answers when you don't know them. Restock on the compassion and patience.
Here's our customer support checklist, hope it helps:
Listen patiently to the client's report of their problem or request - let them get it out. Try not to interrupt. Try not to make them feel stupid; they're already having to beg for your help.
Restate the problem from their point of view. Even if it's painful. Even if it sounds stoopid.
Document their problem - being nice doesn't mean you don't have to cover your butt. Also, acknowledging receipt can ease so much frustration. Finally, somebody is LISTENING out there!
Respond with explicit, step-by-step instructions and/or status, in respectful tone.
Don't include ANY more explanation than absolutely necessary; instead, offer more info if they want it.
Invite them to contact you for a walkthrough, and to contact us again if they need further assistance.
Charge your time!
Best of luck. The world needs SysAdmins with skill and compassion.
-Susan
The main piece of advice I have for new SysAdmins is that if you're doing your job properly, you probably won't have everyone liking you. It's your job to say no and make the unpopular decision sometimes so you'll need to have a thick skin.
Joe Wright hit the nail on the head.
I would consider adding these comments:
9. Read Trade Magazines. Read Trade Websites. Listen to Trade Podcasts.
(Substitute your OS of choice for "Trade" above.)
10. Read a copy of "Debugging" by David Agans. Good debugging skills will take you far.
11. Search for "The Limoncelli Test". Read it and then use it as a litmus test for any site that you work at.
- Bill
In addition to the other great comments so far - one peice of advice I'd share - document everything. Even if you think it's no big deal, having it documented if you need to undo your changes or repeat it at 3:00 a.m. on a Sunday night is a lifesaver. With the proliferation of wikis, keeping documentation up to date is much easier than it used to be.
It's certainly not the most glamourous part of the job, but it's extremely valuable when needed!
Happy Birthday Major.
For Sysadmins: Please be able to say, "I don't know, but I will find out"
Happy Birthday Major!
Real experience is about doing not reading: the importance of hands-on learning
Do not misinterpret this; reading books is excellent and I recommend that all sysadmins and programmers read plenty of books on their field. But as you read, please don't forget to apply the knowledge you're learning.
If you're reading a book about linux, you certainly better have a linux distribution installed on your machine or running in a VM because just reading and understanding the commands is not enough. You have to use what you're reading and learning to let it really sink in. Too many college courses and even certification exams rely on multiple choice tests! However, the certifications you really want (RHCE for example) will engage you in a way that requires real experience that only comes from hands-on learning.
New sysadmin... listen up. Your mamma ain't around to wipe your nose any more so the first thing you'll need as a sysadmin is thick skin. When (not if) things go wrong, somebody's gonna bring down the thunder and sparks will fly. You will be accused and spit upon (OK, maybe not literally) and you'll have to learn to 'take it' and faithfully do your job, day after day, anyway. Don't expect anyone except your fellow sysadmins to remember you on SysAdmin day and don't expect any pats on the back for keeping the backups current. In so many ways, it's a thankless job but one we love and will do without thanks, cause that's not what it's about.
Tip number 1: Don't ever let your boss or a user bring you news about a system being offline that you already didn't know about. You server monitoring and notification system has got to be rock solid.
Happy birthday, Major
Remember why you got into this line of work. There will be days where you wish you were born Amish, but they will go by. There will also be days that you wish you were born as the last human after the computers take over. Yea... there will be more of these days. But when everything comes together in sweet, sweet harmony, you can take on the world. Take charge, assert yourself, be competent, and try not to lose any important 1's or 0's.
Happy birthday Hayden!
I'd like to bring back an old school of thought. Regular reboots! I know so many sysadmins that obsess over uptime. The problem with uptime is that the longer it's been since the last reboot, the bigger chance there is that the system won't come back up once you are finally required to reboot.
These days with increased capacity to scale services, having load balanced standby systems it actually makes sense to reboot the systems regularly as it also helps you test the failover procedures. There is nothing worse than having the primary system fail only to learn that failover hasn't actually been configured right and isn't working!
Enjoy your cake!
The advice I'd give new (or new-ish) sysadmins: learn to know, love, and understand monitoring systems.
They can't tell you everything, but they can give you a heads up when things are trending in the wrong direction.
There's a lot of well-deserved love thrown in the direction of things like Nagios, Icinga, Munin, and Cacti for monitoring the computer systems in your server room. What do you have to monitor the server room itself?
It's all well and good until you discover (the hard way, at 2am) that the HVAC unit in your server room failed about 4:30 pm, and the temperature slowly drifted upwards until such time as the compressor just totally shut down, thus spiking the temperature above 110 degrees F behind the server racks.
That lovely, metallic wall-mounted dial thermometer/hygrometer combo from the hardware store isn't going to send you an email if either temperature or humidity climbs above some threshold. And unless it's a paper-chart recording sort of system, it's not going to show trends. In fact, it's probably on the wall of the server room to cover a hole a predecessor punched into the wall.
Some may say: "But we don't need to see historical trends for the temperature and humidity!!!"
I speak from experience here--- show a systems administrator a graph of memory usage over time, along with a graph of network traffic over time? It gives them a starting point in their diagnosis of problems.
Similarly-- if you have a temperature graph with suitable detail and resolution, when something starts to go wrong, there's going to be a trend.... ( It's not always this obvious. (and yes, we knew the RH meter wasn't scaled correctly. ) )
You may think you only need to know how to troubleshoot slow queries in MySQL, or DNS errors, and the like--- but do you know how to tell if the compressor in your server room's HVAC unit has stopped? Can you tell if it's a breaker issue, or if it's just overheated? In the event it's an overheat, do you know how to reset the compressor? ( Have someone SHOW you how to do this. )
It took my boss and I several months to convince our higher-ups that we needed to bring in Outside Help to deal with our HVAC unit-- they were insistent on using only our (non-certified or trained) physical plant worker to deal with it. Temperature graphs helped us show that (a) something was wrong and (b) they weren't fixing it.
If you're a systems administrator, you need to understand that the physical environment of a server is just as important as the desktop environment.
Hello,
It is always a good idea to understand the business side of your job as well. You should not take your job as someone who is just executing commands. In this market keep the attitude of becoming expert generalist, and try to learn all sides of your job whether it's DB, Networking or scripting. Always remember that lazy Administrators are the good ones and there are no Heroes in Administrators world. People will not remember you once the outage is fixed and you will always get 5 minutes of fame when you are on an outage call.
If you are trying to become Linux Administrator then the following may work for you.
1. Learn scripting if you want to stand out. It could be Bash, Perl or Python...
2. Try to master tools your company has in place for you. But always remember companies may change them or go out of business so develop your skills as rehire-able by the world. Company is Business and they can fail anytime or change their business plans so make sure you develop real Administrators skills which can be utilized by any company not just your company.
3. Go on Dice.com or Indeed.com and try to search for jobs related to your skills and master the skills commonly expected from your line of work. This will help you if you ever loose your job.
4. Linux Administrators are an OS Administrators, so they are expected to know all pieces of puzzles. You will be expected to have skills of great troubleshooter and as well as DNS, Mail, HA configuration etc. You will be expected to become Jack of All Traders as an Administrator.
And please read the word of wisdom in the following links.
http://serverfault.com/questions/13597/linux-knowledge-a-junior-cannot-miss
http://serverfault.com/questions/28915/advice-and-tips-for-a-junior-sysadmin-straight-out-of-college
http://serverfault.com/questions/9766/what-a-beginner-should-know-learn-for-sysadmin-job
http://serverfault.com/questions/5132/what-makes-a-good-or-great-administrator
http://www.karan.org/blog/index.php/2010/09/28/getting-started-with-linux-and-sysadmin
Try to visit best blogs on line for Administrators. My favorites are below.
http://www.thegeekstuff.com/
http://www.rackerhacker.com
http://www.cyberciti.biz/
http://www.karan.org
http://bitfieldconsulting.com/
Best Regards,
Amar
There is an abundance of content on the hard aspects of system administration but be mindful of the soft aspects as well. Consider the skill of persuasion a critical part of your toolset and actively develop it. The appropriate use of persuasion will influence the quality of your work environment just as much as properly managing the systems you are responsible for. Also, system administrators get to work on some very cool stuff. Have fun!
It looks like the contest is already over, but I figured it couldn't hurt to add my story for anyone that might read this.
Now, this depends on the route of system administration you go, but in my case I'm not always behind the scenes - I'm a technical manager at a web hosting company and handle a team of people, so aside from the administration aspects there's the customer part as well.
One thing I can't stress enough... never stop caring about the customer. If your responses to a customer become flat, you no longer offer suggestions on how they improve a situation, and you only respond with the bare minimum - then it might be time to pursue a different career path.
Here's an example:
"Hello,
This problem is now fixed. If we can of further assistance please let us know."
vs
"Hello Roger,
I've gone ahead and fixed the PATH in your .bash_profile. You were pointing to the wrong drectory for your bin - it was previously $HOME/bin, but it looks like you have everything in $HOME/local/bin. Once that was corrected and I sourced your .bash_profile everything was working correctly.
Should you have any further issues, questions or concerns please don't hesitate to let us know!"
Try to be personable - people don't want to feel like they're talking to a robot. It's also fine to explain things to them. If you're dealing with a user that isn't very technical - provide detailed info, explain what you're talking about, or provide links they can read in order to expand their knowledge. You should want to educate your customers and not just think "This guy's stupid" and not bother.
Aside from that, many of what I'd say on the technical side has already been said - but, it can't be said enough - never stop learning! Always find new topics of interest and work on your weak areas. There is no perfect admin - remember that!
Thanks for reading,
-RJC
Skype: ronald_craft
In my humble opinion you have to be passionate about what you are doing. Sys Admins many times work more hours than anyone. And if they are not passionate about what they do it will quickly burn you out. I really enjoy my job and putting in 30 minutes from home on a scheduled day off didn't phase me one bit. Mind you I was not out of town or anything like that, just a normal day off at home. On the flip side, when times are good it will appear to everyone else that you are not doing a thing. The reality is you have planned ahead, automated tasks and have an effective alerting system in place for potential issues.
Desire to learn new systems since the IT industry does not sit still. This is a profession that takes the desire to always learn. If you don't you get left behind and will find the systems you are responsible for outdated and falling behind.
Be respectful to your peers and those who work on the front lines. No one has a lock on all knowledge and listening to what others say can go a long way. This doesn't mean you have to agree all the time. After all argument is the basis for change. Treat arguments as opportunities to bond with your co-workers and the rewards will follow.
Making mistakes is part of learning. Don't be afraid to make a mistake and always own your mistakes. Never try to pass the buck. If you keep good notes and document what you are doing, it should not be too hard to recover from almost any mistake. If you have seasoned peers around you, all the better to learn from them when a mistake does happen. We have all been there or know someone who has.
In the past good documentation was hard to find. I'm dating myself a bit here but as the Internet was in its beginnings documentation was not so easily available. Now a good search from almost any search engine will reveal the information you need for almost every situation. Patience and luck are your friends when searching for solutions. If you don't find it the first go around, re-word the query, narrow it down to specifics and you will be rewarded.
The last thing I can add is to enjoy yourself. Life is too short and even being a Sys Admin is simply a job. If you ever find youself not enjoying the job it is time for self reflection. Your quality of life is more important and when qulaity of life is there, it shows in how well you perform at any job.
Happy birthday.. Keep up the good work...