Most linux distributions use some type of mechanism to gracefully stop daemons and unmount storage volumes during a reboot or shutdown. It's most commonly done via scripts that will wait for each daemon to shut down gracefully before proceeding to the next daemon.
As we know, sometimes servers misbehave due to things put them through, and you can quickly end up in a situation where things are going badly. I'm talking about the type of situation where you're connected via SSH to a server that controls phone lines for five million people and it sits in a tiny building 400 miles away from the nearest human being. We're talking bad. If you issue a plain reboot command, it might not even make it that far. Once SSH stops running, you're going to be out of luck.
If you find yourself in this situation (and I hope you won't!), you have some options to get your way with a misbehaving server remotely. You can force an immediate reboot with the following:
echo 1 > /proc/sys/kernel/sysrq echo b > /proc/sysrq-trigger
WHOA THERE! This is pretty much the same as pressing the reset button on the server (if equipped). No daemons will be shut down gracefully, no filesystem sync will occur, and you may get the wrath of a fsck (or worse, a non-booting server) upon reboot. To do things a little more carefully, read on.
These are called magic commands, and they're pretty much synonymous with holding down Alt-SysRq and another key on older keyboards. Dropping 1 into /proc/sys/kernel/sysrq tells the kernel that you want to enable SysRq access (it's usually disabled). The second command is equivalent to pressing Alt-SysRq-b on a QWERTY keyboard.
There's a better way of rebooting a misbehaving server that Wikipedia shows with the mnemonic "Reboot Even If System Utterly Broken":
unRaw (take control of keyboard back from X), tErminate (send SIGTERM to all processes), kIll (send SIGKILL to all processes), Sync (flush data to disk), Unmount (remount all filesystems read-only), reBoot.
I can't vouch for this actually working, but I'm interested to try it. UPDATE: I've been told that doing this series of commands with ReiserFS is a very bad idea.
If you want to shut the machine down entirely (please think about it before using this on a remote system):
echo 1 > /proc/sys/kernel/sysrq echo o > /proc/sysrq-trigger
If you want to keep SysRq enabled all the time, you can do that with an entry in your server's sysctl.conf:
kernel.sysrq = 1
Good post!
"I can vouch for this actually working, but I’m interested to try it." -- this sentence doesn't make much sense.
With modern linuxes, I've never had an issue using:
sync
sync
reboot -t 0
Over SSH.
@Kyle: On machines with defunct or dead processes (say a hung NFS mount or disk full problems) "reboot" will not always work. sysrq is a last resort but a welcome one for those rare cases.
One of the useful things to do with this tip, is to use it to test your HA setup (such as with MySQL & DRBD / Heartbeat) worked as expected.
It's always interesting to know that you got the data back that you were supposed to - and you managed to solve the durability problem that is the 'D' in ACID.
Note: you don't need to turn system request on if you're echoing into sysrq-trigger, it will always work as long as you are root. You only need to turn it on for key combos.
Thank you! This worked when shutdown -r now, reboot, and init 6 wouldn't. One of the disks failed and the system was locked up forever trying to unmount it. Fortunately it was just a backup disk and no data was lost on the others.
"With modern linuxes, I've never had an issue using:
sync
sync
reboot -t 0
Over SSH."
I just did! No 5 million user phone switch here, but a remote box at my parents house, I just logged in and it was good and screwed (it's a bad sign when you see "/bin/ls: I/O error". Reboot command got me "reboot: command not found". It seems /dev/sda dropped right off the SATA bus. Of course, I *SERIOUSLY* doubt it's going to sucessfully reboot (probably, the disk has either crashed, or will need an actual power cycle to show back up) but at least I could attempt a reboot with these instructions, the machine was doing nothing as it stood.