Racker Hacker

If you find that /dev/null is no longer a block device, and it causes issues during init on Red Hat boxes, you will need to follow these steps to return things to normal:

• Reboot the server
• When grub appears, edit your kernel line to include init=/bin/bash at the end
• Allow the server to boot into the emergency shell
• Run the following three commands

# rm -rf /dev/null
# mknod /dev/null c 1 3
# chmod 666 /dev/null

You should be back to normal. Make sure that the root users on your server don’t use cp or mv with /dev/null as this will cause some pretty ugly issues.

If you find that someone has done a recursive chmod or chown on a server, don’t fret. You can set almost everything back to its original permissions and ownership by doing the following:

rpm -qa | xargs rpm --setperms --setugids

Depending on how many packages are installed as well as the speed of your disk I/O, this may take a while to complete.

If up2date throws some horrible Python errors and rpm says “rpmdb: Lock table is out of available locker entries”, you can restore your system to normality with the following:

The errors:
rpmdb: Lock table is out of available locker entries
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm

Make a backup of /var/lib/rpm in case you break something:
tar cvzf rpmdb-backup.tar.gz /var/lib/rpm

Remove the Berkeley databases that rpm uses:
rm /var/lib/rpm/__db.00*

Make rpm rebuild the databases from scratch (may take a short while):
rpm --rebuilddb

Now, check rpm to make sure everything is okay:
rpm -qa | sort

Why does this happen?
When rpm accesses the Berkeley database files, it makes temporary locker entries within the tables while it searches for data. If you control-c your rpm processes often, this issue will occur much sooner because the locks are never cleared.

In the event that a Fedora/RHEL/CentOS box won’t perform the init (which comes right after the initial kernel load), don’t fret - it can be fixed. Make a note of the missing libraries or executables. Simply boot onto a livecd or rescuecd and chroot into your main installation. Once you’re chrooted, just forcefully install any RPM’s which might contain files that are missing when the init is loaded.

Forcing the installation of an RPM:

rpm -ivh –force filename.rpm

Listing the files that an installed RPM contains:

rpm -ql rpmname

Listing the files that an RPM file contains:

rpm -qpl filename.rpm

Finding the RPM that contains a certain file/executable:

rpm -qf /usr/bin/filename

Figuring out what might be wrong with files already installed from an RPM:

rpm -V rpmname