For my environments, I follow a strategy like this: I have some backups immediately accessible, others that are accessible very quickly (but not instantly), and others that are offsite and may take a bit more time to access.

Immediately accessible backups
One of the easiest way to have an immediately accessible backup is to have multiple machines online running the same versions of code or databases in a high availability group. If you have a node which fails, the remaining nodes should be able to handle the requests immediately. You may not consider this to be a backup under the traditional definition of what a backup should be, but it's functionally similar.

Backups that are accessible quickly
This second level of backups should be stored very close to your environment or within the environment itself. If you have multiple database and web server nodes, you could consider storing your web backups on the database servers and vice versa. For those who run very sensitive applications, this may violate the provisions of different certifications and regulations. A server dedicated to holding backups may be a viable alternative for additional security.

Offsite backup storage
These are the backups that need to be geographically distant from your main environment. Also, you should always consider storing these backups on more than one medium with more than one company.

For example, if your hosting providers offers a storage service, it's fine to store one set of your backups there, but consider storing them with a competitor as well. If you store your backups with your hosting provider in multiple places, you could be caught be a provider issue and lose access to your backups entirely. Hosting with multiple providers will allow you to access at least one copy of your backups even if there are billing or technical issues with a particular provider.

Another thing to keep in mind with offsite backup storage is how long it will take to transfer the backups to your hosting environment in case of an emergency. If your hosting environment is in Texas, but your backups are stored in Australia, you're going to have a longer wait when you transfer your data back.

A specific example
My environments are all in Dallas, Texas and I have a highly available environment with multiple instances. My second layer of backups are stored within the environment as well as in Rackspace's Cloud Files in Dallas. My third layer of backups are stored with Amazon S3 via Jungle Disk and at my home on a RAID array.

While I hope you never need to access your backups under duress, these tips should help to reduce your stress if you need to restore data in a hurry.

Strategies for storing backups is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Let's say you cloned data from a drive using something like this:

# dd if=/dev/sda of=/mnt/nfs/backup/harddrive.img

Once that's finished, you should end up with your partition table as well as the grub data from the MBR in your image file. If you run file against the image file you made, you should see something like this:

# file harddrive.img
harddrive.img: x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, stage2 
address 0x2000, stage2 segment 0x200, GRUB version 0.97; partition 1: ID=0x83, 
active, starthead 1, startsector 63, 33640047 sectors, code offset 0x48

What if you want to pull some files from this image without writing it out to another disk? Mounting it like a loop file isn't going to work:

# mount harddrive /mnt/temp
mount: you must specify the filesystem type

The key is to mount the file with an offset specified. In the output from file, there is a particular portion of the output that will help you:

... startsector 63 ...

This means that the filesystem itself starts on sector 63. You can also view this with fdisk -l:

# fdisk -l harddrive.img
                    Device Boot      Start         End      Blocks   Id  System
harddrive.img                *          63    33640109    16820023+  83  Linux

Since we need to scoot 63 sectors ahead, and each sector is 512 bytes long, we need to use an offset of 32,256 bytes. Fire up the mount command and you'll be on your way:

# mount -o ro,loop,offset=32256 harddrive.img /mnt/loop
# mount | grep harddrive.img
/root/harddrive.img on /mnt/loop type ext3 (ro,loop=/dev/loop1,offset=32256)

If you made this image under duress (due to a failing drive or other emergency), you might have to check and repair the filesystem first. Doing that is easy if you make a loop device:

# losetup --offset 32256 /dev/loop2 harddrive.img
# fsck /dev/loop2

Once that's complete, you can save some time and mount the loop device directly:

# mount /dev/loop2 /mnt/loop

Mounting a raw partition file made with dd or dd_rescue in Linux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

pv allows a user to see the progress of data through a pipeline, by giving information such as time elapsed, percentage completed (with progress bar), current throughput rate, total data transferred, and ETA.

The usage certainly isn't complicated:

To use it, insert it in a pipeline between two processes, with the appropriate options. Its standard input will be passed through to its standard output and progress will be shown on standard error.

A great application of pv is when you're restoring large amounts of data into MySQL, especially if you're restoring data under duress due to an accidentally-dropped table or database. (Who hasn't been there before?) The standard way of restoring data is something we're all familiar with:

# mysql my_database < database_backup.sql

The downside of this method is that you have no idea how quickly your restore is working or when it might be done. You could always open another terminal to monitor the tables and databases as they're created, but that can be hard to follow.

Toss in pv and that problem is solved:

# pv database_backup.sql | mysql my_database
96.8MB 0:00:17 [5.51MB/s] [==>                                ] 11% ETA 0:02:10

When it comes to MySQL, your restore rate is going to be different based on some different factors, so the ETA might not be entirely accurate.

Monitor MySQL restore progress with pv is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

[root@f11 ~]# /etc/init.d/httpd start
Starting httpd: [Fri Aug 14 17:05:24 2009] [crit] (22)Invalid argument: alloc_listener: failed to get a socket for (null)
Syntax error on line 134 of /etc/httpd/conf/httpd.conf:
Listen setup failed
                                                           [FAILED]

The issue is related to three kernel calls that are used in apr-1.3.8-1: accept4(), dup3() and epoll_create1(). Without these calls, apache is unable to start.

Update on August 17, 2009: the Fedora team has pushed apr-1.3.8-2 into the stable repositories for Fedora 11, which eliminates the need for the temporary fix shown below.

Deprecated solution: There is a bug open with the Fedora team, and there is a temporary fix available:

yum --enablerepo=updates-testing update apr

Fedora 11 httpd: alloc_listener: failed to get a socket for (null) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

kernel: sdb : READ CAPACITY failed.
kernel: sdb : status=0, message=00, host=4, driver=00 
kernel: sdb : sense not available. 
kernel: sdb: Write Protect is off
kernel: sdb: Mode Sense: 00 00 00 00
kernel: sdb: asking for cache data failed
kernel: sdb: assuming drive cache: write through
kernel:  sdb:<6>sd 1:0:0:0: SCSI error: return code = 0x00040000
kernel: end_request: I/O error, dev sdb, sector 0
kernel: Buffer I/O error on device sdb, logical block 0
kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
kernel: end_request: I/O error, dev sdb, sector 0
kernel: Buffer I/O error on device sdb, logical block 0
kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
kernel: end_request: I/O error, dev sdb, sector 0

The errors show that the SCSI bus is having issues bringing the new drive online, and it won't be seen by the OS until the SCSI controller is pleased. You can force the controller to re-scan the drives attached to it, and this should correct the problem:

cd /sys/class/scsi_host/hostX
echo "- - - " > scan

Replace the X with the proper controller number of your SCSI controller. If you're not sure which controller is which, try running:

# cat /sys/class/scsi_host/host0/proc_name
sata_nv

Credit for this find goes to Tony Dolan

Re-scan the SCSI bus in Linux after hot-swapping a drive is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

As we know, sometimes servers misbehave due to things put them through, and you can quickly end up in a situation where things are going badly. I'm talking about the type of situation where you're connected via SSH to a server that controls phone lines for five million people and it sits in a tiny building 400 miles away from the nearest human being. We're talking bad. If you issue a plain reboot command, it might not even make it that far. Once SSH stops running, you're going to be out of luck.

If you find yourself in this situation (and I hope you won't!), you have some options to get your way with a misbehaving server remotely. You can force an immediate reboot with the following:

echo 1 > /proc/sys/kernel/sysrq 
echo b > /proc/sysrq-trigger

WHOA THERE! This is pretty much the same as pressing the reset button on the server (if equipped). No daemons will be shut down gracefully, no filesystem sync will occur, and you may get the wrath of a fsck (or worse, a non-booting server) upon reboot. To do things a little more carefully, read on.

These are called magic commands, and they're pretty much synonymous with holding down Alt-SysRq and another key on older keyboards. Dropping 1 into /proc/sys/kernel/sysrq tells the kernel that you want to enable SysRq access (it's usually disabled). The second command is equivalent to pressing Alt-SysRq-b on a QWERTY keyboard.

There's a better way of rebooting a misbehaving server that Wikipedia shows with the mnemonic "Reboot Even If System Utterly Broken":

unRaw      (take control of keyboard back from X),  
 tErminate (send SIGTERM to all processes),
 kIll      (send SIGKILL to all processes), 
  Sync     (flush data to disk),
  Unmount  (remount all filesystems read-only),
reBoot.

I can't vouch for this actually working, but I'm interested to try it. UPDATE: I've been told that doing this series of commands with ReiserFS is a very bad idea.

If you want to shut the machine down entirely (please think about it before using this on a remote system):

echo 1 > /proc/sys/kernel/sysrq 
echo o > /proc/sysrq-trigger

If you want to keep SysRq enabled all the time, you can do that with an entry in your server's sysctl.conf:

kernel.sysrq = 1

Linux: emergency reboot or shutdown with magic commands is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

There are a few processes on the server that keep it fairly busy. Two of the processes are vital to the server's operation - if they are stopped, lots of work is required to get them running properly again. I found that a certain java process was being killed by OOM killer regularly, and another perl process was being killed occasionally.

Naturally, my disdain for java made me think that the java process was the source of the issue. The process was configured to use a small amount of RAM, so it was ruled out. The other perl process used even less memory, so it was ruled out as well. When I checked the sysstat data with sar, I found that the server was only using about 2-3GB out of 4GB of physical memory at the time when OOM killer was started. At this point, I was utterly perplexed.

I polled some folks around the office and gathered some ideas. After putting some ideas together, I found that the server was actually caching too much data in the ext3_inode_cache and dentry_cache. These caches hold recently accessed files and directories on the server, and they're purged as the files and directories become stale. Since the operations on the server read and write large amounts of data locally and via NFS, I knew these caches had to be gigantic. If you want to check your own caches, you can use the slabtop command. For those who like things more difficult, you can also cat the contents of /proc/slabinfo and grep for the caches that are important to you.

An immense amount of Googling revealed very little, but I discovered a dirty hack to fix the issue (don't run this yet):

echo 1 > /proc/sys/vm/drop_caches  # free pagecache
     [OR]
echo 2 > /proc/sys/vm/drop_caches  # free dentries and inodes
     [OR]
echo 3 > /proc/sys/vm/drop_caches  # free pagecache, dentries and inodes
sync  # forces the dump to be destructive

There are huge consequences to dumping these caches and running sync. If you are writing data at the time you run these commands, you'll actually be dumping the data out of the filesystem cache before it reaches the disk, which could lead to very bad things.

While discussing the issue with a coworker, he found a different method for correcting the issue that was much safer. You can echo values into /proc/sys/vm/vfs_cache_pressure to tell the kernel what priority it should take when clearing out the inode/dentry caches. LinuxInsight explains the range of values well:

At the default value of vfs_cache_pressure = 100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

In short, values less than 100 won't reduce the caches very much as all. Values over 100 will signal to the kernel that you want to clear out the caches at a higher priority. I found that no matter what value you use, the kernel clears the caches at a slow rate. I've been using a value of 10000 on the server I talked about earlier in the article, and it has kept the caches down to a reasonable level.

Reducing inode and dentry caches to keep OOM killer at bay is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

EXT3-fs warning (device sda5): ext3_dx_add_entry: Directory index full!

After a thorough amount of searching, I couldn't find out what the error actually meant. As with most errors starting with EXT3-fs warning, I figured that a fsck might be the best option.

During the fsck, several inodes were repaired and the check completed after 10-15 minutes. I jotted down some notes about the directories that popped up on the screen during the fsck. The server rebooted it came up without any problems.

I reviewed the directories that appeared during the fsck and they were full of files. Some of the directories contained upwards of 200,000 files. Many of the files were moved into lost+found after the fsck, so they had to be restored from their backups. I still don't know what caused the original issue as the hardware checked out fine. If you run into this error, a fsck should help, but make sure that you have backups handy.

ext3_dx_add_entry: Directory index full! is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

EXT3-fs error (device hda3) in start_transaction: Journal has aborted

Basically, the system is telling you that it's detected a filesystem/journal mismatch, and it can't utilize the journal any longer. When this situation pops up, the filesystem gets mounted read-only almost immediately. To fix the situation, you can remount the partition as ext2 (if it isn't your active root partition), or you can commence the repair operations.

If you're working with an active root partition, you will need to boot into some rescue media and perform these operations there. If this error occurs with an additional partition besides the root partition, simply unmount the broken filesystem and proceed with these operations.

Remove the journal from the filesystem (effectively turning it into ext2):

# tune2fs -O ^has_journal /dev/hda3

Now, you will need to fsck it to correct any possible problems (throw in a -y flag to say yes to all repairs, -C for a progress bar):

# e2fsck /dev/hda3

Once that's finished, make a new journal which effectively makes the partition an ext3 filesystem again:

# tune2fs -j /dev/hda3

You should be able to mount the partition as an ext3 partition at this time:

# mount -t ext3 /dev/hda3 /mnt/fixed

Be sure to check your dmesg output for any additional errors after you're finished!

EXT3-fs error (device hda3) in start_transaction: Journal has aborted is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

[root@server ~]# service psa start
Key file: /opt/drweb/drweb32.key - Key file not found!
A path to a valid license key file does not specified.
Plesk authorization failed: HTTP request error [7]
Error: Plesk Software not running.
                                                           [FAILED]

(Although I included the text from the drweb failure, I later found that it was not related to the issue. However, since it might appear in your logs prior to the HTTP request error, I included it anyways.)

This was a perfectly working server that had no other issues besides this peculiar Plesk issue. Another technician had upgraded the license a few weeks prior, and it was verified at the the time to be working properly. After a bit of Google searching, I found that the solution was to completely stop Plesk and its related services and then start it all up again.

[root@server ~]# service psa stopall
/usr/local/psa/admin/bin/httpsdctl stop: httpd stopped
Stopping Plesk:                                            [  OK  ]
Stopping named:                                            [  OK  ]
service psa startStopping MySQL:                           [  OK  ]
Stopping : Stopping Courier-IMAP server:
   Stopping imap                                           [  OK  ]
   Stopping imap-ssl                                       [  OK  ]
   Stopping pop3                                           [  OK  ]
   Stopping pop3-ssl                                       [  OK  ]

Stopping postgresql service:                               [  OK  ]
Shutting down psa-spamassassin service:                    [  OK  ]
Stopping httpd:                                            [  OK  ]

[root@server ~]# service psa start
Starting named:                                            [  OK  ]
Starting MySQL:                                            [  OK  ]
Starting qmail:                                            [  OK  ]
Starting Courier-IMAP server:
   Starting imapd                                          [  OK  ]
   Starting imap-ssl                                       [  OK  ]
   Starting pop3                                           [  OK  ]
   Starting pop3-ssl                                       [  OK  ]

Starting postgresql service:                               [  OK  ]
Starting psa-spamassassin service:                         [  OK  ]
Processing config directory: /usr/local/psa/admin/conf/httpsd.*.include
/usr/local/psa/admin/bin/httpsdctl start: httpd started
Starting Plesk:                                            [  OK  ]
Starting up drwebd:                                        [  OK  ]

I couldn't nail down anything within the Plesk log files that would explain the cause of the problem, but this solution corrected the issue instantly.

This issue occurred with Plesk 8.1.1 on Red Hat Enterprise Linux 4 Update 5

Plesk authorization failed: HTTP request error [7] is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Rackspace Information Center
TechCrunch - Quick, Plug The Internet Back In: Major Rackspace Outage
Laughing Squid - Massive Power Outage At Rackspace’s Dallas Data Center
37Signals - Downtime Explanation
ValleyWag - Truck driver in Texas kills all the websites you really use

I don't know of any additional information besides what is contained within these articles and blog posts. However, I can tell you that I've never worked for a company before that pulled together in such large numbers to get on the phones and respond to tickets long after shifts should have been over. Obviously, it was a horrible "perfect storm" type of situation, and no one would wish for it to happen to anyone.

Over the last five years, I've had dedicated servers and VPS accounts with seven companies. Out of those seven, five have had major outages. After those outages, I can honestly say I received a timely and courteous response from one of the companies in that list. In one situation with a certain Texas hosting company, I had no network connectivity for almost 72 hours with no response to phone calls or trouble tickets.

After it's all said and done (and it's not done yet), I find myself to be very proud of the company for which I work. Server parts will eventually fail, as will networks, generators and power grids - it's inevitable. The important part is that the will of those who are providing the support never fails.

Our will is strong, and it continues to stay that way.

Rackspace Outage is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Enforcing mode requested but no policy loaded. Halting now.
Kernel panic - not syncing: Attempted to kill init!

This usually means that you've set SELINUX in enforcing mode within /etc/sysconfig/selinux or /etc/selinux/selinux.conf but you don't have the appropriate SELINUX packages installed. To fix the issue, boot the server into the Red Hat rescue environment and disable SELINUX until you can install the proper packages that contain the SELINUX targeted configuration.

This kernel panic appeared on a Red Hat Enterprise Linux 4 Update 5 server.

Enforcing mode requested but no policy loaded. Halting now. is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Warning: DB function failed with error number 1033
Incorrect information in file: './database_name/table_name.frm' SQL=SELECT col1, col2 FROM table_name WHERE col3 = 'some_value' ORDER BY col1 ASC

MySQL is telling you that the table structure it has within data files doesn't match the structure in the .frm file that's on the disk. There's only a few scenarios where this can happen:

Different version of the .frm files
If the .frm files from an older or later version of the table are placed in MySQL's data directory, MySQL will become confused and it won't be able to determine the proper database structure.

Pending table alteration
A pending database operation that ran an ALTER TABLE may not have written changes to the disk. MySQL may have stopped running abruptly or the entire server may have crashed. The normal operation for MySQL is to make changes in memory first and then perform disk operations.

Complete wierdness
I cannot explain it, and I can't figure out the logic that would allow it to happen, but some web application vulnerabilities can cause this problem. I've seen it happen with Joomla! sites running on fairly secure servers, and there was no Apache privilege escalation used to modify the .frm files directly.

How is it fixed? The only way to repair it is to import the table again from a mysqldump backup, find the correct .frm file and restore it on the server, or run an ALTER TABLE to bring the table back to its original state.

DB function failed with error number 1033 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Within the Apache error logs, this message appeared over and over:

[emerg] (28)No space left on device: Couldn't create accept lock

Apache is basically saying "I want to start, but I need to write some things down before I can start, and I have nowhere to write them!" If this happens to you, check these items in order:

1. Check your disk space
This comes first because it's the easiest to check, and sometimes the quickest to fix. If you're out of disk space, then you need to fix that problem.

2. Review filesystem quotas
If your filesystem uses quotas, you might be reaching a quota limit rather than a disk space limit. Use repquota / to review your quotas on the root partition. If you're at the limit, raise your quota or clear up some disk space. Apache logs are usually the culprit in these situations.

3. Clear out your active semaphores
Semaphores? What the heck is a semaphore? Well, it's actually an apparatus for conveying information by means of visual signals. But, when it comes to programming, semaphores are used for communicating between the active processes of a certain application. In the case of Apache, they're used to communicate between the parent and child processes. If Apache can't write these things down, then it can't communicate properly with all of the processes it starts.

I'd assume if you're reading this article, Apache has stopped running. Run this command as root:

# ipcs -s

If you see a list of semaphores, Apache has not cleaned up after itself, and some semaphores are stuck. Clear them out with this command:

# for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done

Now, in almost all cases, Apache should start properly. If it doesn't, you may just be completely out of available semaphores. You may want to increase your available semaphores, and you'll need to tickle your kernel to do so. Add this to /etc/sysctl.conf:

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

And then run sysctl -p to pick up the new changes.

Further reading:
Wikipedia: Semaphore (Programming)
Apache accept lock fix

Apache: No space left on device: Couldn't create accept lock is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

/usr/sbin/mysqld: File './mysql_bin.000025' not found (Errcode: 2)
[ERROR] Failed to open log (file './9531_mysql_bin.000025', errno 2)
[ERROR] Could not open log file
[ERROR] Can't init tc log
[ERROR] Aborting

InnoDB: Starting shutdown...
InnoDB: Shutdown completed; log sequence number 0 2423986213
[Note] /usr/sbin/mysqld: Shutdown complete

Basically, MySQL is looking in the mysql-bin.index file and it cannot find the log files that are listed within the index. This will keep MySQL from starting, but the fix is quick and easy. You have two options:

Edit the index file
You can edit the mysql-bin.index file in a text editor of your choice and remove the references to any logs which don't exist on the filesystem any longer. Once you're done, save the index file and start MySQL.

Take away the index file
Move or delete the index file and start MySQL. This will cause MySQL to reset its binary log numbering scheme, so if this is important to you, you may want to choose the previous option.

So how do you prevent this from happening? Use the PURGE MASTER LOGS statement and allow MySQL to delete its logs on its own terms. If you're concerned about log files piling up, adjust the expire_logs_days variable in your /etc/my.cnf.

Further reading:
12.6.1.1. PURGE MASTER LOGS Syntax
5.2.3 System Variables

MySQL couldn't find log file is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First off, check for two files in /var/qmail/control called dh512.pem and dh1024.pem. If they are present, well, then this article won't be able to help you. You have a different issue that is causing increased CPU load (check for swap usage and upgrade your disk's speed).

If the files aren't there, do the following:

# cd /var/qmail/control
# cp dhparam512.pem dh512.pem
# cp dhparam1024.pem dh1024.pem
# /etc/init.d/qmail restart
# /etc/init.d/xinetd restart

At this point, your CPU load should be reduced once the currently running processes for qmail clear out.

So why is this fix required? Without dh512.pem and dh1024.pem, qmail has to create certificate and key pairs when other mail servers or mail users connect to qmail via TLS. If qmail is forced to generate them on the fly, you will get a big performance hit, and your load will be much higher than it could be. By copying the dhparam files over, you will pre-populate the SSL key and certificate for qmail to use, and all it has to do is pick it up off the file system rather than regenerating it each time.

Further reading:
SWsoft Forums: Qmail-smtpd spawning many processes, using full cpu

Qmail-smtpd spawns many processes and uses 100% of CPU is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Operating System:
Red Hat Enterprise Linux 4 Update 5

What happened:
The server was abruptly powered down, disassembled, and re-assembled.

Work done:
Ran a fsck from the rescue environment and it eventually completed (after much corruption was found), but the server would not boot properly.

End result:
The damage to the filesystem could not be adequately repaired as the errors were very extensive. The RAID array had to be rebuilt and a new OS was installed.

EXT3-fs error: ext3_check_descriptors / group descriptors corrupted is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Just run this one SQL statement on the table:

ALTER TABLE brokentablename AUTO_INCREMENT=9501;

If you run a quick insert and then run SELECT last_insert_id(), the correct key number should be returned (9501 in this case).

Repair auto_increment in MySQL is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

• Reboot the server
• When grub appears, edit your kernel line to include init=/bin/bash at the end
• Allow the server to boot into the emergency shell
• Run the following three commands

# rm -rf /dev/null
# mknod /dev/null c 1 3
# chmod 666 /dev/null

You should be back to normal. Make sure that the root users on your server don't use cp or mv with /dev/null as this will cause some pretty ugly issues.

Corrupt /dev/null is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

rpm -qa | xargs rpm --setperms --setugids

Depending on how many packages are installed as well as the speed of your disk I/O, this may take a while to complete.

Rebuild RPM file permissions and ownerships is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The errors:

rpmdb: Lock table is out of available locker entries
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm

Make a backup of /var/lib/rpm in case you break something:

tar cvzf rpmdb-backup.tar.gz /var/lib/rpm

Remove the Berkeley databases that rpm uses:

rm /var/lib/rpm/__db.00*

Make rpm rebuild the databases from scratch (may take a short while):

rpm --rebuilddb

Now, check rpm to make sure everything is okay:

rpm -qa | sort

Why does this happen?
When rpm accesses the Berkeley database files, it makes temporary locker entries within the tables while it searches for data. If you control-c your rpm processes often, this issue will occur much sooner because the locks are never cleared.

rpmdb: Lock table is out of available locker entries is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Forcing the installation of an RPM:

rpm -ivh --force filename.rpm

Listing the files that an installed RPM contains:

rpm -ql rpmname

Listing the files that an RPM file contains:

rpm -qpl filename.rpm

Finding the RPM that contains a certain file/executable:

rpm -qf /usr/bin/filename

Figuring out what might be wrong with files already installed from an RPM:

rpm -V rpmname

Fedora/RHEL/Centos Won't Init is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.