Racker Hacker

Let’s say you have a user who can’t receive e-mail. Each time they send a message to the server, this pops up in the mail logs:

postfix/smtpd[23897]: NOQUEUE: reject: RCPT from remotemailserver.com[10.0.0.2]: 554 <user@domain.com>: Relay access denied; from=<user@otherdomain.com> to=<user@domain.com> proto=ESMTP helo=<remotemailserver.com>

This is happening because Postfix is receiving e-mail for a domain for which it doesn’t expect to handle mail. Add the domains to the mydestination parameter in /etc/postfix/main.cf:

mydestination = domain.com, domain2.com, domain3.com

If you have a lot of domains to add, create a mydomains hash file and change the mydestination parameter:

mydestination = hash:/etc/postfix/mydomains

Create /etc/postfix/mydomains:

localhost OK
localmailserver.com OK
domain.com OK

Then run:

# postmap /etc/postfix/mydomains

This will create the hash file (mydomains.db) within /etc/postfix. If you’ve just added the directive to the main.cf, run postfix reload. However, if the directive was already there, but you just adjusted the mydomains and ran postmap, then there is nothing left to do.

Lots of PCI Compliance and vulnerability scan vendors will complain about TRACE and TRACK methods being enabled on your server. Since most providers run Nessus, you’ll see this fairly often. Here’s the rewrite rules to add:

RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

These directives will need to be added to each VirtualHost.

Further reading:
Apache Debugging Guide

If you find yourself in a pinch and you need a temporary fix when your primary IP is blacklisted, use the following iptables rule:

/sbin/iptables -t nat -A POSTROUTING -p tcp --dport 25 -j SNAT --to-source [desired outgoing ip]

Keep in mind, however, that you will need to adjust any applicable SPF records for your domains since your e-mail will appear to be leaving via one of the secondary IP’s on your server. Also, remember that this is only a temporary fix - you should find out why you were blacklisted and eliminate that problem as soon as possible.

One of these errors might appear on your website without warning:

Warning: DB function failed with error number 1033
Incorrect information in file: './database_name/table_name.frm' SQL=SELECT col1, col2 FROM table_name WHERE col3 = 'some_value' ORDER BY col1 ASC

MySQL is telling you that the table structure it has within data files doesn’t match the structure in the .frm file that’s on the disk. There’s only a few scenarios where this can happen:

Different version of the .frm files
If the .frm files from an older or later version of the table are placed in MySQL’s data directory, MySQL will become confused and it won’t be able to determine the proper database structure.

Pending table alteration
A pending database operation that ran an ALTER TABLE may not have written changes to the disk. MySQL may have stopped running abruptly or the entire server may have crashed. The normal operation for MySQL is to make changes in memory first and then perform disk operations.

Complete wierdness
I cannot explain it, and I can’t figure out the logic that would allow it to happen, but some web application vulnerabilities can cause this problem. I’ve seen it happen with Joomla! sites running on fairly secure servers, and there was no Apache privilege escalation used to modify the .frm files directly.

How is it fixed? The only way to repair it is to import the table again from a mysqldump backup, find the correct .frm file and restore it on the server, or run an ALTER TABLE to bring the table back to its original state.

This error completely stumped me a couple of weeks ago. Apparently someone was adjusting the Apache configuration, then they checked their syntax and attempted to restart Apache. It went down without a problem, but it refused to start properly, and didn’t bind to any ports.

Within the Apache error logs, this message appeared over and over:

[emerg] (28)No space left on device: Couldn’t create accept lock

Apache is basically saying “I want to start, but I need to write some things down before I can start, and I have nowhere to write them!” If this happens to you, check these items in order:

1. Check your disk space
This comes first because it’s the easiest to check, and sometimes the quickest to fix. If you’re out of disk space, then you need to fix that problem.

2. Review filesystem quotas
If your filesystem uses quotas, you might be reaching a quota limit rather than a disk space limit. Use repquota / to review your quotas on the root partition. If you’re at the limit, raise your quota or clear up some disk space. Apache logs are usually the culprit in these situations.

3. Clear out your active semaphores
Semaphores? What the heck is a semaphore? Well, it’s actually an apparatus for conveying information by means of visual signals. But, when it comes to programming, semaphores are used for communicating between the active processes of a certain application. In the case of Apache, they’re used to communicate between the parent and child processes. If Apache can’t write these things down, then it can’t communicate properly with all of the processes it starts.

I’d assume if you’re reading this article, Apache has stopped running. Run this command as root:

# ipcs -s

If you see a list of semaphores, Apache has not cleaned up after itself, and some semaphores are stuck. Clear them out with this command:

# for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done

Now, in almost all cases, Apache should start properly. If it doesn’t, you may just be completely out of available semaphores. You may want to increase your available semaphores, and you’ll need to tickle your kernel to do so. Add this to /etc/sysctl.conf:

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

And then run sysctl -p to pick up the new changes.

Further reading:
Wikipedia: Semaphore (Programming)
Apache accept lock fix

This error will pop up when binary logging is enabled, and someone thought it was a good idea to remove binary logs from the filesystem:

/usr/sbin/mysqld: File './mysql_bin.000025' not found (Errcode: 2)
[ERROR] Failed to open log (file ‘./9531_mysql_bin.000025′, errno 2)
[ERROR] Could not open log file
[ERROR] Can’t init tc log
[ERROR] Aborting

InnoDB: Starting shutdown...
InnoDB: Shutdown completed; log sequence number 0 2423986213
[Note] /usr/sbin/mysqld: Shutdown complete

Basically, MySQL is looking in the mysql-bin.index file and it cannot find the log files that are listed within the index. This will keep MySQL from starting, but the fix is quick and easy. You have two options:

Edit the index file
You can edit the mysql-bin.index file in a text editor of your choice and remove the references to any logs which don’t exist on the filesystem any longer. Once you’re done, save the index file and start MySQL.

Take away the index file
Move or delete the index file and start MySQL. This will cause MySQL to reset its binary log numbering scheme, so if this is important to you, you may want to choose the previous option.

So how do you prevent this from happening? Use the PURGE MASTER LOGS statement and allow MySQL to delete its logs on its own terms. If you’re concerned about log files piling up, adjust the expire_logs_days variable in your /etc/my.cnf.

Further reading:
12.6.1.1. PURGE MASTER LOGS Syntax
5.2.3 System Variables

When connecting to your server’s POP3 service, your client might provide this error just after authentication:

The connection to the server was interrupted.

Your best bet is to check the mail log and see exactly what the problem is:

web pop3-login: Login: john [192.168.0.5]
pop3(john): Invalid mbox file /var/spool/mail/john: No such file or directory
pop3(john): Failed to create storage with data: mbox:/var/spool/mail/john
dovecot: child 29864 (pop3) returned error 89

In this case, the mbox file has become corrupt (possible from malformed ‘From’ headers). You have the option of repairing the issues within the file, or you can simply create a new mail spool for the user.

It’s not abnormal for qmail act oddly at times with Plesk, and sometimes it can use 100% of the CPU. However, if you find qmail’s load to be higher than usual with a small volume of mail, there may be a fix that you need.

First off, check for two files in /var/qmail/control called dh512.pem and dh1024.pem. If they are present, well, then this article won’t be able to help you. You have a different issue that is causing increased CPU load (check for swap usage and upgrade your disk’s speed).

If the files aren’t there, do the following:

# cd /var/qmail/control
# cp dhparam512.pem dh512.pem
# cp dhparam1024.pem dh1024.pem
# /etc/init.d/qmail restart
# /etc/init.d/xinetd restart

At this point, your CPU load should be reduced once the currently running processes for qmail clear out.

So why is this fix required? Without dh512.pem and dh1024.pem, qmail has to create certificate and key pairs when other mail servers or mail users connect to qmail via TLS. If qmail is forced to generate them on the fly, you will get a big performance hit, and your load will be much higher than it could be. By copying the dhparam files over, you will pre-populate the SSL key and certificate for qmail to use, and all it has to do is pick it up off the file system rather than regenerating it each time.

Further reading:
SWsoft Forums: Qmail-smtpd spawning many processes, using full cpu

If you have to use short e-mail usernames in Plesk (which is a bad idea), and someone accidentally sets the server to use full usernames, you can force Plesk to go back. You can’t do this in the interface, however. Plesk realizes that duplicate mail names exist, and it wont allow the change.

Plesk will say something like:

Unable to allow the use of short mail names for POP3/IMAP accounts. There are mail names matching the encrypted passwords.

Forcing it back is easy with one SQL statement:

# mysql -u admin -p`cat /etc/psa/.psa.shadow` psa
mysql> UPDATE misc set val='enabled' where param='allow_short_pop3_names';

Keep in mind that users logging in with shortnames will get into the same mailbox if they have the same username and password.

Additional reading:
How can I change back the option “Use of short and full POP3/IMAP mail account names is allowed” forcedly?

While running into MySQL’s open files limit will manifest itself into various error messages, this is the standard one that you’ll receive during a mysqldump:

mysqldump: Got error: 29: File './databasename/tablename.MYD' not found (Errcode: 24) when using LOCK TABLES

The best way to get to the bottom of the error is to find out what it means:

$ perror 24
OS error code 24: Too many open files

There’s two ways to fix the problem. First, if you find that you only hit the limit during mysqldumps and never during normal database operation, just add --single-transaction to your mysqldump command line options. This will cause mysql to keep only one table open at a time.

However, if this happens while backups aren’t running, you may want to increase the open_files_limit in your MySQL configuration file. By default, the variable is set to 1,024 open files.

For further reading:
5.2.3. System Variables
7.13. mysqldump - A Database Backup Program