Apache: No space left on device: Couldn't create accept lock

This error completely stumped me a couple of weeks ago. Apparently someone was adjusting the Apache configuration, then they checked their syntax and attempted to restart Apache. It went down without a problem, but it refused to start properly, and didn't bind to any ports.

Within the Apache error logs, this message appeared over and over:

[emerg] (28)No space left on device: Couldn't create accept lock

Apache is basically saying "I want to start, but I need to write some things down before I can start, and I have nowhere to write them!" If this happens to you, check these items in order:

1. Check your disk space
This comes first because it's the easiest to check, and sometimes the quickest to fix. If you're out of disk space, then you need to fix that problem. :-)

2. Review filesystem quotas
If your filesystem uses quotas, you might be reaching a quota limit rather than a disk space limit. Use repquota / to review your quotas on the root partition. If you're at the limit, raise your quota or clear up some disk space. Apache logs are usually the culprit in these situations.

3. Clear out your active semaphores
Semaphores? What the heck is a semaphore? Well, it's actually an apparatus for conveying information by means of visual signals. But, when it comes to programming, semaphores are used for communicating between the active processes of a certain application. In the case of Apache, they're used to communicate between the parent and child processes. If Apache can't write these things down, then it can't communicate properly with all of the processes it starts.

I'd assume if you're reading this article, Apache has stopped running. Run this command as root:

# ipcs -s

If you see a list of semaphores, Apache has not cleaned up after itself, and some semaphores are stuck. Clear them out with this command:

# for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done

Now, in almost all cases, Apache should start properly. If it doesn't, you may just be completely out of available semaphores. You may want to increase your available semaphores, and you'll need to tickle your kernel to do so. Add this to /etc/sysctl.conf:

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

And then run sysctl -p to pick up the new changes.

Further reading:
Wikipedia: Semaphore (Programming)
Apache accept lock fix

Digg This
Reddit This
Stumble Now!
Buzz This
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Kick It on DotNetKicks.com
Shout it
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Printed from: http://rackerhacker.com/2007/08/24/apache-no-space-left-on-device-couldnt-create-accept-lock/ .
© Major Hayden 2010.

6 Comments   »

  • Wim says:

    Okay, this worked for me. But the question for today is: why?
    What is the cause of this problem? Is it a bug in PHP or Apache?

  • Ese says:

    Hey,

    this was a lifesaver. thanks for putting it up!

    E.

  • jenny says:

    I use clear command "for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done" which can't clear semaphores

  • Well, you are a damn GENIUS!

    I don't know what the heck is a semaphore is either, but I'll tell you this much -- you're idea did the trick.

    This is all started as the result of modifying apache's and PHP's config (was installing PDO, PDO_MYSQL, and PDO_SQLITE), which of course botched everything.

    I went back and removed PDO, et al by using pecl and that did the trick, because before that apache was generating a massive number of errors, and "php -v" gave me a "segmentation fault" so I completely backed off that idea.

    There was only one thing I had to change, and I think it will answer Jenny's question above.

    I'm working in a debian distro, so I had to modify this line:

    for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done

    To this instead:

    for i in `ipcs -s | awk '/www-data/ {print $2}'`; do (ipcrm -s $i); done

    After that I got this (direct from error.log):

    [Thu Dec 31 05:28:09 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch16 configured -- resuming normal operations

    Of course that was music to my ears (er...eyes), and then I checked using "ps aux | grep apache2" and got this:

    www-data 28829 0.0 1.3 29200 8044 ? S 05:28 0:00 /usr/sbin/apache2 -k start
    www-data 28830 0.0 0.7 28808 4756 ? S 05:28 0:00 /usr/sbin/apache2 -k start
    www-data 28831 0.0 0.7 28808 4676 ? S 05:28 0:00 /usr/sbin/apache2 -k start
    www-data 28832 0.0 0.7 28808 4676 ? S 05:28 0:00 /usr/sbin/apache2 -k start
    www-data 28833 0.0 1.0 28904 6728 ? S 05:28 0:00 /usr/sbin/apache2 -k start
    www-data 28837 0.0 0.6 28672 4220 ? S 05:28 0:00 /usr/sbin/apache2 -k start
    www-data 28838 0.0 0.6 28672 4220 ? S 05:28 0:00 /usr/sbin/apache2 -k start
    www-data 28839 0.0 0.6 28672 4220 ? S 05:28 0:00 /usr/sbin/apache2 -k start
    www-data 28840 0.0 0.6 28672 4220 ? S 05:28 0:00 /usr/sbin/apache2 -k start
    www-data 28841 0.0 0.6 28672 4220 ? S 05:28 0:00 /usr/sbin/apache2 -k start
    root 28859 0.0 0.0 1644 508 pts/0 R+ 05:33 0:00 grep apache2

    Naturally, I was a happy camper, and so was my client for that matter -- thanks dude! =0)

  • Dws says:

    Thanks guy you are F*****´ genius

    I have to love people like you

    Thnks again

  • Mohan says:

    Thanks for the tip, it saved so much time,

    but I needed to change httpd to apache inorder for the for loop to work.

    for i in `ipcs -s | awk '/apache/ {print $2}'`; do (ipcrm -s $i);

    cheers!

Trackbacks/Pingbacks

  1. apache [emerg] (28)No space left on device: Couldn't create accept lock - Foros de CHW
  2. FractalizeR’s WebSite » [Solution] Apache: No space left on device: Couldn’t create accept lock

RSS feed for comments on this post , TrackBack URI

Leave a Reply