Last week, I found myself with a server under low load, but it couldn't make or receive network connections. When I ran dmesg, I found the following line repeating over and over:
ip_conntrack: table full, dropping packet
I'd seen this message before, but I headed over to Red Hat's site for more details. It turns out that the server was running iptables, but it was under a very heavy load and also handling a high volume of network connections. Generally, the ip_conntrack_max is set to the total MB of RAM installed multiplied by 16. However, this server had 4GB of RAM, but ip_conntrack_max was set to 65536:
# cat /proc/sys/net/ipv4/ip_conntrack_max 65536
I logged into another server with 1GB of RAM (RHES 5, 32-bit) and another with 2GB of RAM (RHES 4, 64-bit), and both had ip_conntrack_max set to 65536. I'm not sure if this is a known Red Hat issue, or if it's just set to a standard value out of the box.
If you want to check your server's current tracked connections, just run the following:
# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
If you want to adjust it (as I did), just run the following as root:
# echo 131072 > /proc/sys/net/ipv4/ip_conntrack_max
This solves the problem at this moment, but after a reboot the initial value will be restored.
To make this persistent you have to add a line like 'net.ipv4.ip_conntrack_max=131072' to /etc/sysctl.conf
Some distros apparently don't have that file; there you should use the echo-line in a boot script
This just saved my (*#&$(*#. Thank you.
wtf, your article is great, saved me from ultimate &**&(**)
huge thanks.
Thanks a lot..
Been puzzled by this
Thanks this is great article... veary helpfull
Some readers may be interested to know what ip_conntrack is in the first place, and why it fills up. If you run an iptables firewall, and have rules that act upon the state of a packet, then the kernel uses ip_conntrack to keep track of what state what connections are in so that the firewall rule logic can be applied against them. If you have a system that's getting a lot of network activity (high rates of connections, lots of concurrent connections, etc) then the table will accumulate entries.
The entries remain until an RST packet is sent from the original IP address. If you have a flaky network somewhere between you, and the clients accessing your server, it can cause the RST packets to be dropped due to the packet loss, and leave orphaned entries in your ip_conntrack table. This can also happen if you have a malfunctioning switch or NIC card... not necessarily a routing problem out on the internet somewhere.
Typically when I've seen this trouble crop up is when a server is the target of a DDoS attack. Filling up the ip_conntrack table is a relatively easy way to knock a server off line, and attackers know this.
As Major suggested, you can get short term relief by increasing the size of the table. However, these entries are held in memory by the kernel. The bigger you make the table, the more memory it will consume. That memory could be used by your server to serve requests if you really don't need the stateful firewall capability. Don't waste resources on this feature if you really don't need it.
Another option to consider is turning OFF iptables rules that use ip_conntrack so the state able is not used at all. Anything with "-m state" or "-t nat" can be turned off. If you want to just flush all your iptables rules you can do an "iptables -P" to set a default allow policy and "iptables -F" to flush all the rules. On an RHEL or CentOS system you can just do "service iptables stop".
Once iptables is no longer using ip_conntrack, you can reclaim the memory the table was using by unloading the related kernel modules.
rmmod ipt_MASQUERADE
rmmod iptable_nat
rmmod ipt_state
rmmod ip_conntrack
Then you will have an empty ip_conntrack that will stay empty. I mention this because a lot of sysadmins have hordes of iptables rules installed as a matter of course, and don't recognize the downside of having them present. You can still use iptables, but to avoid the use of ip_conntrack simply don't use rules based on stateful logic.
One other aspect to consider when raising your max conntrack setting is the depth of the memory objects used to track these connections, henceforth referred to as "buckets".
On RedHat the default hashsize for the conntrack module is 8192. The rule of thumb is to allow for no more than 8 connections per bucket so you would set your conntrack size to be equal to 8 * hashsize. This is why RedHat defaults the ip_conntrack_max to 65536.
You can tweak these settings by adjusting not just the ip_conntrack_max setting but the hashsize option to the ip_conntrack module.
So, for example, if you were to set your ip_conntrack_max to 131072 without modifying the default hashsize of 8k, you are allowing for a bucket depth of 16 entries. Thus the kernel has to dig deeper, potentially, to find that one connection object in it's bucket.
There are a number of schools of thought on how best to address this but in practice I have found that, given the resources, a shallower bucket is better.
For a server that does extremely heavy network traffic, and of course has the memory to spare, you would want to keep the average bucket depth to 2 or 4.
Hashsize, to my knowledge, isn't a dynamic setting so you will need to load the ip_conntrack module with the option:
hashsize =
So in Major's example above, if you want to double your server's capacity for tracked connections while not doubling the lookups you would reload the module with:
options ip_conntrack hashsize=16384
This keeps the items per bucket to 8. I have seen machines with a depth of beyond 8 get completely cowed under heavy network load and since memory is relatively plentiful nowadays you can increase the efficiency of the lookups by making this 4 connections per bucket or even 2 by just doing simple math and reloading the module with the right options.
Hope that helps.
Here is some relatively dated yet still applicable information on the subject:
http://www.wallfire.org/misc/netfilter_conntrack_perf.txt
darien and adrian, thank you kindly for sharing the knowhow.
i recently found a bunch of
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
in my dmesg, this article, plus your advice, will hopefully take care of this
thanks!
oh worth mentioning, if you wish to change the hashsize... it can be done without reloading the module / using options (according to http://www.wallfire.org/misc/netfilter_conntrack_perf.txt):
Since 2.6.14, it is possible to set hashsize dynamically at runtime,
after boot and module load.
Between 2.6.14 and 2.6.19 (included), use:
# echo $HASHSIZE > /sys/module/ip_conntrack/parameters/hashsize
Since 2.6.20, use:
# echo $HASHSIZE > /sys/module/nf_conntrack/parameters/hashsize
Nice article
Thanks for the post. Really helped. I was down to restarting server few times a day before reading this.
Briprad - I'm glad it was able to help!