For example, adding and then removing from the nat table is enough to cause the DLKM to be loaded up.

I expand more on this here - the short answer is that you'll need to rmmod conntrack and friends out of your kernel to completely prevent the kernel from attempting to track connections.

And this my configuration tunning server to could opposed (/etc/sysctl.conf)
By way increase connection, decrease time, ... your server can good load/response. Remember, read document for all config below to understand them

net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.ip_conntrack_max = 524288

net.core.somaxconn = 4096

net.ipv4.netfilter.ip_conntrack_max = 524288
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 5
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 108000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 15
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 3

net.ipv4.ip_local_port_range = 10000 65536

It seems as if it's a common consensus that once IP_CONNTRACK is full, the server goes "offline", or rejects packets.

I'm sure a lof of sysadmins who stumble across this article would find it helpful like I did.

Thanks!

Since 2.6.14, it is possible to set hashsize dynamically at runtime,
after boot and module load.

Between 2.6.14 and 2.6.19 (included), use:
# echo $HASHSIZE > /sys/module/ip_conntrack/parameters/hashsize

Since 2.6.20, use:
# echo $HASHSIZE > /sys/module/nf_conntrack/parameters/hashsize

i recently found a bunch of

ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.
ip_conntrack: table full, dropping packet.

in my dmesg, this article, plus your advice, will hopefully take care of this

thanks!

On RedHat the default hashsize for the conntrack module is 8192. The rule of thumb is to allow for no more than 8 connections per bucket so you would set your conntrack size to be equal to 8 * hashsize. This is why RedHat defaults the ip_conntrack_max to 65536.

You can tweak these settings by adjusting not just the ip_conntrack_max setting but the hashsize option to the ip_conntrack module.

So, for example, if you were to set your ip_conntrack_max to 131072 without modifying the default hashsize of 8k, you are allowing for a bucket depth of 16 entries. Thus the kernel has to dig deeper, potentially, to find that one connection object in it's bucket.

There are a number of schools of thought on how best to address this but in practice I have found that, given the resources, a shallower bucket is better.

For a server that does extremely heavy network traffic, and of course has the memory to spare, you would want to keep the average bucket depth to 2 or 4.

Hashsize, to my knowledge, isn't a dynamic setting so you will need to load the ip_conntrack module with the option:

hashsize =

So in Major's example above, if you want to double your server's capacity for tracked connections while not doubling the lookups you would reload the module with:

options ip_conntrack hashsize=16384

This keeps the items per bucket to 8. I have seen machines with a depth of beyond 8 get completely cowed under heavy network load and since memory is relatively plentiful nowadays you can increase the efficiency of the lookups by making this 4 connections per bucket or even 2 by just doing simple math and reloading the module with the right options.

Hope that helps.

Here is some relatively dated yet still applicable information on the subject:
http://www.wallfire.org/misc/netfilter_conntrack_perf.txt

The entries remain until an RST packet is sent from the original IP address. If you have a flaky network somewhere between you, and the clients accessing your server, it can cause the RST packets to be dropped due to the packet loss, and leave orphaned entries in your ip_conntrack table. This can also happen if you have a malfunctioning switch or NIC card... not necessarily a routing problem out on the internet somewhere.

Typically when I've seen this trouble crop up is when a server is the target of a DDoS attack. Filling up the ip_conntrack table is a relatively easy way to knock a server off line, and attackers know this.

As Major suggested, you can get short term relief by increasing the size of the table. However, these entries are held in memory by the kernel. The bigger you make the table, the more memory it will consume. That memory could be used by your server to serve requests if you really don't need the stateful firewall capability. Don't waste resources on this feature if you really don't need it.

Another option to consider is turning OFF iptables rules that use ip_conntrack so the state able is not used at all. Anything with "-m state" or "-t nat" can be turned off. If you want to just flush all your iptables rules you can do an "iptables -P" to set a default allow policy and "iptables -F" to flush all the rules. On an RHEL or CentOS system you can just do "service iptables stop".

Once iptables is no longer using ip_conntrack, you can reclaim the memory the table was using by unloading the related kernel modules.

rmmod ipt_MASQUERADE
rmmod iptable_nat
rmmod ipt_state
rmmod ip_conntrack

Then you will have an empty ip_conntrack that will stay empty. I mention this because a lot of sysadmins have hordes of iptables rules installed as a matter of course, and don't recognize the downside of having them present. You can still use iptables, but to avoid the use of ip_conntrack simply don't use rules based on stateful logic.