But when I place the php application and the mysql server together on 1 server (connect to localhost) I never got the problem. Even its just 1 server running with the same amount of client (1000 connected client). Thread connected is 2 or 3.

How this could happen?
Connect to other server : Thread connected is above 1000.
Connect to localhost : Thread connected is 2 or 3.

That is, of course, assuming that MySQL hasn't started swapping to disk to satisfy the configuration (i.e. opening as many threads/connections as the config allows). This is where things become wonky -- high load and iowait, etc. You will be waiting a long time to SSH in and wrangle the box back to normalcy, or connect via MySQL remotely if you have that configured.

@thornibr was dead on with this:
"You should set global max_connections to a value that allows your sever to handle the connections - not increase global max_connections when your server cannot handle the current amount of connections." <--- exactly.

It should be noted that MySQLTuner is very helpful in preventing these problems with its memory usage calculations vs installed RAM.

Good post, Major. =]

Today the database I mange gave me the 1040 error which was the first time I've ever come across it in a year. It is strange though because I would have expected it a year ago. I've moved the DB onto a PowerEdge 1950 and have done a fair amount of query optimization during the year so that our site is very responsive. Also, I'm about 95% sure that our traffic didn't skyrocket (will have to check analytics tomorrow).

I guess my question is, how can the things you mention above affect the max_connections issue? Is it that the suggestions you make are intended to connect/disconnect as quickly as possible in order to avoid the max_connections limit? (That seems to be the only logical thing in my mind.)

Also, what are your thoughts on setting the max_connections, of a dedicated mysql, server to 0?

GRANT ALL ON *.* to XXX..

And you are silly enough to use that user within your application.

If you create a user for a database and as a result have security in mind, you can still get in with your 'admin' account as you point out.

But as major points out, if you are seeing max_connnections errors, chances are your schema, buffers or queries are at fault.

What everyone should really do is create user accounts and set max_connections for that specific user to a value, say 10. And then set global max_connections to be upto the sum of every users max_connections.

Setting max_connections on a per user basis will prevent (or at very least reduce) a single site/database on a shared hosting system from hogging all of the database resources.

You should set global max_connections to a value that allows your sever to handle the connections - not increase global max_connections when your server cannot handle the current amount of connections.

"You won’t even be able to connect to MySQL to find out what is causing the connections to be used up, so you will be forced to restart the MySQL daemon to troubleshoot the issue."

According to the documentation MySQL always reserves one extra connection so a user with the SUPER privilege can login and see what's going on. See here:

mysqld actually allows max_connections+1 clients to connect. The extra connection is reserved for use by accounts that have the SUPER privilege. By granting the SUPER privilege to administrators and not to normal users (who should not need it), an administrator can connect to the server and use SHOW PROCESSLIST to diagnose problems even if the maximum number of unprivileged clients are connected. See Section 12.5.4.21, “SHOW PROCESSLIST Syntax”.