Here's a sample of the latest statistics:

• Hits per day: 1.8 million (about 21 hits/second)
• Unique IP addresses per day: 25,555
• Hits per day from IPv6 addresses: 1,069 (a little sad)
• Bandwidth used per day: ~ 400MB

The site is now running on multiple Cloud Servers at Rackspace behind a load balancer cluster. In addition, the DNS records are hosted with Rackspace's Cloud DNS service.

This should allow the site to reply more quickly and reliably. If you have suggestions for other improvements, let me know!

Performance and redundancy boost for icanhazip.com is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Why do we need a JSON API for MySQL?
The real need sprang from a situation I was facing daily at Rackspace. We have a lot of production and pre-production environments which are in flux but we need a way to query data from various MySQL servers for multiple purposes. Some folks need data in ruby or python scripts while others need to drag in data with .NET and Java. Wrestling with the various adapters and all of the user privileges on disparate database servers behind different firewalls on different networks was less than enjoyable.

That's where this bridge comes in.

The bridge essentially gives anyone the ability to talk to multiple database servers across different environments by talking to a single endpoint with easily configurable security and encryption. As long as the remote user can make an HTTP POST and parse some JSON, they can query data from multiple MySQL endpoints.

How does it work?
It all starts with a simple HTTP POST. I've become a big fan of the Python requests module. If you're using it, this is all you need to submit a query:

import requests
payload = {'sql': 'SELECT * FROM some_tables WHERE some_column=some_value'}
url = "http://localhost:5000/my_environment/my_database"
r = requests.post(url, data=payload)
print r.text

The bridge takes your query and feeds it into the corresponding MySQL server. When the results come back, they're converted to JSON and returned via the same HTTP connection.

What technology does it use?
Flask does the heavy lifting for the HTTP requests and Facebook's Tornado database class wraps the MySQLdb module in something a little more user friendly. Other than those modules, PyYAML and requests are the only other modules not provided by the standard Python libraries.

Is it fast?
Yes. I haven't done any detailed benchmarks on it yet, but the overhead is quite low even with a lot of concurrency. The biggest slowdowns come from network latency between you and the bridge or between the bridge and the database server. Keep in mind that gigantic result sets will take a longer time to transfer across the network and get transformed into JSON.

I found a bug. I have an idea for an improvement. You're terrible at Python.
All feedback (and every pull request) is welcome. I'm still getting the hang of Python (hey, I've only been writing in it seriously for a few weeks!) and I'm always eager to learn a new or better way to accomplish something. Feel free to create an issue in GitHub or submit a pull request with a patch.

mysql-json-bridge: a simple JSON API for MySQL is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Photo credit: Will Clayton

As I mentioned already, the blog started out just as a place to stuff my notes from the things I learned at work. I figured that it would be nice to store it in a searchable format but it would also be great if I could link other people to certain posts if they needed more information to fix a problem. It was a way to retain knowledge but yet give it back to the people around me who needed it.

The blog has hit 456 posts (this one is #457) and it's gone from a few page views per day to just over 20,000 per day. Here are the top five most accessed posts (since I've been keeping stats):

1. Syncing an iPhone with a new Mac without hassles
2. ip_conntrack: table full, dropping packet
3. Delete a single iptables rule
4. Increase MySQL connection limit
5. MySQL Error 1040: too many connections

I'd like to send out a big thanks to the people who read this blog, add comments (or complaints!), and suggest new topics. You are the reason why I take the time to keep this blog going.

Five years of rackerhacker.com is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

openssl s_client -connect encrypted.google.com:443

You'll see the chain of certificates back to the original certificate authority where Google bought its certificate at the top, a copy of their SSL certificate in plain text in the middle, and a bunch of session-related information at the bottom.

This works really well when a site has one SSL certificate installed per IP address (this used to be a hard requirement). With Server Name Indication (SNI), a web server can have multiple SSL certificates installed on the same IP address. SNI-capable browsers will specify the hostname of the server they're trying to reach during the initial handshake process. This allows the web server to determine the correct SSL certificate to use for the connection.

If you try to connect to rackerhacker.com with s_client, you'll find that you receive the default SSL certificate installed on my server and not the one for this site:

$ openssl s_client -connect rackerhacker.com:443
Certificate chain
 0 s:/C=US/ST=Texas/L=San Antonio/O=MHTX Enterprises/CN=*.mhtx.net
   i:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
 1 s:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
   i:/C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority

Add on the -servername argument and s_client will do the additional SNI negotiation step for you:

$ openssl s_client -connect rackerhacker.com:443 -servername rackerhacker.com
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=rackerhacker.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
 2 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

You may be asking yourself this question:

Why doesn't the web server just use the Host: header that my browser sends already to figure out which SSL certificate to use?

Keep in mind that the SSL negotiation must occur prior to sending the HTTP request through to the remote server. That means that the browser and the server have to do the certificate exchange earlier in the process and the browser wouldn't get the opportunity to specify which site it's trying to reach. SNI fixes that by allowing a Host: header type of exchange during the SSL negotiation process.

Using OpenSSL's s_client command with web servers using Server Name Indication (SNI) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

• pypi.mhtx.net

Python packages are updated moments after the commit is merged into the repositories under OpenStack's github account.

Although the packages will contain the latest code available, rest assured that the code has passed an initial code review (by humans), unit tests, and varying levels of functional or integrated testing. There may still be a bug or two cropping up after that, so be aware of that as you utilize these packages.

The package versions utilize a standard format:

[package]-[version]-[git commit count]-[short commit hash]

If you need to check the git log up to that particular commit, just run git log:

git log [short commit hash]

Instructions for configuring pip or easy_install are provided within the repository.

In addition, the repository is accessible via IPv4 and IPv6.

OpenStack bleeding-edge Python packages are now available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

icanhazip.com traffic data - March 2011

The count of unique visitors certainly increased (by about 10-11x), but the overall hits didn't increase by much. I'd imagine that most visitors accessed the site, noticed that it displayed their public IP, and then they went on their way. As I've said before, this site is easy to re-create and will really only serve a niche segment of internet users.

On most days, I'll receive a very high number of hits from a relatively small number of unique IP addresses. There are quite a few people who check their public-facing IP address every second, but it seems like the majority stick to a more reasonable interval of 5-30 minutes. I've yet to find the value in checking my public IP address once per second, but there are obviously some folks out there who find it valuable (or they aren't good at implementing sleeps in their scripts).

Here's a bit of trivia about the site for those who are interested:

• Almost 40% of the traffic to the site is from Eastern European and Asian countries
• The average user on the site generates about 45 hits per day
• Linux users make up 91% of the traffic on the site (based on user agent strings)
• Over 88% of the hits to the site are requests made with curl or wget
• Most traffic is received between 4-5PM CDT
• Almost 98% of the visitors who reach the site do so via a direct link without a referrer

Lifehacker effect on icanhazip.com is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

While many of the users have reasonable usage patterns, there are some users that query the site more than once per second from the same IP address. If you haven't used the site before, all it does is return your public IP address in plain text. Unless your IP changes rapidly, you may not need to query the site more than a few times an hour.

I added the following to my icanhazip.com virtual host definition to get the message across to those users that abuse the service:

ErrorDocument 403 "No can haz IP. Stop abusing this service. \
    Contact major at mhtx dot net for details."
RewriteEngine On
RewriteCond %{REMOTE_ADDR} ^12.23.34.45$ [OR]
RewriteCond %{REMOTE_ADDR} ^98.87.76.65$
RewriteRule .* nocanhaz [F]

The users that are caught on the business end of these 403 responses will see something like this:

$ curl -i icanhazip.com
HTTP/1.1 403 Forbidden
Date: Wed, 17 Nov 2010 13:42:55 GMT
Server: Apache
Content-Length: 84
Connection: close
Content-Type: text/html; charset=iso-8859-1
 
No can haz IP. Stop abusing this service. Contact major at mhtx dot net for details.

Throwing thoughtful "403 Forbidden" responses with apache is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Instead of using GlusterFS as I had before in my redundant cloud hosting guide, I decided to use DRBD in dual-primary mode with OCFS2 as the clustering filesystem on top of it. The performance is quite good so far:

Pingdom Response Time Graph for rackerhacker.com

I switched over the DNS late last night and the response time has fallen from the two to three second range (during times of low load) to right around one second per request. In addition to the reduced load times, I can support higher concurrency without significant performance degradation.

Don't worry - I'll make a detailed post on this topic later along with a guide on how to set it up yourself.

Switching from GlusterFS to DRBD and OCFS2 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The guide: Redundant Cloud Hosting Guide

As always, if you find anything in the guide that needs improvement, I'm all ears.

A simple guide to redundant cloud hosting is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

There has been quite a bit of interest in GlusterFS as of late and I've been inundated with questions from coworkers, other system administrators and developers. Most folks want to know about its reliability and performance in demanding production environments. I'll try to do my best to cover the big points in this post.

First off, here's now I'm using it in production: I have two web nodes that keep content in sync for various web sites. They each run a GlusterFS server instance and they also mount their GlusterFS share. I'm using the replicate translator to keep both web nodes in sync with client side replication.

Here are my impressions after a month:

I/O speed is often tied heavily to network throughput
This one may seem obvious, but it's not always true in all environments. If you deal with a lot of small files like I do, a 40mbit/sec link between the Xen guests is plenty. Adding extra throughput didn't add any performance to my servers. However, if you wrangle large files on your servers regularly, you may want to consider higher throughput links between your servers. I was able to push just under 900mbit/sec by using dd to create a large file within a GlusterFS mount.

Network and I/O latency are big factors for small file performance
If you have a busy network and the latency creeps up from time to time, you'll find that your small file performance will drop significantly (especially with the replicate translator). Without getting too nerdy (you're welcome to read the technical document on replication), replication is an intensive process. When a file is accessed, the client goes around to each server node to ensure that it not only has a copy of the file being read, but that it has the correct copy. If a server didn't save a copy of a file (due to disk failure or the server being offline when the file was written), it has to be synced across the network from one of the good nodes.

When you write files on replicated servers, the client has to roll through the same process first. Once that's done, it has to lock the file, write to the change log, then do the write operation, drop the change log entries, and then unlock the file. All of those operations must be done on all of the servers. High latency networks will wreak havoc on this process and cause it to take longer than it should.

It's quite obvious that if you have a fast, low-latency network between your servers, slow disks can still be a problem. If the client is waiting on the server nodes' disks to write data, the read and write performance will suffer. I've tested this in environments with fast networks and very busy RAID arrays. Even if the network was very underutilized, slow disks could cut performance drastically.

Monitoring GlusterFS isn't easy
When the client has communication problems with the server nodes, some weird things can happen. I've seen situations where the client loses connections to the servers (see the next section on reliability) and the client mount simply hangs. In other situations, the client has been knocked offline entirely and the process is missing from the process tree by the time I logged in. Your monitoring will need to ensure that the mount is active and is responding in a timely fashion.

There's a handy script which allows you to monitor GlusterFS mounts via nagios that Ian Rogers put together. Also, you can get some historical data with acrollet's munin-glusterfs plugin.

GlusterFS 3.x is pretty reliable
When I first started working with GlusterFS, I was using a version from the 2.x tree. The Fedora package maintainer hadn't updated the package in quite some time, but I figured it should work well enough for my needs. I found that the small file performance was lacking and the nodes often had communication issues when many files were being accessed or written simultaneously. This improved when I built my own RPMs of 3.0.4 (and later 3.0.5) and began using those instead.

I did some failure testing by hard cycling the server and client nodes and found some interesting results. First off, abruptly pulling clients had no effects on the other clients or the server nodes. The connection eventually timed out and the servers logged the timeout as expected.

Abruptly pulling servers led to some mixed results. In the 2.x branch, I saw client hangs and timeouts when I abruptly removed a server. This appears to be mostly corrected in the 3.x branch. If you're using replicate, it's important to keep in mind that the first server volume listed in your client's volume file is the one that will be coordinating the file and directory locking. Should that one fall offline quickly, you'll see a hiccup in performance for a brief moment and the next server will be used for coordinating the locking. When your original server comes back up, the locking coordination will shift back.

Conclusion
I'm really impressed with how much GlusterFS can do with the simplicity of how it operates. Sure, you can get better performance and more features (sometimes) from something like Lustre or GFS2, but the amount of work required to stand up that kind of cluster isn't trivial. GlusterFS really only requires that your kernel have FUSE support (it's been in mainline kernels since 2.6.14).

There are some things that GlusterFS really needs in order to succeed:

• Documentation - The current documentation is often out of date and confusing. I've even found instances where the documentation contradicts itself. While there are some good technical documents about the design of some translators, they really ought to do some more work there.
• Statistics gathering - It's very difficult to find out what GlusterFS is doing and where it can be optimized. Profiling your environment to find your bottlenecks is nearly impossible with the 2.x and 3.x branches. It doesn't make it easier when some of the performance translators actually decrease performance.
• Community involvement - This ties back into the documentation part a little, but it would be nice to see more participation from Gluster employees on IRC and via the mailing lists. They're a little better with mailing list responses than other companies I've seen, but there is still room for improvement.

If you're considering GlusterFS for your servers but you still have more questions, feel free to leave a comment or find me on Freenode (I'm 'rackerhacker').

One month with GlusterFS in production is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Typical configuration for a
proxy-type load balancer

• client makes a request
• load balancer receives the request
• load balancer sends request to a web node
• the web server sends content back to the load balancer
• the load balancer responds to the client

If you're not familiar with load balancing, here's an analogy. Consider a fast food restaurant. When you walk up to the counter and place an order, you're asking the person at the counter (the load balancer) for a hamburger. The person at the counter is going to submit your order, and then a group of people (web nodes) are going to work on it. Once your hamburger (web request) is ready, your order will be given to the person at the counter and then back to you.

This style of organization can become a problem as your web nodes begin to scale. It requires you to ensure that your load balancers can keep up with the requests and sustain higher transfer rates that come from having more web nodes serving a greater number of requests. Imagine the fast food restaurant where you have one person taking the orders but you have 30 people working on the food. The person at the counter may be able to take orders very quickly, but they may not be able to keep up with the orders coming out of the kitchen.

LVS allows for application servers
to respond to clients directly

• client makes a request
• load balancer receives the request
• load balancer sends request to a web node
• the web server sends the response directly to the client

The key difference is that the load balancer sends the unaltered request to the web server and the web server responds directly to the client. Here's the fast food analogy again. If you ask the person at the counter (the load balancer) for a hamburger, that person is going to take your order and give it to the kitchen staff (the web nodes) to work on it. This time around, the person at the counter is going to advise the kitchen staff that the order needs to go directly to you once it's complete. When your hamburger is ready, a member of the kitchen staff will walk to the counter and give it directly to you.

In the fast food analogy, what are the benefits? As the number of orders and kitchen staff increases, the job of the person at the counter doesn't drastically increase in difficulty. While that person will have to handle more orders and keep tabs on which of the kitchen staff is working on the least amount of orders, they don't have to worry about returning food to customers. Also, the kitchen staff doesn't need to waste time handing orders to the person at the counter. Instead, they can pass these orders directly to the customer that ordered them.

In the world of servers, this is a large benefit. Since the web servers' responses no longer pass through the load balancer, they can spend more time on what they do best -- balancing traffic. This allows for smaller, lower-powered load balancing servers from the beginning. It also allows for increases in web nodes without big changes for the load balancers.

There are three main implementations of LVS to consider:

LVS-DR: Direct Routing
The load balancer receives the request and sends the packet directly to a waiting real server to process. LVS-DR has the best performance, but all of your servers must be on the same network subnet and they have to be able to share the same router (with no other routing devices in between them).

LVS-TUN: Tunneling
This is very similar to the direct routing approach, but the packets are encapsulated and sent directly to the real servers once the load balancer receives them. This removes the restriction that all of the devices must be on the same network. Thanks to encapsulation, you can use this method to load balance between multiple datacenters.

LVS-NAT: Network Address Translation
Using NAT for LVS yields the least performance and scaling of all of the implementation options. In this configuration, the incoming requests are rewritten so that they will be transported correctly in a NAT environment. This puts a bigger burden on the load balancer as it must rewrite the requests quickly while still keeping up with how much work is being done by each web server.

A modern implementation and explanation of Linux Virtual Server (LVS) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

127.0.0.1 - - [21/Aug/2008:12:00:10 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:11 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:13 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:14 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:15 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"

On servers with ipv6 enabled, you might see a line like this one:

::1 - - [21/Aug/2008:12:00:15 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"

I began to wonder why Apache was making these connections back onto itself and initiating a GET /. Apache's documentation had the following:

When the Apache HTTP Server manages its child processes, it needs a way to wake up processes that are listening for new connections. To do this, it sends a simple HTTP request back to itself. This request will appear in the access_log file with the remote address set to the loop-back interface (typically 127.0.0.1 or ::1 if IPv6 is configured). If you log the User-Agent string (as in the combined log format), you will see the server signature followed by "(internal dummy connection)" on non-SSL servers. During certain periods you may see up to one such request for each httpd child process.

These requests are perfectly normal and you do not, in general, need to worry about them. They can simply be ignored.

Sure, I could easily ignore the requests, but the requests were increasing the load on my server more than I liked. Apache's documentation suggested omitting the lines from the logs by adding the following to the Apache configuration:

SetEnvIf Remote_Addr "127\.0\.0\.1" loopback

And then adding env=!loopback to your CustomLog lines ensures that the data won't show up in your access logs. However, you'll still end up with Directory index forbidden by Options directive: /var/www/html/ filling up your error_logs. A quick search revealed a handy mod_rewrite rule to get rid of these requests as quickly as possible with the lowest effort required from Apache:

RewriteCond %{HTTP_USER_AGENT} ^.*internal\ dummy\ connection.*$ [NC]
RewriteRule .* - [F,L]

At this point, the requests to the localhost should receive a 403 immediately. Since you can't keep Apache from sending all of these requests to itself, the best you can do is respond to them in a manner that requires the lowest possible resources.

Apache 2.2: internal dummy connection is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Luckily, mod_deflate makes this easy, and the Apache documentation has a handy initial configuration available:

<Location />
SetOutputFilter DEFLATE
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
Header append Vary User-Agent env=!dont-vary
</Location>

This configuration will compress everything except for images. Of course, you can't test this with curl, but you can test it with Firefox and LiveHTTPHeaders. If you don't have Firefox handy, you can try a very handy web application that will give you the statistics about the compression of your site's data.

Compress your web content for better performance is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

It's always been a bit of a challenge to disable TRACE and TRACK methods with Plesk. The only available options were to create a ton of vhost.conf files or adjust the httpd.include files and prevent modifications with chattr (which is a bad idea on many levels).

Luckily, Parallels has made things easier with a new knowledge base article.

Plesk: Disabling TRACE/TRACK methods globally is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Luckily, you can remove the news feeds pretty easily. Open the following file in your favorite text editor:

/usr/share/psa-horde/templates/portal/sidebar.inc

Once the file is open, drop down to line 102 and comment out the entire if() statement (lines 102-117).

NOTE: If you upgrade Plesk, this change will most likely be reversed.

Removing news feeds in Horde is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

This blog runs on a server with dual Xeon Woodcrest CPU's, 64-bit CentOS 4.5 and a 100mbit network connection. Here's the first test with WP-Cache turned off:

$ http_load -parallel 10 -seconds 30 urltocheck.txt
346 fetches, 10 max parallel, 1.78616e+07 bytes, in 30 seconds
51623.2 mean bytes/connection
11.5333 fetches/sec, 595387 bytes/sec
msecs/connect: 15.1661 mean, 16.97 max, 14.922 min
msecs/first-response: 445.984 mean, 2328.82 max, 189.62 min
HTTP response codes:
code 200 -- 346

346 fetches in 30 seconds is not a very exciting performance number for me. That's just over 10 fetches per second, and on a busy day, I sometimes reach that number. Also, while this test ran, the server's CPU usage was extremely high and over 80% of all four cores were in use. The iowait was about 20% across the board.

I decided to turn on WP-Cache and give it another go with the same test:

$ http_load -parallel 10 -seconds 30 urltocheck.txt
3482 fetches, 10 max parallel, 1.79671e+08 bytes, in 30 seconds
51600 mean bytes/connection
116.067 fetches/sec, 5.98904e+06 bytes/sec
msecs/connect: 15.2259 mean, 18.257 max, 14.891 min
msecs/first-response: 20.7297 mean, 69.39 max, 18.861 min
HTTP response codes:
code 200 -- 3482

Wow, that's a 10-fold improvement, and I can handle over 100 requests per second with 10 parallel requests. Also, the iowait dropped to 5%, and overall CPU usage remained under 8%.

I kicked it up to 20 parallel connections and tried again:

$ http_load -parallel 20 -seconds 30 urltocheck.txt
5817 fetches, 20 max parallel, 3.02176e+08 bytes, in 30 seconds
51947 mean bytes/connection
193.9 fetches/sec, 1.00725e+07 bytes/sec
msecs/connect: 17.9175 mean, 30.831 max, 14.911 min
msecs/first-response: 24.5352 mean, 97.475 max, 18.978 min
HTTP response codes:
code 200 -- 5817

Almost 194 connections served per second! Also, the CPU usage was only at about 14% during the duration of the test.

I decided to tempt fate and see if I could blow the roof off the test with 50 parallel connections:

$ http_load -parallel 50 -seconds 30 urltocheck.txt
5794 fetches, 50 max parallel, 2.99718e+08 bytes, in 30 seconds
51729 mean bytes/connection
193.133 fetches/sec, 9.99059e+06 bytes/sec
msecs/connect: 43.286 mean, 63.878 max, 14.942 min
msecs/first-response: 68.967 mean, 202.854 max, 20.014 min
HTTP response codes:
code 200 -- 5794

The performance suffered a bit, but the server was still pumping out almost 200 connections per second, and I'm okay with that. Well, unless anyone has a spare Cisco 11501 laying around that I could have. And, of course, one additional server.

Just as a sidenote, I installed Zend Optimizer v3.3 on the server and performance actually dropped by 1%-3% for each test. I found that a bit surprising.

I used http_load to perform the benchmarks after I found it on Caleb's blog.

Why you should use caching for WordPress blogs is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

To fix the problem, create a file called /etc/httpd/conf.d/zz050a_horde_php_workaround.conf and put the following inside it:

<DirectoryMatch /usr/share/psa-horde>
php_admin_value include_path "/usr/share/psa-horde/lib:/usr/share/psa-horde:/usr/share/psa-horde/pear:."
</DirectoryMatch>

Reload the Apache configuration and your Horde installation should work properly with PHP 5.2.5.

Credit for this fix goes to Kevin M.

Fixing Horde problems in Plesk 8.1.x/8.2.x with PHP 5.2.5 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

unable to call function somefunction on undefined value

Of course, replace somefunction with your function of choice. To correct the issue, you can force CPAN to bring back a more sane version of Scalar::Util:

# perl -MCPAN -e shell
cpan> force install Scalar::Util

Red Hat Perl Issues: unable to call function somefunction on undefined value is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First, adjust the AddDefaultCharset directive in /etc/httpd/conf/httpd.conf:

#AddDefaultCharset UTF-8
AddDefaultCharset ISO-8859-1

Then, reload Apache and check your headers:

# /etc/init.d/httpd reload
# curl -I localhost
HTTP/1.1 403 Forbidden
Date: Thu, 08 Nov 2007 22:18:14 GMT
Server: Apache/2.0.52 (Red Hat)
Accept-Ranges: bytes
Content-Length: 3985
Connection: close
Content-Type: text/html; charset=ISO-8859-1

This was tested on Red Hat Enterprise Linux 4 Update 5

Change the default Apache character set is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Error opening /var/lib/squirrelmail/prefs/default_pref
Could not create initial preference file!
/var/lib/squirrelmail/prefs/ should be writable by user apache
Please contact your system administrator and report this error.

No matter what you do to the /var/lib/squirrelmail/prefs/default_pref file, even if you chmod 777 the file, you will still get the error. If you check the /etc/php.ini, you will normally find safe_mode set to on.

;
; Safe Mode
;
safe_mode = Off

Simply change safe_mode to off and reload Apache. If you try to log into squirrelmail again, it should complete successfully. I've tested this on Red Hat Enterprise Linux 4:

# rpm -q squirrelmail
squirrelmail-1.4.8-4.0.1.el4

Plesk: Error opening /var/lib/squirrelmail/prefs/default_pref is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Too many filters / Giant whitelists and blacklists
This is the biggest cause that I've seen. Some users will create gigantic white and black lists (upwards of 5,000 is my record that I've seen) and this makes Horde compare each and every message in the inbox against these lists upon login. This also applies to filters as Plesk does not use sieve/procmail for mail delivery. Horde is forced to do all of the filtering upon login (in some versions) and this can cause extreme delays.

Mailbox is gigantic
I've seen Horde logins take quite a while in mailboxes that are over 500MB in size. If the size of your e-mails is large, and you have a large mailbox with fewer e-mails, Horde can normally work quickly. But, if your inbox is full of tiny e-mails, Horde takes a long time to fully index your mail and display the list (even though it only displays 25-30 at a time).

Too many users logged into Horde simultaneously
In my opinion, Horde's CPU and memory requirements are too large for a webmail application. I've seen 30-40 simultaneous Horde sessions bring a dual-core box with 2-4GB of RAM and SCSI disks to its knees. Consider installing squirrelmail or roundcube webmail for some of your users and urge them to use it instead.

IOwait caused by something else
Sometimes the server can simply be bogged down with other requests from other daemons, and this slows Horde down. Make sure that your MySQL installation is tuned properly, and that users are not abusing scripts running through Apache.

Slow Horde login process with Plesk is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Unfortunately, enabling this in conjunction with Plesk will cause problems with the Plesk web statistics. Since the PHP parsing is disabled for the /plesk-stat/ directories, Apache will mark the page as a PHP page and your browser will attempt to download it rather than display it.

To fix this issue, simply add the following LocationMatch to the bottom of your Apache configuration:

AddType application/x-httpd-php .php .html

<LocationMatch "/plesk-stat/(.*)">
AddType text/html .html
</LocationMatch>

This will force Apache to serve HTML files from /plesk-stat/ as text/html rather than application/x-http-php. Your web statistics will display in the browser rather than downloading as a PHP file.

Parsing HTML through PHP in Plesk is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

You may see an error message in Horde that looks like this:

Your Internet Address has changed since the beginning of your Mail session. To protect your security, you must login again.

You'll normally have this variable in /etc/psa-horde/horde/conf.php:

# $conf['auth']['checkip'] = true;

You can disable this ip check functionality which breaks sessions for AOL users by setting it to false:

# $conf['auth']['checkip'] = false;

Session problems with Horde in Plesk with AOL is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If your private key and certificate do not contain the same modulus, then Apache will sometimes refuse to start or it may not respond properly to SSL requests. You can check the modulus of your private key and SSL certificate with these commands:

# openssl rsa -noout -modulus -in server.key | openssl md5
# openssl x509 -noout -modulus -in server.crt | openssl md5

If the MD5 checksums match, then the certificate and key will work together. However, if they are different, then you cannot use them together. Generally, this means that you used the wrong CSR (that corresponded to some other private key) when you obtained/created your SSL certificate.

Check the modulus of an SSL certificate and key with openssl is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

To hunt down these connections when the spidering is happening, simply run this command:

netstat -plan | grep :80 | awk '{print $5}' | sed 's/:.*$//' | sort | uniq -c | sort -rn

The IP's that are making the most connections will appear at the top of the list, and from there, you can find out which unwelcome spider is scraping your site.

Hunting down annoying web spiders is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.