Racker Hacker

I found myself wrestling with a server where the Plesk interface suddenly became unavailable without any user intervention. An attempt to start the service was less than fruitful:

[root@server ~]# service psa start
Key file: /opt/drweb/drweb32.key - Key file not found!
A path to a valid license key file does not specified.
Plesk authorization failed: HTTP request error [7]
Error: Plesk Software not running.
                                                           [FAILED]

(Although I included the text from the drweb failure, I later found that it was not related to the issue. However, since it might appear in your logs prior to the HTTP request error, I included it anyways.)

This was a perfectly working server that had no other issues besides this peculiar Plesk issue. Another technician had upgraded the license a few weeks prior, and it was verified at the the time to be working properly. After a bit of Google searching, I found that the solution was to completely stop Plesk and its related services and then start it all up again.

[root@server ~]# service psa stopall
/usr/local/psa/admin/bin/httpsdctl stop: httpd stopped
Stopping Plesk:                                            [  OK  ]
Stopping named:                                            [  OK  ]
service psa startStopping MySQL:                           [  OK  ]
Stopping : Stopping Courier-IMAP server:
   Stopping imap                                           [  OK  ]
   Stopping imap-ssl                                       [  OK  ]
   Stopping pop3                                           [  OK  ]
   Stopping pop3-ssl                                       [  OK  ]

Stopping postgresql service:                               [  OK  ]
Shutting down psa-spamassassin service:                    [  OK  ]
Stopping httpd:                                            [  OK  ]

[root@server ~]# service psa start
Starting named:                                            [  OK  ]
Starting MySQL:                                            [  OK  ]
Starting qmail:                                            [  OK  ]
Starting Courier-IMAP server:
   Starting imapd                                          [  OK  ]
   Starting imap-ssl                                       [  OK  ]
   Starting pop3                                           [  OK  ]
   Starting pop3-ssl                                       [  OK  ]

Starting postgresql service:                               [  OK  ]
Starting psa-spamassassin service:                         [  OK  ]
Processing config directory: /usr/local/psa/admin/conf/httpsd.*.include
/usr/local/psa/admin/bin/httpsdctl start: httpd started
Starting Plesk:                                            [  OK  ]
Starting up drwebd:                                        [  OK  ]

I couldn’t nail down anything within the Plesk log files that would explain the cause of the problem, but this solution corrected the issue instantly.

This issue occurred with Plesk 8.1.1 on Red Hat Enterprise Linux 4 Update 5

I’ve received a lot of IM’s and e-mails from friends and readers of this blog about the Rackspace outages on November 11th and 12th. I work for a company that believes in full disclosure, so if you want the facts, they’re already available to the public:

Rackspace Information Center
TechCrunch - Quick, Plug The Internet Back In: Major Rackspace Outage
Laughing Squid - Massive Power Outage At Rackspace’s Dallas Data Center
37Signals - Downtime Explanation
ValleyWag - Truck driver in Texas kills all the websites you really use

I don’t know of any additional information besides what is contained within these articles and blog posts. However, I can tell you that I’ve never worked for a company before that pulled together in such large numbers to get on the phones and respond to tickets long after shifts should have been over. Obviously, it was a horrible “perfect storm” type of situation, and no one would wish for it to happen to anyone.

Over the last five years, I’ve had dedicated servers and VPS accounts with seven companies. Out of those seven, five have had major outages. After those outages, I can honestly say I received a timely and courteous response from one of the companies in that list. In one situation with a certain Texas hosting company, I had no network connectivity for almost 72 hours with no response to phone calls or trouble tickets.

After it’s all said and done (and it’s not done yet), I find myself to be very proud of the company for which I work. Server parts will eventually fail, as will networks, generators and power grids - it’s inevitable. The important part is that the will of those who are providing the support never fails.

Our will is strong, and it continues to stay that way.

Create a strong CSR and private key
openssl req -new -nodes -newkey rsa:2048 -out server.crt -keyout server.key

Parsing out the data within a certificate
openssl asn1parse -in server.crt

Checking a certificate/key modulus to see if they correspond
openssl rsa -in server.key -modulus -noout | openssl md5
openssl x509 -in server.crt -modulus -noout | openssl md5

Convert a key from PEM -> DER
openssl rsa -inform PEM -in key.pem -outform DER -out keyout.der

Convert a key from DER -> PEM
openssl rsa -inform DER -in key.der -outform PEM -out keyout.pem

Remove the password from an encrypted private key
openssl rsa -in server.key -out server-nopass.key

Reviewing a detailed SSL connection
openssl s_client -connect 10.0.0.1:443

I’ve updated MySQLTuner to revision 20 tonight, and it has the following improvements:

* Added NetBSD support (thanks to Dave Burgess)
* Switched username/password prompts to STDERR so they won’t appear in printouts

Downloads are available, or you can do a new subversion checkout.

I’ve had a request on a detailed article about when one should opt for MyISAM or InnoDB when using MySQL. First off, there’s a few deal-breakers that need to be discussed:

MyISAM Limitations

• No foreign keys and cascading deletes/updates
• No transactional integrity (ACID compliance)
• No rollback abilities
• Row limit of 4,284,867,296 rows (2³²)
• Maximum of 64 indexes per row

InnoDB Limitations

• No full text indexing
• Cannot be compressed for fast, read-only

If none of those deal-breakers apply, then you can keep reviewing the pros and cons. Here’s some things to keep in mind:

Read to write ratio
My rule of thumb is to calculate the ratio of queries that read data to those that write data. If you find that the amount of writes climbs over 10-15%, you should consider InnoDB, but if it climbs over 20%, you will get a big performance gain by moving to InnoDB.

This all stems from how the storage engines handle locking. MyISAM uses table-level locking. That means that if you run a query against the table, the whole table will be locked. The only exception is that you can insert a row if there is an active read lock on the table, and the table is not fragmented (called concurrent insertion). So if a user is running a really slow select on a table, your updates or deletes will have to wait until the select is completely finished. This delay can reduce the overall efficiency of your application drastically if your MySQL server is busy and over 20% of your queries are writes.

InnoDB uses a row-level locking scheme, which means that waiting on locks is greatly reduced. When MySQL wants to lock something for a query in an InnoDB table, only the row that needs to be accessed is actually locked. For example, if you run a select that examines rows 5-20 in a table, but another user wants to write to row 24 at the same time, both queries will be executed simultaneously. Of course, transactions will change this behavior a bit, but I’ll leave that out for simplicity’s sake.

Available hardware
InnoDB requires additional hardware in order to function as efficiently as MyISAM. This means that you will need additional CPU power and RAM to get good performance out of InnoDB tables. MyISAM tables can also be compressed with myisampack so that they can handle queries quickly (at the expense of being read-only).

Table growth
MyISAM (by default) is limited to 2³² rows, and this can be a pain when you reach the limit. You can increase it manually when you reach the limit, but this causes the row pointer to become gigantic, and this reduces performance. InnoDB tables have no limits, and they can grow over 2GB even on systems that don’t allow files to grow over 2GB. InnoDB tables can also use raw partitions to store data, and this ends up being quite fast relative to simple ibdata files.

Isolation and integrity
When it comes to transactional integrity, InnoDB holds all the cards. You can adjust your isolation levels so that you see the exact data you want, even when data is changing from other transactions when your transaction runs. Transactions can have save points to which you can roll back if something does not work out. Also, if you have the need to run several queries at once that need to run uninterrupted, transactions are required for this functionality.

I’ve struggled at times to get a decent-looking terminal on my desktop, and I believe I’ve found a good one. Toss this into your ~/.Xdefaults:

aterm*loginShell:true
aterm*transparent:true
aterm*shading:40
aterm*background:Black
aterm*foreground:White
aterm*scrollBar:true
aterm*scrollBar_right:true
aterm*transpscrollbar:true
aterm*saveLines:32767
aterm*font:*-*-fixed-medium-r-normal--*-110-*-*-*-*-iso8859-1
aterm*boldFont:*-*-fixed-bold-r-normal--*-*-110-*-*-*-*-iso8859-1

Then load up the changes and start aterm:

$ xrdb -load .Xdefaults
$ aterm

Of course, if you like rxvt better for your Unicode needs, just use this configuration:

rxvt*loginShell:true
rxvt*transparent:true
rxvt*shading:40
rxvt*background:Black
rxvt*foreground:White
rxvt*scrollBar:true
rxvt*scrollBar_right:true
rxvt*transpscrollbar:true
rxvt*saveLines:32767
rxvt*font:*-*-fixed-medium-r-normal--*-110-*-*-*-*-iso8859-1
rxvt*boldFont:*-*-fixed-bold-r-normal--*-*-110-*-*-*-*-iso8859-1

I’ve been on a bit of a vacation lately, and that’s why there’s been a lack of updates. On October 20th, I married the best woman on the planet, and we we’ve been on our honeymoon since then.

New updates should start again this Monday!