openssl s_client -connect encrypted.google.com:443

You'll see the chain of certificates back to the original certificate authority where Google bought its certificate at the top, a copy of their SSL certificate in plain text in the middle, and a bunch of session-related information at the bottom.

This works really well when a site has one SSL certificate installed per IP address (this used to be a hard requirement). With Server Name Indication (SNI), a web server can have multiple SSL certificates installed on the same IP address. SNI-capable browsers will specify the hostname of the server they're trying to reach during the initial handshake process. This allows the web server to determine the correct SSL certificate to use for the connection.

If you try to connect to rackerhacker.com with s_client, you'll find that you receive the default SSL certificate installed on my server and not the one for this site:

$ openssl s_client -connect rackerhacker.com:443
Certificate chain
 0 s:/C=US/ST=Texas/L=San Antonio/O=MHTX Enterprises/CN=*.mhtx.net
   i:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
 1 s:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
   i:/C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority

Add on the -servername argument and s_client will do the additional SNI negotiation step for you:

$ openssl s_client -connect rackerhacker.com:443 -servername rackerhacker.com
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=rackerhacker.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
 2 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

You may be asking yourself this question:

Why doesn't the web server just use the Host: header that my browser sends already to figure out which SSL certificate to use?

Keep in mind that the SSL negotiation must occur prior to sending the HTTP request through to the remote server. That means that the browser and the server have to do the certificate exchange earlier in the process and the browser wouldn't get the opportunity to specify which site it's trying to reach. SNI fixes that by allowing a Host: header type of exchange during the SSL negotiation process.

Using OpenSSL's s_client command with web servers using Server Name Indication (SNI) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

• pypi.mhtx.net

Python packages are updated moments after the commit is merged into the repositories under OpenStack's github account.

Although the packages will contain the latest code available, rest assured that the code has passed an initial code review (by humans), unit tests, and varying levels of functional or integrated testing. There may still be a bug or two cropping up after that, so be aware of that as you utilize these packages.

The package versions utilize a standard format:

[package]-[version]-[git commit count]-[short commit hash]

If you need to check the git log up to that particular commit, just run git log:

git log [short commit hash]

Instructions for configuring pip or easy_install are provided within the repository.

In addition, the repository is accessible via IPv4 and IPv6.

OpenStack bleeding-edge Python packages are now available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

However, certain folks may find the need to host a local PyPi repository for their own packages. You may need it to store Python code which you don't plan to release publicly or you may need to add proprietary patches to upstream Python packages. Regardless of the reason to have it, a local PyPi repository is relatively easy to configure.

You'll need to start with a base directory for your PyPi repository. For this example, I chose /var/pypi. The directory structure should look something like this:

/var/pypi/simple/[package_name]/[package_tarball]

For a package like pip, you'd make a structure like this:

/var/pypi/simple/pip/pip-1.0.2.tar.gz

Once you have at least one package stored locally, it's time to configure apache. Here's a snippet from the virtual host I configured:

DocumentRoot /var/pypi/
ServerName pypi.example.com
 
Options +Indexes
 
RewriteEngine On
RewriteRule ^/robots.txt - [L]
RewriteRule ^/icons/.* - [L]
RewriteRule ^/index\..* - [L]
 
RewriteCond /var/pypi/$1 !-f
RewriteCond /var/pypi/$1 !-d
RewriteRule ^/(.*)/?$ http://pypi.python.org/$1 [R,L]

The last set of rewrite directives check to see if the request refers to an existing file or directory under your document root. If it does, your server will reply with a directory listing or with the actual file to download. If the directory or file doesn't exist, apache will send the client a redirection to the main PyPi site.

Reload your apache configuration to bring in your new changes. Let's try to download the pip tarball from our local server in the example I mentioned above:

$ curl -I http://pypi.example.com/simple/pip/
HTTP/1.1 200 OK
 
$ curl -I http://pypi.example.com/simple/pip/pip-1.0.2.tar.gz
HTTP/1.1 200 OK

I've obviously snipped a bit of the response above, but you can see that apache is responding with 200's since it has the directories and files that I was trying to retrieve via curl. Let's try to get something we don't have locally, like kombu:

$ curl -I http://pypi.example.com/simple/kombu/
HTTP/1.1 302 Found
Location: http://pypi.python.org/simple/kombu/

Our local PyPi repository doesn't have kombu so it will refer our Python tools over to the official PyPi repository to get the listing of available package versions for kombu.

Now we need to tell pip to use our local repository. Edit ~/.pip/pip.conf and add:

[global]
index-url = http://pypi.example.com/simple/

If you'd rather use easy_install, edit ~/.pydistutils.cfg and add:

[easy_install]
index_url = http://pypi.example.com/simple/

Once your tools are configured, try installing a package you have locally and try to install one that you know you won't have locally. You can add -v to pip install to watch it retrieve different URL's to get the packages it needs. If you spot any peculiar behavior or unexpected redirections, double-check your mod_rewrite rules in your apache configuration and check the spelling of your directories under your document root.

Create a local PyPi repository using only mod_rewrite is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

[Thu Sep 08 03:23:40 2011] [error] [client 11.22.33.44] PHP Warning:  
Unknown: Failed to write session data (memcached). Please verify that 
the current setting of session.save_path is correct (127.0.0.1:11211) 
in Unknown on line 0

I ran through my usual list of checks:

• netstat showed memcached bound to the correct ports/interfaces
• memcached was running and I could reach it via telnet
• memcached-tool could connect and pull stats from memcached
• double-checked my php.ini
• tested memcached connectivity via a PHP and ruby script -- they worked

Even after all that, I still couldn't figure out what was wrong. I ran strace on memcached while I ran a curl against the page which creates a session and I found something significant -- memcached wasn't seeing any connections whatsoever at that time. A quick check of the lo interface with tcpdump showed the same result. Just before I threw a chair, I remembered one thing:

SELinux.

A quick check for AVC denials showed the problem:

# aureport --avc | tail -n 1
4021. 09/08/2011 03:23:38 httpd system_u:system_r:httpd_t:s0 42 tcp_socket name_connect system_u:object_r:memcache_port_t:s0 denied 31536

I'm far from being a guru on SELinux, so I leaned on audit2allow for help:

# grep memcache /var/log/audit/audit.log | audit2allow
 
#============= httpd_t ==============
#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_relay, httpd_can_network_memcache, httpd_can_network_connect
 
allow httpd_t memcache_port_t:tcp_socket name_connect;

The boolean we're looking for is httpd_can_network_memcache. Flipping the boolean can be done in a snap:

# setsebool -P httpd_can_network_memcache 1
# getsebool httpd_can_network_memcache
httpd_can_network_memcache --> on

After adjusting the boolean, apache was able to make connections to memcached without a hitch. My page which created sessions loaded quickly and I could see data being stored in memcached. If you want to check the status of all of the apache-related SELinux booleans, just use getsebool:

# getsebool -a | grep httpd | grep off$
allow_httpd_anon_write --> off
allow_httpd_mod_auth_ntlm_winbind --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_sys_script_anon_write --> off
httpd_can_check_spam --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_read_user_content --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_tmp_exec --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off

If you're interested in SELinux, a good way to get your feet wet is to head over to the CentOS Wiki and review their SELinux Howtos

Getting apache, PHP, and memcached working with SELinux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

While many of the users have reasonable usage patterns, there are some users that query the site more than once per second from the same IP address. If you haven't used the site before, all it does is return your public IP address in plain text. Unless your IP changes rapidly, you may not need to query the site more than a few times an hour.

I added the following to my icanhazip.com virtual host definition to get the message across to those users that abuse the service:

ErrorDocument 403 "No can haz IP. Stop abusing this service. \
    Contact major at mhtx dot net for details."
RewriteEngine On
RewriteCond %{REMOTE_ADDR} ^12.23.34.45$ [OR]
RewriteCond %{REMOTE_ADDR} ^98.87.76.65$
RewriteRule .* nocanhaz [F]

The users that are caught on the business end of these 403 responses will see something like this:

$ curl -i icanhazip.com
HTTP/1.1 403 Forbidden
Date: Wed, 17 Nov 2010 13:42:55 GMT
Server: Apache
Content-Length: 84
Connection: close
Content-Type: text/html; charset=iso-8859-1
 
No can haz IP. Stop abusing this service. Contact major at mhtx dot net for details.

Throwing thoughtful "403 Forbidden" responses with apache is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The guide: Redundant Cloud Hosting Guide

As always, if you find anything in the guide that needs improvement, I'm all ears.

A simple guide to redundant cloud hosting is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Many cloud providers allow you access to a private network environment where you can exchange data with other instances or other services offered by the provider. The convenience of this access comes with a price: other instances can access your instance on the private network just as easily as they could on the public interface.

Here are some security tips for your private interfaces:

Disable the private interface
This one is pretty simple. If you have only one instance or server, and you don't need to communicate privately with any other instances, just disable the interface. Remember to configure your networking scripts to leave the interface disabled after reboots.

Use packet filtering
The actual mechanism will vary based on your operating system, but filtering packets is the one of the simplest ways to secure your private interface. You can take some different approaches with them, but I find the easiest method is to allow access from your other instances and reject all other traffic.

For additional security, you can limit access based on ports as well as source IP addresses. This could prevent an attacker from having easy access to your other instances if they're able to break into one of them.

Configure your daemons to listen on the appropriate interfaces
If there are services that don't need to be listening on the private network, don't allow them to listen on your private interface. For example, MySQL might need to listen on the private interface so the web server can talk to it, but apache won't need to listen on the private interface. This reduces the profile of your instance on the private network and makes it a less likely target for attack.

Use hosts.allow and hosts.deny
Many new systems administrators forget about how handy tcpwrappers can be for limiting access. If your firewall is down in error, host.allow and hosts.deny could be an extra layer of protection. It's important to ensure that the daemons you are attempting to control are build with tcpwrappers support. Daemons like sshd support it, but apache and MySQL do not.

Encrypt all traffic on the private network
Just because it's called a "private" network doesn't mean that your traffic can traverse the network privately. You should always err on the side of caution and encrypt all traffic traversing the private network. You can use ssh tunnels, stunnel, or the built-in SSL features found in most daemons.

This also brings up an important point: you should know how your provider's private network works. Are there safeguards to prevent sniffing? Could someone else possibly ARP spoof your instance's private IP addresses? Is your private network's subnet shared among many customers?

With all of that said, it's also very important to have proper change control policies so that administrators working after you are fully aware of the security measures in place and why they are important. This will ensure that all of the administrators on your instances will understand the security of the system and they should be able to make sensible adjustments later for future functionality.

Private network interfaces: the forgotten security hole is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

How do I become a better systems administrator?

The best way to become a better systems administrator is to fully understand the theory of what's happening in your server's environment.

What do I mean by that? Learn why things aren't happening as you expected and think about all of the factors that could possibly be involved. Instead of thinking purely about cause and effect, you'll find it much easier and rewarding to consider everything inside and outside your environment before you make any changes.

This still may be a little difficult to fully understand, so he's an example. Let's say you're handling an issue where a customer can't reach a website hosted on their server. When you ask them for more details, they might give you the dreaded reply: "It's not coming up." Start by making a mental list of the problems that are easiest to check:

• Is the web server daemon running?
• If a database server is being used, is it running and accessible?
• Is there a software/hardware firewall blocking port 80?
• Is a script stuck on the server tying up resources?
• Could there be a DNS resolution problem?
• Is the server up?
• Did a switch fail?
• Is the server's hard disk out of space?
• Can the customer reach other websites like Google or Yahoo?
• If SELinux is involved, have the appropriate contexts been set?
• Could the site be a target of a denial of service attack?
• Has the server reached its connection tracking limit?

Of course, this is a relatively short list, but these are all easy to check. If you're thinking about cause and effect, you might only consider the web server daemon and some basic network issues. By considering all of the other factors that may be related, you've ensured that all of the basics are covered before you consider more complex problems.

Most systems administrators have taken an error message and tossed it in en masse into Google before. Occasionally, no results will appear for the search. If you find yourself in this situation, try to understand the individual parts of the error message. Work outward from what you know already. You should know which daemon said it, and you may have an idea of what the application was doing when the error occurred. Take time to consider what the daemon is trying to tell you within the context of what it was doing at the time.

One of the easiest ways to force yourself to be immersed into this way of thinking is to host applications for non-technical people. You'll find that many customers want things done differently, and they're all at different levels of technical aptitude. Some may find it a frustrating experience at first, but you'll think yourself later. It will force you to consider all aspects of how a server operates since you might not always know what's happening within a customer's application.

As always, if you find yourself stumbling, remember to ask your peers and colleagues. Even if they haven't seen the particular issue, they will probably be able to guide you closer to the solution you seek.

A New Year System Administrator Inspiration is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

[root@f11 ~]# /etc/init.d/httpd start
Starting httpd: [Fri Aug 14 17:05:24 2009] [crit] (22)Invalid argument: alloc_listener: failed to get a socket for (null)
Syntax error on line 134 of /etc/httpd/conf/httpd.conf:
Listen setup failed
                                                           [FAILED]

The issue is related to three kernel calls that are used in apr-1.3.8-1: accept4(), dup3() and epoll_create1(). Without these calls, apache is unable to start.

Update on August 17, 2009: the Fedora team has pushed apr-1.3.8-2 into the stable repositories for Fedora 11, which eliminates the need for the temporary fix shown below.

Deprecated solution: There is a bug open with the Fedora team, and there is a temporary fix available:

yum --enablerepo=updates-testing update apr

Fedora 11 httpd: alloc_listener: failed to get a socket for (null) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Toss this into /etc/logrotate.d/yourrailsapplication:

/var/www/yourrailsapp/log/*.log {
  daily
  missingok
  rotate 30
  compress
  delaycompress
  sharedscripts
  postrotate
    touch /var/www/yourrailsapp/tmp/restart.txt
  endscript
}

For a detailed explanation, see the post on Overstimulate.

Rotating rails logs when using Phusion Passenger is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

127.0.0.1 - - [21/Aug/2008:12:00:10 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:11 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:13 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:14 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:15 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"

On servers with ipv6 enabled, you might see a line like this one:

::1 - - [21/Aug/2008:12:00:15 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"

I began to wonder why Apache was making these connections back onto itself and initiating a GET /. Apache's documentation had the following:

When the Apache HTTP Server manages its child processes, it needs a way to wake up processes that are listening for new connections. To do this, it sends a simple HTTP request back to itself. This request will appear in the access_log file with the remote address set to the loop-back interface (typically 127.0.0.1 or ::1 if IPv6 is configured). If you log the User-Agent string (as in the combined log format), you will see the server signature followed by "(internal dummy connection)" on non-SSL servers. During certain periods you may see up to one such request for each httpd child process.

These requests are perfectly normal and you do not, in general, need to worry about them. They can simply be ignored.

Sure, I could easily ignore the requests, but the requests were increasing the load on my server more than I liked. Apache's documentation suggested omitting the lines from the logs by adding the following to the Apache configuration:

SetEnvIf Remote_Addr "127\.0\.0\.1" loopback

And then adding env=!loopback to your CustomLog lines ensures that the data won't show up in your access logs. However, you'll still end up with Directory index forbidden by Options directive: /var/www/html/ filling up your error_logs. A quick search revealed a handy mod_rewrite rule to get rid of these requests as quickly as possible with the lowest effort required from Apache:

RewriteCond %{HTTP_USER_AGENT} ^.*internal\ dummy\ connection.*$ [NC]
RewriteRule .* - [F,L]

At this point, the requests to the localhost should receive a 403 immediately. Since you can't keep Apache from sending all of these requests to itself, the best you can do is respond to them in a manner that requires the lowest possible resources.

Apache 2.2: internal dummy connection is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Luckily, mod_deflate makes this easy, and the Apache documentation has a handy initial configuration available:

<Location />
SetOutputFilter DEFLATE
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
Header append Vary User-Agent env=!dont-vary
</Location>

This configuration will compress everything except for images. Of course, you can't test this with curl, but you can test it with Firefox and LiveHTTPHeaders. If you don't have Firefox handy, you can try a very handy web application that will give you the statistics about the compression of your site's data.

Compress your web content for better performance is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First, adjust the AddDefaultCharset directive in /etc/httpd/conf/httpd.conf:

#AddDefaultCharset UTF-8
AddDefaultCharset ISO-8859-1

Then, reload Apache and check your headers:

# /etc/init.d/httpd reload
# curl -I localhost
HTTP/1.1 403 Forbidden
Date: Thu, 08 Nov 2007 22:18:14 GMT
Server: Apache/2.0.52 (Red Hat)
Accept-Ranges: bytes
Content-Length: 3985
Connection: close
Content-Type: text/html; charset=ISO-8859-1

This was tested on Red Hat Enterprise Linux 4 Update 5

Change the default Apache character set is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Within the Apache error logs, this message appeared over and over:

[emerg] (28)No space left on device: Couldn't create accept lock

Apache is basically saying "I want to start, but I need to write some things down before I can start, and I have nowhere to write them!" If this happens to you, check these items in order:

1. Check your disk space
This comes first because it's the easiest to check, and sometimes the quickest to fix. If you're out of disk space, then you need to fix that problem.

2. Review filesystem quotas
If your filesystem uses quotas, you might be reaching a quota limit rather than a disk space limit. Use repquota / to review your quotas on the root partition. If you're at the limit, raise your quota or clear up some disk space. Apache logs are usually the culprit in these situations.

3. Clear out your active semaphores
Semaphores? What the heck is a semaphore? Well, it's actually an apparatus for conveying information by means of visual signals. But, when it comes to programming, semaphores are used for communicating between the active processes of a certain application. In the case of Apache, they're used to communicate between the parent and child processes. If Apache can't write these things down, then it can't communicate properly with all of the processes it starts.

I'd assume if you're reading this article, Apache has stopped running. Run this command as root:

# ipcs -s

If you see a list of semaphores, Apache has not cleaned up after itself, and some semaphores are stuck. Clear them out with this command:

# for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done

Now, in almost all cases, Apache should start properly. If it doesn't, you may just be completely out of available semaphores. You may want to increase your available semaphores, and you'll need to tickle your kernel to do so. Add this to /etc/sysctl.conf:

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

And then run sysctl -p to pick up the new changes.

Further reading:
Wikipedia: Semaphore (Programming)
Apache accept lock fix

Apache: No space left on device: Couldn't create accept lock is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.