However, certain folks may find the need to host a local PyPi repository for their own packages. You may need it to store Python code which you don't plan to release publicly or you may need to add proprietary patches to upstream Python packages. Regardless of the reason to have it, a local PyPi repository is relatively easy to configure.

You'll need to start with a base directory for your PyPi repository. For this example, I chose /var/pypi. The directory structure should look something like this:

/var/pypi/simple/[package_name]/[package_tarball]

For a package like pip, you'd make a structure like this:

/var/pypi/simple/pip/pip-1.0.2.tar.gz

Once you have at least one package stored locally, it's time to configure apache. Here's a snippet from the virtual host I configured:

DocumentRoot /var/pypi/
ServerName pypi.example.com
 
Options +Indexes
 
RewriteEngine On
RewriteRule ^/robots.txt - [L]
RewriteRule ^/icons/.* - [L]
RewriteRule ^/index\..* - [L]
 
RewriteCond /var/pypi/$1 !-f
RewriteCond /var/pypi/$1 !-d
RewriteRule ^/(.*)/?$ http://pypi.python.org/$1 [R,L]

The last set of rewrite directives check to see if the request refers to an existing file or directory under your document root. If it does, your server will reply with a directory listing or with the actual file to download. If the directory or file doesn't exist, apache will send the client a redirection to the main PyPi site.

Reload your apache configuration to bring in your new changes. Let's try to download the pip tarball from our local server in the example I mentioned above:

$ curl -I http://pypi.example.com/simple/pip/
HTTP/1.1 200 OK
 
$ curl -I http://pypi.example.com/simple/pip/pip-1.0.2.tar.gz
HTTP/1.1 200 OK

I've obviously snipped a bit of the response above, but you can see that apache is responding with 200's since it has the directories and files that I was trying to retrieve via curl. Let's try to get something we don't have locally, like kombu:

$ curl -I http://pypi.example.com/simple/kombu/
HTTP/1.1 302 Found
Location: http://pypi.python.org/simple/kombu/

Our local PyPi repository doesn't have kombu so it will refer our Python tools over to the official PyPi repository to get the listing of available package versions for kombu.

Now we need to tell pip to use our local repository. Edit ~/.pip/pip.conf and add:

[global]
index-url = http://pypi.example.com/simple/

If you'd rather use easy_install, edit ~/.pydistutils.cfg and add:

[easy_install]
index_url = http://pypi.example.com/simple/

Once your tools are configured, try installing a package you have locally and try to install one that you know you won't have locally. You can add -v to pip install to watch it retrieve different URL's to get the packages it needs. If you spot any peculiar behavior or unexpected redirections, double-check your mod_rewrite rules in your apache configuration and check the spelling of your directories under your document root.

Create a local PyPi repository using only mod_rewrite is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

While many of the users have reasonable usage patterns, there are some users that query the site more than once per second from the same IP address. If you haven't used the site before, all it does is return your public IP address in plain text. Unless your IP changes rapidly, you may not need to query the site more than a few times an hour.

I added the following to my icanhazip.com virtual host definition to get the message across to those users that abuse the service:

ErrorDocument 403 "No can haz IP. Stop abusing this service. \
    Contact major at mhtx dot net for details."
RewriteEngine On
RewriteCond %{REMOTE_ADDR} ^12.23.34.45$ [OR]
RewriteCond %{REMOTE_ADDR} ^98.87.76.65$
RewriteRule .* nocanhaz [F]

The users that are caught on the business end of these 403 responses will see something like this:

$ curl -i icanhazip.com
HTTP/1.1 403 Forbidden
Date: Wed, 17 Nov 2010 13:42:55 GMT
Server: Apache
Content-Length: 84
Connection: close
Content-Type: text/html; charset=iso-8859-1
 
No can haz IP. Stop abusing this service. Contact major at mhtx dot net for details.

Throwing thoughtful "403 Forbidden" responses with apache is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

127.0.0.1 - - [21/Aug/2008:12:00:10 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:11 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:13 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:14 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"
127.0.0.1 - - [21/Aug/2008:12:00:15 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"

On servers with ipv6 enabled, you might see a line like this one:

::1 - - [21/Aug/2008:12:00:15 -0400] "GET / HTTP/1.0" 200 2269 "-" "Apache/2.2.3 (Red Hat) (internal dummy connection)"

I began to wonder why Apache was making these connections back onto itself and initiating a GET /. Apache's documentation had the following:

When the Apache HTTP Server manages its child processes, it needs a way to wake up processes that are listening for new connections. To do this, it sends a simple HTTP request back to itself. This request will appear in the access_log file with the remote address set to the loop-back interface (typically 127.0.0.1 or ::1 if IPv6 is configured). If you log the User-Agent string (as in the combined log format), you will see the server signature followed by "(internal dummy connection)" on non-SSL servers. During certain periods you may see up to one such request for each httpd child process.

These requests are perfectly normal and you do not, in general, need to worry about them. They can simply be ignored.

Sure, I could easily ignore the requests, but the requests were increasing the load on my server more than I liked. Apache's documentation suggested omitting the lines from the logs by adding the following to the Apache configuration:

SetEnvIf Remote_Addr "127\.0\.0\.1" loopback

And then adding env=!loopback to your CustomLog lines ensures that the data won't show up in your access logs. However, you'll still end up with Directory index forbidden by Options directive: /var/www/html/ filling up your error_logs. A quick search revealed a handy mod_rewrite rule to get rid of these requests as quickly as possible with the lowest effort required from Apache:

RewriteCond %{HTTP_USER_AGENT} ^.*internal\ dummy\ connection.*$ [NC]
RewriteRule .* - [F,L]

At this point, the requests to the localhost should receive a 403 immediately. Since you can't keep Apache from sending all of these requests to itself, the best you can do is respond to them in a manner that requires the lowest possible resources.

Apache 2.2: internal dummy connection is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Go to your domain that you want to adjust, and click Setup. Make sure the CGI and FastCGI options are enabled. Pick a name for your application and make the directory for your application in the httpdocs directory. Upload your files to that directory.

Once you've done that, create an .htaccess file in the httpdocs directory with the following text inside:

RewriteEngine On
RewriteRule ^$ /public/index.html [L]
RewriteCond % !^/railsapp/public
RewriteRule ^(.*)$ /public/$1 [L]
RewriteCond % !-f
RewriteRule ^(.*)$ public/dispatch.fcgi/$1 [QSA,L]

Remove the .htaccess file within the public directory of your application and add a file called dispatch.fcgi to that directory which contains:

#!/usr/bin/ruby

You should be able to access your application at http://domain.com/railsapp/.

Enabling Ruby on Rails support for a domain in Plesk is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.