• pypi.mhtx.net

Python packages are updated moments after the commit is merged into the repositories under OpenStack's github account.

Although the packages will contain the latest code available, rest assured that the code has passed an initial code review (by humans), unit tests, and varying levels of functional or integrated testing. There may still be a bug or two cropping up after that, so be aware of that as you utilize these packages.

The package versions utilize a standard format:

[package]-[version]-[git commit count]-[short commit hash]

If you need to check the git log up to that particular commit, just run git log:

git log [short commit hash]

Instructions for configuring pip or easy_install are provided within the repository.

In addition, the repository is accessible via IPv4 and IPv6.

OpenStack bleeding-edge Python packages are now available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

However, certain folks may find the need to host a local PyPi repository for their own packages. You may need it to store Python code which you don't plan to release publicly or you may need to add proprietary patches to upstream Python packages. Regardless of the reason to have it, a local PyPi repository is relatively easy to configure.

You'll need to start with a base directory for your PyPi repository. For this example, I chose /var/pypi. The directory structure should look something like this:

/var/pypi/simple/[package_name]/[package_tarball]

For a package like pip, you'd make a structure like this:

/var/pypi/simple/pip/pip-1.0.2.tar.gz

Once you have at least one package stored locally, it's time to configure apache. Here's a snippet from the virtual host I configured:

DocumentRoot /var/pypi/
ServerName pypi.example.com
 
Options +Indexes
 
RewriteEngine On
RewriteRule ^/robots.txt - [L]
RewriteRule ^/icons/.* - [L]
RewriteRule ^/index\..* - [L]
 
RewriteCond /var/pypi/$1 !-f
RewriteCond /var/pypi/$1 !-d
RewriteRule ^/(.*)/?$ http://pypi.python.org/$1 [R,L]

The last set of rewrite directives check to see if the request refers to an existing file or directory under your document root. If it does, your server will reply with a directory listing or with the actual file to download. If the directory or file doesn't exist, apache will send the client a redirection to the main PyPi site.

Reload your apache configuration to bring in your new changes. Let's try to download the pip tarball from our local server in the example I mentioned above:

$ curl -I http://pypi.example.com/simple/pip/
HTTP/1.1 200 OK
 
$ curl -I http://pypi.example.com/simple/pip/pip-1.0.2.tar.gz
HTTP/1.1 200 OK

I've obviously snipped a bit of the response above, but you can see that apache is responding with 200's since it has the directories and files that I was trying to retrieve via curl. Let's try to get something we don't have locally, like kombu:

$ curl -I http://pypi.example.com/simple/kombu/
HTTP/1.1 302 Found
Location: http://pypi.python.org/simple/kombu/

Our local PyPi repository doesn't have kombu so it will refer our Python tools over to the official PyPi repository to get the listing of available package versions for kombu.

Now we need to tell pip to use our local repository. Edit ~/.pip/pip.conf and add:

[global]
index-url = http://pypi.example.com/simple/

If you'd rather use easy_install, edit ~/.pydistutils.cfg and add:

[easy_install]
index_url = http://pypi.example.com/simple/

Once your tools are configured, try installing a package you have locally and try to install one that you know you won't have locally. You can add -v to pip install to watch it retrieve different URL's to get the packages it needs. If you spot any peculiar behavior or unexpected redirections, double-check your mod_rewrite rules in your apache configuration and check the spelling of your directories under your document root.

Create a local PyPi repository using only mod_rewrite is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

One of the first things I wanted to tackle was understanding how a build request flows through Nova to a XenServer hypervisor. Following this process through the code is a bit tricky (I'm still learning python, so that could explain it). Here are the basic steps:

• Client requests a build via the API.
• The API runs some checks (quotas, auth, etc) and hands the build off to the scheduler.
• The scheduler figures out where the instance should go.
• The scheduler drops a message in queue specific to one compute node (where the instance will be built).
• The API responds to the client and the client is now unblocked and free to do other things.
• The compute node updates the database with the instance details and calls to the hypervisor to assemble block devices for the instance.
• A message is dropped into the network node's queue (from the compute node) to begin assembling networks for the instance. The compute node blocks and waits while this step completes.
• Once the networking details come back (via the queue), the compute node does the remaining adjustments on the hypervisor and starts up the actual instance.
• When the instance starts successfully (or fails to do so), the database is updated and a message is dropped onto another message queue as a notification that the build is complete.

Click on the thumbnail on the right to see the flow chart I created to explain this process.

Please note: This information should be accurate to the Nova code as of November 1, 2011. There could be some refactoring of these build processes before Essex is released.

Tracing a build through OpenStack Compute (Nova) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The day started out with Jared Smith's "State of Fedora" address. The audio has already been posted on the wiki, but the speech was very positive overall. He talked about some of the struggles that have happened in the past and how they'll probably happen again in some form or another. It was pretty inspirational and you could obviously tell that people in the room were energized by it.

After the address, all of the talks were pitched in BarCamp format. It was a very efficient and entertaining way to create a schedule for the conference. Everyone had 15-20 seconds to present their talk and then they had to rush outside to post their topic on the wall. We all had the opportunity to go outside and vote for the talks that sounded interesting. Once the votes were tallied, the schedule was set and the conference was fully underway.

The first talk for me was about Marek Goldmann's BoxGrinder. (Note: If you Google for BoxGrinder, make sure that you enter it as a single word. You'll get some wild unrelated results if you use two words.) In short, BoxGrinder gives you the ability to have a kickstart-ish method for automatically building images for virtual machine environments. It's completely plugin-based, so you can have different platform and delivery plugins depending on where your VM needs to be deployed. For example, you could deploy a VM with BoxGrinder that is in a format for VMWare (platform) and is delivered to the target server via SFTP (delivery). The public cloud plugins are only compatible with Amazon's products, but I'm eager to change that during one of the upcoming hackfests.

The Sheepdog talk started up right after lunch and although it was interesting, I think it left most people with quite a few questions when it was over. However, I think people are generally apprehensive when anyone tries to do anything innovative with storage. Losing data due to a bug is a big concern and many of the questions went deeper into data safety than performance and functionality.

Next up was Dave Malcolm's talk about the different implementations of python. This was definitely an eye-opening talk for my coworker and I. Dave covered CPython, Jython, PyPy and various other implementations and compared their advantages and disadvantages. I'm still pretty new to Python (I'm clutching on to ruby, PHP and perl still), but this talk really had me thinking about which implementations are best for a particular environment or task. It was quite a bit of fun to learn about some of the deep underpinnings of Python and how they differ depending on the specific implementation.

Jeff Darcy's talk about CloudFS was very intriguing. I've been a fan of GlusterFS recently, but I eventually moved away due to a lack of enterprise features and degrading performance. Jeff is working to add in encryption and authentication without rewriting the filesystem itself. There are quite a few tricky problems involved in the encryption portion due to partial writes and general security during the handshake process. CloudFS could potentially be a network filesystem which could be shared by multiple tenants with their own individual namespaces and segregated UID's. This could be a big win for providers as they could offer up large amounts of storage in an organized fashion without too many management headaches.

We wrapped up the day of talks with Chris Lalancette's presentation about Deltacloud. In short, it's a bag of daemons that allow you to manage multiple public or private clouds. Everything from image management to provisioning are included in the project. Questions were raised about whether another application was needed since vendor-specific libraries are abundant and libcloud offers many of the same features in a simpler package.

Tonight's social event was FUDPub at ASU's Memoral Union building. The food and drinks were excellent (thanks to Rackspace!) and it was a great opportunity to relax and talk with other Fedora users and developers. We had the opportunity to meet people from around the world while playing round after round of bowling and billiards. The discussions were extremely valuable, but as I said before, it was quite tiring.

I've compiled the FUDCon photos I've taken into a Flickr photo set.

That's the end of today's summary. I'll try to keep this going tomorrow as well. Thanks for reading!

FUDCon 2011: Day One is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The errors:

rpmdb: Lock table is out of available locker entries
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm

Make a backup of /var/lib/rpm in case you break something:

tar cvzf rpmdb-backup.tar.gz /var/lib/rpm

Remove the Berkeley databases that rpm uses:

rm /var/lib/rpm/__db.00*

Make rpm rebuild the databases from scratch (may take a short while):

rpm --rebuilddb

Now, check rpm to make sure everything is okay:

rpm -qa | sort

Why does this happen?
When rpm accesses the Berkeley database files, it makes temporary locker entries within the tables while it searches for data. If you control-c your rpm processes often, this issue will occur much sooner because the locks are never cleared.

rpmdb: Lock table is out of available locker entries is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.