Here's Kerberos in a nutshell within a two-server environment: One server is a Kerberos key distribution center (KDC) and the other is a Kerberos client. The KDC has the list of users and their passwords. Consider a situation where a user tries to ssh into the Kerberos client:

• sshd calls to pam to authenticate the user
• pam calls to the KDC for a ticket granting ticket (TGT) to see if the user can authenticate
• the KDC replies to the client with a TGT encrypted with the user's password
• pam (on the client) tries to decrypt the TGT with the password that the user provided via ssh
• if pam can decrypt the TGT, it knows the user is providing the right password

Now that the client has a a TGT for that user, it can ask for tickets to access other network services. What if the user who just logged in wants to access another Kerberized service in the environment?

• client calls the KDC and asks for a ticket to grant access to the other service
• KDC replies with two copies of the ticket:
  • one copy is encrypted with the user's current TGT
  • a second copy is encrypted with the password of the network service the user wants to access
• the client can decrypt the ticket which was encrypted with the current TGT since it has the TGT already
• client makes an authenticator by taking the decrypted ticket and encrypting it with a timestamp
• client passes the authenticator and the second copy of the ticket it received from the KDC
• the other network service decrypts the second copy of the ticket and verifies the password
• the other network service uses the decrypted ticket to decrypt the authenticator it received from the client
• if the timestamp looks good, the other network service allows the user access

Okay, that's confusing. Let's take it one step further. Enabling pre-authentication requires that clients send a request containing a timestamp encrypted with the user's password prior to asking for a TGT. Without this requirement, an attacker can ask for a TGT one time and then brute force the TGT offline. Pre-authentication forces the client to send a timestamped request encrypted with the user's password back to the KDC before they can ask for a KDC. This means the attacker is forced to try different passwords when encrypting the timestamp in the hopes that they'll get a TGT to work with eventually. One would hope that you have something configured on the KDC to set off an alarm for multiple failed pre-authentication attempts.

Oh, but we can totally kick it up another notch. What if an attacker is able to give a bad password to a client but they're also able to impersonate the KDC? They could reply to the TGT request (as the KDC) with a TGT encrypted with whichever password they choose and get access to the client system. Enabling mutual authentication stops this attack since it forces the client to ask the KDC for the client's own host principal password (this password is set when the client is configured to talk to the KDC). The attacker shouldn't have any clue what that password is and the attack will be thwarted.

By this point, you're either saying "Oh man, I don't ever want to do this." or "How do I set up Kerberos?". Stay tuned if you're in the second group. I'll have a dead simple (or as close to dead simple as one can get with Kerberos) how-to on the blog shortly.

In the meantime, here are a few links for extra Kerberos bedtime reading:

• Kerberos on Wikipedia
• MIT's "Why Kerberos" [PDF]
• How Kerberos Authentication Works

Kerberos for haters is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

• pypi.mhtx.net

Python packages are updated moments after the commit is merged into the repositories under OpenStack's github account.

Although the packages will contain the latest code available, rest assured that the code has passed an initial code review (by humans), unit tests, and varying levels of functional or integrated testing. There may still be a bug or two cropping up after that, so be aware of that as you utilize these packages.

The package versions utilize a standard format:

[package]-[version]-[git commit count]-[short commit hash]

If you need to check the git log up to that particular commit, just run git log:

git log [short commit hash]

Instructions for configuring pip or easy_install are provided within the repository.

In addition, the repository is accessible via IPv4 and IPv6.

OpenStack bleeding-edge Python packages are now available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

However, certain folks may find the need to host a local PyPi repository for their own packages. You may need it to store Python code which you don't plan to release publicly or you may need to add proprietary patches to upstream Python packages. Regardless of the reason to have it, a local PyPi repository is relatively easy to configure.

You'll need to start with a base directory for your PyPi repository. For this example, I chose /var/pypi. The directory structure should look something like this:

/var/pypi/simple/[package_name]/[package_tarball]

For a package like pip, you'd make a structure like this:

/var/pypi/simple/pip/pip-1.0.2.tar.gz

Once you have at least one package stored locally, it's time to configure apache. Here's a snippet from the virtual host I configured:

DocumentRoot /var/pypi/
ServerName pypi.example.com
 
Options +Indexes
 
RewriteEngine On
RewriteRule ^/robots.txt - [L]
RewriteRule ^/icons/.* - [L]
RewriteRule ^/index\..* - [L]
 
RewriteCond /var/pypi/$1 !-f
RewriteCond /var/pypi/$1 !-d
RewriteRule ^/(.*)/?$ http://pypi.python.org/$1 [R,L]

The last set of rewrite directives check to see if the request refers to an existing file or directory under your document root. If it does, your server will reply with a directory listing or with the actual file to download. If the directory or file doesn't exist, apache will send the client a redirection to the main PyPi site.

Reload your apache configuration to bring in your new changes. Let's try to download the pip tarball from our local server in the example I mentioned above:

$ curl -I http://pypi.example.com/simple/pip/
HTTP/1.1 200 OK
 
$ curl -I http://pypi.example.com/simple/pip/pip-1.0.2.tar.gz
HTTP/1.1 200 OK

I've obviously snipped a bit of the response above, but you can see that apache is responding with 200's since it has the directories and files that I was trying to retrieve via curl. Let's try to get something we don't have locally, like kombu:

$ curl -I http://pypi.example.com/simple/kombu/
HTTP/1.1 302 Found
Location: http://pypi.python.org/simple/kombu/

Our local PyPi repository doesn't have kombu so it will refer our Python tools over to the official PyPi repository to get the listing of available package versions for kombu.

Now we need to tell pip to use our local repository. Edit ~/.pip/pip.conf and add:

[global]
index-url = http://pypi.example.com/simple/

If you'd rather use easy_install, edit ~/.pydistutils.cfg and add:

[easy_install]
index_url = http://pypi.example.com/simple/

Once your tools are configured, try installing a package you have locally and try to install one that you know you won't have locally. You can add -v to pip install to watch it retrieve different URL's to get the packages it needs. If you spot any peculiar behavior or unexpected redirections, double-check your mod_rewrite rules in your apache configuration and check the spelling of your directories under your document root.

Create a local PyPi repository using only mod_rewrite is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Later on, I found that one of my servers (which I'd previously secured quite thoroughly) had some rogue processes running that were spawned through httpd. Had I actually been using SELinux in enforcing mode, those processes would have probably never even started.

If you're trying to get started with SELinux but you're not sure how to do it without completely disrupting your server's workflow, these tips should help:

Get some good reporting and monitoring
Two of the most handy SELinux tools are setroubleshoot and setroubleshoot-server. If you're running a server without X, you can use my guide for configuring setroubleshoot-server. You will receive email alerts within seconds of an AVC denial and the emails should contain tips on how to resolve the denial if the original action should be allowed. If the AVC denial caught something you didn't expect, you'll know about the potential security breach almost immediately.

Start out with SELinux in permissive mode
If you're overly concerned about SELinux getting in your way, or if you're enabling SELinux on a server that has been running without SELinux since it was installed, start out with SELinux in permissive mode. To make the change effective immediately, just run:

# setenforce 0
# getenforce
Permissive

Edit /etc/sysconfig/selinux to make it persistent across reboots:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=permissive

Adjust booleans before adding your own custom modules
There are a lot of booleans you can toggle to get the functionality you need without adding your own custom SELinux modules with audit2allow. If you wanted to see all of the applicable booleans for httpd, just use getsebool:

# getsebool -a | grep httpd
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_network_connect --> on
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> on
httpd_can_sendmail --> on
... and so on ...

Toggling booleans is easy with togglesebool:

# togglesebool httpd_can_network_memcache
httpd_can_network_memcache: active

Now httpd can talk to memcache. You can also use setsebool if you want to be specific about your setting (this is good for scripts):

# setsebool httpd_can_network_memcache on

Tracking your history of AVC denials
All of your AVC denals are logged by auditd in /var/log/audit/audit.log but it's not the easiest file to read and parse. That's where aureport comes in:

# aureport --avc | tail -n 5
45. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 4 fifo_file getattr system_u:object_r:postfix_public_t:s0 denied 1061
46. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 2 fifo_file write system_u:object_r:postfix_public_t:s0 denied 1062
47. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 2 fifo_file open system_u:object_r:postfix_public_t:s0 denied 1062
48. 01/24/2012 14:01:58 sendmail unconfined_u:system_r:httpd_t:s0 160 process setrlimit unconfined_u:system_r:httpd_t:s0 denied 1123
49. 01/24/2012 14:01:58 postdrop unconfined_u:system_r:httpd_t:s0 4 dir search system_u:object_r:postfix_public_t:s0 denied 1124

Summary
There's no need to be scared of or be annoyed by SELinux in your server environment. While it takes some getting used to (and what new software doesn't?), you'll have an extra layer of security and access restrictions which should let you sleep a little better at night.

Getting started with SELinux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Before you get started, go through the steps to disable GPT. That post also explains an optional adjustment to get a larger root partition (which I would recommend). You cannot complete the steps in this post if your XenServer installation uses GPT.

You should have three partitions on your first disk after the installation:

# fdisk -l /dev/sda
-- SNIP --
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        2611    20971520   83  Linux
/dev/sda2            2611        5222    20971520   83  Linux
/dev/sda3            5222       19457   114345281   8e  Linux LVM

Here's a quick explanation of your partitions:

• /dev/sda1: the XenServer root partition
• /dev/sda2: XenServer uses this partition for temporary space during upgrades
• /dev/sda3: your storage repository should be in this logical volume

We need to replicate the same partition structure across each of your drives and the software RAID volume will span the across the third partition on each disk. Copying the partition structure from disk to disk is done easily with sfdisk:

WHOA THERE! NO TURNING BACK! This step is destructive! If your other disks have any data on them, this step will make it (relatively) impossible to retrieve data on those disks again. Back up any data on the other disks in your XenServer machine before running these next commands.

sfdisk -d /dev/sda | sfdisk --force /dev/sdb
sfdisk -d /dev/sda | sfdisk --force /dev/sdc
sfdisk -d /dev/sda | sfdisk --force /dev/sdd

If you have only two disks, stop with /dev/sdb and you'll be making a RAID 1 array. My machine has four disks and I'll be making a RAID 10 array.

We need to destroy the main storage repository, but we need to unplug the physical block device first. Get the storage repository uuid first, then use it to find the corresponding physical block device. Once the physical block device is unplugged, the storage repository can be destroyed:

# xe sr-list name-label=Local\ storage | head -1
uuid ( RO)                : 75264965-f981-749e-0f9a-e32856c46361
# xe pbd-list sr-uuid=75264965-f981-749e-0f9a-e32856c46361 | head -1
uuid ( RO)                  : ff7e9656-c27c-1889-7a6d-687a561f0ad0
# xe pbd-unplug uuid=ff7e9656-c27c-1889-7a6d-687a561f0ad0 
# xe sr-destroy uuid=75264965-f981-749e-0f9a-e32856c46361

All of the LVM data from /dev/sda3 should now be gone:

# lvdisplay && vgdisplay && pvdisplay
#

Change the third partition on each physical disk to be a software RAID partition type:

echo -e "t\n3\nfd\nw\n" | fdisk /dev/sda
echo -e "t\n3\nfd\nw\n" | fdisk /dev/sdb
echo -e "t\n3\nfd\nw\n" | fdisk /dev/sdc
echo -e "t\n3\nfd\nw\n" | fdisk /dev/sdd

Stop here and reboot your XenServer box to pick up the new partition changes. Once the server comes back from the reboot, start up a software RAID volume with mdadm:

// RAID 1 for two drives
mdadm --create /dev/md0 -l 1 -n 2 /dev/sda3 /dev/sdb3
// RAID 10 for four drives
mdadm --create /dev/md0 -l 10 -n 4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

Check to see that your RAID array is building:

# cat /proc/mdstat 
Personalities : [raid10] 
md0 : active raid10 sdd3[3] sdc3[2] sdb3[1] sda3[0]
      228690432 blocks 64K chunks 2 near-copies [4/4] [UUUU]
      [>....................]  resync =  0.3% (694272/228690432) finish=16.4min speed=231424K/sec

Although you don't have to wait for the resync to complete, just be aware that XenServer doesn't do well with a lot of disk I/O within dom0. You may notice unusually slow performance in dom0 until it finishes. Save the array's configuration for reboots:

mdadm --detail --scan > /etc/mdadm.conf

Edit the /etc/mdadm.conf file and append auto=yes to the end of the line (but leave everything on one line):

ARRAY /dev/md0 level=raid10 num-devices=4 metadata=0.90 \
  UUID=2876748c:5117eed5:ce4d62d3:9592bd84 auto=yes

Create a new storage repository on the RAID volume with thin provisioning (thanks to Spherical Chicken for the command):

xe sr-create content-type=user type=ext device-config:device=/dev/md0 shared=false name-label="Local storage"

This command takes some time to complete since it makes logical volumes and then makes an ext3 filesystem for the new storage repository. Bigger RAID arrays will take more time and it's guaranteed to take longer than you'd expect if your RAID array is still building. As soon as it completes, you'll be given the uuid of your new storage repository and it should appear within the XenCenter interface.

TIP: If you run into any problems during reboots, open /boot/extlinux.conf and remove splash and quiet from the label xe boot section. This removes the framebuffer during boot-up and it causes a lot more output to be printed to the console. It won't affect the display once your XenServer box has fully booted.

XenServer 6: Storage repository on software RAID is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I'd rather use MBR partition tables and get a larger root partition. If you want to make these adjustments in your XenServer 6 installation, follow these steps after booting into the XenServer 6 install disc:

When the installer initially boots, press F2 to access the advanced installation options.

Type shell and press enter. The installer should begin booting into a pre-installation shell where you can make your adjustments.

Once you've booted into the pre-installation shell, type vi /opt/xensource/installer/constants.py and press enter.

Change GPT_SUPPORT = True to GPT_SUPPORT = False to disable GPT and use MBR partition tables. Adjust the value of root_size from 4096 (the default) to a larger number to get a bigger root partition. The size is specified in MB, so 4096 is 4GB. Save the file and exit vim.

Type exit and the installer should start.

Once the installation is complete, you should have a bigger root partition on a MBT partition table:

# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              20G  1.8G   17G  10% /
# fdisk -l /dev/sda
 
Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        2611    20971520   83  Linux
/dev/sda2            2611        5222    20971520   83  Linux
/dev/sda3            5222       19457   114345281   8e  Linux LVM

XenServer 6: Disable GPT and get a larger root partition is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Get informed about what the US government is trying to accomplish with SOPA and PIPA. Get involved and do what you can to prevent it from moving any further.

• WordPress > Help Stop SOPA/PIPA
• Reddit - SOPA
• Stop American Censorship
• Wikipedia - Stop Online Piracy Act
• EFF - Take Action
• Why Rackspace opposes the Stop Online Piracy Act
• Mozilla - Protect The Internet

You may need a little humor after all of that reading. Head on over to Know Your Meme (warning: NSFW language in certain areas) for a laugh.

Fight SOPA & PIPA is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I recently added a RB493G (~ $230 USD) to sit in front of a pair of colocated servers. The majority of the setup routine was the same as with my previous devices except for the IPv6 configuration.

In the past, I've set up IPv6 tunnels with Hurricane Electric and it's been mostly a cut-and-paste operation from the sample configuration in their IPv6 tunnel portal. Setting up native IPv6 involved a little more legwork.

If your provider will give you two /64's or an entire /48, getting IPv6 connectivity for your WAN/LAN interfaces is simple. However, if you can only get one /64, you'll have to see if your provider can route it to you via your Mikrotik's link local interface (I wouldn't recommend this for many reasons).

I split my Mikrotik into two interfaces: wan and lanbridge. The lanbridge bridge joins all of the LAN ethernet ports (ether2-9 on the RB493G) and the wan interface connects to the upstream switch.

My configuration:

/ipv6 address
add address=2001:DB8:0:1::2/64 advertise=yes disabled=no eui-64=no interface=wan
add address=2001:DB8:0:2::1/64 advertise=yes disabled=no eui-64=no interface=lanbridge
/ipv6 route
add disabled=no distance=1 dst-address=::/0 gateway=2001:DB8:0:1::1 scope=30 \
  target-scope=10
/ipv6 nd
add advertise-dns=no advertise-mac-address=yes disabled=no hop-limit=64 \ 
  interface=all managed-address-configuration=no mtu=unspecified \
  other-configuration=no ra-delay=3s ra-interval=3m20s-10m ra-lifetime=30m \ 
  reachable-time=unspecified retransmit-interval=unspecified
/ipv6 nd prefix default
set autonomous=yes preferred-lifetime=1w valid-lifetime=4w2d

Explanation:

/ipv6 address
add address=2001:DB8:0:1::2/64 advertise=yes disabled=no eui-64=no interface=wan
add address=2001:DB8:0:2::1/64 advertise=yes disabled=no eui-64=no interface=lanbridge

These two lines configure the IPv6 addresses for the firewall's interfaces. My provider's router holds the 2001:DB8:0:1::1/64 address and routes the remainder of that /64 to me via 2001:DB8:0:1::2/64. The second /64 is on the lanbridge interface and my LAN devices take their IP addresses from that block. My provider routes that second /64 to me via the 2001:DB8:0:1::2/64 IP on my wan interface.

/ipv6 route
add disabled=no distance=1 dst-address=::/0 gateway=2001:DB8:0:1::1 scope=30 \
  target-scope=10

I've set a gateway for IPv6 traffic so that the Mikrotik knows where to send internet-bound IPv6 traffic (in this case, to my ISP's core router).

/ipv6 nd
add advertise-dns=no advertise-mac-address=yes disabled=no hop-limit=64 \ 
  interface=lanbridge managed-address-configuration=no mtu=unspecified \
  other-configuration=no ra-delay=3s ra-interval=3m20s-10m ra-lifetime=30m \ 
  reachable-time=unspecified retransmit-interval=unspecified
/ipv6 nd prefix default
set autonomous=yes preferred-lifetime=1w valid-lifetime=4w2d

These last two lines configure the neighbor discovery on my lanbridge interface. This allows my LAN devices to do stateless autoconfiguration (which gives them an IPv6 address as well as the gateway).

Want to read up on IPv6?

• Linux IPv6 HOWTO
• IPv6 on Wikipedia
• IPv6 Cheat Sheet [PDF]
• IPv6 Subnetting Card

Native IPv6 connectivity in Mikrotik's RouterOS is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

echo "user@example.com" > /root/.forward

With SELinux, you'll end up getting an AVC denial each time your mail server tries to read the contents of the .forward file:

type=AVC msg=audit(1325543823.787:7416): avc:  denied  { open } for  pid=9850 
  comm="local" name=".forward" dev=md0 ino=17694734 
  scontext=system_u:system_r:postfix_local_t:s0 
  tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file

The reason is that your .forward file doesn't have the right SELinux contexts. You can set the correct contest quickly with restorecon:

# ls -Z /root/.forward 
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 /root/.forward
# restorecon -v /root/.forward
restorecon reset /root/.forward context unconfined_u:object_r:admin_home_t:s0->system_u:object_r:mail_forward_t:s0
# ls -Z /root/.forward 
-rw-r--r--. root root system_u:object_r:mail_home_t:s0 /root/.forward

Try to send another e-mail to root and you should see the mail server forward the e-mail properly without any additional AVC denials.

SELinux and .forward files is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

That led me to pick up a CradlePoint PHS-300 (discontinued by the manufacturer, but available from Amazon for about $35). It's compatible with my AT&T USBConnect Mercury (a.k.a. Sierra Wireless Compass 885/885U) USB stick.

Configuring the PHS-300 was extremely easy since I could just associate with the wireless network and enter the password printed on the bottom of the unit. However, getting the 3G stick to work was an immense pain. If you're trying to pair up these products, these steps should help:

• Access the PHS-300's web interface
• Click the Modem tab
• Click Settings on the left
• Click Always on under Reconnect Mode
• Uncheck Aggressive Modem Reset
• Put the following into the AT Dial Script text box:

  ATE0V1&F&D2&C1S0=0
  ATDT*99***1#

• Add ISP.CINGULAR to the Access Point Name (APN) box
• Flip the Connect Mode under Dual WiMAX/3G Settings to 3G Only
• Scroll up and push Save Settings and then Reboot Now

Once the PHS-300 reboots, the USB stick may light up, then turn off, and the display on the PHS-300 might show a red light for the 3G card. Wait about 10-15 seconds for the light to turn green. The lights on the 3G stick should be glowing and blinking as well.

So how did I figure this out?

After scouring Google search results, Sierra Wireless FAQ's, CradlePoint's support pages, and trolling through minicom (yes, minicom), I thought I'd try connecting with my MacBook Pro using the 3G Watcher application provided by Sierra Wireless. Before connecting, I opened up Console.app and watched the ppp.log file. Sure enough, two lines popped up that were quite relevant to my interests:

Fri Dec 16 00:37:51 2011 : Initializing phone: ATE0V1&F&D2&C1S0=0
Fri Dec 16 00:37:51 2011 : Dialing: ATDT*99***1#

I didn't have the exact initialization string in the PHS-300 and that was the cause of the failure the entire time.

If you'd like to talk to your USBConnect Mercury stick with minicom, just install minicom from macports (sudo port -v install minicom) and start it up like so:

sudo minicom -D /dev/cu.sierra04

For other Sierra Wireless cards and adapters, there's a helpful page on Sierra Wireless' site for Eee PC users.

Getting online with a CradlePoint PHS-300 and an AT&T USBConnect Mercury is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Getting it to behave like Red Hat Enterprise Linux and CentOS is a painless process. Just install the sl6x repository with yum:

yum install yum-conf-sl6x

Check to ensure that you're getting updates from the new repository:

# yum repolist
repo id            repo name                                              status
sl                 Scientific Linux 6.1 - x86_64                          6,251
sl-security        Scientific Linux 6.1 - x86_64 - security updates         548
sl6x               Scientific Linux 6x - x86_64                           6,251
sl6x-security      Scientific Linux 6x - x86_64 - security updates          548
repolist: 13,598

Automatically upgrading to new point releases of Scientific Linux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The IOUSBFamily is having trouble enumerating a USB device that has been plugged in.  It will keep retrying.  (Port 4 of Hub at 0xfa100000)
The IOUSBFamily was not able to enumerate a device.
The IOUSBFamily is having trouble enumerating a USB device that has been plugged in.  It will keep retrying.  (Port 4 of Hub at 0xfa100000)
The IOUSBFamily was not able to enumerate a device.
The IOUSBFamily is having trouble enumerating a USB device that has been plugged in.  It will keep retrying.  (Port 4 of Hub at 0xfa100000)
The IOUSBFamily gave up enumerating a USB device after 10 retries.  (Port 4 of Hub at 0xfa100000)
The IOUSBFamily was not able to enumerate a device.

The solution is a bit goofy, but here's what you can do:

1. Unplug the adapter from the USB port.
2. Disconnect the DVI cable from the DisplayLink adapter.
3. Power off the display you normally use with the adapter.
4. Connect the USB cable between your computer and the DisplayLink adapter.
5. Wait for your displays to flash (as if a new display was connected).
6. The light on your DisplayLink adapter should be on now.
7. Connect the DVI cable to the DisplayLink adapter.
8. Wait a few seconds and then power on the display connected to the adapter.

If this process doesn't work, try a reboot and repeat the process once Finder finishes starting up.

DisplayLink USB to DVI issues in OS X Lion is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Upgrading Fedora via yum is not the recommended method. Your first choice for upgrading Fedora should be to use preupgrade. Seriously.

This begs the question: When should you use another method to upgrade Fedora? What other methods are there?

You have a few other methods to get the upgrade done:

• Toss in a CD or DVD: You can upgrade via the anaconda installer provided on the CD, DVD or netinstall media. My experiences with this method for Fedora (as well as CentOS, Scientific Linux, and Red Hat) haven't been too positive, but your results may vary.
• Download the newer release's fedora-release RPM, install it with rpm, and yum upgrade: This is the really old way of doing things. Don't try this (read the next bullet).
• Use yum's distro-sync functionality: If you can't go the preupgrade route, I'd recommend giving this a try. However, leave plenty of time to fix small glitches after it's done (and after your first reboot).

Personal anecdote time (Keep scrolling for the meat and potatoes)
I have a dedicated server at Joe's Datacenter (love those folks) with IPMI and KVM-over-LAN access. The preupgrade method won't work for me because my /boot partition is on a software RAID volume. There's a rat's nest of a Bugzilla ticket over on Red Hat's site about this problem. I'm really only left with a live upgrade using yum.

Live yum upgrade process
Before even beginning the upgrade, I double-checked that I'd applied all of the available updates for my server. Once that was done, I realized I was one kernel revision behind and I rebooted to ensure I was in the latest Fedora 15 kernel.

A good practice here is to run package-cleanup --orphans (it's in the yum-utils package) to find any packages which don't exist on any Fedora mirrors. In my case, I had two old kernels and a JungleDisk package. I removed the two old kernels (probably wasn't necessary) and left JungleDisk alone (it worked fine after the upgrade). If you have any external repositories, such as Livna or RPMForge, you may want to disable those until the upgrade is done. Should the initial upgrade checks bomb out, try adding as few repositories back in as possible to see if it clears up the problem.

Once you make it this far, just follow the instructions available in Fedora's documentation: Upgrading Fedora using yum. I set SELinux to permissive mode during the upgrade just in case it caused problems.

I'd recommend skipping the grub2-install portion since your original grub installation will still be present after the upgrade. If your server has EFI (not BIOS), don't use grub2 yet. Keep an eye on the previously mentioned documentation page to see if the problems get ironed out between grub2 and EFI.

Before you reboot, be sure to get a list of your active processes and daemons. After your reboot, some old SysVinit scripts will be converted into Systemd service scripts. They might not start automatically and you might need to enable and/or start some services.

New to Systemd? This will be an extremely handy resource: SysVinit to Systemd Cheatsheet.

I haven't seen too many issues after cleaning up some daemons that didn't start properly. There is a problem between asterisk and SELinux that I haven't nailed down yet but it's not a showstopper.

Good luck during your upgrades. Keep in mind that Fedora 15 could be EOL'd as early as May or June 20102 when Fedora 17 is released.

Live upgrade Fedora 15 to Fedora 16 using yum is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

One of the first things I wanted to tackle was understanding how a build request flows through Nova to a XenServer hypervisor. Following this process through the code is a bit tricky (I'm still learning python, so that could explain it). Here are the basic steps:

• Client requests a build via the API.
• The API runs some checks (quotas, auth, etc) and hands the build off to the scheduler.
• The scheduler figures out where the instance should go.
• The scheduler drops a message in queue specific to one compute node (where the instance will be built).
• The API responds to the client and the client is now unblocked and free to do other things.
• The compute node updates the database with the instance details and calls to the hypervisor to assemble block devices for the instance.
• A message is dropped into the network node's queue (from the compute node) to begin assembling networks for the instance. The compute node blocks and waits while this step completes.
• Once the networking details come back (via the queue), the compute node does the remaining adjustments on the hypervisor and starts up the actual instance.
• When the instance starts successfully (or fails to do so), the database is updated and a message is dropped onto another message queue as a notification that the build is complete.

Click on the thumbnail on the right to see the flow chart I created to explain this process.

Please note: This information should be accurate to the Nova code as of November 1, 2011. There could be some refactoring of these build processes before Essex is released.

Tracing a build through OpenStack Compute (Nova) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

-E, -S, -save-temps and -M options are not allowed with multiple -arch flags

Sure enough, when I looked at the lines in the output, both x86_64 and i386 were passed to the compiler:

... -pipe -O2 -arch x86_64 -arch i386 -fno-common ...

I opened a ticket in trac and began looking for a workaround. Another trac ticket (from four years ago) on the MacPorts site gave some pointers on how to work around the bug for a previous version.

I changed up the instructions a bit since we're not dealing with the ppc architecture any longer:

sudo port -v clean irssi +perl
sudo port -v configure irssi +perl
cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_irc_irssi/irssi/work/
sudo find . -type f -exec sed -i "" -e "s/-arch i386//g" {} \;
cd
sudo port -v install irssi +perl

The build worked!

$ irssi -v
irssi 0.8.15 (20100403 1617)

Installing irssi via MacPorts on OS X Lion 10.7.1 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

For example, all onboard ethernet adapters are labeled as emX (em1, em2...) and all PCI ethernet adapters are labeled as pXpX (p[slot]p[port], like p7p1 for port 1 on slot 7). Ethernet devices within Xen virtual machines aren't adjusted.

This may make sense to people who swap out the chassis on servers regularly and they don't want to mess with hard-coding MAC addresses in network configuration files. Also, it should give users predictable names even if a running system's drives are inserted into a newer hardware revision of the same server.

However, I don't like this on my personal dedicated servers and I prefer to revert back to the old way of doing things. Getting back to eth0 is pretty simple and it only requires a few configuration files to be edited followed by a reboot.

First, add biosdevname=0 to your grub.conf on the kernel line:

title Fedora (2.6.40.4-5.fc15.x86_64)
	root (hd0,0)
	kernel /boot/vmlinuz-2.6.40.4-5.fc15.x86_64 ro root=/dev/md0 SYSFONT=latarcyrheb-sun16 KEYTABLE=us biosdevname=0 quiet LANG=en_US.UTF-8
	initrd /boot/initramfs-2.6.40.4-5.fc15.x86_64.img

Open /etc/udev/rules.d/70-persistent-net.rules in your favorite text editor (create it if it doesn't exist) and add in the following:

# Be sure to put your MAC addresses in the fields below
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:10", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:11", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Be sure to rename your ifcfg-* files in /etc/sysconfig/network-scripts/ to match the device names you've assigned. Just for good measure, I add in the MAC address in /etc/sysconfig/network-scripts/ifcfg-ethX:

...
HWADDR=00:11:22:33:44:10
...

Reboot the server and you should be back to eth0 and eth1 after a reboot.

Getting back to using eth0 in Fedora 15 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The purpose behind setroubleshoot is to let users know when access has been denied, help them resolve it if necessary, and to reduce overall frustration while working through tight security restrictions in the default SELinux policies. The GUI frontend for setroubleshoot is great for users who run Linux desktops or those who run servers with a display attached. Don't worry, you can configure setroubleshoot on remote servers to send alerts elsewhere when a GUI alert isn't an option.

Install a few packages to get started:

yum install setroubleshoot{-server,-plugins,-doc}

Open /etc/setroubleshoot/setroubleshoot.conf in your favorite text editor and adjust the [email] section to fit your server:

recipients_filepath = /var/lib/setroubleshoot/email_alert_recipients
smtp_port = 25
smtp_host = localhost
from_address = selinux@myserver.com
subject = [MyServer] SELinux AVC Alert

You could probably see it coming, but you need to put the e-mail addresses for your recipients into /var/lib/setroubleshoot/email_alert_recipients:

echo "selinux@mycompany.com" >> /var/lib/setroubleshoot/email_alert_recipients

You'll notice that setroubleshoot doesn't have an init script and it doesn't exist in systemd in Fedora 15. It runs through the dbus-daemon and a quick bounce of the messagebus via its init script brings in the necessary components to run setroubleshoot:

service messagebus restart

A really easy (and safe) test is to ask sshd to bind to a non-standard port. Simply define an additional port on in your /etc/ssh/sshd_config like this:

Port 22
Port 222

When you restart sshd, it will bind to port 22 with success, but it won't be allowed to bind to port 222 (since that's blocked by SELinux as a non-standard port for the ssh_port_t port type). DON'T WORRY! Your sshd server will still be listening on port 22. If you wait a moment, you'll get an e-mail (perhaps two) that not only notify you of the denial, but they make suggestions for how to fix it:

SELinux is preventing /usr/sbin/sshd from name_bind access on the tcp_socket port 222.
 
*****  Plugin bind_ports (99.5 confidence) suggests  *************************
 
If you want to allow /usr/sbin/sshd to bind to network port 222
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 222
   where PORT_TYPE is one of the following: ...

For this particular example, the quick fix would be to run:

semanage port -a -t ssh_port_t -p tcp 222

Receive e-mail reports for SELinux AVC denials is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

[Thu Sep 08 03:23:40 2011] [error] [client 11.22.33.44] PHP Warning:  
Unknown: Failed to write session data (memcached). Please verify that 
the current setting of session.save_path is correct (127.0.0.1:11211) 
in Unknown on line 0

I ran through my usual list of checks:

• netstat showed memcached bound to the correct ports/interfaces
• memcached was running and I could reach it via telnet
• memcached-tool could connect and pull stats from memcached
• double-checked my php.ini
• tested memcached connectivity via a PHP and ruby script -- they worked

Even after all that, I still couldn't figure out what was wrong. I ran strace on memcached while I ran a curl against the page which creates a session and I found something significant -- memcached wasn't seeing any connections whatsoever at that time. A quick check of the lo interface with tcpdump showed the same result. Just before I threw a chair, I remembered one thing:

SELinux.

A quick check for AVC denials showed the problem:

# aureport --avc | tail -n 1
4021. 09/08/2011 03:23:38 httpd system_u:system_r:httpd_t:s0 42 tcp_socket name_connect system_u:object_r:memcache_port_t:s0 denied 31536

I'm far from being a guru on SELinux, so I leaned on audit2allow for help:

# grep memcache /var/log/audit/audit.log | audit2allow
 
#============= httpd_t ==============
#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_relay, httpd_can_network_memcache, httpd_can_network_connect
 
allow httpd_t memcache_port_t:tcp_socket name_connect;

The boolean we're looking for is httpd_can_network_memcache. Flipping the boolean can be done in a snap:

# setsebool -P httpd_can_network_memcache 1
# getsebool httpd_can_network_memcache
httpd_can_network_memcache --> on

After adjusting the boolean, apache was able to make connections to memcached without a hitch. My page which created sessions loaded quickly and I could see data being stored in memcached. If you want to check the status of all of the apache-related SELinux booleans, just use getsebool:

# getsebool -a | grep httpd | grep off$
allow_httpd_anon_write --> off
allow_httpd_mod_auth_ntlm_winbind --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_sys_script_anon_write --> off
httpd_can_check_spam --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_read_user_content --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_tmp_exec --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off

If you're interested in SELinux, a good way to get your feet wet is to head over to the CentOS Wiki and review their SELinux Howtos

Getting apache, PHP, and memcached working with SELinux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Thanks to @pinojo for snapping the photo.

You might see me on CNN Money soon is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you're not technical, you might be asking yourself: "I need to e-mail technical people and they need to take what I say seriously? How do I do it?" It's not impossible, but the rest of this blog post should help.

Brevity is key

There are some people who thrive on receiving e-mail, sending e-mail, and talking about e-mail that they've sent or received. Most nerds don't feel this way.

You need to get your point across concisely and succinctly so that your e-mail is seen as less of a distraction. Avoid adding a lot of context where it isn't needed and try to summarize business needs and processes unless details are absolutely critical. If you need to send your e-mail to multiple recipients and some of those recipients need additional details, provide an abstract at the beginning of the e-mail.

Learn the ways of TL;DR

I've heard quite a few conversations like these around the office:

Nerd 1: "Did you get that e-mail from [name here]?"
Nerd 2: "The six page one with four PDF files attached?"
Nerd 1: "Yeah. That one."
Nerd 2: "TL;DR dude, seriously. Did you read it?"
Nerd 1: "Nah. I might read it later."

If someone's ever mentioned "TL;DR" (too long; didn't read) when your e-mail was mentioned, don't fret. It's a quick fix. Just add a quick summary to the top of your e-mail prefaced with "TL;DR". Provide a really brief summary (bulleted lists are a plus) of your e-mail in the section and then start your e-mail right afterwards. Here's an example:

TL;DR
  * next software release deploys Monday
  * two bugs remaining to fix
  * we will get started at 8AM Saturday, yeaaaaah

Missed the joke? Head over to Wikipedia.

If one of the summary points interests a recipient, they'll scan your e-mail for the pertinent sections. Some recipients may only need to see what's in the summary and they won't bother reading the remainder. Either way, the effectiveness of your e-mail increases by leaps and bounds.

Plain text

Users of mutt prefer plain text e-mails

Here are a few more tips under this category:

• Don't use Outlook stationery.
• Never send e-mails with an image as the e-mail itself.
• No Comic Sans at any time. Period.
• Avoid graphical e-mail signatures (more on that in a moment).

E-mail signatures

Brevity can definitely be applied to e-mail signatures, too. How many times have you seen e-mails that end like this:

Frank Frankelton MCSE, RHCSA, RHCE, CCNA, RHCA, LPIC-3, Ph.D., M.D., Esq., CMDBA
Systems Adminstrator Extraordinaire, Database Administrator, All-around great guy
Office: 210-555-1212
Mobile: 210-555-1213
Other Mobile: 210-555-1214
Fax: 210-555-1215
VOIP: 210-555-1216
AIM: frankeltonia
Twitter: @frankyfrank
Jabber: frankfurter@frankeltonisinthehouse.com
Big Company, Inc

You might think that nobody would ever send out e-mails with a signature like the one above, but I've seen some that are actually worse. Keep the signature short and only put in the information that people really need to know. Generally, your name and title or department is sufficient for e-mail signatures (unless your local/federal laws require otherwise). Always preface it with a double dash "--" on a line by itself to signify that the remainder of the e-mail is the signature.

Summary

Keep it simple, keep it brief, and keep it relevant. While the suggestions above might not apply to every business or every person, following the suggestions will increase the effectiveness of your e-mails and ensure that your voice is heard on the other end.

I'm really interested to hear your comments. Are there some suggestions you have that I missed in the post? Did I make some suggestions which didn't make sense or don't apply to you? Let me know!

How to write e-mails to nerds (that they will actually read) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Here are the list of winners:

• Grand Prize ($50 at ThinkGeek): Dan Udey
• Runners-Up ($25 at ThinkGeek): Joe Wright, Susan Price, and Giovanni Tirloni

Dan's comment rang true with me since much of a sysadmin's job involves responding to crises regardless of how much planning you put forth:

Keep a cool head. Focus. Work methodically. Figure out what to do and get it done, and people will remember you as the person who performs under pressure. Once you can do that, you're a sysadmin.

Joe touched on a critical point about system administration:

Tell the truth. If you break something, 'fess up and fix it. If you don't know how to do something, admit it and learn how to do the task. Create your own culture of honesty on the job; others will respect and follow your example.

Susan offered some inspiration for system administrators stuck in frustrating situations:

I know, I know - dumb users, RTFM. Believe me, I've been there. In fact - one of your strategies should be to establish a trusted community where you can VENT about these issues, and get support for yourself. Ask for answers when you don't know them. Restock on the compassion and patience.

Giovanni talked about the basics and what every system administrator should know to get started in a career. We probably take this for granted, but this is critical to keep in mind:

If you are starting in the system administration area, don't praise yourself only because you (blindly?) fixed an issue or helped that friend with his/her server. Ask yourself: Why what I did fixed the issue? Why that was happening in the first place? And more importantly, how to avoid it for all eternity? You won't but it doesn't hurt to aim high.

Even though it isn't a runner-up, Paul's comment certainly deserves an honorable mention. His comment is actually a true story (with a slight amount of embellishment, of course) and it serves as a reminder that system administrators and developers must stand up for their beliefs even if it goes against the beliefs of their superiors. If your managers don't value the feedback, it might be a sign that a career change is in order.

Once again, a big thanks goes out to everyone who submitted a comment. I'll reach out to the winners today and get the gift certificates sent out to them.

Contest winners from the "Inspire a sysadmin" contest is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Today is my birthday and I'm doing things in reverse -- you are getting gifts today. I'm giving away four $25 gift certificates to ThinkGeek today (yep, that's $100 out of my pocket) but you'll have to do something to earn them.

I'm looking for words of wisdom and guidance from the readers of my blog for system administrators who are just getting started. I talk to brand new sysadmins and college graduates regularly and they're all hungry for what the seasoned folks in the industry know. They're not specifically on the hunt for hard facts and how-to's; they're looking for guidance on how to gain experience, reduce errors and learn efficiently.

Let's get to the important stuff: How does this contest work?

• Write a comment. Put an inspirational story, anecdote, or random words of wisdom for system administrators who are new to the industry in a comment on this post. Although it doesn't have to be extraordinarily lengthy, try to write more than just a sentence or two.
• Give me a way to contact you. Add something to your comment so I can contact you if you're the winner.
• Do it soon. The contest ends at 11:59PM CDT tonight.

I'll be the judge of the comments and I'm going to choose the winners based on the content of the comment. The more inspirational and profound your comment is, the better chance you have of winning. Any comment written in LOLCats caption style will lose points immediately.

One last thing: This contest isn't affiliated with my employer or ThinkGeek. I'm doing this on my own. However, I'm a big fan of both my employer and ThinkGeek, but that's irrelevant right now.

Inspire a sysadmin, get a ThinkGeek gift certificate is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

To change the setting in Firefox, open up a new Firefox tab and go to about:config in the browser. Paste word_select.stop in the search bar that appears below your tab bar and double click the layout.word_select.stop_at_punctuation line. It should become bold and the value on the end will flip from true to false.

Go back to another tab and open a web page which displays an IP address. Double click on any portion of the IP address and Firefox should highlight the entire address.

Highlight IP addresses with a double click in Firefox is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The feeling I had at last year's summit is that Xen was on the verge of losing traction in the market. Very few distributions still had Xen support going forward and much of the discussion was around the lack of dom0 support in upstream Linux kernels. Distribution vendors were hesitant to drag patches forward into modern kernels and this made it much more difficult to get Xen working for many people.

This year was quite different. The number of attendees was up, the venue was much better, and there was an obvious buzz of energy in the room. As many of the presenters noted, this excitement stemmed from the upstream dom0 support in Linux 3.0. This inclusion is a huge win and it helps to drive Xen forward since the developers don't have to worry about dragging patches forward. They can focus on improving performance, adding features, and tightening security.

Many of the discussions this year focused on security and performance. Ian Pratt discussed Xen's ability to view memory pages of virtual machines via an API to detect malware running inside the instance. Memory pages could be identified and marked as not executable or applications could be triggered when a VM attempts to touch a particular memory page. Also, the whole VM could be frozen if needed.

There's also a big push to bring code out of the dom0 and push it into utility VM's. Driver domains could manage the network or I/O infrastructure and this would further reduce the amount of privileged code actively running in dom0. There is already very little code required for the Xen hypervisor itself (much much less than the Linux kernel -- I'm looking at you, KVM) and this reduces the attack surface for potential compromises of the hypervisor. Some projects even aim to restart driver domains multiple times per minute to ensure that any malicious code injected into those virtual machines can't exist for long periods.

Pradeep Vincent from Amazon talked about how Amazon uses Xen and the pain points they have with its current architecture. Much of his discussion was around scaling problems (and we see many of the same issues at Rackspace). Higher performance could easily be gained by multi-threaded operations in dom0 when attaching block devices and creating virtual network interfaces. He also saw some areas for performance gains in the pvops I/O code.

Quite a few of the talks centered on the ARM architecture and what Xen is able to do on those systems after Samsung published their port in 2008. HVM is on the way for ARM and it might even show up in Xen 4.2. Some demos of Xen on mobile phones from Samsung were amazing. They showed how an attacker could compromise the web browser on the phone with a keylogger, but that application was running in a VM. Once the user switched back to the phone's main menu, the keylogger couldn't access the keystrokes any longer. After that, a simple close of the browser killed the VM and destroyed the malicious code.

Xen 4.2 should be available in early 2012 and the feature list is staggering. Improvements to libxenlight, pvops performance (even in HVM), and guest memory sharing should be available with the new release. Nested virtualization (run a hypervisor inside a hypervisor) is also coming in Xen 4.2 and I'm sure Xzibit will be a huge fan. This should streamline hypervisor testing, allow for embedded hypervisor options and extend the capabilities of client hypervisors. Remus should be available in 4.2 as well, but it might be marked as experimental. OVMF will be added as a BIOS option for UEFI (along with the standard SeaBIOS) and this should allow for Mac OS X guests. UEFI allows Windows to boot faster since it switches to PV mode sooner and it allows for simpler platform certification for software vendors.

Mike McClurg's presentation on XCP was pretty important to me since Rackspace is a big consumer of XenServer. If you're not familiar with XCP, it's basically open-source XenServer which runs on bleeding edge (and sometimes unstable) components. XCP 1.5 and XenServer 6 should be available in November with Xen 4.1 and Linux 2.6.32. GPU passthrough, up to 1TB RAM, and disaster recovery will be available. Another goal for the XCP team is to work closely with OpenStack via Project Olympus. Mike's vision is to have XCP become the configuration of choice for open source clouds. Project Kronos was also extremely interesting. It's essentially XCP's XenAPI stack running on Debian and Ubuntu. You'd be able to install either OS on a physical server and run XCP's services on it for a fully OSS hypervisor.

Konrad Wilk gave an update on Linux pvops and it appears there is a shift to get Xen working well on a desktop. This includes 3D graphics support, S3/hibernate capabilities and various bug fixes. There's also a push to get PV functionality into HVM and get HVM functionality into PV. Driver/device domains were discussed again in Patrick Kolp's talk and he had plenty of graphs showing performance changes when regularly restarting device domains. The performance dips were almost negligible with 10 second restarts and the security gains were significant.

There were several other great presentations on other topics like GlusterFS, OpenStack Nova, and Linpicker (from the NSA!). If these types of things interests you, keep your eyes peeled for Xen Summit 2012 next year. The weather in the bay area is well worth the trip.

Xen Summit 2011: My Takeways is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Something else you might not have noticed is that the Fedora kernel team has quietly slipped Linux 3.0 into Fedora 15's update channels in disguise. Click that link, scroll down, and you'll see "Rebase to 3.0. Version reports as 2.6.40 for compatibility with older userspace." Although I'm not a fan of calling something what it isn't (2.6.40 doesn't exist on kernel.org), I can understand some of the reasoning behind the choice.

This change makes the Xen installation on Fedora 15 pretty trivial. To get started, update your kernel to the latest if you're not already on Fedora's 2.6.40 kernels:

yum -y upgrade kernel

We need three more packages (quite a few dependencies will roll in with them):

yum -y install xen libvirt python-virtinst

The xen package reels in the hypervisor itself along with libraries and command line tools (like xl and xm). Libvirt gives us easy access to VM management with the virsh command and python-virtinst gives us the handy virt-install command to make OS installations easy.

Once those packages are installed, we need to make some adjustments in your grub configuration. Open /boot/grub/menu.lst in your text editor of choice and add something like this at the bottom:

title Fedora + Xen (2.6.40-4.fc15.x86_64)
        root (hd0,1)
	kernel /boot/xen.gz
        module /boot/vmlinuz-2.6.40-4.fc15.x86_64 ro root=/dev/sda1
        module /boot/initramfs-2.6.40-4.fc15.x86_64.img

Ensure that the root (hd0,1) is applicable to your system (adjust it if it isn't). Also, check the kernel version to ensure it matches your installed kernel and adjust the root= portion to match your root volume. Flip the default line to a value which will boot your new grub entry and ensure the timeout is set to a reasonable number if you need to temporarily switch back to your original grub entry at boot time. (Hey, we all make mistakes.)

I take one extra precaution and change the UPDATEDEFAULT=yes line to no in /etc/sysconfig/kernel. This ensures that future kernel updates don't trample the entry you've just made. Keep in mind that you'll need to manually update your grub configuration when you do kernel upgrades later.

Cross your fingers and reboot. If your system doesn't reboot properly, reboot it again and choose your old kernel from the grub menu. Double-check your configuration for fat-fingering and give it another try. If your system boots and pings but you have no output via a monitor, don't fret. There's a patch for the problem which should appear soon in Linux 3.0. The impatient can snag a kernel source RPM, add the patch file, and build a local kernel (or you can download my local build from when I did it).

Log in and verify that you booted into the dom0:

[root@xenbox ~]# xm dmesg | head -n 5
 __  __            _  _    _   _   ____     __      _ ____  
 \ \/ /___ _ __   | || |  / | / | |___ \   / _| ___/ | ___| 
  \  // _ \ '_ \  | || |_ | | | |__ __) | | |_ / __| |___ \ 
  /  \  __/ | | | |__   _|| |_| |__/ __/ _|  _| (__| |___) |
 /_/\_\___|_| |_|    |_|(_)_(_)_| |_____(_)_|  \___|_|____/

Once you're done with that, make sure libvirtd is running:

/etc/init.d/libvirtd start; chkconfig libvirtd on

Try installing a VM:

virt-install \
  --paravirt \
  --name=testvm \
  --ram=512 \
  --vcpus=4 \
  --file /dev/vmstorage/testvm \
  --graphics vnc,port=5905 --noautoconsole \
  --autostart --noreboot \
  --location=http://mirrors.kernel.org/debian/dists/squeeze/main/installer-amd64/

You should have a VM installation underway pretty quickly and it will be visible via port 5905 on the local host. Enjoy the power and freedom of your brand new type 1 hypervisor.

Xen 4.1 on Fedora 15 with Linux 3.0 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.