Racker Hacker

Last week, I found myself with a server under low load, but it couldn’t make or receive network connections. When I ran dmesg, I found the following line repeating over and over:

ip_conntrack: table full, dropping packet

I’d seen this message before, but I headed over to Red Hat’s site for more details. It turns out that the server was running iptables, but it was under a very heavy load and also handling a high volume of network connections. Generally, the ip_conntrack_max is set to the total MB of RAM installed multiplied by 16. However, this server had 4GB of RAM, but ip_conntrack_max was set to 65536:

# cat /proc/sys/net/ipv4/ip_conntrack_max
65536

I logged into another server with 1GB of RAM (RHES 5, 32-bit) and another with 2GB of RAM (RHES 4, 64-bit), and both had ip_conntrack_max set to 65536. I’m not sure if this is a known Red Hat issue, or if it’s just set to a standard value out of the box.

If you want to check your server’s current tracked connections, just run the following:

# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count

If you want to adjust it (as I did), just run the following as root:

# echo 131072 > /proc/sys/net/ipv4/ip_conntrack_max

If your system abruptly loses power, or if a RAID card is beginning to fail, you might see an ominous message like this within your logs:

EXT3-fs error (device hda3) in start_transaction: Journal has aborted

Basically, the system is telling you that it’s detected a filesystem/journal mismatch, and it can’t utilize the journal any longer. When this situation pops up, the filesystem gets mounted read-only almost immediately. To fix the situation, you can remount the partition as ext2 (if it isn’t your active root partition), or you can commence the repair operations.

If you’re working with an active root partition, you will need to boot into some rescue media and perform these operations there. If this error occurs with an additional partition besides the root partition, simply unmount the broken filesystem and proceed with these operations.

Remove the journal from the filesystem (effectively turning it into ext2):

# tune2fs -O ^has_journal /dev/hda3

Now, you will need to fsck it to correct any possible problems (throw in a -y flag to say yes to all repairs, -C for a progress bar):

# e2fsck /dev/hda3

Once that's finished, make a new journal which effectively makes the partition an ext3 filesystem again

# tune2fs -j /dev/hda3

You should be able to mount the partition as an ext3 partition at this time:

# mount -t ext3 /dev/hda3 /mnt/fixed

Be sure to check your dmesg output for any additional errors after you’re finished!

Apparently, a recent Red Hat Enterprise Linux update for ES3, 4 and 5 caused some Perl applications to throw errors like these:

unable to call function somefunction on undefined value

Of course, replace somefunction with your function of choice. To correct the issue, you can force CPAN to bring back a more sane version of Scalar::Util:

# perl -MCPAN -e shell
cpan> force install Scalar::Util

Create a strong CSR and private key
openssl req -new -nodes -newkey rsa:2048 -out server.crt -keyout server.key

Parsing out the data within a certificate
openssl asn1parse -in server.crt

Checking a certificate/key modulus to see if they correspond
openssl rsa -in server.key -modulus -noout | openssl md5
openssl x509 -in server.crt -modulus -noout | openssl md5

Convert a key from PEM -> DER
openssl rsa -inform PEM -in key.pem -outform DER -out keyout.der

Convert a key from DER -> PEM
openssl rsa -inform DER -in key.der -outform PEM -out keyout.pem

Remove the password from an encrypted private key
openssl rsa -in server.key -out server-nopass.key

Reviewing a detailed SSL connection
openssl s_client -connect 10.0.0.1:443

I’ve struggled at times to get a decent-looking terminal on my desktop, and I believe I’ve found a good one. Toss this into your ~/.Xdefaults:

aterm*loginShell:true
aterm*transparent:true
aterm*shading:40
aterm*background:Black
aterm*foreground:White
aterm*scrollBar:true
aterm*scrollBar_right:true
aterm*transpscrollbar:true
aterm*saveLines:32767
aterm*font:*-*-fixed-medium-r-normal--*-110-*-*-*-*-iso8859-1
aterm*boldFont:*-*-fixed-bold-r-normal--*-*-110-*-*-*-*-iso8859-1

Then load up the changes and start aterm:

$ xrdb -load .Xdefaults
$ aterm

Of course, if you like rxvt better for your Unicode needs, just use this configuration:

rxvt*loginShell:true
rxvt*transparent:true
rxvt*shading:40
rxvt*background:Black
rxvt*foreground:White
rxvt*scrollBar:true
rxvt*scrollBar_right:true
rxvt*transpscrollbar:true
rxvt*saveLines:32767
rxvt*font:*-*-fixed-medium-r-normal--*-110-*-*-*-*-iso8859-1
rxvt*boldFont:*-*-fixed-bold-r-normal--*-*-110-*-*-*-*-iso8859-1

A few days ago, I began to install a group of packages with up2date, and the person next to me was surprised that up2date even had this functionality. I use it regularly, but I realized that many users might not be familiar with it.

You can install package groups using an at-sign (@) in front of the group name:

# up2date -i "@X Window System"

This will tell up2date to install all of the packages that are marked within the “X Window System” package group. That would include X drivers, the X libraries/binaries, and twm (among many other packages). If you’re not sure which groups are available, just pass the --show-groups flag and review the list:

# up2date --show-groups
Administration Tools
Arabic Support
Assamese Support
Authoring and Publishing
Base
Bengali Support
Brazilian Portuguese Support
British Support
Bulgarian Support
Catalan Support
Chinese Support
Compatibility Arch Development Support
Compatibility Arch Support
Core
Cyrillic Support
Czech Support
DNS Name Server
Danish Support
Development Libraries
Development Tools
Dialup Networking Support
Dutch Support
Editors
Emacs
Engineering and Scientific
Estonian Support
FTP Server
Finnish Support
French Support
GNOME
GNOME Desktop Environment
GNOME Software Development
Games and Entertainment
German Support
Graphical Internet
Graphics
Greek Support
Gujarati Support
Hebrew Support
Hindi Support
Hungarian Support
ISO8859-2 Support
ISO8859-9 Support
Icelandic Support
Italian Support
Japanese Support
KDE
KDE (K Desktop Environment)
KDE Software Development
Korean Support
Legacy Network Server
Legacy Software Development
Mail Server
Miscellaneous Included Packages
MySQL Database
Network Servers
News Server
Norwegian Support
Office/Productivity
Polish Support
Portuguese Support
PostgreSQL Database
Printing Support
Punjabi Support
Romanian Support
Ruby
Russian Support
Serbian Support
Server
Server Configuration Tools
Slovak Support
Slovenian Support
Sound and Video
Spanish Support
Swedish Support
System Tools
Tamil Support
Text-based Internet
Turkish Support
Ukrainian Support
Web Server
Welsh Support
Windows File Server
Workstation Common
X Software Development
X Window System
XEmacs

In the event that your system is running out of file descriptors, or you simply want to know what your users are doing, you can review their count of open files by running this command:

lsof | grep ' root ' | awk '{print $NF}' | sort | wc -l

Of course, if you want to drop the count and show the actual processes, you can run:

lsof | grep ' root '

With RHEL 5 ditching up2date for yum, many Red Hat users might find themselves confused with the new command line flags. Red Hat has published a document detailing the new changes and their old counterparts.

Red Hat Knowledgebase: What are the yum equivalents of former up2date common tasks?

When you create a CSR and private key to obtain an SSL certificate, the private key has some internal data called a modulus. This is integral to the security of your SSL encryption, but for this specific post, we will focus on one specific aspect.

If your private key and certificate do not contain the same modulus, then Apache will sometimes refuse to start or it may not respond properly to SSL requests. You can check the modulus of your private key and SSL certificate with these commands:

# openssl rsa -noout -modulus -in server.key | openssl md5
# openssl x509 -noout -modulus -in server.crt | openssl md5

If the MD5 checksums match, then the certificate and key will work together. However, if they are different, then you cannot use them together. Generally, this means that you used the wrong CSR (that corresponded to some other private key) when you obtained/created your SSL certificate.

When you find yourself in a pinch, and you don’t know the limits of a certain Red Hat Enterprise Linux version, you can find this information in one place. Whether you want to know RHEL’s CPU or memory limitations, you can find them here:

http://www.redhat.com/rhel/compare/