Racker Hacker

Just in case some of you out there enjoy nomenclature and theory behind Linux filesystems, here’s some things to keep in mind. The modification time (mtime) of a file describes when the actual data blocks that hold the file changed. The changed time (ctime) of a file describes when the metadata was last changed.

Also, metadata is stored within a different location than the data blocks. The metadata fits in the inode while the file’s data goes within data blocks. The inode information contains the owner, owner’s group, time related data (atime, ctime, mtime), and the mode (permissions).

The name of the file itself is actually stored within the file that makes up the directory. And, the directory is simply a file that masquerades as a directory once the filesystem is mounted and read.

A question I’m asked daily is “How can I find out what is generating iowait on my server?” Sure, you can dig through pages of lsof output, restart services, or run strace, but it can be a frustrating process. I saw a process on this blog post, and I changed the regexes to fit Red Hat and CentOS systems a bit better:

# /etc/init.d/syslog stop
# echo 1 > /proc/sys/vm/block_dump
# dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)’ | sort | uniq -c | sort -rn | head
1526 mysqld
819 httpd
429 kjournald
35 qmail
27 in
7 imapd
6 irqbalance
5 pop
4 pdflush
3 spamc

In my specific situation, it looks like MySQL is the biggest abuser of my disk, followed by Apache and the filesystem journaling. As expected, qmail is a large contender, too.

Don’t forget to set things back to their normal state when you’re done!

# echo 0 > /proc/sys/vm/block_dump
# /etc/init.d/syslog start

I recently came across a server that was throwing this error into its message log:

ntpd_initres[2619]: ntpd returns a permission denied error!

It would only appear about every five minutes on the server, and restarting ntpd didn’t correct the issue. I stopped ntpd entirely, but the error still appeared a few minutes later.

After examining the running processes, I found that there was a lonely ntpd process that was running using a non-standard method. I killed that process, started the default instance of ntpd using the init scripts, and the issue went away.

It turns out that ntpd daemon that was started manually was unable to access some of the required paths and sockets that is necessary for ntpd to run properly. These configuration items are set up in the init scripts, but they’re not included when ntpd is running manually.

This was tested on Red Hat Enterprise Linux 4.

By setting a certain bash environment variable, you can limit which commands are kept in the .bash_history file. The following options can be passed to the HISTCONTROL environmental variable:

ignorespace - omits commands beginning with a space
ignoredups - omits commands that match the previously run command
ignoreboth - combines ignorespace and ignoredups
erasedups - removes previous lines that match the line that was just run

To set it, simply run the following from the command line, or add it to the .bashrc or a single user’s .bash_profile:

export HISTCONTROL=ignorespace

If no value is set, then all commands will be saved regardless of their content.

Installing new hardware may mean that new kernel need to be loaded when your server boots up. There’s a two step process to making a new initrd file:

First, add the appropriate line to your /etc/modules.conf or /etc/modprobe.conf which corresponds to your new kernel module.

Next, rebuild the initial ram disk after making a backup of the current one:

# cp /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img.bak
# mkinitrd -f initrd-`uname -r`.img `uname -r`

Reboot the server now and make sure the new driver is loaded properly.

If your system abruptly loses power, or if a RAID card is beginning to fail, you might see an ominous message like this within your logs:

EXT3-fs error (device hda3) in start_transaction: Journal has aborted

Basically, the system is telling you that it’s detected a filesystem/journal mismatch, and it can’t utilize the journal any longer. When this situation pops up, the filesystem gets mounted read-only almost immediately. To fix the situation, you can remount the partition as ext2 (if it isn’t your active root partition), or you can commence the repair operations.

If you’re working with an active root partition, you will need to boot into some rescue media and perform these operations there. If this error occurs with an additional partition besides the root partition, simply unmount the broken filesystem and proceed with these operations.

Remove the journal from the filesystem (effectively turning it into ext2):

# tune2fs -O ^has_journal /dev/hda3

Now, you will need to fsck it to correct any possible problems (throw in a -y flag to say yes to all repairs, -C for a progress bar):

# e2fsck /dev/hda3

Once that's finished, make a new journal which effectively makes the partition an ext3 filesystem again

# tune2fs -j /dev/hda3

You should be able to mount the partition as an ext3 partition at this time:

# mount -t ext3 /dev/hda3 /mnt/fixed

Be sure to check your dmesg output for any additional errors after you’re finished!

Apparently, a recent Red Hat Enterprise Linux update for ES3, 4 and 5 caused some Perl applications to throw errors like these:

unable to call function somefunction on undefined value

Of course, replace somefunction with your function of choice. To correct the issue, you can force CPAN to bring back a more sane version of Scalar::Util:

# perl -MCPAN -e shell
cpan> force install Scalar::Util

Create a strong CSR and private key
openssl req -new -nodes -newkey rsa:2048 -out server.crt -keyout server.key

Parsing out the data within a certificate
openssl asn1parse -in server.crt

Checking a certificate/key modulus to see if they correspond
openssl rsa -in server.key -modulus -noout | openssl md5
openssl x509 -in server.crt -modulus -noout | openssl md5

Convert a key from PEM -> DER
openssl rsa -inform PEM -in key.pem -outform DER -out keyout.der

Convert a key from DER -> PEM
openssl rsa -inform DER -in key.der -outform PEM -out keyout.pem

Remove the password from an encrypted private key
openssl rsa -in server.key -out server-nopass.key

Reviewing a detailed SSL connection
openssl s_client -connect 10.0.0.1:443

I’ve struggled at times to get a decent-looking terminal on my desktop, and I believe I’ve found a good one. Toss this into your ~/.Xdefaults:

aterm*loginShell:true
aterm*transparent:true
aterm*shading:40
aterm*background:Black
aterm*foreground:White
aterm*scrollBar:true
aterm*scrollBar_right:true
aterm*transpscrollbar:true
aterm*saveLines:32767
aterm*font:*-*-fixed-medium-r-normal--*-110-*-*-*-*-iso8859-1
aterm*boldFont:*-*-fixed-bold-r-normal--*-*-110-*-*-*-*-iso8859-1

Then load up the changes and start aterm:

$ xrdb -load .Xdefaults
$ aterm

Of course, if you like rxvt better for your Unicode needs, just use this configuration:

rxvt*loginShell:true
rxvt*transparent:true
rxvt*shading:40
rxvt*background:Black
rxvt*foreground:White
rxvt*scrollBar:true
rxvt*scrollBar_right:true
rxvt*transpscrollbar:true
rxvt*saveLines:32767
rxvt*font:*-*-fixed-medium-r-normal--*-110-*-*-*-*-iso8859-1
rxvt*boldFont:*-*-fixed-bold-r-normal--*-*-110-*-*-*-*-iso8859-1

A few days ago, I began to install a group of packages with up2date, and the person next to me was surprised that up2date even had this functionality. I use it regularly, but I realized that many users might not be familiar with it.

You can install package groups using an at-sign (@) in front of the group name:

# up2date -i "@X Window System"

This will tell up2date to install all of the packages that are marked within the “X Window System” package group. That would include X drivers, the X libraries/binaries, and twm (among many other packages). If you’re not sure which groups are available, just pass the --show-groups flag and review the list:

# up2date --show-groups
Administration Tools
Arabic Support
Assamese Support
Authoring and Publishing
Base
Bengali Support
Brazilian Portuguese Support
British Support
Bulgarian Support
Catalan Support
Chinese Support
Compatibility Arch Development Support
Compatibility Arch Support
Core
Cyrillic Support
Czech Support
DNS Name Server
Danish Support
Development Libraries
Development Tools
Dialup Networking Support
Dutch Support
Editors
Emacs
Engineering and Scientific
Estonian Support
FTP Server
Finnish Support
French Support
GNOME
GNOME Desktop Environment
GNOME Software Development
Games and Entertainment
German Support
Graphical Internet
Graphics
Greek Support
Gujarati Support
Hebrew Support
Hindi Support
Hungarian Support
ISO8859-2 Support
ISO8859-9 Support
Icelandic Support
Italian Support
Japanese Support
KDE
KDE (K Desktop Environment)
KDE Software Development
Korean Support
Legacy Network Server
Legacy Software Development
Mail Server
Miscellaneous Included Packages
MySQL Database
Network Servers
News Server
Norwegian Support
Office/Productivity
Polish Support
Portuguese Support
PostgreSQL Database
Printing Support
Punjabi Support
Romanian Support
Ruby
Russian Support
Serbian Support
Server
Server Configuration Tools
Slovak Support
Slovenian Support
Sound and Video
Spanish Support
Swedish Support
System Tools
Tamil Support
Text-based Internet
Turkish Support
Ukrainian Support
Web Server
Welsh Support
Windows File Server
Workstation Common
X Software Development
X Window System
XEmacs