Also, I don't really hate Kerberos. It's a bit of an inside joke with my coworkers who are studying for some of the RHCA exams at Rackspace. The additional security provided by Kerberos is quite good but the setup involves a lot of small steps. If you miss one of the steps or if you get something done out of order, you may have to scrap the whole setup and start over unless you can make sense of the errors in the log files. A lot of my dislikes for Kerberos comes from the number of steps required in the setup process and the difficulty in tracking down issues when they crop up.

To complete this guide, you'll need the following:

• two CentOS, Red Hat Enterprise Linux or Scientific Linux 5 servers or VM's
• some patience

Here's how I plan to name my servers:

• kdc.example.com - the Kerberos KDC server at 192.168.250.2
• client.example.com - the Kerberos client at 192.168.250.3

CRITICAL STEP: Before getting started, ensure that both systems have their hostnames properly set and both systems have the hostnames and IP addresses of both systems in /etc/hosts. Your server and client must be able to know the IP and hostname of the other system as well as themselves.

First off, we will need NIS working to serve up the user information for our client. Install the NIS server components on the KDC server:

[root@kdc ~]# yum install ypserv

Set the NIS domain and set a static port for ypserv to make it easier to firewall off. Edit /etc/sysconfig/network on the KDC server:

NISDOMAINNAME=EXAMPLE.COM
YPSERV_ARGS="-p 808"

Manually set the NIS domain on the KDC server and add it to /etc/yp.conf:

[root@kdc ~]# nisdomain EXAMPLE.COM
[root@kdc ~]# echo "domain EXAMPLE.COM server kdc.example.com" >> /etc/yp.conf

Adjust /var/yp/securenets on the KDC server for additional security:

[root@kdc ~]# echo "255.0.0.0 127.0.0.0" >> /var/yp/securenets
[root@kdc ~]# echo "255.255.255.0 192.168.250.0" >> /var/yp/securenets

Start the NIS server and generate the NIS maps:

[root@kdc ~]# /etc/init.d/ypserv start; chkconfig ypserv on
[root@kdc ~]# make -C /var/yp

I usually like to prepare my iptables rules ahead of time so I ensure that it doesn't derail me later on. Paste this into the KDC's terminal:

iptables -N SERVICES
iptables -I INPUT -j SERVICES
iptables -A SERVICES -p tcp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p udp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p tcp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p udp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p tcp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 464 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p tcp --dport 749 -j ACCEPT -m comment --comment "kerberos"
/etc/init.d/iptables save

We need our time in sync for Kerberos to work properly. Install NTP on both nodes, start it, and ensure it comes up at boot time:

[root@kdc ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start
[root@client ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start

Now we're ready to set up Kerberos. Start by installing some packages on the KDC:

[root@kdc ~]# yum install krb5-server krb5-workstation

We will need to make some edits to /etc/krb5.conf on the KDC to set up our KDC realm. Ensure that the default_realm is set:

default_realm = EXAMPLE.COM

The [realms] section should look like this:

[realms]
EXAMPLE.COM = {
	kdc = 192.168.250.2:88
	admin_server = 192.168.250.2:749
}

The [domain_realm] section should look like this:

[domain_realm]
kdc.example.com = EXAMPLE.COM
client.example.com = EXAMPLE.COM

Add validate = true within the pam { } block of the [appdefaults] section:

[appdefaults]
 pam = {
   validate = true

Adjust /var/kerberos/krb5kdc/kdc.conf on the KDC:

[realms]
EXAMPLE.COM = {
	master_key_type = des-hmac-sha1
	default_principal_flags = +preauth
}

There's one last configuration file to edit on the KDC! Ensure that /var/kerberos/krb5kdc/kadm5.acl looks like this:

*/admin@EXAMPLE.COM	    *

We're now ready to make a KDC database to hold our sensitive Kerberos data. Create the database and set a good password which you can remember. This command also stashes your password on the KDC so you don't have to enter it each time you start the KDC:

kdb5_util create -r EXAMPLE.COM -s

On the KDC, create a principal for the admin user as well as user1 (which we'll create shortly). Also, export the admin details to the kadmind key tab. You'll get some extra output after each one of these commands but I've snipped it to reduce the length of the post.

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc root/admin
kadmin.local:  addprinc user1
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/admin
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/changepw
kadmin.local:  exit

Let's start the Kerberos KDC and kadmin daemons:

[root@kdc ~]# /etc/init.d/krb5kdc start; /etc/init.d/kadmin start
[root@kdc ~]# chkconfig krb5kdc on; chkconfig kadmin on

Now that the administration work is done, let's create a principal for our KDC server and stick it in it's keytab:

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc -randkey host/kdc.example.com
kadmin.local:  ktadd host/kdc.example.com

Transfer your /etc/krb5.conf from the KDC server to the client. Hop onto the client server, install the Kerberos client package and add some host principals:

[root@client ~]# yum install krb5-workstation
[root@client ~]# kadmin.local
kadmin.local:  addpinc --randkey host/client.example.com
kadmin.local:  ktadd host/kdc.example.com

There aren't any daemons on the client side, so the configuration is pretty much wrapped up there for Kerberos. However, we now need to tell both servers to use Kerberos for auth and your client servers needs to use NIS to get user data.

• On the KDC:
  • run authconfig-tui
  • choose Use Kerberos from the second column
  • press Next
  • don't edit the configuration (authconfig got the data from /etc/krb.conf)
  • press OK
• On the client:
  • run authconfig-tui
  • choose Use NIS and Use Kerberos
  • press Next
  • enter your NIS domain (EXAMPLE.COM) and NIS server (kdc.example.com or 192.168.250.2)
  • press Next
  • don't edit the Kerberos configuration (authconfig got the data from /etc/krb.conf)
  • press OK

Got NIS problems? If the NIS connection stalls on the client, ensure that you have the iptables rules present on the KDC that we added near the beginning of this guide. Also, if you forgot to add both hosts to both servers' /etc/hosts, go do that now.

Let's make our test user on the KDC. Don't add this user to the client -- we'll get the user information via NIS and authenticate via Kerberos shortly. We'll also rebuild our NIS maps after adding the user:

[root@kdc ~]# useradd user1
[root@kdc ~]# passwd user1
[root@kdc ~]# make -C /var/yp/

On the client, see if you can get the password hash for the user1 account via NIS:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:$1$sUlSTlCv$riK5El3z8N4y.mi5Fe3Q60:500:500::/home/user1:/bin/bash

You can see why NIS isn't a good way to authenticate users. Someone could easily pull the hash for any account and brute force the hash on their own server. Go back to the KDC and lock out the user account:

[root@kdc ~]# usermod -p '!!' user1

Go back to the client and try to pull the password hash now:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:!!:500:500::/home/user1:/bin/bash

On the plus side, the user's password hash is now gone. On the negative side, you've just prevented this user from logging in locally or via NIS. Don't worry, the user can log in via Kerberos now. Let's prepare a home directory on the client for the user:

[root@client ~]# mkdir /home/user1
[root@client ~]# cp -av /etc/skel/.bash* /home/user1/
[root@client ~]# chown -R user1:user1 /home/user1/

Note: In a real-world scenario, you'd probably want to export this user's home directory via NFS so they didn't get a different home directory on every server.

While you're still on the client, try to log into the client via the user. Use the password that you used when you created the user1 principal on the KDC.

[root@client ~]# ssh user1@localhost
user1@localhost's password:
[user1@client ~]$ whoami
user1

List your Kerberos tickets and you should see one for your user principal:

[user1@client ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500_fCKPnZ
Default principal: user1@EXAMPLE.COM
 
Valid starting     Expires            Service principal
02/05/12 14:18:53  02/06/12 00:18:53  krbtgt/EXAMPLE.COM@EXAMPLE.COM
	renew until 02/05/12 14:18:53

Your KDC should have a couple of lines in its /var/log/krb5kdc.log showing the authentication:

Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): AS_REQ (12 etypes {18 17 16 23 1 3 2 11 10 15 12 13}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=16 ses=16}, user1@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=18 ses=18}, user1@EXAMPLE.COM for host/client.example.com@EXAMPLE.COM

The first line shows that the client asked for a Authentication Server Request (AS_REQ) and the second line shows that the client then asked for a Ticket Granting Server Request (TGS_REQ). In layman's terms, the client first asked for a ticket-granting ticket (TGT) so it could authenticate to other services. When it actually tried to log in via ssh it asked for a ticket (and received it).

YOU JUST CONFIGURED KERBEROS!

From here, the sky's the limit. Another popular implementation of Kerberos is encrypted NFSv4. You can even go crazy and use Kerberos with apache.

Let me know if you have any questions about this post or if you spot any errors. With this many steps, there's bound to be a typo or two in this guide. Keep in mind that there are some obvious spots for network-level and service-level security improvements. This guide was intended to give you the basics and it doesn't cover all of the security implications involved with a Kerberos implementation.

The Kerberos-hater's guide to installing Kerberos is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Here's Kerberos in a nutshell within a two-server environment: One server is a Kerberos key distribution center (KDC) and the other is a Kerberos client. The KDC has the list of users and their passwords. Consider a situation where a user tries to ssh into the Kerberos client:

• sshd calls to pam to authenticate the user
• pam calls to the KDC for a ticket granting ticket (TGT) to see if the user can authenticate
• the KDC replies to the client with a TGT encrypted with the user's password
• pam (on the client) tries to decrypt the TGT with the password that the user provided via ssh
• if pam can decrypt the TGT, it knows the user is providing the right password

Now that the client has a a TGT for that user, it can ask for tickets to access other network services. What if the user who just logged in wants to access another Kerberized service in the environment?

• client calls the KDC and asks for a ticket to grant access to the other service
• KDC replies with two copies of the ticket:
  • one copy is encrypted with the user's current TGT
  • a second copy is encrypted with the password of the network service the user wants to access
• the client can decrypt the ticket which was encrypted with the current TGT since it has the TGT already
• client makes an authenticator by taking the decrypted ticket and encrypting it with a timestamp
• client passes the authenticator and the second copy of the ticket it received from the KDC
• the other network service decrypts the second copy of the ticket and verifies the password
• the other network service uses the decrypted ticket to decrypt the authenticator it received from the client
• if the timestamp looks good, the other network service allows the user access

Okay, that's confusing. Let's take it one step further. Enabling pre-authentication requires that clients send a request containing a timestamp encrypted with the user's password prior to asking for a TGT. Without this requirement, an attacker can ask for a TGT one time and then brute force the TGT offline. Pre-authentication forces the client to send a timestamped request encrypted with the user's password back to the KDC before they can ask for a TGT. This means the attacker is forced to try different passwords when encrypting the timestamp in the hopes that they'll get a TGT to work with eventually. One would hope that you have something configured on the KDC to set off an alarm for multiple failed pre-authentication attempts.

Oh, but we can totally kick it up another notch. What if an attacker is able to give a bad password to a client but they're also able to impersonate the KDC? They could reply to the TGT request (as the KDC) with a TGT encrypted with whichever password they choose and get access to the client system. Enabling mutual authentication stops this attack since it forces the client to ask the KDC for the client's own host principal password (this password is set when the client is configured to talk to the KDC). The attacker shouldn't have any clue what that password is and the attack will be thwarted.

By this point, you're either saying "Oh man, I don't ever want to do this." or "How do I set up Kerberos?". Stay tuned if you're in the second group. I'll have a dead simple (or as close to dead simple as one can get with Kerberos) how-to on the blog shortly.

In the meantime, here are a few links for extra Kerberos bedtime reading:

• Kerberos on Wikipedia
• MIT's "Why Kerberos" [PDF]
• How Kerberos Authentication Works

Kerberos for haters is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The purpose behind setroubleshoot is to let users know when access has been denied, help them resolve it if necessary, and to reduce overall frustration while working through tight security restrictions in the default SELinux policies. The GUI frontend for setroubleshoot is great for users who run Linux desktops or those who run servers with a display attached. Don't worry, you can configure setroubleshoot on remote servers to send alerts elsewhere when a GUI alert isn't an option.

Install a few packages to get started:

yum install setroubleshoot{-server,-plugins,-doc}

Open /etc/setroubleshoot/setroubleshoot.conf in your favorite text editor and adjust the [email] section to fit your server:

recipients_filepath = /var/lib/setroubleshoot/email_alert_recipients
smtp_port = 25
smtp_host = localhost
from_address = selinux@myserver.com
subject = [MyServer] SELinux AVC Alert

You could probably see it coming, but you need to put the e-mail addresses for your recipients into /var/lib/setroubleshoot/email_alert_recipients:

echo "selinux@mycompany.com" >> /var/lib/setroubleshoot/email_alert_recipients

You'll notice that setroubleshoot doesn't have an init script and it doesn't exist in systemd in Fedora 15. It runs through the dbus-daemon and a quick bounce of the messagebus via its init script brings in the necessary components to run setroubleshoot:

service messagebus restart

A really easy (and safe) test is to ask sshd to bind to a non-standard port. Simply define an additional port on in your /etc/ssh/sshd_config like this:

Port 22
Port 222

When you restart sshd, it will bind to port 22 with success, but it won't be allowed to bind to port 222 (since that's blocked by SELinux as a non-standard port for the ssh_port_t port type). DON'T WORRY! Your sshd server will still be listening on port 22. If you wait a moment, you'll get an e-mail (perhaps two) that not only notify you of the denial, but they make suggestions for how to fix it:

SELinux is preventing /usr/sbin/sshd from name_bind access on the tcp_socket port 222.
 
*****  Plugin bind_ports (99.5 confidence) suggests  *************************
 
If you want to allow /usr/sbin/sshd to bind to network port 222
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 222
   where PORT_TYPE is one of the following: ...

For this particular example, the quick fix would be to run:

semanage port -a -t ssh_port_t -p tcp 222

Receive e-mail reports for SELinux AVC denials is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Here are my favorite techniques listed from most effective to least effective:

SSH key pairs
By disabling password-based authentication and requiring ssh key pairs, you reduce the chances of compromise via a brute force attack. This can also help you protect against weak account passwords since a valid private key is required to gain access to the server. However, a weak account password is still a big problem if you allow your users to use sudo.

If you're new to using ssh keys, there are many great guides that can walk you through the process.

Firewall
Limiting the source IP addresses that can access your server on port 22 is simple and effective. However, if you travel on vacation often or your home IP address changes frequently, this may not be a convenient way to limit access. Acquiring a server with trusted access through your firewall would make this method easier to use, but you'd need to consider the security of that server as well.

The iptables rules would look something like this:

iptables -A INPUT -j ACCEPT -p tcp --dport 22 -s 10.0.0.20
iptables -A INPUT -j ACCEPT -p tcp --dport 22 -s 10.0.0.25
iptables -A INPUT -j DROP -p tcp --dport 22

Use a non-standard port
I'm not a big fan of security through obscurity and it doesn't work well for ssh. If someone is simply scanning a subnet to find ssh daemons, you might not be seen the first time. However, if someone is targeting you specifically, changing the ssh port doesn't help at all. They'll find your ssh banner quickly and begin their attack.

If you prefer this method, simply adjust the Port configuration parameter in your sshd_config file.

Limit users and groups
If you have only certain users and groups who need ssh access to your server, setting user or group limits can help increase security. Consider a server which needs ssh access for developers and a manager. Adding this to to your sshd_config would allow only those users and groups to access your ssh daemon:

AllowGroups developers
AllowUsers jsmith pjohnson asamuels

Keep in mind that any users or groups not included in the sshd_config won't be able to access your ssh server.

TCP wrappers
While TCP wrappers are tried and true, I consider them to be a bit old-fashioned. I've found that many new systems administrators may not think of TCP wrappers when they diagnose server issues and this could possibly cause delays when adjustments need to be made later.

If you're ready to use TCP wrappers to limit ssh connections, check out Red Hat's extensive documentation.

fail2ban and denyhosts
For those systems administrators who want to take a bit more active stance on blocking brute force attacks, there's always fail2ban or denyhosts. Both fail2ban and denyhosts monitor your authentication logs for repeated failures, but denyhosts can only work with your ssh daemon. You can use fail2ban with other applications like web servers and FTP servers.

The only downside of using these applications is that if a valid user accidentally tries to authenticate unsuccessfully multiple times, they may be locked out for a period of time. This could be a big problem if you're in the middle of a server emergency.

A quick search on Google will give you instructions on fail2ban configuration as well as denyhosts configuration.

Port knocking
Although port knocking is another tried and true method to prevent unauthorized access, it can be annoying to use unless you have users who are willing to jump through additional hoops. Port knocking involves a "knock" on an arbitrary port that then allows the ssh daemon to be exposed to the user who sent the original knock.

Linux Journal has a great article explaining how port knocking works and it provides some sample configurations as well.

Conclusion
The best way to secure your ssh daemon is to apply more than one of these methods to your servers. Weighing security versus convenience of access isn't an easy task and it will be different for every environment. Regardless of the method or methods you choose, ensure that the rest of your team is comfortable with the changes and capable of adapting to them efficiently.

Securing your ssh server is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

You may be wondering: "Why not just use ssh in a for loop?" Sure, you could do something like this in bash:

for i in`cat ~/myhosts.txt`; do ssh $i 'uptime'; done

But dsh allows you to do this:

dsh -g myhosts 'uptime'

In addition, dsh allows you to run the commands concurrently (-c) or one after the other (-w). You can tell it to prepend each line with the machine's name (-M) or it can omit the machine name from the output (-H). If you need to pass extra options, such as which ssh key to use, or an alternative port, you can do that as well (-o). All of these command line options can be tossed into a configuration file if you have a default set of options you prefer.

Another thing that makes dsh more powerful is the groups feature. Let's say you have three groups of servers - some are in California, others in Texas, and still others in New York. You could make three files for the groups:

• ~/.dsh/group/california
• ~/.dsh/group/texas
• ~/.dsh/group/newyork

Inside each file, you just need to list the hosts one after the other. Here's the ~/.dsh/group/texas group file:

db1.tx.mydomain.com
db2.tx.mydomain.com
web1.tx.mydomain.com
web2.tx.mydomain.com
#web3.tx.mydomain.com

As you can see, dsh handles comments in the hosts file. In the above example, the web3 server will be skipped since it's prepended with a comment. Let's say you want to check the uptime on all of the Texas servers as fast as possible:

dsh -c -g texas 'uptime'

That will run the uptime command on all of the servers in the Texas group concurrently. If you need to run it on two groups at once, just pass another group (eg. -g texas -g california) as an argument. You can also run the commands against all of your groups (-a).

The dsh command can really help you if you need to gather information or run simple commands on many remote servers. If you find yourself using it often for systems management, you may want to consider something like puppet.

Crash course in dsh is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

First off, credit for this fix on OS X 10.6.2 goes to Geoff Watts from his two tweets.

If you're using Snow Leopard, you'll find that the current version of MacFusion refuses to complete a connection to a remote server. You can fix this in two steps:

First, quit MacFusion.

Second, open System Preferences and then open the MacFUSE pane. Check the "Show Beta Versions" box and click "Check For Updates". Go ahead and update MacFUSE.

Third, open up a terminal and do the following:

rm /Applications/Macfusion.app/Contents/PlugIns/sshfs.mfplugin/Contents/Resources/sshnodelay.so

Your MacFusion installation should now be working on Snow Leopard. I've tested SSH and FTP connectivity so far, and they both appear to be working. Thanks again to Geoff for the fix!

Fix MacFusion on Snow Leopard is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Create a simple SOCKS proxy with SSH by using the -D flag:

ssh -D 2400 username@some.host.com

That command will open up a SOCKS proxy on your workstation on port 2400. If you configure your application to use the local SOCKS proxy, any traffic using the proxy will be sent through an encrypted SSH connection to your remote server and out to the internet. Inbound traffic through the proxy is encrypted through the same connection.

You can pair that with autossh to keep your proxy connected at all times:

autossh -f -M 20000 -D 2400 username@some.host.com -N

Simple SOCKS proxy using SSH is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.