MySQL: Restore Master from Slave

I had a power-outage in one of my Linux clusters. The master MySQL server was corrupted and wouldn't start properly. After trying a bunch of bin-log/relay fixes and other failed attempts, I finally decided to uninstall/re-install and that was so quick and simple and everything worked perfectly. Here are the steps for CentOS/RHEL:

 service mysqld stop
 mv /etc/my.cnf /root/
 mv /var/lib/mysql /var/lib/mysql.orig/
 rpm -e mysql-server
 yum -y install mysql-server
 service mysqld start
 
# set pw and take defaults
 /usr/bin/mysql_secure_installation 
 mysqldump -h working-mysql-server -u root --all-databases --quick --lock-all-tables | mysql -u root -p
 mv /root/my.cnf /etc/
 service mysqld restart

Now, check /var/log/mysqld.log for errors. If all is well, run:

 mysql -e 'show slave status\G'

To get the current status.

Apache: How to Automatically Null Route Abusive Hosts

I notice tons of suspicious requests in my apache log files. I simply don't like this, so if someone or something makes a request for a file that doesn't exist on one of my systems, that earns them a lifetime ban. Here's how we do that. First, modify /etc/httpd/conf/httpd.conf and find this line: DocumentRoot "/var/www/html" Add this, directly after:

RewriteEngine on
RewriteCond %{REQUEST_URI} !/myfile1.html$
RewriteCond %{REQUEST_URI} !/myfile2.html$
RewriteRule ^(.*)$ /block.cgi

Where myfile1.html and myfile2.html are files that actually exist. This will route all other requests to a file called 'block.cgi'. Now, create a file, called: /var/www/html/block.cgi which contains:

#!/usr/bin/perl 
#
# The null router..   There is a redirect in http.conf which executes
# this script by default. Because really, if someone is hitting us
# with a GET request on /, they're up to no good so bye-bye.
#
# McLovin

use strict;
use CGI qw(:standard);

my $date = `date`;
chomp $date;
my $ip = $ENV{'REMOTE_ADDR'};

open FH, ">> /root/blocked_hosts.txt";
print FH "#$date\n/sbin/route add $ip gw 127.0.0.1 lo\n";
close FH;

print header;
print start_html("Environment");
system "sudo /sbin/route add $ip gw 127.0.0.1 lo"; 
print end_html;

Make sure that file is executable, next, make sure that the apache user can execute that script by adding the following to sudoers:

apache ALL = NOPASSWD: /sbin/route

One final touch, I log each blocked host, so that file needs to be owned by user: apache.

touch /root/blocked_hosts.txt && chown apache:apache /root/blocked_hosts.txt

Now, test that out and make sure it's working. After a failed request, you should see the IP in the output of netstat, like this:

netstat -rn|grep lo$
222.209.132.155 127.0.0.1       255.255.255.255 UGH       0 0          0 lo
202.53.8.82     127.0.0.1       255.255.255.255 UGH       0 0          0 lo

CentOS SSHFS Howto

I have a need to remotely mount a file system over the Internet. I'd like to do so without doing a VPN tunnel and all the encryption myself. So, I gave sshfs a shot and turn's out, it's super easy. Here's how on CentOS:

# Install EPEL
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

# Install sshfs
yum -y install sshfs fuse

# Load FUSE Kernel Module
modprobe fuse

# Mount remote file system.
mkdir /mnt/sshfs/
sshfs root@my.server.com:/export/opt/ /mnt/sshfs/

Now if you want to get fancy, you can push your ssh key and auto mount on boot, like this:

# Push identity
ssh-copy-id user@my.server.com

# Edit /etc/fstab and add:
root@my.server.com:/export/opt/ /mnt/sshfs fuse.sshfs defaults,noauto,user 0 0

# Mount it up
mount /mnt/sshfs/

Using IPTables to Blackhole Large Set's of IP's

I found a host where I was getting a bunch of POST's in my apache server log files which looked to be malicious. I wanted to go through and just block all IP's which were trying to post to my web server, since I don't have anything but static content on it. So, I came up with this little one liner:

grep POST /var/log/apache2/*log* | perl -lane 'print $1 if (/^.*?:(.*?)\s/)'|sort | uniq | perl -lane 'system "iptables -A INPUT -s @F[0] -j DROP"'

This is useful for any group of IP's you wish to black-hole.

Creating an LVM Volume on EBS with XFS

Well that's mouth full.. I wanted to add some storage to a VM but wanted to be able to add to it later if I got to that point. So, I attached a 100G EBS volume, and here's how I formatted it. It's EBS as the physical volume, then LVM formatted as XFS. You can attach another EBS volume to this VM, then add it to the volume group later, striped for increased capacity. Anyway, here's how it's done:

rpm -q xfsprogs &> /dev/null || yum -y instal xfsprogs
mkdir /export
pvcreate /dev/vdb
vgcreate VolGroup_EBS /dev/vdb
lvcreate -I 2M -l 100%FREE -n export VolGroup_EBS
mkfs.xfs /dev/mapper/VolGroup_EBS-export
mount /dev/mapper/VolGropu_EBS-export /export/

Finally, add this to /etc/fstab to get it mounted when your system boots:

/dev/mapper/VolGroup_EBS-export /export  xfs defaults 1 1

Multi Hop SSH SOCKS Proxy

From a corporate network I had a need to jump to one system, and through it to another in order to have an open web proxy to another 'internal' network - or in this case it was a lab network where I had to hit a openstack Horizon dashboard. It's a somewhat simple concept and I was sure SSH could do it but I had some trouble figuring out how. The scenario looks like this:

my macbook -> ssh server <- INTERNET -> second-ssh-server -> browse internal network

Note that the second ssh server was running sshd on port 22000, you probably don't need that. The command I came up with to accomplish this was:

ssh -t -t -v -L9999:localhost:9932 root@ssh-server ssh -t -D 9932 root@second-ssh-server -p 22000

A whole blog-post for one command? Yes, it was that cool!

Find the Fastest Mirror in Bash

I needed a script that would determine the fastest mirror in a list of addresses. In this example I'm using the mighty 'www.google.com' and port 80 but it's obviously configurable. I also needed to use nano seconds rather than seconds because all the results (for google) came back in less than 1 second. It should actually be a bash function but I'll leave that to you, fine reader. You're welcome!

mirror='www.google.com'
port='80'
iplist=()

for ip in $(dig $mirror A | perl -lane 'print $1 if (/A\s+(.*?)$/)')
do
   # Give me a resonable starting point (Using epoch in nano seconds).
   a=$(($(date +'%s * 1000 + %-N / 1000000')))
   nc -w 2 -z $ip $port &> /dev/null &
   b=$(($(date +'%s * 1000 + %-N / 1000000')))
   msec=`expr $b - $a`
   iplist+=("$msec $ip")
done

# Grab the fastest one..
fastest=$(for a in "${iplist[@]}"; do echo "$a"; done | sort -n | head -1 | awk '{print $2}')

echo $fastest

sshwatch using geoiptool.com

I've written a script called 'sshwatch' that will process the default /var/log/secure file, associate the folks that have logged in against a a geoip lookup (using geoiptool.com). The idea is that you should be wary of people logging in from countries that are not expected. It will only look up each IP address once, so if you have multiple logins from the same IP, we only do a single geoip lookup. If you put this script in /etc/cron.daily/sshwatch - you'll get an e-mail each night about who's been on your box. I think it only works on RHEL and CentOS right now. Enjoy:

#!/usr/bin/perl
# Send a little report of who's been loggin in and from where.
# Joey 

use strict;
my %userblob;
$|++;

for (`cat /var/log/secure*`) {

   if (/Accepted/) { # Somebody logged in..

      my ( $user, $ip) = ( $1, $2 ) if (/for (.*?) from (.*?) /);
      $userblob{$ip}->{'IP'} = $ip;

      unless ( $userblob{$ip}->{'COUNTRY'} ) {
         $userblob{$ip}->{'COUNTRY'} = get_country($ip);
      }

      my $seen;
      for ( @{$userblob{$ip}->{'USER'}} ) {
         $seen ++ if ( $_ eq $user );
      }
      push @{$userblob{$ip}->{'USER'}}, $user unless ($seen)
   }
}

my @mail;
while ( my ($ip, $ref) = each %userblob ) {
   push @mail, "$ip :: $ref->{'COUNTRY'} :: @{$ref->{'USER'}}\n";
}

send_mail(@mail);

sub get_country() {

   my $ip = shift;
   print "Looking up: $ip: ";
   my $url = "http://www.geoiptool.com/en/?IP=$ip";
   my $data = `GET "$url"`;
   my $country = $1 if ( $data =~ /Country:.*\n.*\> (.*?)<\/a/m );
   print "$country\n";

   return $country if ($country);
   return undef;
}

sub send_mail() {

   my @body = @_;

   open  MAIL, "|/usr/sbin/sendmail.postfix -t";
   print MAIL "to: your.e-mail\@domain.com\n"
            . "from: your.mama\@domain.com\n"
            . "Subject: SSHWatch Report\n\n";

   map { print MAIL "$_" }@body;
   close(MAIL);
}

Linux: Splitting a Large File into Small Files

Recently I was trying to transfer a large ISO file across a horribly unstable VPN. The transfer would fail at various amounts of transfer percentages. So, I thought I'd best split the file up into 10MB chunks, then rsync those over and stitch it back together.

That way if it failed 90% of the way through, I wouldn't have to resend all the data, just that last 10%. The way I managed to do this, was to:

split --bytes=10m file.iso file_part 

What happens now is, you have a bunch of 10MB files called

file_partaa
file_partab
flie_partac
file_partad
...

So, just rsync all those files to the destination:

rsync -e ssh -a --progress file_part* user@desination.host.com:

When that completes, login to the remote host and put them back together:

cat file_part* > orig_file.iso

Done and done.

Gluster Setup in the Cloud: Simple, Easy

Here, I'll spin up two instances and configure distributed, replicated storage between them. I'll do two instances but I've actually done it for up to six instances. I have no idea how far it would scale but I'm guessing 30 or more would work fine using this method.

First I fired up two CentOS 5 instances which will be talking to each other over the private network. I'll name them node-1 and node-2 and add entries for both to /etc/hosts, like so:

10.19.128.5             node-1
10.19.128.6             node-2

They are in the same security group, so they can see each other. After you add the entries, ping the other side, like this:

[root@node-1 ~]# ping node-2
PING node-2 (10.19.128.6) 56(84) bytes of data.
64 bytes from node-2 (10.19.128.6): icmp_seq=1 ttl=64 time=0.140 ms
64 bytes from node-2 (10.19.128.6): icmp_seq=2 ttl=64 time=0.138 ms

--- node-2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.138/0.139/0.140/0.001 ms

Next, grab and install the software:

wget http://download.gluster.com/pub/gluster/glusterfs/LATEST/RHEL/glusterfs-core-3.1.3-1.x86_64.rpm
wget http://download.gluster.com/pub/gluster/glusterfs/LATEST/RHEL/glusterfs-fuse-3.1.3-1.x86_64.rpm

rpm -Uvh gluster*.rpm
rm gluster*.rpm

Then, load the fuse module:

modprobe fuse

Then, start glusterd

/etc/init.d/glusterd start

Pick some directories to use, in this case, gluster will use /export/queue-data and you and your apps will use /queue. So, don't every access files in /export/queue-data, gluster owns that directory:

mkdir /queue /export/queue-data

Setup the clients so they can see/talk to each other, run this on each system. From node-1:

[root@node-1 ~]# gluster peer probe node-2
Probe successful

[root@node-2 ~]# gluster peer probe node-1
Probe successful

Next, create your directories on both systems:

[root@node-1 ~]# mkdir -p /queue /export/queue-data/
[root@node-2 ~]# mkdir -p /queue /export/queue-data/

Now, create your volume:

[root@node-1 ~]# gluster volume create queue-data replica 2 node-1:/export/queue-data node-2:/export/queue-data
Creation of volume queue-data has been successful. Please start the volume to access data.

Ready to start the volume export:

[root@node-1 ~]# gluster volume start queue-data
Starting volume queue-data has been successful

To manually mount the volume, run:

[root@node-1 ~]# mount -t glusterfs 127.0.0.1:queue-data /queue

You can see that it's been mounted here:

[root@node-1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             2.0G  1.5G  445M  77% /
/dev/sda2             7.9G  152M  7.4G   2% /opt
none                  256M     0  256M   0% /dev/shm
glusterfs#127.0.0.1:queue-data
                      2.0G  1.5G  445M  77% /queue

To mount it automatically on boot, run:

[root@node-2 ~]# echo "127.0.0.1:queue-data    /queue     glusterfs defaults,_netdev 0 0" >> /etc/fstab
[root@node-2 ~]# mount -a

If you're doing something different and want to be able to run VM's off your glusterfs, add this to fstab:

127.0.0.1:queue-data          /queue                   glusterfs direct-io-mode=disable,_netdev 0 0

And you can see that on node-2 it's also been mounted:

[root@node-2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             2.0G  1.5G  445M  77% /
/dev/sda2             7.9G  152M  7.4G   2% /opt
none                  256M     0  256M   0% /dev/shm
glusterfs#127.0.0.1:queue-data
                      2.0G  1.5G  445M  77% /queue

Now, let's make sure it works. I'll create a file on node-1 and then make sure it exists on node-2:

[root@node-1 ~]# cd /queue && dd if=/dev/zero of=output.dat bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.572936 seconds, 18.3 MB/s

And, here I see it on node-2:

[root@node-2 ~]# ls -al /queue
total 10276
drwxr-xr-x  2 root root     4096 Apr  5 12:49 .
drwxr-xr-x 25 root root     4096 Apr  5 12:43 ..
-rw-r--r--  1 root root 10485760 Apr  5 12:49 output.dat

And, there you have it. If you require iptables rules, which I don't because I'm already behind ec2's ACL's, add something like this to node-1 and changing the IP for node-2:

# /etc/sysconfig/iptables
-A INPUT -m state --state NEW -p tcp --dport 24007 --source 10.19.128.6 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport 24008 --source 10.19.128.6 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport 24009 --source 10.19.128.6 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport 24010 --source 10.19.128.6 -j ACCEPT

You could also just do something like this:

A INPUT -m state --state NEW -p tcp --dport 24007:24010 --source 10.19.128.6 -j ACCEPT

Ruby and Rails and MySQL and Apache and Passenger and Git: Install on CentOS 5

My New Years resolution was to learn ruby on rails, the popular web application framework. It just so happens I've been assigned a ruby project at work. Go figure..

I've been watching some screen-cast stuff my buddy at Pivotal lent me and the first thing to do was to install the stuff. Looks like it's pretty much dominated by people that develop on their Mac, then push apps to some public Rails hosting company. That's fine but I have a hankerin to install in my own CentOS 5 environment. It was actually a lot more tinkering than it should be, which is weird. So, I figured I'd better lay it out so I don't forget how it's done. Hopefully this'll help others. The latest version of ruby comes with ruby gems, the ruby package manager. So you don't need to install that separately.

Something that is interesting to note, Passenger is really mod_ruby - an apache module. They don't call it mod_ruby but that's what it is. Similar in theory to mod_perl and mod_php.



You'll get the latest stable ruby as of this writing (ruby-1.9.2-p136) + rails 3.0.3.

Another note, I'm using MySQL as my back end database but to do default 'bundle installs', you need sqlite so I'm installing that as well.

#

# Install deps
# Install the latest version of Ruby and Rails and Git to run in Apache + Passenger.
# Includes MySQL and sqlite.

yum -y install httpd zlib-devel curl-devel openssl-devel httpd-devel apr-devel apr-util-devel

wget ftp://ftp.ruby-lang.org//pub/ruby/1.9/ruby-1.9.2-p136.tar.gz
tar -xvf ruby-1.9.2-p136.tar.gz
cd ruby-1.9.2-p136
./configure --enable-shared --enable-pthread
make && make install
cd ext/zlib
ruby extconf.rb --with-zlib-include=/usr/include --with-zlib-lib=/usr/lib64
cd ../..
make
make install
ruby --version

# Installing sqlite
wget http://www.sqlite.org/sqlite-autoconf-3070400.tar.gz
tar -zxf sqlite-autoconf-3070400.tar.gz
cd sqlite-autoconf-3070400
./configure && make && make install


gem install rails
gem install passenger
gem install mysql -- --with-mysql-conf=/usr/bin/mysql --with-mysql-lib=/usr/lib64/mysql


wget http://kernel.org/pub/software/scm/git/git-1.7.3.2.tar.bz2
tar xvf git*.*
cd git*
make prefix=/usr/local all
make prefix=/usr/local install


# run this by hand because you have to answer some questions.
# passenger-install-apache2-module

Next, you'll want to configure a hosting environment with a custom http config:


vi /etc/httpd/conf.d/rails_app.conf

<VirtualHost *:80>
    ServerName www.domain.com
    DocumentRoot /opt/rails/myapp/public
    <Directory /opt/rails/myapp/public>
        Allow from all
        Options -MultiViews
    </Directory>
</VirtualHost>

Creating a Virtual Private Cluster with OpenVPN

I'd like to change gears and look at using the cloud in a bit of a new way for me. That is, as a direct extension of the LAN. This post represents a bunch of research on my part. It's what I spent the 2010 Christmas holiday working on. If you get bored with all the tech stuff here, make sure you read the last couple of paragraphs in this post so you can see what the end-goal really is. Overwhelming possibilities is all I can say. It's more than the title implies - so keep an open mind after it's all setup and working.

In this post, I'll create a new security group and connect it to my office network via a VPN tunnel. Then, I'll route packets to instances running in that security group through the tunnel, just as if it were on my local office LAN. You just need a single port open to the security group to create the tunnel. I'll also mention that you can spend a lot of money and pay for something similar, however by rolling your own you have way more flexibility. It's an investment to understand how it all works together to be sure but once I got it down I feel like I'm on another, higher level of understanding networking. For me it was a really great project.

So, let's get started. First, I'm going to create a new security group and call it vpc for virtual private cloud:

ec2-add-group VPC -d "Virtual Private Cloud"
GROUP VPC Virtual Private Cloud

Now, I'm going to authorize two ports for my group, 1194 for the tunnel and 22000 for sshd while I get everything configured. Keep in mind I usually run sshd on a port other than 22 because of annoying scan-bots. I'll de-authorize port 22000 when I'm up and running.

# ec2auth -Ptcp -p22000 VPC
# ec2auth -Pudp -p1194 VPC

Now, I'll spin up an instance, install openvpn and configure the service. The following instance is an AMI I created that doesn't have OpenVPN installed.

# ec2run -k joeyssh -g VPC pmi-182a79e7

Ok, I'm going to start another instance in that VPC so we can test end-to-end connectivity once we get the tunnel setup. Same command as above.

Now I can check the IP's of my instances and login.

# ec2din

Ok, now I'm logged into one of the instances I'm going to rename openvpn-server because it's going to be my OpenVPN server system.

# hostname openvpn-server

Next, I'm going to add the RPMForge repos to this system and install it. The nice thing about doing it this way is that you'll solve all of OpenVPN's dependencies at the same time. They are stuff like openssl-devel, lzo-devel, pam-devel, etc.

So, install the RPMForge RPM for your architecture:

# rpm -Uhv http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS//rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm

Then, install OpenVPN:

# yum -y install openvpn

Now that the software is installed, let's configure the OpenVPN server. We're going to create a set of certificates for authenticating connections. We're going to create 3 certs. One is the certificate authority, or CA. This will be used to sign both the server certificate and the client certificate. Next, I'll create the server and client certs that will be used to authenticate and encrypt the tunnel. In order to do this, I'll use the 'EasyRSA' software that comes with openvpn. This is specific to CentOS, your distro's methods may vary.

Copy easy-rsa into /etc/openvpn/

# rsync -a /usr/share/doc/openvpn-2.1.4/easy-rsa/2.0/ /etc/openvpn/easy-rsa/

Before we create our certificates and keys, I'm going to edit the vars file which contains the default values for my certificates. They're at the very bottom of the file, look for KEY_COUNTRY through KEY_EMAIL and customize them for your environment:

# vi /etc/openvpn/easy-rsa/vars

This is a bit of a hack but for some reason all the shell scripts that easy-rsa provides aren't marked as executable. So, to fix that I ran:

# cd /etc/openvpn/easy-rsa/
# file * | perl -lane 'system "chmod 755 $1" if (/(.*?):.*?Bourne.*?/)'

Next, source the vars file and build the Certificate Authority certificate:

# cd /etc/openpvn/easy-rsa/
# . vars
# ./clean-all
# ./build-ca

Ok, next we'll create the openpvn server's certificate:

# ./build-key-server openvpn-server

Next, we build the Diffie-Hellman key:

# ./build-dh

And finally, we'll build our client certificate which will be installed on a PC at the office to setup the tunnel.

# ./build-key openvpn-client

Now, we'll configure the openvpn server. Before doing so, we need to know two things. One, the network information for my office LAN and the network information for the AWS LAN. In my case, they are:

Office: 172.20.0.0/24
AWS: 10.19.237.80/28

That plays into my config file, notice the customizations, "push route" and "route". Note also that for the tunnel itself I'm using 172.16.130.0/24. You can just leave that alone, or change it. It doesn't really matter what you use there. It's less confusing to use something that doesn't overlap with either your office or AWS networks though. Create a file (/etc/openvpn/vpc.conf) and add the following:

# OpenVPN Server Config
port 1194
proto udp
dev tun

# Certificates
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/openvpn-server.crt
key /etc/openvpn/easy-rsa/keys/openvpn-server.key # This file should be kept secret
dh /etc/openvpn/easy-rsa/keys/dh1024.pem

server 172.16.130.0 255.255.255.0
push "route 10.19.237.80 255.255.255.240"
log-append  /var/log/openvpn.vpc.log
verb           3
status         /etc/openvpn/vpc_status.log
keepalive 20 100
persist-tun
persist-key

push "dhcp-option DNS 8.8.8.8"
client-to-client
client-config-dir /etc/openvpn/ccd
route 172.20.0.0 255.255.255.0

user nobody
group nobody
comp-lzo 

We haven't configured OpenVPN to use pre-shared (aka static) keys because that's less secure. Static keys are bad because if a system with the shared key is lost or stolen, the shared key must be regenerated and replaced on all systems running OpenVPN. We're running asymmetric (or two-way) encryption to ensure the identity of the VPN partner. The way that works, is the client and server both have a public and private key. They trade public keys when the connection starts and start encrypting traffic for the partner with the public key. They decrypt traffic with the private key. Only the recipient's private can decrypt data encoded by his public key. If a system is lost or stolen an administrator can simply revoke the certificates (on openvpn-server) that belong to the client in question.

In our configuration we're using asymmetric public key encryption to establish a session, then OpenVPN negotiates a static key between hosts for tunnel encryption. These expire on a regular basis and are regenerated. I see things in my logs like:

TLS: tls_process, killed expiring key

I think this is because if there's a man-in-the-middle type of attack, by the time they decrypt that static key and can actually eavesdrop on the session, the key has expired and been regenerated. By default, the time is 60 seconds - which seems good to me? :)

Now, before we start the service, it's important that both systems have accurate clocks. If the time is off by more than 5 minutes on either side, the tunnel goes away and doesn't come back until the clocks are back in sync. The easiest way to manage that is just run ntpd:

yum -y install ntp
chkconfig ntpd on
service ntpd start

Now, let's fire up OpenVPN:

# chkconfig openvpn on
# service openvpn start

You should now see openvpn running as user nobody and listening on port 1194:

# lsof -i:1194
COMMAND  PID   USER   FD   TYPE DEVICE SIZE NODE NAME
openvpn 1448 nobody    4u  IPv4   4866       UDP *:openvpn 

If you have any trouble starting the service, look in: /var/log/openvpn.vpc.log for errors.

Ok, so that's the server side config. Now, we need an OpenVPN client on your LAN to create our tunnel. So, pick a box on the network and install OpenVPN. In my case, I'm going to use a Xen instance on an existing infrastructure system but you can use a desktop system or an old PC or whatever.

If you're running CentOS x86_64, use the instructions above to install RPMForge and OpenVPN.

I've called this system 'openvpn-client' and installed the software I need. So now, I need to copy down the keys I generated for this host from the OpenVPN Server in the cloud. To do this, simply:

# mkdir /etc/openvpn/keys/
# cd /etc/openvpn/keys/
# scp -P 22000 -i ~/.ssh/mykey root@publicipofopenvpnserver:/etc/openvpn/easy-rsa/keys/openvpn-client.crt . 
# scp -P 22000 -i ~/.ssh/mykey root@publicipofopenvpnserver:/etc/openvpn/easy-rsa/keys/openvpn-client.key .
# scp -P 22000 -i ~/.ssh/mykey root@publicipofopenvpnserver:/etc/openvpn/easy-rsa/keys/ca.crt .

Ok, now we need the OpenVPN client configuration. The changes to this file you'll need to make are the name/location of the certificates and the PUBLIC IP of your OpenVPN server. Create a file on the openvpn-client system (/etc/openvpn/vpc.conf) and add the following:

client
dev tun
proto udp

remote 204.231.109.96 1194
resolv-retry 10
nobind
persist-key
persist-tun
ca   /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/openvpn-client.crt
key  /etc/openvpn/keys/openvpn-client.key

verb           3
status-version 2
log-append  /var/log/openvpn.log

syslog
mute 10

# Enable Compression
comp-lzo

Now, fire up the service on openvpn-client, which will create the tunnel:

# chkconfig openvpn on
# service openvpn start

On the client, you should see that a new virtual tun interface has been created, tun0:

# ip addr show tun0
4: tun0: -POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP- mtu 1500 qdisc pfifo_fast qlen 100
    link/[65534] 
    inet 172.16.130.6 peer 172.16.130.5/32 scope global tun0

So, we've established a connection to the VPN server, which is great. That's the magic part, it makes the systems in the cloud only one hop away through the virtual network:

# traceroute 172.16.130.1
traceroute to 172.16.130.1 (172.16.130.1), 30 hops max, 40 byte packets
 1  172.16.130.1 (172.16.130.1)  110.496 ms  111.075 ms  111.100 ms

Now, you'll also notice that I have a route, automagically added by the openvpn software that makes the cloud LAN, local (in a sense). In this example, the cloud systems have LAN addresses of 10.19.237.80/28 and I have a route for that network now, through my new tunnel:

# route -n | grep ^10
10.19.237.80    172.16.130.5    255.255.255.240 UG    0      0        0 tun0

So, in theory, I should be able to ping the eth0 interface of my cloud instance, which you can see below:

# ping 10.19.237.82
PING 10.19.237.82 (10.19.237.82) 56(84) bytes of data.
64 bytes from 10.19.237.82: icmp_seq=1 ttl=64 time=202 ms
64 bytes from 10.19.237.82: icmp_seq=2 ttl=64 time=131 ms

And of course I can now access that instance directly from my LAN. No need to hit the big bad internet:

ssh -l root -i ~/.ssh/yoursshkey root@10.19.237.82 -p 22000

That's a beautiful thing! Now, I can turn off (or revoke) the ACL that allows ssh access into port 22000 from my security group:

ec2-revoke VPC -Ptcp -p22000

We're still not done yet, we need to turn on packet forwarding on both openpvn-server and openvpn-client. Simply do this on both systems:

# echo 1 > /proc/sys/net/ipv4/ip_forward

Now, you're going to need to add a route, on your office gateway system. That route is going to tell all the hosts on your network that if they want to get to the AWS LAN network, they need to go through the openvpn-client system. In my case, it's something like:

# ip route add 10.19.237.80/28 via 172.20.0.9

That's because my gateway is a Linux box but all the DSL/Cable routers have the ability to add routes. Just figure out how to do that on your gateway router.

In addition, you're going to need to add routes to each instance in AWS that tells each system how to get to your office LAN. So in my case I have two instances currently running in my VPC security group. The openvpn-server and a subversion server, called svn. So, I log into the svn server and run:

# ip route add 172.16.130.0/24 via 10.19.237.82

Where 10.19.237.82 is the eth0 IP of my openvpn-server system. So now, I can ping my SVN server from my LAN. Alternatively, you could make your office the default route for all the AWS systems which may make sense in some situations.

That's pretty much scratching the surface here..

Now, what's really pretty cool about this setup, is that if you had say a San Francisco office, you could build out another openvpn-client system in SF and route packets between sites, securely through your cloud instances. Not only that but you could just make the cloud, your default route for both office networks and install Snort or whatever IDS you want and maybe a firewall distro on an instance if you want.. how about a PBX like trixbox? Total control over your networks. This concept really offers overwhelming possibilities. An office SMB share mounted on an EBS drive for example - basically you can move your IT infrastructure to the cloud.

Another possibility is that because OpenVPN supports HTTP and SOCKS proxies, you could tunnel a network from your office, through the HTTP proxy and have the public AWS systems be local to your LAN there by making those systems totally 'greenside', this is if you worked for say, some telecommunications giant with a tightly controlled internal network, for example.

If you were to run OpenVPN in bridge mode (using the tap interface) you could even route broadcast traffic and even IPX and non-IP packets to the other locations. And, there's more, traffic shaping isn't only possible it's built in to OpenVPN. In addition, if you loose your IP on the DSL office line and re-up to your Internet provider, the tunnel is re-established so quickly that none of your SSH sessions will die, they'll simply freeze up for some seconds while one of the partners gets a new IP, and then continue working normally.

Ok, also now that I've showed you how to do all this by hand, there are some dedicated firewall/gateway distro's, like Shorewall that I haven't used but that have built in support for OpenVPN, so the client config and routing stuff are done in a WebGUI. Might be a fun experiment at some point.

Then of course you can have your telecommuters or road warriors connect to OpenVPN from the road and have access to the corporate network. Again, the possibilities are overwhelming.

I'll have to do a post on getting Snort going, that would be really fun. As usual, post questions and or problems, thanks for stopping by!

Using the Mighty IPTables to Prevent an HTTP(s) DoS Attack

Using this procedure, the kernel netfilter will deny (and log to /var/log/messages) packets to ports 80,443 from hosts that exceed 20 requests in 5 seconds. IPTables will then DROP packets for 5 seconds, then allow them back to. This has the benefit of not blocking legitimate traffic, only slowing it to a reasonable amount.
So, let's get started, install iptables:

yum -y install iptables

IPTables, by default only timestamps and tracks up to 20 connections. Which isn't very many. This means that by default if you use --hitcount 21 you'll error out. You can control the limit by updating /etc/modprob.d/modprobe.conf:

options ipt_recent ip_pkt_list_tot=50

Then, reload the ipt_recent kernel module:

rmmod ipt_recent
modprobe ipt_recent

Next, create the script that will add the rules (vi /tmp/limit.sh):

# Create a LOGDROP chain to log and drop packets
iptables -N LOGDROP
iptables -A LOGDROP -j LOG
iptables -A LOGDROP -j DROP

iptables -A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -m recent --set --name "limit-http" --rsource
iptables -A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -m recent --update --seconds 5 --hitcount 20 --name "limit-http" --rsource -j LOGDROP
iptables -A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT

iptables -A INPUT -p tcp -m tcp --dport 443 -m state --state NEW -m recent --set --name "limit-https" --rsource
iptables -A INPUT -p tcp -m tcp --dport 443 -m state --state NEW -m recent --update --seconds 5 --hitcount 20 --name "limit-https" --rsource -j LOGDROP
iptables -A INPUT -p tcp -m tcp --dport 443 -m state --state NEW -j ACCEPT

Execute that script:

sh /tmp/limit.sh

Now, we want these rules to be applied on each reboot, so only do this if you have nothing in /etc/sysconfig/iptables - which most ec2 clients don't even have. If you have stuff in there, just bake the above in. so do (on CentOS/RHEL derivatives):

iptables-save > /etc/sysconfig/iptables

To check, do:

service iptables stop
service iptables start
service iptables status

Helpful NOTES:

You can see what the packet filter is doing, in real time, like this:

watch iptables -nvx -L

Also, you can use apache's benchmarking program, ab to trip the filter for testing purposes, like this:

ab -n 1000 -c 5 http://IP/index.html

Where 1000 is the total number of requests and 5 is the number of concurrent requests.


So, to test it out, point that at your webserver and tail /var/log/messages, and you'll find that you start dropping packets from the client running apache bench.


You're welcome! :)

Running OpenVAS Security Scanner: Ubuntu 10.10

The nessus project is now a for-pay company. I think you can get a free home license but if you want to scan your infrastructure at work, they no likey. A project called OpenVAS is a fork of the Nessus project that's all open source and free.

I want to continually scan some systems and generate reports I can diff to see if any jokers have added services or change rules that expose network services I don't want exposed. So, I'm installing OpenVAS on a VM instance that I'm going to use to scan my infrastructure. I'm not going to use the GUI client because I want this to be scripted. The way OpenVAS works is, you have an OpenVAS server, which you connect to with clients and tell it what to do. So, you could install the software on a system in a data center or EC2 or whatever, then run the client from your desktop and have it do you're bidding.

In this case, my client - is the command line client which is going to run on the same system as the server.

To install on Ubuntu 10.10, simply do:

# Update your distro
apt-get update && apt-get dist-upgrade

# Install openvas server and client software + security plugins
apt-get install openvas-server openvas-client \
   openvas-plugins-base openvas-plugins-dfsg

# Update the vuln. database
openvas-nvt-sync

Add a user that you're going to use from the client, to login:

openvas-adduser

Here, you'll add a user/pass combination.

When prompted to add a 'rule' - I allow my user to do everything. The rules allow/disallow scanning of hosts. If you want you can let bob scan 192.168.0.0/24 or whatever. I want my user to scan all, so when prompted, simply enter

default accept

Now, fire up the server. Note that the first time you run, it loads all those checks into memory so it takes a LONG time for the server to actually start.

/etc/init.d/openvas-server start

Now, you can start scanning. Create a file with IP's and/or hostnames that your client will feed to the server to scan. Something like this:

192.168.1.5
www.mydomain.com
dns.mydomain.com
10.1.19.0/24

etc.


The server listens on port: 9390 by default so you'll want to tell your client to connect there. Once you have the file created, you can kick off your scan like this:

OpenVAS-Client -q 127.0.0.1 9390 admin scanme.txt -T html \
     ~/Desktop/openvas-output-`date`.html 

You'll be prompted to accept the SSL certificate, go ahead, it's automagically created by the pkg when it's installed. Then, open that file in a browser when it's done and start going through it. Be warned, scanning is very hostile so you should really only scan your own systems.. and those of your enemies.

Deploying a Set of HAProxy Servers as EC2 Instances

If you want high availability in EC2, one option is to deploy a couple of load balancers in front of a bunch of application servers, all in the same security group. I really like HAProxy. It's fast, and very configurable. I remember we got a couple of F5's back in the day and I wanna say that was like 60k for two. Which is a lot if you're a startup. Not that you can deploy them in EC2 but it's just kinda cool how all this stuff trickles on down to the cloud.

I think it makes complete sense. Anyway, here's how I did it recently. Since you can't get a floater IP in EC2, or more than one IP per instance, I setup DNS round robin for the proxy addresses. So, in this case we have two dedicated ec2 instances, running CentOS. Each one will have haproxy installed and configured. So, once you get them spun up, you'll want to get an elastic IP for each, then configure DNS to point to both.

www.domain.com { 1.2.3.4, 1.2.3.5 }

Each IP is the public address of each of your HAProxy servers. To test, you can just setup a dummy hostname, like test.domain.com pointing to those IP's and do the cut over when you're sure you're happy with the setup.

So, next, login to each instance and run:

yum -y install haproxy

HAProxy supports two modes, tcp and http. You can't do SSL in http mode so, this deployment was in tcp mode. HTTP mode has some really cool and interesting features with HAProxy's recent acl additions. Google around for HAProxy and ACL. You can get super granular on which app server handles what kind of traffic to include or where to direct certain kinds of request.. for example: go here for SSL, here for dynamic content and here for static HTML and images. It's really pretty cool and new in 1.3, I think. Maybe I'll try all that out someday.

Next, edit /etc/haproxy/haproxy.cfg and enter the following:

# HA Proxy Configuration
defaults
balance roundrobin

global
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     20000
    user        haproxy
    group       haproxy
    daemon

defaults
    mode        tcp
    log         127.0.0.1       local0
    log         127.0.0.1       local1 notice
    option      dontlognull
    option      redispatch
    timeout connect 2000 # default 2 second time out if a backend is not found
    timeout client 300000
    timeout server 300000
    maxconn     60000
    retries     3

# This is what we're listening on.
frontend haproxy *:443
   mode tcp
   maxconn 20480
   default_backend app_servers

# This is who we send requests to
backend app_servers
   mode tcp
   server app1 10.19.127.30:443
   server app2 10.19.127.49:443
   server app3 10.19.127.21:443
   server app4 10.19.127.19:443

So, in this example we have 4 app servers. I feel like it's so simple and self explanatory that you can just get in there and edit and test it out. Both HAProxy instances have the exact same configuration file - assuming you've deployed everything to the same security group.

The only other thing i did was to add a snippit to /etc/syslog-ng/syslog-ng.conf to log all HAProxy's messages via syslog. That's here /etc/syslog-ng/syslog-ng.conf:

source s_udp {
       udp(ip(127.0.0.1) port(514));
};
destination d_haproxy { file("/var/log/haproxy"); };
filter f_local0 { facility(local0); };
log { source(s_udp); filter(f_local0); destination(d_haproxy); };

Now, just fire it all up:

service haproxy start
chkconfig haproxy on
service syslog-ng restart

HAProxy has a stats interface which I haven't enabled here. If I do, I'll edit the above with the stat's info config.

How To Create A Gluster Roll for Rocks Clusters

If you're here you probably already know what rocks is and why it's awesome. In this post, I'm going to download the Gluster 3.0.5 code and build a roll I can use to deploy to all my compute nodes. So, first off gluster is a super cool product. My only complaint is, it's missing some tools for troubleshooting and status checks. Thus, I've written some nagios checks that have been working out quite well though, so I get alerts if any badness goes down.

The thing I LOVE about Gluster? No meta-nodes! Seriously, that is KEY. If you manage a storage platform and deal with meta-nodes you know what I'm talking about. Clustered meta-nodes? Even more annoying. You have replicated storage, why would you need a clustered meta node?! Stuff all that information into the storage cloud and spread bits around. Not to mention, meta-nodes require a physical RAID, etc, which for one is oh, about $10k. Another thing I like about Gluster? You can run it between ec2 instances - no meta-nodes! That's another posting, how to get Gluster going on a bunch of cloud instances.. So anyway, let's jump right in, shall we? Also, sorry about the formatting for my code and cmds. Blogger might not be the best platform for me. Oh well....

Login to your frontend..

In order to compile Gluster, you'll need to install the following:

yum -y install flex libibverbs-devel

Next, download the Gluster source:

cd /root/
mkdir gluster-3.0.5/
cd $!
wget http://download.gluster.com/pub/gluster/glusterfs/3.0/3.0.5/glusterfs-3.0.5.tar.gz

Next, we're going to turn that code into some RPM's to stuff in our roll.

rpmbuild -ta glusterfs-3.0.5.tar.gz

That rpmbuild will untar the archive, compile and build the software and bundle it into RPM's for us. Dope.

When it's done, the RPM's will be in

/usr/src/redhat/RPMS/x86_64/

However, for all the performance goodness, we should also download Gluster's fuse code, as it uses fuse for bridging user/kernel land. Pretty much the same process, although fuse is a kernel module, so you're going to need the kernel sources to compile it against. In MY case, I'm running the Xen kernel, so I need the kernel-xen-devel package, you probably don't but it doesn't hurt to install both:

yum -y install kernel-devel 
OR
yum -y install kernel-xen-devel

Now, download, compile and package:

cd /root/gluster-3.0.5/
wget http://download.gluster.com/pub/gluster/glusterfs/fuse/fuse-2.7.4glfs11.tar.gz
rpmbuild -ta fuse-2.7.4glfs11.tar.gz

Ok, now you'll have all the fuse and gluster software bundled into RPM's:

[root@yomama gluster-3.0.5]# dir -1 /usr/src/redhat/RPMS/x86_64/
fuse-2.7.4glfs11-1.x86_64.rpm
fuse-devel-2.7.4glfs11-1.x86_64.rpm
fuse-libs-2.7.4glfs11-1.x86_64.rpm
glusterfs-client-3.0.5-1.x86_64.rpm
glusterfs-common-3.0.5-1.x86_64.rpm
glusterfs-devel-3.0.5-1.x86_64.rpm
glusterfs-server-3.0.5-1.x86_64.rpm

Ok, so now, simply create a new (empty) roll:

cd /export/site-roll/rocks/src/roll/
rocks create new roll gluster-3.0.5 version=5.3 color=brown

I use 5.3 just since all the other rolls are 5.3, you can use whatever. Also, the color is what's displayed in the software graph, so that is optional.

Ok, so now..

cd gluster-3.0.5/
rm -rf src/
mkdir RPMS/
cp /usr/src/redhat/RPMS/x86_64/gluster* RPMS/
cp /usr/src/redhat/RPMS/x86_64/fuse* RPMS/

Ok, so almost done. We simply need a couple the node and graph files.

Here's my node file. Note that I have my compute nodes grab the gluster_setup.sh file off the frontend when the Gluster roll is installed on the node. The reason is that you don't have to rebuild your roll every time you make a change to the gluster_setup.sh script, which you'll likely want to do as it's somewhat hardware dependent, in terms of your HD layout and also, the IP address of your frontend will be different. Once you're happy with the gluster_setup.sh script, you can add it to the post section of your node file. It executes either way, so it's more of a cosmetic issue.

Edit: /export/site-roll/rocks/src/roll/gluster-3.0.5/nodes/gluster-3.0.5.xml

<?xml version="1.0" standalone="no"?>

<kickstart>


        <description>
        Gluster 3.0.5
        </description>

        <copyright>
        Copyright (c) 2010
        </copyright>

        <changelog>
        $Log: gluster-3.0.5.xml,v $
        Revision 1.17  2010 joey
        Let's rock 
        </changelog>



        <package>glusterfs-client</package>
        <package>glusterfs-common</package>
        <package>glusterfs-devel</package>
        <package>glusterfs-server</package>

        <package>fuse</package>
        <package>fuse-devel</package>
        <package>fuse-libs</package>
        <package>libibverbs</package>
        <package>openib</package>

        <post>
           # This is somewhat hardware dependent.. so I don't put it 
           # in the nodes file.
           cd /tmp/
           wget http://10.19.0.1/gluster_setup.sh
           sh gluster_setup.sh
        </post>

</kickstart>

and here's my graph file. You'll probably want to change vm-container to compute nodes.

Edit: /export/site-roll/rocks/src/roll/gluster-3.0.5/graphs/default/gluster-3.0.5.xml

<?xml version="1.0" standalone="no"?>

<graph>

        <description>
        The gluster 3.0.5 Roll
        </description>

        <copyright>
        Copyright (c) 2010
        All rights reserved. 
        </copyright>

        <changelog>
        $Log: gluster.xml,v $
        </changelog>

        # Install on all compute nodes, the frontend and vm-container nodes.
        <edge from="vm-container-client" to="gluster-3.0.5" />
        <edge from="compute" to="gluster-3.0.5" />
        <edge from="server" to="gluster-3.0.5" />

</graph>

So, that's it for the roll. Just build it:

cd /export/site-roll/rocks/src/roll/gluster-3.0.5
make roll

Now, add your roll to the FE and enable it:

rocks add roll gluster-3.0.5-5.3-0.x86_64.disk1.iso
rocks enable roll gluster-3.0.5
rocks list roll gluster-3.0.5
cd /export/rocks/install && rocks create distro

Now, this script is something you'll have to edit and modify to YOUR environment. This is an example of one of my clusters. In this case, I'm taking all of /dev/sdb on the vm-containers and dedicating that to gluster. In my case, it's a 1TB drive. Note that this wacks all the data on sdb but you already know that cause you're a rocks admin and we're reimaging all the compute nodes.

You'll clearly want to modify this to suit your needs. This is the install script referenced in the node xml above. Now, note that the reason we have to do this, is because we need to know about all the nodes prior to configuring Gluster. New to Gluster in version 3.1 which has been released recently is the ability to add nodes dynamically - which will mean mean a fully functioning cluster share when the systems are imaged. Very nice.

Edit: /var/www/html/gluster_setup.sh

# Install and configure gluster
# Joey 

mkdir /gluster/
mkdir /etc/glusterfs/
mkdir /glusterfs/

# Prepare sdb to be mounted as /gluster/
/sbin/fdisk -l /dev/sdb | perl -lane 'print "Wacking: /dev/sdb$1" and system "parted /dev/sdb rm $1" if (/\/dev\/sdb(\d+)\s/)'
/sbin/parted -s /dev/sdb mkpart primary ext3 0 1000200
sleep 5
/sbin/mkfs.ext3 /dev/sdb1

# Get sdb and the glusterfs loaded into /etc/fstab
echo "/dev/sdb1               /gluster                ext3    defaults        0 1" >> /etc/fstab
echo "/etc/glusterfs/glusterfs.vol  /glusterfs/ glusterfs  defaults  0  0" >> /etc/fstab

# Create the list of volume participants.
cd /etc/glusterfs/

# Replicated
glusterfs-volgen --name glusterfs --raid 1 \
   vm-container-0-0:/gluster \
   vm-container-0-1:/gluster \
   vm-container-0-2:/gluster \
   vm-container-0-3:/gluster \
   vm-container-0-4:/gluster \
   vm-container-0-5:/gluster

cp vm-container-0-0-glusterfs-export.vol glusterfsd.vol
cp glusterfs-tcp.vol glusterfs.vol
rm -rf *glusterfs-export.vol
rm -rf *.sample
rm -rf glusterfs-tcp.vol

echo "modprobe fuse && mount -a" >> /etc/rc.local

Re-image your vm-containers/compute nodes and you'll have gluster mounted up on

How to Create a CentOS 5 AMI to run on EC2 or Eucalyptus

Building a CentOS 5 AMI

First of all, a bit of background and why I love the cloud. Previously, I have worked for large ISP's (UUNet, Level 3, British Telecom) and a couple of startups. In some cases, when I'd provision a new system, I'd go to the vendor website, spec out the box, order it, wait for it to arrive, rack and console it. Then, I'd provision switch ports, etc and through an OS and apps on it. Then, I'd unrack it, box it up and drive it to the data center. Re-rack, cable, test, switch port configure it, etc. Took a huge amount of effort. With ec2 and cloud computing, you run a few commands and you're up and running in like, oh.. 20 minutes or so. So, that's awesome, convenient, etc.

You can either run a public AMI, an image created by some random person, or you can build and run you're own. Here's how to do it my way.

First a couple of assumptions, you're running this sequence on an existing CentOS system. You have the fuse module loaded:

modprobe fuse && lsmod | grep fuse

Need that to do loopback mounts.

Ok, let's get started. Note that this whole thing can be scripted but I figured I'd give a little explanation on each step.

So, first let's create a directory to work in:

mkdir -p ~/ami/centos/ && cd ~/ami/

Now, we're going to create an empty disk image. If you know about anaconda, this is just like the downloading stage2 process (well, kinda). Mine is going to be 2G because I like to have a bunch of stuff in there. Change count to 1024 if you want it to be smaller.

dd if=/dev/zero of=centos.fs bs=1M count=2048

Now, create a file system on it:

mke2fs -F -j centos.fs

Good, now we're going to open it up to be written to by mounting it:

mount -o loop centos.fs ~/ami/centos/

Ok, now we're going to turn this thing into a Linux system you can boot up. The kernel likes to have this stuff. The directory ~/ami/centos/ is the root directory / on your new instance.

Make the /dev/ directory:

mkdir ~/ami/centos/dev
   /sbin/MAKEDEV -d ~/ami/centos/dev/ -x console
   /sbin/MAKEDEV -d ~/ami/centos/dev/ -x null
   /sbin/MAKEDEV -d ~/ami/centos/dev/ -x zero

Create /etc/

mkdir ~/ami/centos/etc/

Now, we're going to use yum - just the way the default installer does to add a bunch of software to your new system. We create a yum.conf on the local file system to use to install OS and software.

vi ~/ami/yum.conf

Add this to the file:

[main]
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
exclude=*-debuginfo
gpgcheck=0
obsoletes=1
pkgpolicy=newest
distroverpkg=redhat-release
tolerant=1
exactarch=1
reposdir=/dev/null
metadata_expire=1800
[base]
name=CentOS-5.5 Base
baseurl=http://mirror.centos.org/centos/5.5/os/x86_64/
gpgcheck=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-centos5.5
priority=1
protect=1
#released updates
[update]
name=CentOS-5.5 Updates
baseurl=http://mirror.centos.org/centos/5.5/updates/x86_64/
gpgcheck=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-centos5.5
priority=1
protect=1
#packages used/produced in the build but not released
[addons]
name=CentOS-5.5 Addons
baseurl=http://mirror.centos.org/centos/5.5/addons/x86_64/
gpgcheck=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-centos5.5
priority=1
[extras]
name=CentOS 5.5 Extras $releasever $basearch
baseurl=http://mirror.centos.org/centos/5.5/extras/x86_64/
enabled=1

So that pulls stuff from the official CentOS mirror list. GPG checking is off because we don't have the keys installed.

Next, create the proc file system and mount it up. This is where the kernel keeps track of all the stuff it's doing.

mkdir ~/ami/centos/proc
   mount -t proc none ~/ami/centos/proc/

Ok, here's where the magic begins to happen. We're going to start loading os packages:

yum -c ~/ami/yum.conf --installroot=/root/ami/centos -y groupinstall Core

So using that command, you can install all the stuff you want. The following is my list, you probably don't need the group 'Development Tools'. That's a BUNCH of stuff you only really need if you're doing development. I like to have it on some instances, some it never gets used. So, you should probably ignore it. A good way to see what's available to install is to run:

yum grouplist | less

and that'll tell you what package groups exist. If you're creating a DNS server, you'll want to add the 'DNS Name Server' group. Pick through the list below and install what you want/need. Not everybody is going to want JDK for example.

yum -c ~/ami/yum.conf --installroot=/root/ami/centos -y groupinstall 'Text-based Internet'
   yum -c ~/ami/yum.conf --installroot=/root/ami/centos -y groupinstall Ruby
   yum -c ~/ami/yum.conf --installroot=/root/ami/centos -y groupinstall 'Web Server'
   yum -c ~/ami/yum.conf --installroot=/root/ami/centos -y groupinstall 'Development Tools'
   yum -c ~/ami/yum.conf --installroot=/root/ami/centos -y groupinstall 'Java'
   yum -c ~/ami/yum.conf --installroot=/root/ami/centos -y groupinstall 'MySQL Database'
   yum -c ~/ami/yum.conf --installroot=/root/ami/centos -y install curl wget rsync sudo mlocate lsof man tcpdump bc iptables

Once all your software is installed, configure sshd. Now, this is just an old sysadmin tip. Don't run sshd on port 22. It gets scanned 24x7x365 with all kinds of brute force attacks and everything else. I always, always, always run it on another port. You can do tcpwrappers and other tricks (iptables, etc) but just running it on some higher port saves you tons of problems.
So, in this example, I'm running on port 55000.

vi ~/ami/centos/etc/ssh/sshd_config

and add something like this:

Port 55000
Protocol 2
SyslogFacility AUTHPRIV

PermitRootLogin yes
MaxAuthTries 4
PasswordAuthentication no
ChallengeResponseAuthentication no

GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes

AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL
X11Forwarding yes

Subsystem sftp /usr/libexec/openssh/sftp-server

Create a resolv.conf file with valid name servers. I pretty much always use google's because it's fast and anycasted so it's going to be fast no matter where you are, and it's google.

vi ~/ami/centos/etc/resolv.conf

and add:

# Google's public DNS servers.
nameserver 8.8.8.8
nameserver 8.8.4.4

Now, configure you motd. This is the banner that gets displayed whenever anyone logs in:

vi ~/ami/centos/etc/motd

Put whatever you want in there.. here's an example:

________________________________________
/ Unauthorized users will be killed and  \
\ eaten.                                 /
 ----------------------------------------
        \   ^__^
         \  (xx)\_______
            (__)\       )\/\
             U  ||----w |
                ||     ||

This is optional too but I like to have ec2tools installed on my instances:

cd ~/ami/centos/ && wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm*
   chroot /root/ami/centos rpm -Uvh ec2-ami-tools.noarch.rpm

vi /etc/profile.d/ec2tools.sh

export EC2_HOME=/opt/ec2-tools
   export PATH=$EC2_HOME/bin:$PATH

Now, configure DHCP for networking:

mkdir -p /etc/sysconfig/network-scripts/
   vi /etc/sysconfig/network-scripts/ifcfg-eth0

Enter:

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=no
PERSISTENT_DHCLIENT=yes

That PERSISTENT_DHCLIENT is very nice to have. It means that if for some reason the DHCP server bites the dust, keep the lease you have. Otherwise your instance could loose it's IP at which point, it's game over.

Turn on networking:

vi /etc/sysconfig/network

add:

NETWORKING=yes

Now, configure fstab to mount everything up:

vi /etc/fstab

Add the following:

/dev/sda1               /                       ext3    defaults 1 1
none                    /dev/pts                devpts  gid=5,mode=620 0 0
none                    /dev/shm                tmpfs   defaults 0 0
none                    /proc                   proc    defaults 0 0
none                    /sys                    sysfs   defaults 0 0
/dev/sdc1               /mnt                    ext3    defaults 0 0
/dev/sdc2               swap                    swap    defaults 0 0

Next, turn on the services you want.

chroot ~/ami/centos/ bash
chkconfig --level 345 sshd on
chkconfig --level 345 httpd on

exit 

Almost done. I also like to add a user account with a password and sudo access, so if for some reason I can't get in with my ssh key(s), I can just login as this user and sudo to root and figure out what's going on. This step is optional but very useful:

chroot ~/ami/centos/ bash
   useradd -g users mclovin
   passwd mclovin

Then add your user to the sudoers file:

visudo

find this line:

root    ALL=(ALL)       ALL

and add your new user:

mclovin    ALL=(ALL)       ALL

Then, exit the chroot'd env:

exit

umount ~/ami/centos/

That's pretty much it holmes. Once that's all done you can unmount the image file. Bundle, upload and run.

CentOS 5 Icecast Server HowTo

So, you wanna stream some tunes, aye? Ok well here's mine, click to listen:

   Jah Radio

Please feel free to tune in, currently it's pretty much all Bob Marley. I'm using some pretty schweet software called icecast. You can pretty much stream any kind of audio you want. The icecast server takes a feed and retransmits, so you'll need a source. That can either be local MP3 files, or a stream from your desktop player, or someone else's stream. Which is what I'm doing. I'm just grabbing some other dude's stream and rebroadcasting it.

Here's how ya do that on a fresh CentOS 5 install:


First, grab the source:

wget http://downloads.xiph.org/releases/icecast/icecast-2.3.2.tar.gz

Next, add the RPMforge repo (make sure it matches your architecture, goober):

rpm -Uhv http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS//rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm

Install the depdencies:

yum -y install libvorbis-devel libogg-devel curl-devel libxml2-devel libxslt-devel libtheora-devel speex-devel

If you don't have development tools installed, you'll need them to build, compile and bundle, so just run:

yum groupinstall 'Development Tools'

Otherwise, if you already have gcc and all that jazz, run:

rpmbuild -ta icecast-2.3.2.tar.gz

That takes the icecast source and builds you an RPM which you can install. I like doing it that way so you can make a copy of that RPM and use it for other deployments.

Now, once the rpm(s) are build, you can install them like so:

rpm -Uvh /usr/src/redhat/RPMS/x86_64/icecast*.rpm

I like to stuff to log to /var/log - some folks don't. Whatever, it's your call, if your doing it my way (which you should because I'm awesome) do:

mkdir /var/log/icecast/
chown -R nobody:nobody /var/log/icecast/

Now, move their default /etc/icecast.xml and use mine:

mv /etc/icecast.xml /etc/icecast.xml.orig

This is a fully, ready to roll config file which will have you rebroadcasting a popular shoutcast Bob Marley stream:

<icecast>
<limits>
<clients>100</clients>
<sources>2</sources>
<threadpool>5</threadpool>
<queue-size>524288</queue-size>
<client-timeout>30</client-timeout>
<header-timeout>15</header-timeout>
<source-timeout>10</source-timeout>
<burst-on-connect>1</burst-on-connect>
<burst-size>65535</burst-size>
</limits>

<authentication>

<source-password>nanana</source-password>
<relay-password>nanana</relay-password>
<admin-user>admin</admin-user>
<admin-password>nanana</admin-password>
</authentication>

<hostname>tunes.cloud21cn.com</hostname>
<listen-socket>
<port>8000</port>
</listen-socket>
<relay>
<server>88.191.16.115</server>
<port>8000</port>
<mount>/</mount>
<local-mount>/JahRadio</local-mount>
<on-demand>0</on-demand>
<relay-shoutcast-metadata>0</relay-shoutcast-metadata>
</relay>
<fileserve>1</fileserve>
<paths>
<basedir>/usr/share/icecast</basedir>
<logdir>/var/log/icecast</logdir>
<webroot>/usr/share/icecast/web</webroot>
<adminroot>/usr/share/icecast/admin</adminroot>
<alias source="/" dest="/status.xsl"/>
</paths>

<logging>
<accesslog>access.log</accesslog>
<errorlog>error.log</errorlog>
<loglevel>3</loglevel> <!-- 4 Debug, 3 Info, 2 Warn, 1 Error -->
<logsize>10000</logsize> <!-- Max size of a logfile -->
</logging>

<security>

<chroot>0</chroot>
<changeowner>
<user>nobody</user>
<group>nobody</group>
</changeowner>
</security>
</icecast>

You'll have to edit that file slightly for the hostname. Also, find your own music here:

Shoutcast

and update the relay stanza in the above config.

Now, just make sure you have port 8000 open for TCP connects and hit:

http://your.url.com:8000/

Relax and enjoy.

By the way, I had to encode all that XML above to get it to look right in Blogger. The way to do that is to use a site like this: http://centricle.com/tools/html-entities/

Worked great.

Increasing File Handle Limits in CentOS

So, I pretty much use CentOS for all the systems I deploy to production. I futz around with others but I've been doing Redhat since, oh like 1996 and I just know it and it's solid and there's rpmforge for packages that aren't in Base and Extras.

I notice Java applications like to use a lot of file handles. Also, busy TCP network services (udp only uses a single socket) use up lots of sockets, which are counted as a file handle. Think, a very busy Java Message Bus..... so I have no clue why the default is 1024. But if you find your applications (or system applications that are logging to syslog) tell you that you're out of file handles, you can check to see how many you have like this:


[root@elvira ~]# ulimit -n
1024

That should be more like 65535 for a system with, oh 8G or memory. So, you need to set it in a couple of spots. When a user logs into a system, the PAM system comes into play. The limits.conf file is PAM configuration file and affects user processes and shells. The other file is sysctl.conf which sets kernel values and thus affects pretty much everything else. Here's how it's done:

#!/bin/sh
#
# Increase file descriptor limit to 65535

cat<< EOF >> /etc/sysctl.conf

# Increase file handle limit 

fs.file-max = 65535

EOF

cat<< EOF >> /etc/security/limits.conf

*       soft    nofile  65535
*       hard    nofile  65535

EOF

sysctl -p /etc/sysctl.conf

Before running that, check the files and make sure to delete any existing entries for the above, so you don't end up with duplicates.

Now, simply log out of your shell and log back in. You should now see that your ulimit is set to 65535.



[root@elvira~]# ulimit -n
65535

I'm really doing it this time..

Howdy, well it's time for me to start blogging. I've been threatening myself for years. Blog or else. But I have a ton of other stuff going on and ski season is just around the corner. However.. I need a place to put stuff that I've discovered and just ramble on about stuff. Mostly this is going to be hard-core Linux stuff because I have a bunch of random notes from all over that could really help others out, the way I've learned so much from other helpful Internet citizens. Anyhoo, this is it. Welcome.


Hmm.. I just noticed that my URL sysextra looks a little provocative. Bonus!


So, some stuff about me. I use to work for UUNet, where I really lucked out. There were some really smart people that there I learned from. I started out in the support group which kinda turned into me working in a web hosting sys-admin group. After that, I was in product development (devo) which was really operations and development. From there it was off to Level 3 in Colorado, then a couple of start-ups and now I work for BT on a cloud computing platform. Funny how much it reminds me of managed hosting at UUNet, only cooler.


I really love all things operations so I always kinda do it, even when I'm in a development roll. Good sysadmins like to be on the frontlines because that's where the action is. I enjoy writing perl and PHP code and all things Linux. I think that's about it for now, the kids are bugging me to get dinner going. Oh  yea, I'm a big Grateful Dead fan.