MySQL: Restore Master from Slave

I had a power-outage in one of my Linux clusters. The master MySQL server was corrupted and wouldn't start properly. After trying a bunch of bin-log/relay fixes and other failed attempts, I finally decided to uninstall/re-install and that was so quick and simple and everything worked perfectly. Here are the steps for CentOS/RHEL:

 service mysqld stop
 mv /etc/my.cnf /root/
 mv /var/lib/mysql /var/lib/mysql.orig/
 rpm -e mysql-server
 yum -y install mysql-server
 service mysqld start
 
# set pw and take defaults
 /usr/bin/mysql_secure_installation 
 mysqldump -h working-mysql-server -u root --all-databases --quick --lock-all-tables | mysql -u root -p
 mv /root/my.cnf /etc/
 service mysqld restart

Now, check /var/log/mysqld.log for errors. If all is well, run:

 mysql -e 'show slave status\G'

To get the current status.

Apache: How to Automatically Null Route Abusive Hosts

I notice tons of suspicious requests in my apache log files. I simply don't like this, so if someone or something makes a request for a file that doesn't exist on one of my systems, that earns them a lifetime ban. Here's how we do that. First, modify /etc/httpd/conf/httpd.conf and find this line: DocumentRoot "/var/www/html" Add this, directly after:

RewriteEngine on
RewriteCond %{REQUEST_URI} !/myfile1.html$
RewriteCond %{REQUEST_URI} !/myfile2.html$
RewriteRule ^(.*)$ /block.cgi

Where myfile1.html and myfile2.html are files that actually exist. This will route all other requests to a file called 'block.cgi'. Now, create a file, called: /var/www/html/block.cgi which contains:

#!/usr/bin/perl 
#
# The null router..   There is a redirect in http.conf which executes
# this script by default. Because really, if someone is hitting us
# with a GET request on /, they're up to no good so bye-bye.
#
# McLovin

use strict;
use CGI qw(:standard);

my $date = `date`;
chomp $date;
my $ip = $ENV{'REMOTE_ADDR'};

open FH, ">> /root/blocked_hosts.txt";
print FH "#$date\n/sbin/route add $ip gw 127.0.0.1 lo\n";
close FH;

print header;
print start_html("Environment");
system "sudo /sbin/route add $ip gw 127.0.0.1 lo"; 
print end_html;

Make sure that file is executable, next, make sure that the apache user can execute that script by adding the following to sudoers:

apache ALL = NOPASSWD: /sbin/route

One final touch, I log each blocked host, so that file needs to be owned by user: apache.

touch /root/blocked_hosts.txt && chown apache:apache /root/blocked_hosts.txt

Now, test that out and make sure it's working. After a failed request, you should see the IP in the output of netstat, like this:

netstat -rn|grep lo$
222.209.132.155 127.0.0.1       255.255.255.255 UGH       0 0          0 lo
202.53.8.82     127.0.0.1       255.255.255.255 UGH       0 0          0 lo

CentOS SSHFS Howto

I have a need to remotely mount a file system over the Internet. I'd like to do so without doing a VPN tunnel and all the encryption myself. So, I gave sshfs a shot and turn's out, it's super easy. Here's how on CentOS:

# Install EPEL
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

# Install sshfs
yum -y install sshfs fuse

# Load FUSE Kernel Module
modprobe fuse

# Mount remote file system.
mkdir /mnt/sshfs/
sshfs root@my.server.com:/export/opt/ /mnt/sshfs/

Now if you want to get fancy, you can push your ssh key and auto mount on boot, like this:

# Push identity
ssh-copy-id user@my.server.com

# Edit /etc/fstab and add:
root@my.server.com:/export/opt/ /mnt/sshfs fuse.sshfs defaults,noauto,user 0 0

# Mount it up
mount /mnt/sshfs/

Using IPTables to Blackhole Large Set's of IP's

I found a host where I was getting a bunch of POST's in my apache server log files which looked to be malicious. I wanted to go through and just block all IP's which were trying to post to my web server, since I don't have anything but static content on it. So, I came up with this little one liner:

grep POST /var/log/apache2/*log* | perl -lane 'print $1 if (/^.*?:(.*?)\s/)'|sort | uniq | perl -lane 'system "iptables -A INPUT -s @F[0] -j DROP"'

This is useful for any group of IP's you wish to black-hole.

Creating an LVM Volume on EBS with XFS

Well that's mouth full.. I wanted to add some storage to a VM but wanted to be able to add to it later if I got to that point. So, I attached a 100G EBS volume, and here's how I formatted it. It's EBS as the physical volume, then LVM formatted as XFS. You can attach another EBS volume to this VM, then add it to the volume group later, striped for increased capacity. Anyway, here's how it's done:

rpm -q xfsprogs &> /dev/null || yum -y instal xfsprogs
mkdir /export
pvcreate /dev/vdb
vgcreate VolGroup_EBS /dev/vdb
lvcreate -I 2M -l 100%FREE -n export VolGroup_EBS
mkfs.xfs /dev/mapper/VolGroup_EBS-export
mount /dev/mapper/VolGropu_EBS-export /export/

Finally, add this to /etc/fstab to get it mounted when your system boots:

/dev/mapper/VolGroup_EBS-export /export  xfs defaults 1 1

Multi Hop SSH SOCKS Proxy

From a corporate network I had a need to jump to one system, and through it to another in order to have an open web proxy to another 'internal' network - or in this case it was a lab network where I had to hit a openstack Horizon dashboard. It's a somewhat simple concept and I was sure SSH could do it but I had some trouble figuring out how. The scenario looks like this:

my macbook -> ssh server <- INTERNET -> second-ssh-server -> browse internal network

Note that the second ssh server was running sshd on port 22000, you probably don't need that. The command I came up with to accomplish this was:

ssh -t -t -v -L9999:localhost:9932 root@ssh-server ssh -t -D 9932 root@second-ssh-server -p 22000

A whole blog-post for one command? Yes, it was that cool!

Find the Fastest Mirror in Bash

I needed a script that would determine the fastest mirror in a list of addresses. In this example I'm using the mighty 'www.google.com' and port 80 but it's obviously configurable. I also needed to use nano seconds rather than seconds because all the results (for google) came back in less than 1 second. It should actually be a bash function but I'll leave that to you, fine reader. You're welcome!

mirror='www.google.com'
port='80'
iplist=()

for ip in $(dig $mirror A | perl -lane 'print $1 if (/A\s+(.*?)$/)')
do
   # Give me a resonable starting point (Using epoch in nano seconds).
   a=$(($(date +'%s * 1000 + %-N / 1000000')))
   nc -w 2 -z $ip $port &> /dev/null &
   b=$(($(date +'%s * 1000 + %-N / 1000000')))
   msec=`expr $b - $a`
   iplist+=("$msec $ip")
done

# Grab the fastest one..
fastest=$(for a in "${iplist[@]}"; do echo "$a"; done | sort -n | head -1 | awk '{print $2}')

echo $fastest