Categories
Linux

Compile Latest Ganglia for All Versions of Debian and Ubuntu

I have learned a lot since I wrote the original how-to. I do not have time to write a nice story but follow the steps below and you should be fine. This will provide you with the latest version of Ganglia on any Debian based distro.

Objectives:
1. Download latest sources from Ganglia git repository.
2. Compile, install and setup monitor-core.
3. Setup ganglia-web front-end.
4. Setup hsflowd on servers.

On your collector / PHP enabled web-server:

apt-get install build-essential automake autoconf pkg-config gperf libtool rrdtool librrd-dev libconfuse-dev libapr1-dev libpcre3-dev

cd ~
git clone https://github.com/ganglia/monitor-core.git ganglia
cd ganglia
git submodule init
git submodule update

./bootstrap

Update February 1, 2014: You need to install Concurrency Kit before running configure or it will fail.
Instructions:

"git clone http://concurrencykit.org/cgit/cgit.cgi/ck/"; "cd ck"; "./configure"; "make"; "make install"

and then switch back to your ganglia directory and continue.

./configure --with-gmetad
make
make install
ln -s /usr/local/lib64/ganglia /usr/lib/ganglia
useradd --system ganglia
groupadd --system ganglia

nano /etc/ld.so.conf
add line: /usr/local/lib

ldconfig

nano /usr/local/etc/gmetad.conf
change: data_source "your cluster name" 20 localhost
cp ~/ganglia/debian/gmond.conf /usr/local/etc/gmond.conf
nano /usr/local/etc/gmond.conf
change: mute = yes
change: name = "your cluster name"
change:
udp_send_channel {
  port = 8649
  ttl = 1
}
udp_recv_channel {
   port = 8649
}
sflow {
  udp_port = 8649
  accept_vm_metrics = yes
}
tcp_accept_channel {
  port = 8649
}

cd /var/www
git clone https://github.com/ganglia/ganglia-web.git ganglia
cd ganglia
mkdir dwoo/compiled
mkdir dwoo/cache
mkdir -p /var/lib/ganglia/rrds
chown nobody /var/lib/ganglia/rrds
chown -R www-data:www-data /var/www/ganglia
cp conf_default.php.in conf_default.php
cp version.php.in version.php
nano conf_default.php
change: $conf['gweb_confdir'] = "/var/www/ganglia";
change: $conf['gmetad_root'] = "/var/lib/ganglia";

gmond
gmetad

On servers you want to monitor:

apt-get install hsflowd

Note: This package does not appear to be included anymore. Visit their website.

nano /etc/hsflowd.conf
change: DNSSD = off
change: polling = 20
change: sampling = 2000
change:
collector {
ip = (IP running gmond+gmetad)
udpport = 8649
}

/etc/init.d/hsflowd start

Visit your webserver to see your graphs. DONE!

NOTE: There is a mistake in stacked.php as of January 16, 2013 which causes the graph to break. The exact error is: “PHP Parse error: syntax error, unexpected T_FOREACH in /var/www/ganglia/stacked.php”

To fix, search for “$min_index = min(array_keys($hosts))” and add a semi-colon to the end of the line:

nano /var/www/ganglia/stacked.php
change: $min_index = min(array_keys($hosts));

EDIT (8/5/2013): I recently read this article and decided to set my sampling rate to 2000. I was occasionally getting errors where my network speed was being measured in petabytes/s! Here is a second article about the same topic that is also worth a quick read.

Categories
Linux

Setup a KVM VPS Host, LVM on Software RAID1 and a Virtual pfSense Router

We are here today to setup a KVM host on CentOS 6 (or Linux variant). The host will have logical volumes backed by software RAID1 and a virtual pfSense router. With this setup you can securely and reliably host multiple applications on the same server.

I am writing these instructions for Linux users. I do not use Windows on my personal computer. You may need to run a Linux virtual machine or locate the appropriate Windows software.

The server should have a VT-x enabled proccessor, two hard drives (for software RAID1) and an uninterruptible power source. If the server is in a datacenter with reliable or redundant power you may be fine without a UPS.

Install CentOS 6.2 (or latest version) using the minimal disc (CentOS-6.2-x86_64-minimal.iso) and choose custom partitioning. If your provider pre-installed the operating system then skip the next few steps.

At the partition editor create two RAID partitions on each drive. One 500mb and a second to fill the remainder of space. This leads to four partitions.

Next, create a RAID device with mount point “/boot”, filesystem ext4, type RAID1 and select both 500mb partitions. Create another RAID device, select filesystem LVM, type RAID1 and select the two remaining partitions.

Next, create a new logical volume group and call it whatever you want. In the logical volume manager click add, set the mount point to “/”, filesystem ext4 and size of 20000MB. Click add again, set the filesystem to “swap” and make it the same size as your RAM. (I never seem to need the swap but I read it is better to have than not.)

Complete the install and reboot.

Login as root. Let’s disable some services we don’t use, “chkconfig fcoe off; chkconfig ip6tables off; chkconfig iptables off; chkconfig iscsi off; chkconfig iscsid off; chkconfig lldpad off; chkconfig netfs off; chkconfig nfslock off; chkconfig rpcbind off; chkconfig rpcgssd off; chkconfig rpcidmapd off”. If your provider pre-installed your operating system don’t disable iptables unless you have another firewall setup.

Time to setup initial networking.

eth0 is connected to our WAN. We will configure it as a bridge so our pfSense router can use it as an interface. If you have multiple network ports and/or LAN this may be different. “vi /etc/sysconfig/network-scripts/ifcfg-eth0”

DEVICE="eth0"
HWADDR="xx:xx:xx:xx:xx:xx"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
BRIDGE="br0"

Next we will set the IP on the WAN bridge. Later we will use it for SSH to setup pfSense. If you have one IP, don’t worry, we will comment it out before bringing up our router. “vi /etc/sysconfig/network-scripts/ifcfg-br0”

DEVICE=br0
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Bridge
STP=on
DELAY=0
BOOTPROTO=none
IPADDR=x.x.x.x
GATEWAY=x.x.x.x
NETMASK=x.x.x.x
DNS1=x.x.x.x

If you have a LAN port to use for SSH then configure that interface instead. If you would like to allow virtual machines to be on your LAN, in addition to WAN, then replicate the above two configurations for your LAN interface. Make sure to only specify the IP information (address, gateway, netmask, DNS) on one bridge.

We have to make atleast one bridge for virtual machines to connect to. I usually make two. One for the host server and virtualizations that do not run public services. The second for virtualizations that run public services. “vi /etc/sysconfig/network-scripts/ifcfg-br1”

DEVICE=br1
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Bridge
STP=on
DELAY=0
BOOTPROTO=none

“vi /etc/sysconfig/network-scripts/ifcfg-br2”

DEVICE=br2
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Bridge
STP=on
DELAY=0
BOOTPROTO=none

Change the default SSH port. Why use the default port when you don’t have to? “vi /etc/ssh/sshd_config”, uncomment and change “Port 22” to “Port xxx22” (easier to remember) or “Port xxxxx”. Restart the service, “/etc/init.d/sshd restart”.

If you are on a public network apply some iptables rules to protect yourself before or immediately after restarting the network service. Restart the network service. “/etc/init.d/network restart”

If you need a simple firewall to copy and paste, “vi /root/firewall”

#!/bin/sh

# iptables script generated 2011-01-23
# http://www.mista.nu/iptables

IPT="/sbin/iptables"

# Extra stuff
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/all/log_martians
echo 0 > /proc/sys/net/ipv4/tcp_ecn
echo 0 > /proc/sys/net/ipv4/ip_forward

# Flush old rules, old custom tables
$IPT --flush
$IPT --delete-chain

# Set default policies for all three default chains
$IPT -P INPUT DROP
$IPT -P FORWARD DROP
$IPT -P OUTPUT ACCEPT

# Enable free use of loopback interfaces
$IPT -A INPUT -i lo -j ACCEPT
$IPT -A OUTPUT -o lo -j ACCEPT

# All TCP sessions should begin with SYN
$IPT -A INPUT -p tcp ! --syn -m state --state NEW -s 0.0.0.0/0 -j DROP

# Accept inbound TCP packets
$IPT -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A INPUT -p tcp --dport XXX22 -m state --state NEW -s 0.0.0.0/0 -j ACCEPT

# Accept inbound ICMP messages
$IPT -A INPUT -p ICMP --icmp-type 8 -s 0.0.0.0/0 -j ACCEPT
# $IPT -A INPUT -p ICMP --icmp-type 11 -s 0.0.0.0/0 -j ACCEPT

Then, “chmod +x /root/firewall; /root/firewall”.

We are connected to the internet! Let’s update the packages, “yum update -y”.

Next, install KVM, libvirt, ntpd and wget, “yum install kvm libvirt ntp wget -y”.

Enable ntpd on boot, “chkconfig ntpd on”.

Disable SELinux, “vi /etc/selinux/config” and change “SELINUX=enforcing” to “SELINUX=disabled”.

Remove the default libvirt networking, “rm -f /etc/libvirt/qemu/networks/autostart/default.xml; rm -f /etc/libvirt/qemu/networks/default.xml”.

The next numbered items are optional and may improve performance.
1. “vi /etc/fstab”, change the entry for root(/) from “defaults” to “defaults,relatime”.
2. “vi /boot/grub/menu.lst”, at the end of the kernel line add “elevator=deadline”.
3. “vi /etc/sysctl.conf”, at the bottom add “vm.swappiness=0” and on another line “vm.zone_reclaim_mode=0”
The idea is to increase the availability of writing to the disk. Read about it on IBM’s website.
(Another good idea, if you are using LVMs, is to disable caching on your virtual hard drives. Keep this in mind for later.)

Add a line to “/etc/rc.local” (above the last line) for your firewall script. For example, “/root/firewall”. You may also want to add a line to turn off TCP segementation offloading. For example, “ethtool -K eth0 tso off”.

Type “reboot now”.

On your personal computer, create a key pair using “ssh-keygen” and copy the public key to your server using “ssh-copy-id”.

The easiest way to start making virtual machines is to install virt-manager on your personal computer, “sudo yum (or apt-get) install virt-manager”.

Open virt-manager and click “add connection” in the file menu. Tick “connect to remote host” and set hostname to “domain(or IP):port”. If you copied the key it will connect without a password.

Right click on the host entry and click on details. Click on the storage tab. Click on the + at the bottom left corner. Name it whatever, select type “logical: LVM Volume Group” and click next. Set the target path to “/dev/(whatever you named it during installation)”, leave the rest blank and click finish. You will see the name on the left with the percentage of free space.

Open a terminal and SSH into the server. Change directories to the default image location, “cd /var/lib/libvirt/images”. Visit the pfSense mirror page and download the iso, “wget mirror/pfSense-2.0.1-RELEASE-i386.iso.gz”. Decompress it, “gunzip pfSense-2.0.1-RELEASE-i386.iso.gz”.

In virt-manager, right click on the host and click new. Call it whatever, select local installation media and click forward. Browse for the iso you just downloaded (“/var/lib/libvirt/images/pfSense-2.0.1-RELEASE-i386.iso”) and click choose volume. Set the OS type to Unix, version to FreeBSD 8.x and click forward. Allocate atleast 512mb of memory, 1 or 2 CPUs and click forward. Select managed or existing storage and click browse. Select your volume group, click new volume, set the max capcity between 1000 and 4000mb, click ok, click choose volume and click forward. Click advanced options, select eth0/br0 and click finish.

While the pfSense iso is booting push “i” when prompted to start the installer. Install with default options and SMP kernel. Reboot. Before the virtual machine turns back on force the power off.

Go to the virtual machine details. Click add hardware, select network, select host device “eth1/br1”, select device model “e1000” and click finish. Repeat this for each bridge. Write down the MAC assigned to each bridge. Click on IDE cd-rom and remove it. Click boot options, tick “start virtual machine on host boot up” and click apply. Start the virtual machine.

When asked about VLANs choose setup later. Assign the interfaces their appropriate designation.

Go to your terminal that is connected to the host server. In the default image directory download a live CD. I like to use Ubuntu 10.04.4. Here is the mirror list.

Go to virt-manager, right click on the host and click new. Setup a virtual machine to boot from the live CD. The amount of storage doesn’t matter. Under advanced options, select the bridge you assigned to the LAN in pfSense.

On the live desktop open Firefox and go to “http://192.168.1.1”. The default login is admin and password is pfsense. This can be changed under system, user manager.

Go to system, advanced and change the default port. (optional)

Go to system, general settings, set your hostname and domain. You can use your own DNS servers or set them to 8.8.4.4, 8.8.8.8, 208.67.222.222 and 208.67.220.220. Set your time zone and hit save.

Go to interfaces, set the details for each interface and save but do not hit apply. If you have one public IP it is very important that you do not click apply!

Go to services and setup your DHCP servers as desired.

Go to firewall, NAT and click on the + icon. Set the destination port range to whatever you changed it to earlier in system/advanced, redirect target IP to the LAN IP of your router, redirect target port to the same value as destination port and click save.

Click the plus icon, again. Set the destination port to the SSH port you setup earlier, redirect target IP to the LAN IP of your host server (to be set in the next step), redirect target port to the same as destination port and click save.

Go to diagnostics, halt system and click yes.

Open the terminal which is connected to your server and open the br0 config file. Comment or delete the address, gateway, netmask, DNS details and save. Open the br1 (assuming you used br1 for LAN) config file and use the same format as br0 to set a private IP. For example, the router is x.x.x.1 so I made the host server x.x.x.2. Don’t forget to set the gateway, netmask and DNS!

Reboot the host server. When it comes back up the router will autostart. You can visit the web interface and SSH using the WAN IP!

A few more things before you are on your way…

Install grub on the second drive’s master boot record. When the first drive fails use the second to boot. Go to this link and scroll down to “Install Grub on new hard drive MBR”. You can follow exactly what the person wrote there. Test it by rebooting and booting from the second drive!

To monitor your virtual machines’s resource consumption install Host sFlow. It will add and remove virtual machines without manual intervention. Combine with Ganglia for web based reporting. I have written a guide here.

In pfSense, setup firewall rules on the OPT* interfaces to allow internet access. Look at the LAN rules to get started. Also, add rules to block traffic from OPT* interfaces to your LAN, etc.

When rebooting the host set a 2 minute timer (instead of now) and halt the router. I have had problems with the router when resuming from a suspended state.

I hope this helped you setup something cool!! Please leave a comment if you have a suggestion or question.

Categories
Linux Windows

New Server for a Small Office

I recently setup an HP 6200 Pro Small Form Factor PC. It had 12GB of RAM and 2x 250GB 7200RPM disks running software RAID 1 on CentOS 6.

It was for a small office where the server was 10 years old and died. This beauty is dual core with hyperthreading! A perfect candidate to virtualize 3 Windows XP machines and a Linux file server. The printers are setup on the host operating system since they are specific to that site.

The virtualizations can easily be migrated from office to office or accessed over the VPN. Employees and IT no longer need to worry about physically moving their computer from one site to another. Data will remain in a safe location and backups can be made regularly by the administrator.

Some people may be interested to know that there is a parallel port adapter available from HP. It connects directly to a pin-out on the motherboard. I used it to connect a Panasonic KX-P3196. I had to install kmod-lp for my printer to be detected.

To run KVM you will need to enable VT-x in the BIOS. Go through every menu and you WILL find it. If this is not enabled the KVM module for your CPU will not load.

I also setup a Turnkey Linux Core as a central samba server which will backup nightly to Amazon S3.

Three Windows XP virtualizations get one logical processor each and 2GB of RAM. The Turnkey Linux Core server gets 1GB of RAM and one logical processor.

I will update this post if there are any problems with this setup.

Categories
Linux

Reset Supermicro IPMI Password

I forgot my IPMI password. Did you do this too? Do you have root access via SSH to the installed operating system? If you answered yes to both these questions, I have good news for you.

Here is how to do this via SSH to the installed Linux operating system.

Download the IPMICFG utility on the Supermicro FTP.

Unzip it and set the appropriate file as executable. For me it is “ipmicfg-linux.x86_64.static”.

Use –help to view the available options.

I copied the values for the following: -m (show IP and MAC), -k (show subnet mask), -dhcp (get DHCP status), -g (show gateway IP), -vlan (get VLAN status).

Take a deep breath and reset to factory default (-fd). You should see: Reset to the factory default completed!

Update the settings and you should be good to go! The following is what I did to accomplish this.

Turn dhcp off (-dhcp off) and set the IP (-m ###.###.###.###). If you need to set a custom MAC (-a ##:##:##:##:##:##). I checked my IP and MAC (-m) and did not need to set the MAC.

Set the subnet mask (-k ###.###.###.###) and the gateway (-g ###.###.###.###).

Login using IPMI View. The default (case sensitive) login / pass is ADMIN / ADMIN. Go ahead and change the password! If you try to do this from the web interface it will not work.

Enjoy the great features which you have recovered!

Categories
Linux

Logrotate MySQL Cron Error

I have been receiving the following email report:

/etc/cron.daily/logrotate:
error: error running shared postrotate script for '/var/log/mysql.log /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

Most recommendations to fix this problem suggest updating the debian-sys-maint password for the MySQL server to match the one in /etc/mysql/debian.cnf.

This will not fix the problem if MySQL is not logging. Check if logging is turned on in /etc/mysql/my.cnf.

If you choose only to log errors then remove /etc/logrotate.d/mysql-server.

You should no longer get this error!!

Categories
Linux

Setup PSAD in Ubuntu 9+

Psad scans your firewall log in real time. It can be configured to automatically drop packets and more. While reading the guides that are available for this I ran into a problem, there was no /etc/syslog.conf. On Ubuntu’s webpage I found this release note. It says that as of Ubuntu 9.10, syslog has been upgraded with rsyslog. This can make setting up psad a little tricky.

This guide has been tested on Ubuntu 10.04 LTS Server and 10.10 Desktop

The first thing to do is install psad:

sudo apt-get install psad

Now edit the config file:

sudo nano /etc/psad/psad.conf

Change “ENABLE_SYSLOG_FILE Y;” to “ENABLE_SYSLOG_FILE N;”. We will not need psad to read our syslog.

Another setting to review right now depending on your environment is “EMAIL_ALERT_DANGER_LEVEL”.

Set the email at the top of the config file or leave the default, root. I have root’s mail set to forward to my real email address. To forward root (or any user’s) mail: place a file named “.forward” in their home folder. Inside the file enter the email address where the mail is to go.

Restart psad:

sudo /etc/init.d/psad restart

Next: configure iptables to log the non-legitimate packets. The logging rules need to go after the accept rules but before the drop. Confusing? It was for me.

For example, my default policy for INPUT and FORWARD is to DROP. After this my accept rules for specific ports are appended. Meaning our logging rules must go at the end of the file, before they are dropped because the packets were not accepted by any previous rules.

$IPT -A INPUT -j LOG --log-prefix "firewall1 "
$IPT -A FORWARD -j LOG --log-prefix "firewall1 "

The prefix is going to allow rsyslog to filter the messages. After applying the log rules it is possible to view the end of the syslog to see if logging is working.

sudo tail /var/log/syslog

The last step is for rsyslog to send the messages that contain “firewall1” to psad’s pipe.

sudo nano /etc/rsyslog.d/50-default.conf

We are going to place our rules at the top of the file. That way we can stop “firewall1” messages from making it to any other logs.

:msg, contains, "firewall1" |/var/lib/psad/psadfifo
:msg, contains, "firewall1" ~

Note: the ~ means to discard.

That’s it! Restart rsyslog:

sudo restart rsyslog

To view psad’s status:

sudo psad --Status

Comments and suggestions are welcome!!