toic.org - Entries for the tag centos

Reclaiming InnoDB ibdata unused space.

Tue, 04 Dec 2012 23:05:56 +0000

As you well know InnoDB will store its data either in one big system tablespace called ibdata or you can store it in multiple tablespaces (innodb_file_per_table). In either case InnoDB by default will not reclaim unused space gained when deleting data. Using innodb_file_per_table will separate each InnoDB table in its own .ibd file and it can be easily reclaimed by doing optimize table, using system tablespace will get you stuck since there is no real supported method for reducing you ibdata file and reclaiming unused disk space.

One could wonder why would anyone want to have its ibdata in system tablespace? Well there are some "bugs" (in versions prior to 5.5) and performance issues that cause mysql lock-ups when using innodb_file_per_table and doing truncate table or drop table or even drop database queries with large buffer pools. In a nutshell MySQL will do a whole buffer pool lock and scan on each of the above mentioned commands effectively stoping the entire MySQL server until it scans over the entire buffer pool for references to those dropped/truncated tables.

When given choice and when I do know the pattern of data input to my tablespace i will chose system tablespace.

Recently I was a victim of my own system and uncontrolled input of data to my InnoDB table space which leaved me with almost 36Gb of garbage data over one night. Naturally i truncated the data, but it left me with 37Gb ibdata file.

So here are the steps I've taken to reduce this system tablespace.

First of all backup!

I highly recommend Xtrabackup from Percona for this task, yes it will require additional 37Gb of space for backup but this is by far best point in time backup for MySQL.

innobackupex $backupdir

And replace the backupdir with your preferred location

Let's create a working environment, say you have a lots of space on /home

mkdir /home/inno-reduce
cd /home/inno-reduce

Ok, next stop dumping all the databases that have innodb tables inside, for this I've created a small bash script a while ago..

#!/bin/bash
CWD=`pwd`
mysql -e "SELECT distinct(table_schema) FROM information_schema.TABLES where ENGINE = 'InnoDB'" > $CWD/innodb-databases.list
sed "/table_schema/d" $CWD/innodb-databases.list > $CWD/tmp
mv $CWD/tmp $CWD/innodb-databases.list
for database in `cat $CWD/innodb-databases.list`; do
echo "dumping data from $database to $CWD/$database.sql"
/usr/bin/mysqldump --triggers --routines $database > $CWD/$database.sql;
done

Now would be a good time to cut off any application using MySQL (for example turn off Apache, or drop the packets in firewall). Just copy and paste this code in step1-dump.sh file, make it executable and run it. It will produce a .sql dump files in your current directory (hopefully we are still on /home/inno-reduce)

Step2 script will iterate trough that dumped data files and drop all the databases that have innodb tables inside.

#!/bin/bash
CWD=`pwd`
for database in `ls $CWD/*.sql|awk -F'/' '{print $NF}'|cut -d. -f1`; do
echo "droping database $database"
mysqladmin drop $database
done

Again just copy and paste this code in step2-drop.sh, make it executable and run it. At this point There shouldn't be any InnoDB tables in MySQL server.

Now, it is a good time to shutdown your MySQL service and rm ibdata file. On typical CentOS system this would be done with

service mysqld stop
rm -f /var/lib/mysql/ibdata1

If you are not running CentOS and default MySQL install or if you have ibdata file on some other path… please adjust the above. Starting MySQL service at this point would "regenerate" ibdata file with its smallest size increment as defined in my.cnf lets do so:

service mysqld start

And now for the third script

#!/bin/bash
CWD=`pwd`
for database in `ls $CWD/*.sql|awk -F'/' '{print $NF}'|cut -d. -f1`; do
mysqladmin create $database
mysql $database < $CWD/$database.sql
done

As before copy and paste this script into step3-import.sh (remember to place this file at the same directory as other two, and dumps), make it executable and run it.

Voila! your data is back in, ibdata reduced and MySQL up&running.

Remember to restore any access to your MySQL service (starting Apache, clearing firewall drops, etc..)

Wsgi on cPanel improved

Sun, 27 Mar 2011 20:01:02 +0000

There is a long time now from my original post on running Django with python 2.6 on cPanel based servers, and as time passed there was some issues while deploying Django in such a way. So I decided to post another post with some updates regarding deployment of any python based scripts trough mod_wsgi and cPanel.

What's changed since last post?

There is a fix for easy_apache rebuilds, retaining the mod_wsgi and not braking rebuild process
There was some additions in virtualhost include files, enabling python app to run as a user, not nobody
A fix for cpanel's shell fork bomb protection while running python aps as a user and not nobody
Some minor changes regarding media handling, so it get's included in virtualhost includes and not by eating up user's subdomains
And of course python and Django version updates

Disclaimer at the beginning, this setup works for me on latest RELESE cPanel build, running on top of CentOS 5.5. I'm posting this as a way how I managed to get it working, so this shouldn't be considered as a official way of running Django or any other python app on cPanell servers. Although, as per my last post <a href="/blog/2010/django-on-cpanel-with-python2-6-virtualenv-and-mod_wsgi/">Django on cpanel with python2.6, virtualenv and mod_wsgi</a> lot's of people are reporting this is working for them also, and this version of deployment guide will bring some more enhancements.

Installing python and dependencies

mkdir /usr/src/python2.7 && cd /usr/src/python2.7

again, if you need sqlite, now is the time to install it.

wget http://www.sqlite.org/sqlite-autoconf-3070500.tar.gz
tar zxvf sqlite-autoconf-3070500.tar.gz
cd sqlite-autoconf-3070500
./configure
make
make install

Since CentOS 5.5 is still running python 2.4 and if you don't wish to brake your system you should install your python 2.7 in alternate location:

cd /usr/src/python2.7/
wget http://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz
tar zxvf Python-2.7.1.tgz
cd Python-2.7.1
./configure --prefix=/opt/python2.7 --with-threads --enable-shared
./configure --help
make
make install

This will install python 2.7 in /opt/python2.7 directory. To make any use of this alternate install we must do the following:

ln -s /opt/python2.7/bin/python /usr/bin/python2.7
echo '/opt/python2.7/lib'>> /etc/ld.so.conf.d/opt-python2.7.conf
ldconfig

It will make a symbolic link in your path and instruct the system where to find libs for this alternate python install.

Now is a good time to check if everything is working as it should

root@toy2 [/usr/src/python2.7]# /usr/bin/python2.7
Python 2.7.1 (r271:86832, Mar 26 2011, 22:31:33)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>>

Type CTRL+D to exit.

Next thing to do is installation of python-setup tools:

/usr/bin/python2.7
cd /usr/src/python2.7/
wget http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
sh setuptools-0.6c11-py2.7.egg --prefix=/opt/python2.7

In this tutorial, I'm skipping the installation of mysql for python since you will have to install it per user basis in their virtual environments. All we need to do next is to install virtualenv to our python 2.7 so we can distribute them to our users

cd /opt/python2.7/bin/
./easy_install virtualenv

mod_wsgi installation

cd /opt/python2.7/lib/python2.7/config/
ln -s ../../libpython2.7.so .
cd /usr/src/python2.7/
wget http://modwsgi.googlecode.com/files/mod_wsgi-3.3.tar.gz
tar zxvf mod_wsgi-3.3.tar.gz
cd mod_wsgi-3.3
./configure --with-python=/opt/python2.7/bin/python
make
make install

Now this will install mod_wsgi.so file in /usr/local/apache/modules folder, as I experienced during cPanel easy_apache rebuilds it will completely empty this folder, thus any virtualhost includes relying on mod_wsgi will fail and easy_apache will not be able to properly rebuild your httpd.conf file and whole apache or php upgrade you were running will fail and revert back to last good state.

To avoid this we will now copy the mod_wsgi to different folder:

mkdir /usr/local/apache/extramodules
mv /usr/local/apache/modules/mod_wsgi.so /usr/local/apache/extramodules/

Note that this wsgi module is built with python 2.7 and will not work with any other version. If for some strange reason later on you try to run any given python application with system's default python2.4 it will not work.

Anyways, all we need to do next is to include mod_wsgi into the apache configuration:

nano /usr/local/apache/conf/includes/pre_virtualhost_global.conf

and paste:

LoadModule wsgi_module /usr/local/apache/extramodules/mod_wsgi.so
AddHandler wsgi-script .wsgi

Now do configtest:

root@toy2 [~]# service httpd configtest
Syntax OK

If you get Syntax OK at the end, you can safely restart your apache:

/scripts/restartsrv httpd

This is it, our apache can now serve python apps like Django, pylons, etc...

Setting up Django projects

Just like in my previous post I'll use '[username]' for cpanel username reference and '[domain]' for that user's domain. To match your needs you will have to replace those where appropriate.

We will quickly setup user's virtualenv, be aware though some of the python packages requires compile rights, and most of the cPanel setups I've seen so far are forbidding compiler access to their users. You can either make an permanent exception for this user, or you can instruct your user to contact you whenever he needs to install some package that requires compile rights (PIL, mysql...).

To enable compile rights for that specific user you can find that user in WHM WHM->Compiler Access->Allow specific users to use the compilers or if you don't like to exit the shell you can do it like this:

gpasswd -a [username] compiler

to remove it, use the same path in WHM and in shell:

gpasswd -d [username] compiler

For now, lets add this user to the compiler group since we will be installing some python packages for him in the start. Also you will have to enable your user a normal login shell, let's first check what shell this user has.

cat /etc/passwd |grep [username]

it should return something like this:

[username]:x:928:923::/home/[username]:/usr/local/cpanel/bin/jailshell

This user has jailshell enabled to change that into normal shell do this:

usermod -s /bin/bash [username]

If it's already bash, don't change it. You can also do this via WHM-> Manage Shell Access.

So let's finaly create a virtualenv for our user:

cd /home/[username]
/opt/python2.7/bin/virtualenv --no-site-packages --distribute virtualenv
chown -R [username].[username] virtualenv

this will create a new directory /home/[username]/virtualenv with our new virtual python environment for our user. From this point on we will do everything as the user and not root, until told otherwise.

su [username]
source virtualenv/bin/activate

And the promt will change to something like this:

(virtualenv)[username]@[domain] [~]#

that way we know we are loged in as user and we are using his virtualenv. Now let's install Django in this users environment, simply enter:

easy_install django

Now is also a good time to install additional python site-packages like mysql and PIL.

easy_install pil
easy_install mysql-python

So we have the django installed inside the users virtualenv, all we have to do now is start one of the instance. It's probably the best practice to keep django sites outside of public_html folders on cpanel servers. So we will do (still as a user)

cd
mkdir djangosites
cd djangosites
django-admin.py startproject [projectname]

So by now we should have a directory structure like this:

/home/[username]/virtualenv < - python virtual environment
/home/[username]/djangosites <- django sites folder
/home/[username]/djangosites/[projectname] <- django project

Create the wsgi script for that project, usualy [projectname].wsgi in django sites folder

nano /home/[username]/djangosites/[projectname].wsgi

and paste the following code:

import sys
import site
import os
vepath = '/home/[username]/virtualenv/lib/python2.7/site-packages'
prev_sys_path = list(sys.path)
site.addsitedir(vepath)
sys.path.append('/home/[username]/djangosites')
new_sys_path = [p for p in sys.path if p not in prev_sys_path]
for item in new_sys_path:
sys.path.remove(item)
sys.path[:0] = new_sys_path
from django.core.handlers.wsgi import WSGIHandler
os.environ['DJANGO_SETTINGS_MODULE'] = '[projectname].settings'
application = WSGIHandler()

give the scripts execute permissions:

chmod +x /home/[username]/djangosites/[projectname].wsgi

and exit user login with:

exit

now you will have your root propmt at the shell:

root@servername [/home/username]#

If you have decided so, now is the good time to disable the compiler access for this user.

create the apache include folder for that virtualhost:

mkdir -p /usr/local/apache/conf/userdata/std/2/[username]/[domain]/
nano /usr/local/apache/conf/userdata/std/2/[username]/[domain]/django.conf

and add:

<ifmodule mod_wsgi.c>
WSGIScriptAlias / /home/[username]/djangosites/[projectname].wsgi
WSGIDaemonProcess [projectname] user=[username] group=[username] processes=5 threads=15 display-name=%{GROUP}
WSGIProcessGroup [projectname]
WSGIApplicationGroup %{GLOBAL}
</ifmodule>

Ok, so this will create a processgroup for this specific django projects with 5 processes each 15 threads with our user's uid and gid. This is great since we have isolated this user to his own application pool and his own home path, also we are enforcing user quotas since if we omit "user=[username] group=[username]" from WSGIDaemonProcess the deamon process will run ass apache user nobody, and all files made/uploaded via this application will not count into this user's quota. They will also make lot's of problems if the user don't grant global write permissions on some folders. Just like mod_php and php_suexec.

Shell fork bomb exception

On the other hand, running 5 processes each with 15 threads (you can customize this ofcourse), will hit the defaults ulimits enforced by cPanels shell fork bomb protection (if enabled). You could either disable the protection, or you can make a little adjustments to its logic so that you can make exceptions per user allowing them a bit more freedom but still not unlimited.

I have found this method somewhere on cPanel forums (I can't seem to find the exact post right now, but I will update this post when I find it), anyways with little patching of profile, limits and bashrc you can create a list of users that have a slightly higher limits, so our django users won't run into any memory or fork limits normally enforced by fork bomb protection.

I've created a little tarball with installer scripts for patched fork bomb logic, I advise you to check the files prior to installing them. Script will make a copies of your original files, to the same destination with .orig suffixes.

cd /usr/src/
wget https://toic.org/files/django/forkbomb.tar.gz
tar zxvf forkbomb.tar.gz
cd forkbomb
./install.sh

Now all you have to do is put one username per line in /etc/profile.exclude

Each user in that line will have slightly bigger ulimits

Back to the Django

Let's also make admin media paths and media folder paths

System Message: WARNING/2 (<string>, line 323)

Literal block expected; none found.

mkdir /home/[username]/djangosites/[projectname]/media nano /usr/local/apache/conf/userdata/std/2/[username]/[domain]/django-media.conf

and add.

Alias /admin_media/ /home/[username]/virtualenv/lib/python2.7/site-packages/Django-1.2.4-py2.7.egg/django/contrib/admin/media/
<Directory /home/[username]/virtualenv/lib/python2.7/site-packages/Django-1.2.4-py2.7.egg/django/contrib/admin/media>
Order deny,allow
Allow from all
</Directory>
Alias /media/ /home/[username]/djangosites/[projectname]/media/
<Directory /home/[username]/djangosites/[projectname]/media>
Order deny,allow
Allow from all
</Directory>

Just replace your python version, django version and username and projectname in paths. This will create [domain]/media - for site media and [domain]/admin_media/ for admin media

Now verify those apache include files and rebuild apache conf:

/scripts/verify_vhost_includes
/scripts/rebuildhttpdconf

As always you can make subdomains for your media files, you will just need to update your [projectname]/settings.py

And that's it, hope you enjoy your django installation

Django on cpanel with python2.6, virtualenv and mod_wsgi

Sat, 14 Aug 2010 19:59:44 +0000

Here is the simple to follow step by step tutorial on howto run latest Django on cpanel powered servers with python 2.6, virtual env and mod_wsgi.
If you are running cPanel on your server it's most probably a rhel or centos distro, and those will have by default python2.4 installed. You can't just overwrite your python2.4 as many of the system utilities are depended on the python2.4 So what we will do is install an alternate python 2.6 and setup our server to run Django instances with python2.6

First of all let's make a folder where we will download all those source packages

Installing python and dependencies

mkdir -p /usr/src/python26 && cd /usr/src/python26

Next, we have to download and install sqlite3 dependency. It's not needed if your Django instances will not run sqlite databases, but on a safe side it's better to install it than not.

wget http://www.sqlite.org/sqlite-amalgamation-3.6.4.tar.gz
tar zxvf sqlite-amalgamation-3.6.4.tar.gz
cd sqlite-3.6.4
./configure
make
make install

If all went well it's time to install python 2.6 as alternate install.

cd /usr/src/python26
wget http://www.python.org/ftp/python/2.6.5/Python-2.6.5.tgz
tar zxvf Python-2.6.5.tgz
cd Python-2.6.5
./configure --prefix=/opt/python2.6 --with-threads --enable-shared
make
make install

This will install python 2.6 in /opt/python2.6 and will not interfere with system python install. Let's make a symbolic link to our newly installed python:

ln -s /opt/python2.6/bin/python /usr/bin/python2.6<

Also we must instruct our system where it should find the libs for new python.

echo '/opt/python2.6/lib' >> /etc/ld.so.conf.d/opt-python2.6.conf
ldconfig

You shoud check if your new python install is working as it should:

/usr/bin/python2.6
Python 2.6.5 (r265:79063, Jul 15 2010, 16:55:23)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Type CTRL+D to exit. Next thing to do is installation of python-setup tools:

cd /usr/src/python26
wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg
sh setuptools-0.6c11-py2.6.egg --prefix=/opt/python2.6

Make sure to define the prefix on where your new python 2.6 installation is residing. We should make a temporary alias for new python as it will be needed for installation of python mysql package:

alias python="/opt/python2.6/bin/python"

Let's check if everything is working ok:

python
Python 2.6.5 (r265:79063, Jul 15 2010, 16:55:23)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> CTRL + D

So next in line is mysql for python:

cd /usr/src/python26
wget http://netcologne.dl.sourceforge.net/project/mysql-python/mysql-python/1.2.3/MySQL-python-1.2.3.tar.gz
tar zxvf MySQL-python-1.2.3.tar.gz
cd MySQL-python-1.2.3
python setup.py build
python setup.py install

Again let's verify the install:

cd /usr/src/python26
python
Python 2.6.5 (r265:79063, Jul 15 2010, 17:57:29)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> import MySQLdb
>>> CTRL + D

It shouldn't return any errors while issuing import statements. Ok for the end let's install the virtualenv so we can create a virtual python enviroment for each user and his Django instances

cd /opt/python2.6/bin
./easy_install virtualenv

And here we conclude the python instalation. Next order of business is mod_wsgi

mod_wsgi installation

cd /opt/python2.6/lib/python2.6/config/
ln -s ../../libpython2.6.so .
cd /usr/src/python26
wget http://modwsgi.googlecode.com/files/mod_wsgi-3.2.tar.gz
tar zxvf mod_wsgi-3.2.tar.gz
cd mod_wsgi-3.2
./configure –with-python=/opt/python2.6/bin/python
make
make install

This will install mod_wsgi in /usr/local/apache/modules/mod_wsgi.so Note that this wsgi module is built with python 2.6 and will not work with any other version. All we have to do now is add the mod_wsgi in our apache conf. You can do that on cpanel servers via WHM. Log into WHM , click on Apache Configuration then Include Editor then Pre-Virtualhost Include and select All versions Paste this:

LoadModule wsgi_module /usr/local/apache/modules/mod_wsgi.so
AddHandler wsgi-script .wsgi

** Save and restart apache.** And that's it. Our apache is now ready to serve python scripts via mod_wsgi Now all we have to do is install some Django instances for our user.

Setting up Django projects

I'll use '[username] ' for cpanel username reference and '[domain] ' for that user's domain. To match your needs you will have to replace those where appropriate. First of all you will have to decide if you will do empty virtual environments so that the user can decide what python site-packages will he use or will you preinstall some packages so that user can only update his virtualenv with packages that are missing. If you are choosing the first method user will have to have compile rights under security setting of cpanel WHM otherwise he will not be able to install packages as python mysql, PIL and some others. If you choose the former method you can preinstall those packages systemwide for our alternate python install, but whenever you update those packages all virtual environments will be updated with them. It's a matter of personal preference. I for myself rather like the clean virtualenv, and before delivering it to the client you as root preinstall in his virtualenv those packages requiring gcc rights. So let's create a virtualenv for our user:

cd /home/[username]
/opt/python2.6/bin/virtualenv --no-site-packages --distribute virtualenv
chown -R [username].[username] virtualenv

this will create a new directory __ /home/[username]/virtualenv __ with our new virtual python environment for our user. You will have to enable normal shell for this user, let's first check what shell this user has.

cat /etc/passwd |grep [username]

it should return something like this:

[username]:x:928:923::/home/[username]:/usr/local/cpanel/bin/jailshell

This user has jailshell enabled to change that into normal shell do this:

usermod -s /bin/bash [username]

You can also do this via WHM-> Manage Shell Access. From this point on we will do everything as the user and not root, until told otherwise.

su [username]
source virtualenv/bin/activate

And the promt will change to something like this:

(virtualenv)[username]@[domain] [~]#

that way we know we are loged in as user and we are using his virtualenv. Now let's install Django in this users environment, simply enter:

easy_install django

Now is also a good time to install additional python site-packages like mysql and PIL. First temporarily enable compilers for this user via WHM->Compiler Access->Allow specific users to use the compilers do the modules install with:

easy_install [module_name]

disable the compiler access afterwards

cd
mkdir djangosites
cd djangosites
django-admin.py startproject [projectname]

So by now we should have a directory structure like this:

/home/[username]/virtualenv < - python virtual environment
/home/[username]/djangosites <- django sites folder
/home/[username]/djangosites/[projectname] <- django project

Create the wsgi script for that project, usualy [projectname].wsgi in django sites folder

nano /home/[username]/djangosites/[projectname].wsgi

and paste the following code:

import sys
import site
import os
vepath = '/home/[username]/virtualenv/lib/python2.6/site-packages'
prev_sys_path = list(sys.path)
site.addsitedir(vepath)
sys.path.append('/home/[username]/djangosites')
new_sys_path = [p for p in sys.path if p not in prev_sys_path]
for item in new_sys_path:
sys.path.remove(item)
sys.path[:0] = new_sys_path
from django.core.handlers.wsgi import WSGIHandler
os.environ['DJANGO_SETTINGS_MODULE'] = '[projectname].settings'
application = WSGIHandler()

give the scripts execute permissions:

chmod +x /home/[username]/djangosites/[projectname].wsgi

and exit user login with:

exit

now you will have your root propmt at the shell:

root@servername [/home/username]#

create the apache include folder for that virtualhost:

mkdir -p /usr/local/apache/conf/userdata/std/2/[username]/[domain]/
nano /usr/local/apache/conf/userdata/std/2/[username]/[domain]/django.conf

and add:

<ifmodule mod_wsgi.c>
WSGIScriptAlias / /home/[username]/djangosites/[projectname].wsgi
WSGIDaemonProcess [username] processes=7 threads=1 display-name=%{GROUP}
WSGIProcessGroup [username]
WSGIApplicationGroup %{GLOBAL}
</ifmodule>

edit the virtualhost entry for that domain:

nano /usr/local/apache/conf/httpd.conf

and in that virtualhost add:

Include "/usr/local/apache/conf/userdata/std/2/[username]/[domain]/*.conf"

save & exit

test the apache configuration prior to reload:

service httpd configtest

Should say syntax OK at the end, if so reload the apache:

/scripts/restartsrv_httpd

And voila, you have your django instance up& running You can now add via User's cpanel media subdomain for django media files, and define the path to the subdomain root in setings.py media root can live inside public_html folder. You can also create additional subdomain for admin media and copy the admin media content to that subdoman. For example: admin-media.[domain]:

cp -r /home/[username]/virtualenv/lib/python2.6/site-packages/Django-1.2.1-py2.6.egg/django/contrib/admin/media/* /home/[username]/public_html/admin-media/

Munin centralized monitoring on Centos

Fri, 09 Oct 2009 19:56:24 +0000

Munin is a great tool for performance graphing your servers, by default it will graph resources on localhost, however if you wish to monitor multiple servers from single location, then you must deploy a central Munin server.

Having central graphs mean you can have central hub of data about performance of your systems. This can later be better presented, viewed and analyzed,

Let's begin setting up a central server

To get things started we will need one server for centralized graphs. For start this can be a low budget dedicated server or a small vps, if you have large amount of monitored nodes, and large amout of metrics on them, then you will probably want to invest in better disks. Updating huge amounts of rrd files and regenerating html an graphs can be disk IO intensive with large instalations.

Installing munin

This will be a minimal install for a central munin server. I'm using a small vps with minimal centos 5 install.

First let's setup elrepo:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

Now we need to install munin and munin-node (if you wish to monitor this host as well):

yum install -y munin munin-node

by default munin will put its html files into /var/www/html/munin folder If you wish to move that to another place, now is your time. For the sake of simplicity I'll just leave it where it is. Of course we will need apache to access munin html files, so if you don't have apache installed do:

yum install -y httpd

now start the apache:

service httpd start

If you left everything as it is munin html should be available at: http://yourhostname.com/munin/ You may notice that there is nothing there yet, just wait until we configure all other hosts. start the munin-node on this host (if you installed it):

service munin-node start

Make sure your cron is runing:

service crond status

and let's go configure those other hosts.

Installing munin-node

Installing on cPanel

Since lot's of my servers to monitor are with cPanel installed there is an easy way to install munin.

Login to your whm go to: Manage plugins, now find Munin, click a check box, scroll down and click save. After the munin is installed it should appear in your whm at the bottom of the navigation. Go and check up if the munin is installed correctly.

Installing trough cPanel will install munin-node and munin, you can disable the munin graphing later if you like.

Installing on non cPanel

We can install munin on Centos trough Elrepo. first we will setup elrepo:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

and then install a munin-node:

yum install -y munin-node

voila... let's configure nodes on remote servers now.

Configuring munin-node on remote hosts

For both cPanel and non-cPanel servers all we need to do is add allowed host in munin-node.conf:

nano /etc/munin/munin-node.conf

add at the end of the file:

allow ^192\.168\.0\.20$

where 192.168.0.20 is the IP address of you central munin server. restart the munin-node:

service munin-node restart

If you have firewall installed on that host (and I hope you do), allow the incoming tcp port 4949 for the IP of the central node.

In csf add the following line:

tcp:in:d=4949:s=192.168.0.20

to your /etc/csf/csf.allow file

or just run:

iptables -A INPUT -p tcp -s 192.168.0.2 --dport 4949 -m state --state NEW,ESTABLISHED -j ACCEPT

Modify this to your firewall, and don't forget to replace 192.168.0.20 with your munin server IP. Now everything should be ready for data collection from central server

Configuring munin server

We need to configure munin conf file on our central server to collect data from remote servers:

nano /etc/munin/munin.conf

If you didn't change any locations of html files and munin datastore you realy don't need to change that in the conf file.

What we are interested with are the host sections. You will notice there is configuration for our localhost You can change its name now, leave the address field as it is.

To add up a new host just add:

[myhost.mydomain.com]
address 192.168.0.10
use_host_name yes

change the 192.168.0.10 with the IP of the server you wish to monitor. you can now add as many host you like. Make sure that you have enabled outgoing connections on tcp port 4949 on your central munin server. After a while the first results should start to appear.

Configuring multi host display graphs

The real benefit of having all the host graphs and data on one place is you can easily make multi host graphs and compare the loads on the servers. This could help you grasp a bigger picture of individual server workloads and give you an idea what to improve and how to load balance between the machines.

Here is one of the example graphs, showing apache request per second. If the machines were the same hardware configuration that would give indications that some of the machines have higher hit rate and we would need to rewrite our load balancing.

We could do the same thing with load graphs and see which servers have the spikes, and distribute the workload on some less loaded servers.

So how do we configure this?

First you need to find out rrd's name of the data you wish to put on the graph.

for example apache accesses per second:

cd /var/lib/munin/yourdomain.com

ls -lh in the directory and you will find out what data is available to munin. in case of the apache accesses data we will have few files named:

hostname.domainname.com-apache_accesses-accesses80-d.rrd hostname2.domainname.com-apache_accesses-accesses80-d.rrd

what we are interested with are those fields (marked in red) after the domain name separated by dash. Ok let's write a conf in munin.conf for this two hosts:

nano /etc/munin/munin.conf

Go under the host definitions in your conf file and add:

[domainname.com;Totals] update no apacheaccess.graph_title Apache access side by side apacheaccess.graph_order hostname=hostname.domainname.com:apache_accesses.accesses80 hostname1=hostname1.domainname.com.com:apache_accesses.accesses80

Notice the red lines, they are the same as rrd filenames red parts we saw earlier, just replace dash with dot. Green text is to disable updates for this domain declaration since updates are already done at the host declaration in the conf file. Blue is the graph representation name, followed by title in the first line and data in second.

This way you can make all the side by side graphs for all the data munin collected in rrd files. After the changes wait for a next munin update and enjoy the graphs

Firewalling xen bridge

Sun, 19 Apr 2009 19:52:29 +0000

Occasionally you will wish to block certain ports to your DomUs from Dom0. By default you wish to allow any traffic from and to DomU but for some security considerations, I found it to be wise to block some ports to and from my clients DomUs. One such port range is for example IRC. Although it can be routed trough alternate ports, most of automated malicious scripts use default ones. It's quite handy to block them so they ain't able to contact home.

As said by default Xen bridge is open for all traffic from and to DomUs. It's up to DomU admin to firewall their own virtual machine. Unfortunately some just forget to do the proper securing of the system, and as a result you get compromised DomU contacting various botnets, and executing all kind of nasty stuff.

To prevent this we can make a firewall rules in DomU that will by default block some traffic. Since I'm using bridged network, firwalling must be done on bridge. I found this great article on shorewall manuals explaining how to setup bridged network firewall. I installed it and tested it on 32bit Centos 5.2, but it should work on any system.

Fist of all you will need to download and install latest shorewall.

As stated in documentation link above, you must enable bridge support in shorewall.conf:

nano /etc/shorewall/shorewall.conf

Set:

BRIDGING=Yes

Now we have to edit our firewall zones:

nano /etc/shorewall/zones

It should look something like this:

#ZONE TYPE OPTIONS IN OUT
# OPTIONS OPTIONS
fw    firewall
dom0 ipv4
domU ipv4
net ipv4
#LAST LINE - ADD YOUR ENTRIES ABOVE THIS ONE - DO NOT REMOVE

Next thing to do is to define network interfaces, we will be dealing with two network interfaces: virtualized eth0 and bridge:

nano /etc/shorewall/interfaces

And the file should look like this:

#ZONE INTERFACE    BROADCAST    OPTIONS
-    xenbr0 - dhcp
net eth0 detect dhcp
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

Next stop, hosts file:

nano /etc/shorewall/hosts

And the file should look like this:

#ZONE HOST(S) OPTIONS
dom0 xenbr0:vif0.0
domU xenbr0:vif+     routeback
net xenbr0:peth0
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS LINE -- DO NOT REMOVE

Now let's make some policies in our firewall:

nano /etc/shorewall/policy

And the file should look like this:

#SOURCE DEST POLICY LOG LIMIT: CONNLIMIT:
# LEVEL BURST MASK
fw    all ACCEPT
all fw    ACCEPT info
dom0 all ACCEPT
all dom0 ACCEPT info
domU all ACCEPT
all domU     ACCEPT
net net NONE
all all REJECT info
#LAST LINE -- DO NOT REMOVE

This will by default allow any traffic through the bridge. You can also specify DROP policy for your Dom0 and then open necessary ports in rules file. Note that the fw and dom0 are the same but they both need to be declared in policy and rules file. So... for now, this does not block IRC traffic as we started to do, all we need to do now is to setup the rules file:

nano /etc/shorewall/rules

And the file should look like this:

#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE USER/ MARK CONNLIMIT    TIME PORT PORT(S) DEST LIMIT GROUP
#irc
REJECT net domU tcp 6660:6669
REJECT domU net tcp 6660:6669
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

After adding this, it will block all incoming and outgoing traffic from port range 6660 to 6669 for all DomUs. If you wish to add an exception to one DomU you can simply edit the rules file and insert the exception above the REJECT:

#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE USER/ MARK CONNLIMIT    TIME PORT PORT(S) DEST LIMIT GROUP
#DomU exceptions
ACCEPT net domU:192.168.0.10    tcp 6660:6669
ACCEPT domU:192.168.0.10    net tcp 6660:6669
#DomU restrictions
#irc
REJECT net domU tcp 6660:6669
REJECT domU net tcp 6660:6669
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

This way only the DomU with ip 192.168.0.10 will have unblocked IRC ports. Although the above config should work it didn't for me. Centos 5.2 by default comes with:

net.bridge.bridge-nf-call-iptables = 0

so no bridge firewalling is actually done. To enable this edit your sysctl.conf file:

nano /etc/sysctl.conf

and append:

net.bridge.bridge-nf-call-iptables = 1

now run:

sysctl -p

And the bridged firewall for your DomUs should work now.

Multiple network interfaces in Xen

Mon, 06 Oct 2008 19:44:41 +0000

By default xen tools comes with only one network interface enabled for your dom0 and domU machines. So what if you want to add some more? It's actually very simple.

All you need to do is run:

/etc/xen/scripts/network-bridge start vifnum=1 netdev=eth1 bridge=xenbr1

This will effectively create one extra xenbr attached to eth1 interface. You can repeat the above command for all your interfaces, and you can stop them in the same manner, just replace start with stop.

To enable this automatically you can create file named, multi-network-bridge:

vim /etc/xen/scripts/multi-network-bridge

paste this:

#!/bin/sh
/etc/xen/scripts/network-bridge $@ vifnum=0 netdev=eth0 bridge=xenbr0
/etc/xen/scripts/network-bridge $@ vifnum=1 netdev=eth1 bridge=xenbr1

Of course you can add up as many interfaces you like in this script. After you have added your interfaces, you need to edit xend-config.spx file:

vim /etc/xen/xend-config.spx

Find a line defining network script, it should by default look like this:

(network-script network-bridge)

Edit it so it contains your newly created multi network bridge script. In my case it should look like:

(network-script multi-network-bridge)

All you need to do now is restart xend service:

/etc/init.d/xend restart

New network bridge named xenbr1 should be available now.

How can I add up another network bridge to my domU machine?

It's pretty simple actually, you already have defined vif statements for current network interface, all you need to do now is edit that domU config file, precisely vif line and add up another xenbr interface.

For example, if my domU vif line looks like this:

vif = ['ip=xx.xxx.167.4, vifname=vifbran0, rate = 10000KB/s, bridge=xenbr0']

with another xenbr interface it should look like this:

vif = ['ip=89.201.167.4, vifname=vifbran0, rate = 10000KB/s, bridge=xenbr0', 'ip=192.168.1.10, vifname=vifbran1, rate = 10000KB/s, bridge=xenbr1']

Also if you wish to use ip conflict prevention you must add additional mac section in vif configuration. Restart your domU and voila, another network interface is present

Preventing ip conflicts in xen

Mon, 22 Sep 2008 19:07:40 +0000

Lately I was playing with stock xen kernel and virtualization, and I came across one relatively big problem. Let’s say I want to share my guest machines to, let’s say clients. You must give them root… because that’s whats VPS-es all all about, having root access to OS without having to purchase expensive hardware. Having that in mind they are by default untrusted and unpredictable, they can do god knows what in there!

So what caught my eye?

By default xen, and available management tools, don’t really have a way of sorting out IP conflicts in bridged mode. Basically you have bunch of scripts that will provision VPS alongside with IP address. Looking at the conf files you have vif and IP declarations in vm_xen.conf file.

But what is really preventing clients from entering:

ifconfig eth0 xxx.xxx.xxx.xxx

where xxx is ip of some super important server in same netmask?

Luckily, I came across this problem while still in testing, here’s what I found and came up with after 3 days intensive googling.

Xen supports IP declaration in vif statment of domU config file like this:

vif = ['ip=xxx.xxx.xxx.xxx, more parametars here....']

Also you can declare multiple IP's by simply putting space between them, like this:

vif = ['ip=xxx.xxx.xxx.xx1 xxx.xxx.xxx.xx2, more parametars here....']

For the purpose of IP conflict prevention make sure you declare unique mac address in vif section to.

So what does this IP thingy in vif do? Absolutely nothing (at least not yet)!

Next step is to install ebtables (http://ebtables.sourceforge.net/) on your distro. Then all we need to do is patch up a vif-bridge script located in /etc/xen/scripts/

So here’s the diff:

--- vif-bridge-org 2008-07-30 21:26:16.000000000 +0200
+++ vif-bridge 2008-07-30 21:30:59.000000000 +0200
@@ -57,15 +57,35 @@
online)
setup_bridge_port "$vif"
add_to_bridge "$bridge" "$vif"
- ;;
-
+ ebtables -N $vif
+ ebtables -P $vif DROP
+ ebtables -A INPUT -i $vif -j $vif
+ ebtables -A FORWARD -i $vif -j $vif
+ ebtables -A $vif -p ARP --arp-opcode 1 -j ACCEPT
+
+ if [ ! -z "$ip" ]
+ then
+ for oneip in $ip
+ do
+ ebtables -A $vif -p IPv4 --ip-src $oneip -j ACCEPT
+ ebtables -A $vif -p IPv4 --ip-dst $oneip -j ACCEPT
+ ebtables -A $vif -p ARP --arp-opcode 2 --arp-ip-src $oneip -j ACCEPT
+ done
+ ebtables -A $vif --log-prefix="arp-drop" --log-arp -j DROP
+ fi
+ ;;
+
offline)
do_without_error brctl delif "$bridge" "$vif"
do_without_error ifconfig "$vif" down
- ;;
+ do_without_error ebtables -D INPUT -i $vif -j $vif
+ do_without_error ebtables -D FORWARD -i $vif -j $vif
+ do_without_error ebtables -F $vif
+ do_without_error ebtables -X $vif
+ ;;
esac-handle_iptable
+#handle_iptable
log debug "Successful vif-bridge $command for $vif, bridge $bridge."
if [ "$command" == "online" ]

Asuming you use bridging scripts this effectively restricts IP address(es) from "vif = ['ip=xxx.xxx.xxx.xxx']" list to mac addresses in vif list. Restrictoins are done while booting up VPS and removed when powering it off. This way untrusted user is limited only to IP addresses defined in xen guest conf file. Trying to change existing IP address into another IP on network will only render that machine unresponsive.