Opensolaris server with COMSTAR and zfs

I’ve being playing around with Sun’s Opensolaris for a while and I’m quite pleased with it to say at least.
Opensolaris 2009.06 comes with few nifty features and software like dtrace, zfs, COMSTAR, etc.

I tried it a while ago as a desktop system, and was quite presently surprised with ZFS and its snapshots so I played little more with it. I do plan to make another post on this subject alone but mine main intent of this post will be to help you setup an Opensolaris server, without GUI who will use its, I would say best quality, zfs and other set of tools like COMSTAR to provide scalable, high performance, low budget, storage server.

Since I intend to use this post for future reference in some setups I will try to keep it straightforward and just explain basic setup for get the non gui Opensolaris up & running with COMSTAR and additional zfs pool with single simple iSCSI target.

I will be using a Opensolaris 2009.06 (you can download LiveCD image from this link), a simple low budget hardware with three sata hard drives.

Read more

2009
11/08
CATEGORY
Opensolaris
Storage
2 comments

Dealing with memory limits

memory usage limits

So i decided to make a little different post than usual, a little more thoughts based than tutorial alike.
I would like to take a brief overview of todays memory usage for various tasks and scripts. Since large part of my the day is involved in web servers and their management I will mainly focus on memory usage for web applications and scripts.

Not so long ago, having a server with 4Gb of  working memory was a luxury, today we have certain scripts consuming about 512Mb of memory while running. What changed?
Internet boom, popularity of web 2.0 applications, ease of development, bunch of those learn programing in 21 days books and tutorials, that is what happened. In addition there is a whole bunch of people from “I want Internet” generation who still don’t quite grasp the difference between RAM and Disk memory, not to mention they don’t quite grasp the inner mechanics of computer systems.

Sir… the white smoke got out of your metal box… you must refill it with white smoke to make it work again

Well, like it or not those people like to call themselves web masters and web developers.

Read more

2009
10/27
CATEGORY
Linux
PHP
programing
3 comments

Howto create rsync server

There are tons of reasons why would one want to create a rsync server. For example you wish to backup your data to a remote server but you don’t want to backup everything every time.

rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License and is currently being maintained by Wayne Davison.

As you can see rsync is ideal for this. You can use it within ssh protocol, rsh and rsync itself. Creating a rsync server will allow you to create easily accessible storage server, update server for your scripts, etc.

Anyway let’s get started on configuring rsync server which will serve as remote backup server.
Read more

2009
10/21
CATEGORY
Linux
Storage
Write comment

Munin centralized monitoring on Centos

muninSo recently I went nuts having to login onto each server to look at its munin graphs. While you have few servers it’s doable, but managing large farms and checking up on them while having to login into each is just pain in the ass.
So what to do?
Hey… let’s make a central munin server, and let’s hold all the graphs there. That way we can review them all with just one user name and password, we can compare host performances, etc…
To accomplish this we will need one server for centralized graphs (could be a low budget dedicated server or a small vps), apache installed on central munin server, munin-node installed on all other server we wish to monitor.
Read more

2009
10/03
CATEGORY
Linux
1 comment

Tail -f in python, truncate aware

So while doing a little coding I tried to find some tail -f class in python that will recognize when file that we tailing is been truncated. All I found was some tail -f classes that brakes on file truncate or rotate.

Eventually I came up with this:

import time
from os import stat
from os.path import abspath
from stat import ST_SIZE
 
class LogTail:
    def __init__(self, logfile):
        self.logfile = abspath(logfile)
        self.f = open(self.logfile,"r")
        file_len = stat(self.logfile)[ST_SIZE]
        self.f.seek(file_len)
        self.pos = self.f.tell()
    def _reset(self):
        self.f.close()
        self.f = open(self.logfile, "r")
        self.pos = self.f.tell()
    def tail(self):        
        while 1:
            self.pos = self.f.tell()
            line = self.f.readline()
            if not line:
                if stat(self.logfile)[ST_SIZE] < self.pos:
                    self._reset()
                else:
                    time.sleep(1)
                    self.f.seek(self.pos)
            else:
                """print, return or otherwise manipulate
                the tailed line"""
                print line

Maybe it’s a too much overhead to check each time for file size, but you get the general idea.

So anyways here’s a usage exampe:

tail = LogTail("/var/log/messages")
tail.tail()

This will print out any new line appended to /var/log/messages file. If the file gets truncated or log rotated, class will detect it and will return to the start.

2009
08/11
CATEGORY
Linux
Python
programing
Write comment