toic.org - Entries for the tag python

IPv4 country code IP ranges database

Sun, 23 Jun 2013 00:24:58 +0000

For a while now, I was using some third party IP ranges file lists for some restrictions lists based on country codes.

Long story short those third party sites either died or are not properly detecting IP ranges from certain country codes of my interest. I'm trying to put up my own list. In the process my IP ranges zone files will be available for public use on this site. I'll try to announce any changes to the background logic.

For now, the script in the background will be running every 6h rebuilding those files with new info. Current source is RIR-Statistics-Exchange-Format gathered from RIR members

Apache monitoring tool ApTop beta released

Wed, 12 Dec 2012 23:34:19 +0000

This is my first public publish to PyPi, without further delays, here is the full project description

ApTop

ApTop ia a top(1) alike monitoring utility for Apache.

Being daily involved in monitoring system load in shared environment I find it lacking proper tool for monitoring Apache behavior in real time. Granted there are some tools for this job, they are usually of limited options for some monitoring needs.

ApTop was written as a hobby project in python for debugging purposes while investigating several system load situations caused by web applications in shared hosting environment.

ApTop is still an early beta release with some known bugs, nevertheless it still provides a great starting point in monitoring Apache. Every feedback and feature request at this point is more than welcomed.

Description

ApTop will show near real time Apache active working proceses and virtual host connections. By default sleeping / waiting for connections slots will be filtered out.

ApTop is also displaying several screens

Dashboard or Home screen

This screen displays some relevant informations gathered from apache-status page. It is by default sorted by shortest last apache access time "SS" and it displays only active connections. Displayed fields are the same as in server-status page, so you can find detailed explanation for their meaning at the bottom of server-status page.

Displayed fields are:

PID - OS process ID M - Mode of operation CPU - CPU usage, number of seconds SS - Seconds since beginning of most recent request< Req - Milliseconds required to process most recent request Conn - Kilobytes transferred this connection Acc - Number of accesses this connection / this child / this slot Client - Remote client VHost - Local virtualhost Request - Request method and partial URL

Count by vhosts screen

This screen displays virtualhosts in order by highest count of active connections along with connection count.

Very useful for detecting high traffic sites.

Count by clients screen

This screen displays clients in order by highest count of active connections along with connection count.

Very useful for detecting resource exhausters.

Requirements

ApTop should work just fine on any unix based system with python standard packages and lxml.

Make sure you have libxml2 libxslt libxslt-devel libxml2-devel packages instaleld prior to building lxml:

For RHEL based distro you can install those with

yum install libxml2 libxslt libxslt-devel libxml2-devel

After that you can install lxml python package with easy_install lxml or pip install lxml

ApTop is using Apaches mod_status with ExtendedStatus On for gathering data. Please ensure that mod_status is enabled and proper status url is defined in aptop.conf There are some concernes regarding enabled ExtendedStatus On, but I think this links covers this nicely.

Instalation

These installation options are for RHEL based distros, adjust accordingly.

yum install libxml2 libxslt libxslt-devel libxml2-devel python-setuptools

Now you can do

easy_install ApTop

easy_install https://bitbucket.org/btoic/aptop/get/master.tar.gz

You can now run aptop.py

Configuration

ApTop has some built in default configuration directives, but it will also look first in ~/.aptop.conf and than /etc/aptop.conf for overrides.

ApTop uses .ini style configuration files and main cofniguration options should be placed after [aptop] section.

By default ApTop will use http://localhost/server-status url for grabbing data to display, and with default refresh rate of 5 seconds.

Available configuration options:

status_url = URL

This configuration option will require you to enter valid mod_status status url for server you wish to monitor. This defaults to http://localhost/server-status Note that Apache should have ExtendedStatus conf option set to On

refresh = seconds

This option will control ApTop default refresh interval. It defaults to 5 seconds and can't be lower than 1 second.

You can even monitor the remote systems if they granted you access rights to theirs server-status URL. In the future ApTop will provide options to save remote monitored servers to its configuration.

Runtime options

While running ApTop responds to following options:

H - display main screen

V - display vhosts by conn. count

c - display clients by conn. count

R - reverse the current sort order

I - toggle active/inactive slots

Aditional links

http://toic.org

https://bitbucket.org/btoic/aptop

Wsgi on cPanel improved

Sun, 27 Mar 2011 20:01:02 +0000

There is a long time now from my original post on running Django with python 2.6 on cPanel based servers, and as time passed there was some issues while deploying Django in such a way. So I decided to post another post with some updates regarding deployment of any python based scripts trough mod_wsgi and cPanel.

What's changed since last post?

There is a fix for easy_apache rebuilds, retaining the mod_wsgi and not braking rebuild process
There was some additions in virtualhost include files, enabling python app to run as a user, not nobody
A fix for cpanel's shell fork bomb protection while running python aps as a user and not nobody
Some minor changes regarding media handling, so it get's included in virtualhost includes and not by eating up user's subdomains
And of course python and Django version updates

Disclaimer at the beginning, this setup works for me on latest RELESE cPanel build, running on top of CentOS 5.5. I'm posting this as a way how I managed to get it working, so this shouldn't be considered as a official way of running Django or any other python app on cPanell servers. Although, as per my last post <a href="/blog/2010/django-on-cpanel-with-python2-6-virtualenv-and-mod_wsgi/">Django on cpanel with python2.6, virtualenv and mod_wsgi</a> lot's of people are reporting this is working for them also, and this version of deployment guide will bring some more enhancements.

Installing python and dependencies

mkdir /usr/src/python2.7 && cd /usr/src/python2.7

again, if you need sqlite, now is the time to install it.

wget http://www.sqlite.org/sqlite-autoconf-3070500.tar.gz
tar zxvf sqlite-autoconf-3070500.tar.gz
cd sqlite-autoconf-3070500
./configure
make
make install

Since CentOS 5.5 is still running python 2.4 and if you don't wish to brake your system you should install your python 2.7 in alternate location:

cd /usr/src/python2.7/
wget http://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz
tar zxvf Python-2.7.1.tgz
cd Python-2.7.1
./configure --prefix=/opt/python2.7 --with-threads --enable-shared
./configure --help
make
make install

This will install python 2.7 in /opt/python2.7 directory. To make any use of this alternate install we must do the following:

ln -s /opt/python2.7/bin/python /usr/bin/python2.7
echo '/opt/python2.7/lib'>> /etc/ld.so.conf.d/opt-python2.7.conf
ldconfig

It will make a symbolic link in your path and instruct the system where to find libs for this alternate python install.

Now is a good time to check if everything is working as it should

root@toy2 [/usr/src/python2.7]# /usr/bin/python2.7
Python 2.7.1 (r271:86832, Mar 26 2011, 22:31:33)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>>

Type CTRL+D to exit.

Next thing to do is installation of python-setup tools:

/usr/bin/python2.7
cd /usr/src/python2.7/
wget http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
sh setuptools-0.6c11-py2.7.egg --prefix=/opt/python2.7

In this tutorial, I'm skipping the installation of mysql for python since you will have to install it per user basis in their virtual environments. All we need to do next is to install virtualenv to our python 2.7 so we can distribute them to our users

cd /opt/python2.7/bin/
./easy_install virtualenv

mod_wsgi installation

cd /opt/python2.7/lib/python2.7/config/
ln -s ../../libpython2.7.so .
cd /usr/src/python2.7/
wget http://modwsgi.googlecode.com/files/mod_wsgi-3.3.tar.gz
tar zxvf mod_wsgi-3.3.tar.gz
cd mod_wsgi-3.3
./configure --with-python=/opt/python2.7/bin/python
make
make install

Now this will install mod_wsgi.so file in /usr/local/apache/modules folder, as I experienced during cPanel easy_apache rebuilds it will completely empty this folder, thus any virtualhost includes relying on mod_wsgi will fail and easy_apache will not be able to properly rebuild your httpd.conf file and whole apache or php upgrade you were running will fail and revert back to last good state.

To avoid this we will now copy the mod_wsgi to different folder:

mkdir /usr/local/apache/extramodules
mv /usr/local/apache/modules/mod_wsgi.so /usr/local/apache/extramodules/

Note that this wsgi module is built with python 2.7 and will not work with any other version. If for some strange reason later on you try to run any given python application with system's default python2.4 it will not work.

Anyways, all we need to do next is to include mod_wsgi into the apache configuration:

nano /usr/local/apache/conf/includes/pre_virtualhost_global.conf

and paste:

LoadModule wsgi_module /usr/local/apache/extramodules/mod_wsgi.so
AddHandler wsgi-script .wsgi

Now do configtest:

root@toy2 [~]# service httpd configtest
Syntax OK

If you get Syntax OK at the end, you can safely restart your apache:

/scripts/restartsrv httpd

This is it, our apache can now serve python apps like Django, pylons, etc...

Setting up Django projects

Just like in my previous post I'll use '[username]' for cpanel username reference and '[domain]' for that user's domain. To match your needs you will have to replace those where appropriate.

We will quickly setup user's virtualenv, be aware though some of the python packages requires compile rights, and most of the cPanel setups I've seen so far are forbidding compiler access to their users. You can either make an permanent exception for this user, or you can instruct your user to contact you whenever he needs to install some package that requires compile rights (PIL, mysql...).

To enable compile rights for that specific user you can find that user in WHM WHM->Compiler Access->Allow specific users to use the compilers or if you don't like to exit the shell you can do it like this:

gpasswd -a [username] compiler

to remove it, use the same path in WHM and in shell:

gpasswd -d [username] compiler

For now, lets add this user to the compiler group since we will be installing some python packages for him in the start. Also you will have to enable your user a normal login shell, let's first check what shell this user has.

cat /etc/passwd |grep [username]

it should return something like this:

[username]:x:928:923::/home/[username]:/usr/local/cpanel/bin/jailshell

This user has jailshell enabled to change that into normal shell do this:

usermod -s /bin/bash [username]

If it's already bash, don't change it. You can also do this via WHM-> Manage Shell Access.

So let's finaly create a virtualenv for our user:

cd /home/[username]
/opt/python2.7/bin/virtualenv --no-site-packages --distribute virtualenv
chown -R [username].[username] virtualenv

this will create a new directory /home/[username]/virtualenv with our new virtual python environment for our user. From this point on we will do everything as the user and not root, until told otherwise.

su [username]
source virtualenv/bin/activate

And the promt will change to something like this:

(virtualenv)[username]@[domain] [~]#

that way we know we are loged in as user and we are using his virtualenv. Now let's install Django in this users environment, simply enter:

easy_install django

Now is also a good time to install additional python site-packages like mysql and PIL.

easy_install pil
easy_install mysql-python

So we have the django installed inside the users virtualenv, all we have to do now is start one of the instance. It's probably the best practice to keep django sites outside of public_html folders on cpanel servers. So we will do (still as a user)

cd
mkdir djangosites
cd djangosites
django-admin.py startproject [projectname]

So by now we should have a directory structure like this:

/home/[username]/virtualenv < - python virtual environment
/home/[username]/djangosites <- django sites folder
/home/[username]/djangosites/[projectname] <- django project

Create the wsgi script for that project, usualy [projectname].wsgi in django sites folder

nano /home/[username]/djangosites/[projectname].wsgi

and paste the following code:

import sys
import site
import os
vepath = '/home/[username]/virtualenv/lib/python2.7/site-packages'
prev_sys_path = list(sys.path)
site.addsitedir(vepath)
sys.path.append('/home/[username]/djangosites')
new_sys_path = [p for p in sys.path if p not in prev_sys_path]
for item in new_sys_path:
sys.path.remove(item)
sys.path[:0] = new_sys_path
from django.core.handlers.wsgi import WSGIHandler
os.environ['DJANGO_SETTINGS_MODULE'] = '[projectname].settings'
application = WSGIHandler()

give the scripts execute permissions:

chmod +x /home/[username]/djangosites/[projectname].wsgi

and exit user login with:

exit

now you will have your root propmt at the shell:

root@servername [/home/username]#

If you have decided so, now is the good time to disable the compiler access for this user.

create the apache include folder for that virtualhost:

mkdir -p /usr/local/apache/conf/userdata/std/2/[username]/[domain]/
nano /usr/local/apache/conf/userdata/std/2/[username]/[domain]/django.conf

and add:

<ifmodule mod_wsgi.c>
WSGIScriptAlias / /home/[username]/djangosites/[projectname].wsgi
WSGIDaemonProcess [projectname] user=[username] group=[username] processes=5 threads=15 display-name=%{GROUP}
WSGIProcessGroup [projectname]
WSGIApplicationGroup %{GLOBAL}
</ifmodule>

Ok, so this will create a processgroup for this specific django projects with 5 processes each 15 threads with our user's uid and gid. This is great since we have isolated this user to his own application pool and his own home path, also we are enforcing user quotas since if we omit "user=[username] group=[username]" from WSGIDaemonProcess the deamon process will run ass apache user nobody, and all files made/uploaded via this application will not count into this user's quota. They will also make lot's of problems if the user don't grant global write permissions on some folders. Just like mod_php and php_suexec.

Shell fork bomb exception

On the other hand, running 5 processes each with 15 threads (you can customize this ofcourse), will hit the defaults ulimits enforced by cPanels shell fork bomb protection (if enabled). You could either disable the protection, or you can make a little adjustments to its logic so that you can make exceptions per user allowing them a bit more freedom but still not unlimited.

I have found this method somewhere on cPanel forums (I can't seem to find the exact post right now, but I will update this post when I find it), anyways with little patching of profile, limits and bashrc you can create a list of users that have a slightly higher limits, so our django users won't run into any memory or fork limits normally enforced by fork bomb protection.

I've created a little tarball with installer scripts for patched fork bomb logic, I advise you to check the files prior to installing them. Script will make a copies of your original files, to the same destination with .orig suffixes.

cd /usr/src/
wget https://toic.org/files/django/forkbomb.tar.gz
tar zxvf forkbomb.tar.gz
cd forkbomb
./install.sh

Now all you have to do is put one username per line in /etc/profile.exclude

Each user in that line will have slightly bigger ulimits

Back to the Django

Let's also make admin media paths and media folder paths

System Message: WARNING/2 (<string>, line 323)

Literal block expected; none found.

mkdir /home/[username]/djangosites/[projectname]/media nano /usr/local/apache/conf/userdata/std/2/[username]/[domain]/django-media.conf

and add.

Alias /admin_media/ /home/[username]/virtualenv/lib/python2.7/site-packages/Django-1.2.4-py2.7.egg/django/contrib/admin/media/
<Directory /home/[username]/virtualenv/lib/python2.7/site-packages/Django-1.2.4-py2.7.egg/django/contrib/admin/media>
Order deny,allow
Allow from all
</Directory>
Alias /media/ /home/[username]/djangosites/[projectname]/media/
<Directory /home/[username]/djangosites/[projectname]/media>
Order deny,allow
Allow from all
</Directory>

Just replace your python version, django version and username and projectname in paths. This will create [domain]/media - for site media and [domain]/admin_media/ for admin media

Now verify those apache include files and rebuild apache conf:

/scripts/verify_vhost_includes
/scripts/rebuildhttpdconf

As always you can make subdomains for your media files, you will just need to update your [projectname]/settings.py

And that's it, hope you enjoy your django installation

Django on cpanel with python2.6, virtualenv and mod_wsgi

Sat, 14 Aug 2010 19:59:44 +0000

Here is the simple to follow step by step tutorial on howto run latest Django on cpanel powered servers with python 2.6, virtual env and mod_wsgi.
If you are running cPanel on your server it's most probably a rhel or centos distro, and those will have by default python2.4 installed. You can't just overwrite your python2.4 as many of the system utilities are depended on the python2.4 So what we will do is install an alternate python 2.6 and setup our server to run Django instances with python2.6

First of all let's make a folder where we will download all those source packages

Installing python and dependencies

mkdir -p /usr/src/python26 && cd /usr/src/python26

Next, we have to download and install sqlite3 dependency. It's not needed if your Django instances will not run sqlite databases, but on a safe side it's better to install it than not.

wget http://www.sqlite.org/sqlite-amalgamation-3.6.4.tar.gz
tar zxvf sqlite-amalgamation-3.6.4.tar.gz
cd sqlite-3.6.4
./configure
make
make install

If all went well it's time to install python 2.6 as alternate install.

cd /usr/src/python26
wget http://www.python.org/ftp/python/2.6.5/Python-2.6.5.tgz
tar zxvf Python-2.6.5.tgz
cd Python-2.6.5
./configure --prefix=/opt/python2.6 --with-threads --enable-shared
make
make install

This will install python 2.6 in /opt/python2.6 and will not interfere with system python install. Let's make a symbolic link to our newly installed python:

ln -s /opt/python2.6/bin/python /usr/bin/python2.6<

Also we must instruct our system where it should find the libs for new python.

echo '/opt/python2.6/lib' >> /etc/ld.so.conf.d/opt-python2.6.conf
ldconfig

You shoud check if your new python install is working as it should:

/usr/bin/python2.6
Python 2.6.5 (r265:79063, Jul 15 2010, 16:55:23)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Type CTRL+D to exit. Next thing to do is installation of python-setup tools:

cd /usr/src/python26
wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg
sh setuptools-0.6c11-py2.6.egg --prefix=/opt/python2.6

Make sure to define the prefix on where your new python 2.6 installation is residing. We should make a temporary alias for new python as it will be needed for installation of python mysql package:

alias python="/opt/python2.6/bin/python"

Let's check if everything is working ok:

python
Python 2.6.5 (r265:79063, Jul 15 2010, 16:55:23)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> CTRL + D

So next in line is mysql for python:

cd /usr/src/python26
wget http://netcologne.dl.sourceforge.net/project/mysql-python/mysql-python/1.2.3/MySQL-python-1.2.3.tar.gz
tar zxvf MySQL-python-1.2.3.tar.gz
cd MySQL-python-1.2.3
python setup.py build
python setup.py install

Again let's verify the install:

cd /usr/src/python26
python
Python 2.6.5 (r265:79063, Jul 15 2010, 17:57:29)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> import MySQLdb
>>> CTRL + D

It shouldn't return any errors while issuing import statements. Ok for the end let's install the virtualenv so we can create a virtual python enviroment for each user and his Django instances

cd /opt/python2.6/bin
./easy_install virtualenv

And here we conclude the python instalation. Next order of business is mod_wsgi

mod_wsgi installation

cd /opt/python2.6/lib/python2.6/config/
ln -s ../../libpython2.6.so .
cd /usr/src/python26
wget http://modwsgi.googlecode.com/files/mod_wsgi-3.2.tar.gz
tar zxvf mod_wsgi-3.2.tar.gz
cd mod_wsgi-3.2
./configure –with-python=/opt/python2.6/bin/python
make
make install

This will install mod_wsgi in /usr/local/apache/modules/mod_wsgi.so Note that this wsgi module is built with python 2.6 and will not work with any other version. All we have to do now is add the mod_wsgi in our apache conf. You can do that on cpanel servers via WHM. Log into WHM , click on Apache Configuration then Include Editor then Pre-Virtualhost Include and select All versions Paste this:

LoadModule wsgi_module /usr/local/apache/modules/mod_wsgi.so
AddHandler wsgi-script .wsgi

** Save and restart apache.** And that's it. Our apache is now ready to serve python scripts via mod_wsgi Now all we have to do is install some Django instances for our user.

Setting up Django projects

I'll use '[username] ' for cpanel username reference and '[domain] ' for that user's domain. To match your needs you will have to replace those where appropriate. First of all you will have to decide if you will do empty virtual environments so that the user can decide what python site-packages will he use or will you preinstall some packages so that user can only update his virtualenv with packages that are missing. If you are choosing the first method user will have to have compile rights under security setting of cpanel WHM otherwise he will not be able to install packages as python mysql, PIL and some others. If you choose the former method you can preinstall those packages systemwide for our alternate python install, but whenever you update those packages all virtual environments will be updated with them. It's a matter of personal preference. I for myself rather like the clean virtualenv, and before delivering it to the client you as root preinstall in his virtualenv those packages requiring gcc rights. So let's create a virtualenv for our user:

cd /home/[username]
/opt/python2.6/bin/virtualenv --no-site-packages --distribute virtualenv
chown -R [username].[username] virtualenv

this will create a new directory __ /home/[username]/virtualenv __ with our new virtual python environment for our user. You will have to enable normal shell for this user, let's first check what shell this user has.

cat /etc/passwd |grep [username]

it should return something like this:

[username]:x:928:923::/home/[username]:/usr/local/cpanel/bin/jailshell

This user has jailshell enabled to change that into normal shell do this:

usermod -s /bin/bash [username]

You can also do this via WHM-> Manage Shell Access. From this point on we will do everything as the user and not root, until told otherwise.

su [username]
source virtualenv/bin/activate

And the promt will change to something like this:

(virtualenv)[username]@[domain] [~]#

that way we know we are loged in as user and we are using his virtualenv. Now let's install Django in this users environment, simply enter:

easy_install django

Now is also a good time to install additional python site-packages like mysql and PIL. First temporarily enable compilers for this user via WHM->Compiler Access->Allow specific users to use the compilers do the modules install with:

easy_install [module_name]

disable the compiler access afterwards

cd
mkdir djangosites
cd djangosites
django-admin.py startproject [projectname]

So by now we should have a directory structure like this:

/home/[username]/virtualenv < - python virtual environment
/home/[username]/djangosites <- django sites folder
/home/[username]/djangosites/[projectname] <- django project

Create the wsgi script for that project, usualy [projectname].wsgi in django sites folder

nano /home/[username]/djangosites/[projectname].wsgi

and paste the following code:

import sys
import site
import os
vepath = '/home/[username]/virtualenv/lib/python2.6/site-packages'
prev_sys_path = list(sys.path)
site.addsitedir(vepath)
sys.path.append('/home/[username]/djangosites')
new_sys_path = [p for p in sys.path if p not in prev_sys_path]
for item in new_sys_path:
sys.path.remove(item)
sys.path[:0] = new_sys_path
from django.core.handlers.wsgi import WSGIHandler
os.environ['DJANGO_SETTINGS_MODULE'] = '[projectname].settings'
application = WSGIHandler()

give the scripts execute permissions:

chmod +x /home/[username]/djangosites/[projectname].wsgi

and exit user login with:

exit

now you will have your root propmt at the shell:

root@servername [/home/username]#

create the apache include folder for that virtualhost:

mkdir -p /usr/local/apache/conf/userdata/std/2/[username]/[domain]/
nano /usr/local/apache/conf/userdata/std/2/[username]/[domain]/django.conf

and add:

<ifmodule mod_wsgi.c>
WSGIScriptAlias / /home/[username]/djangosites/[projectname].wsgi
WSGIDaemonProcess [username] processes=7 threads=1 display-name=%{GROUP}
WSGIProcessGroup [username]
WSGIApplicationGroup %{GLOBAL}
</ifmodule>

edit the virtualhost entry for that domain:

nano /usr/local/apache/conf/httpd.conf

and in that virtualhost add:

Include "/usr/local/apache/conf/userdata/std/2/[username]/[domain]/*.conf"

save & exit

test the apache configuration prior to reload:

service httpd configtest

Should say syntax OK at the end, if so reload the apache:

/scripts/restartsrv_httpd

And voila, you have your django instance up& running You can now add via User's cpanel media subdomain for django media files, and define the path to the subdomain root in setings.py media root can live inside public_html folder. You can also create additional subdomain for admin media and copy the admin media content to that subdoman. For example: admin-media.[domain]:

cp -r /home/[username]/virtualenv/lib/python2.6/site-packages/Django-1.2.1-py2.6.egg/django/contrib/admin/media/* /home/[username]/public_html/admin-media/

Tail -f in python, truncate aware

Tue, 11 Aug 2009 19:55:19 +0000

While doing a little coding I tried to find some tail -f class in python that will recognize when file that we tailing is been truncated. All I found was some tail -f classes that brakes on file truncate or rotate.

Eventually I came up with this

import time
from os import stat
from os.path import abspath
from stat import ST_SIZE
class LogTail:
def __init__(self, logfile):
self.logfile = abspath(logfile)
self.f = open(self.logfile,"r")
file_len = stat(self.logfile)[ST_SIZE]
self.f.seek(file_len)
self.pos = self.f.tell()
def _reset(self):
self.f.close()
self.f = open(self.logfile, "r")
self.pos = self.f.tell()
def tail(self):
while 1:
self.pos = self.f.tell()
line = self.f.readline()
if not line:
if stat(self.logfile)[ST_SIZE] < self.pos:
self._reset()
else:
time.sleep(1)
self.f.seek(self.pos)
else:
"""print, return or otherwise manipulate
the tailed line"""
print line

Maybe it's a too much overhead to check each time for file size, but you get the general idea.

So anyways here's a usage example

tail = LogTail("/var/log/messages")
tail.tail()

This will print out any new line appended to /var/log/messages file. If the file gets truncated or log rotated, class will detect it and will return to the start.