toic.org - Entries for the category Codinghttps://toic.org/blog/categories/coding/My humble coding experiences en-usZinniaThu, 08 Oct 2015 20:57:30 +0000IPv4 country code IP ranges database https://toic.org/blog/2013/ipv4-country-code-ip-ranges-database/<p>For a while now, I was using some third party IP ranges file lists for some restrictions lists based on country codes.</p> <p>Long story short those third party sites either died or are not properly detecting IP ranges from certain country codes of my interest. I'm trying to put up my own list. In the process my IP ranges zone files will be available for public use on <a href="http://toic.org/network/cc_zones/">this site</a>. I'll try to announce any changes to the background logic.</p> <p>For now, the script in the background will be running every 6h rebuilding those files with new info. Current source is <a class="reference external" href="ftp://ftp.ripe.net/ripe/stats/RIR-Statistics-Exchange-Format.txt">RIR-Statistics-Exchange-Format</a> gathered from <a class="reference external" href="http://www.ripe.net/lir-services/member-support/info/list-of-members/list-of-country-codes-and-rirs">RIR members</a></p> branko@toic.org (branko)Sun, 23 Jun 2013 00:24:58 +0000https://toic.org/blog/2013/ipv4-country-code-ip-ranges-database/CodingApache monitoring tool ApTop beta released https://toic.org/blog/2012/apache-monitoring-tool-aptop-beta-released/<p>This is my first public publish to PyPi, without further delays, here is the full project description</p> <p><h1>ApTop</h1></p> <p>ApTop ia a top(1) alike monitoring utility for Apache.</p> <p>Being daily involved in monitoring system load in shared environment I find it lacking proper tool for monitoring Apache behavior in real time. Granted there are some tools for this job, they are usually of limited options for some monitoring needs.</p> <p>ApTop was written as a hobby project in python for debugging purposes while investigating several system load situations caused by web applications in shared hosting environment.</p> <p>ApTop is still an early beta release with some known bugs, nevertheless it still provides a great starting point in monitoring Apache. Every feedback and feature request at this point is more than welcomed.</p> <div class="section" id="description"> <h1>Description</h1> <p>ApTop will show near real time Apache active working proceses and virtual host connections. By default sleeping / waiting for connections slots will be filtered out.</p> <p>ApTop is also displaying several screens</p> <div class="section" id="dashboard-or-home-screen"> <h2>Dashboard or Home screen</h2> <p>This screen displays some relevant informations gathered from <a class="reference external" href="http://httpd.apache.org/docs/2.2/mod/mod_status.html">apache-status</a> page. It is by default sorted by shortest last apache access time &quot;SS&quot; and it displays only active connections. Displayed fields are the same as in server-status page, so you can find detailed explanation for their meaning at the bottom of server-status page.</p> <p>Displayed fields are:</p> <p>PID - OS process ID M - Mode of operation CPU - CPU usage, number of seconds SS - Seconds since beginning of most recent request&lt; Req - Milliseconds required to process most recent request Conn - Kilobytes transferred this connection Acc - Number of accesses this connection / this child / this slot Client - Remote client VHost - Local virtualhost Request - Request method and partial URL</p> </div> <div class="section" id="count-by-vhosts-screen"> <h2>Count by vhosts screen</h2> <p>This screen displays virtualhosts in order by highest count of active connections along with connection count.</p> <p>Very useful for detecting high traffic sites.</p> </div> <div class="section" id="count-by-clients-screen"> <h2>Count by clients screen</h2> <p>This screen displays clients in order by highest count of active connections along with connection count.</p> <p>Very useful for detecting resource exhausters.</p> </div> </div> <div class="section" id="requirements"> <h1>Requirements</h1> <p>ApTop should work just fine on any unix based system with python standard packages and lxml.</p> <p>Make sure you have libxml2 libxslt libxslt-devel libxml2-devel packages instaleld prior to building lxml:</p> <p>For RHEL based distro you can install those with</p> <pre class="literal-block"> yum install libxml2 libxslt libxslt-devel libxml2-devel </pre> <p>After that you can install lxml python package with <strong>easy_install lxml</strong> or <strong>pip install lxml</strong></p> <p>ApTop is using Apaches <strong>mod_status</strong> with <strong>ExtendedStatus On</strong> for gathering data. Please ensure that <strong>mod_status</strong> is enabled and proper status url is defined in <strong>aptop.conf</strong> There are some concernes regarding enabled ExtendedStatus On, but I think <a class="reference external" href="http://www.philchen.com/2008/06/02/apache-20-mod_status-effects-on-performance-server-resources">this links</a> covers this nicely.</p> </div> <div class="section" id="instalation"> <h1>Instalation</h1> <p>These installation options are for RHEL based distros, adjust accordingly.</p> <blockquote> yum install libxml2 libxslt libxslt-devel libxml2-devel python-setuptools</blockquote> <p>Now you can do</p> <blockquote> easy_install ApTop</blockquote> <p>or</p> <blockquote> easy_install <a class="reference external" href="https://bitbucket.org/btoic/aptop/get/master.tar.gz">https://bitbucket.org/btoic/aptop/get/master.tar.gz</a></blockquote> <p>You can now run aptop.py</p> </div> <div class="section" id="configuration"> <h1>Configuration</h1> <p>ApTop has some built in default configuration directives, but it will also look first in ~/.aptop.conf and than /etc/aptop.conf for overrides.</p> <p>ApTop uses .ini style configuration files and main cofniguration options should be placed after <strong>[aptop]</strong> section.</p> <p>By default ApTop will use <a class="reference external" href="http://localhost/server-status">http://localhost/server-status</a> url for grabbing data to display, and with default refresh rate of 5 seconds.</p> <div class="section" id="available-configuration-options"> <h2>Available configuration options:</h2> <blockquote> status_url = URL</blockquote> <p>This configuration option will require you to enter valid mod_status status url for server you wish to monitor. This defaults to <a class="reference external" href="http://localhost/server-status">http://localhost/server-status</a> Note that Apache should have ExtendedStatus conf option set to On</p> <blockquote> refresh = seconds</blockquote> <p>This option will control ApTop default refresh interval. It defaults to 5 seconds and can't be lower than 1 second.</p> <p>You can even monitor the remote systems if they granted you access rights to theirs server-status URL. In the future ApTop will provide options to save remote monitored servers to its configuration.</p> </div> </div> <div class="section" id="runtime-options"> <h1>Runtime options</h1> <p>While running ApTop responds to following options:</p> <p><strong>H</strong> - display main screen</p> <p><strong>V</strong> - display vhosts by conn. count</p> <p><strong>c</strong> - display clients by conn. count</p> <p><strong>R</strong> - reverse the current sort order</p> <p><strong>I</strong> - toggle active/inactive slots</p> </div> <div class="section" id="aditional-links"> <h1>Aditional links</h1> <p><a class="reference external" href="http://toic.org">http://toic.org</a></p> <p><a class="reference external" href="https://bitbucket.org/btoic/aptop">https://bitbucket.org/btoic/aptop</a></p> </div> branko@toic.org (branko)Wed, 12 Dec 2012 23:34:19 +0000https://toic.org/blog/2012/apache-monitoring-tool-aptop-beta-released/CodingMonitoringWsgi on cPanel improved https://toic.org/blog/2011/wsgi-on-cpanel-improved/<p>There is a long time now from my original post on running Django with python 2.6 on cPanel based servers, and as time passed there was some issues while deploying Django in such a way. So I decided to post another post with some updates regarding deployment of any python based scripts trough mod_wsgi and cPanel.</p> <p><strong>What's changed since last post?</strong></p> <ul> <li><p class="first">There is a fix for easy_apache rebuilds, retaining the mod_wsgi and not braking rebuild process</p> </li> <li><p class="first">There was some additions in virtualhost include files, enabling python app to run as a user, not nobody</p> </li> <li><p class="first">A fix for cpanel's shell fork bomb protection while running python aps as a user and not nobody</p> </li> <li><p class="first">Some minor changes regarding media handling, so it get's included in virtualhost includes and not by eating up user's subdomains</p> </li> <li><p class="first">And of course python and Django version updates</p> <blockquote> <p>Disclaimer at the beginning, this setup works for me on latest RELESE cPanel build, running on top of CentOS 5.5. I'm posting this as a way how I managed to get it working, so this shouldn't be considered as a official way of running Django or any other python app on cPanell servers. Although, as per my last post &lt;a href=&quot;/blog/2010/django-on-cpanel-with-python2-6-virtualenv-and-mod_wsgi/&quot;&gt;Django on cpanel with python2.6, virtualenv and mod_wsgi&lt;/a&gt; lot's of people are reporting this is working for them also, and this version of deployment guide will bring some more enhancements.</p> </blockquote> </li> </ul> <div class="section" id="installing-python-and-dependencies"> <h1>Installing python and dependencies</h1> <pre class="literal-block"> mkdir /usr/src/python2.7 &amp;&amp; cd /usr/src/python2.7 </pre> <p>again, if you need sqlite, now is the time to install it.</p> <pre class="code bash literal-block"> wget http://www.sqlite.org/sqlite-autoconf-3070500.tar.gz tar zxvf sqlite-autoconf-3070500.tar.gz <span class="name builtin">cd </span>sqlite-autoconf-3070500 ./configure make make install </pre> <p>Since CentOS 5.5 is still running python 2.4 and if you don't wish to brake your system you should install your python 2.7 in alternate location:</p> <pre class="code bash literal-block"> <span class="name builtin">cd</span> /usr/src/python2.7/ wget http://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz tar zxvf Python-2.7.1.tgz <span class="name builtin">cd </span>Python-2.7.1 ./configure --prefix<span class="operator">=</span>/opt/python2.7 --with-threads --enable-shared ./configure --help make make install </pre> <p>This will install python 2.7 in <strong>/opt/python2.7</strong> directory. To make any use of this alternate install we must do the following:</p> <pre class="code bash literal-block"> ln -s /opt/python2.7/bin/python /usr/bin/python2.7 <span class="name builtin">echo</span> <span class="literal string single">'/opt/python2.7/lib'</span>&gt;&gt; /etc/ld.so.conf.d/opt-python2.7.conf ldconfig </pre> <p>It will make a symbolic link in your path and instruct the system where to find libs for this alternate python install.</p> <p>Now is a good time to check if everything is working as it should</p> <pre class="code python literal-block"> <span class="name">root</span><span class="name decorator">&#64;toy2</span> <span class="punctuation">[</span><span class="operator">/</span><span class="name">usr</span><span class="operator">/</span><span class="name">src</span><span class="operator">/</span><span class="name">python2</span><span class="operator">.</span><span class="literal number integer">7</span><span class="punctuation">]</span><span class="comment"># /usr/bin/python2.7</span> <span class="name">Python</span> <span class="literal number float">2.7</span><span class="operator">.</span><span class="literal number integer">1</span> <span class="punctuation">(</span><span class="name">r271</span><span class="punctuation">:</span><span class="literal number integer">86832</span><span class="punctuation">,</span> <span class="name">Mar</span> <span class="literal number integer">26</span> <span class="literal number integer">2011</span><span class="punctuation">,</span> <span class="literal number integer">22</span><span class="punctuation">:</span><span class="literal number integer">31</span><span class="punctuation">:</span><span class="literal number integer">33</span><span class="punctuation">)</span> <span class="punctuation">[</span><span class="name">GCC</span> <span class="literal number float">4.1</span><span class="operator">.</span><span class="literal number integer">2</span> <span class="literal number integer">20080704</span> <span class="punctuation">(</span><span class="name">Red</span> <span class="name">Hat</span> <span class="literal number float">4.1</span><span class="operator">.</span><span class="literal number integer">2</span><span class="operator">-</span><span class="literal number integer">48</span><span class="punctuation">)]</span> <span class="name">on</span> <span class="name">linux2</span> <span class="name">Type</span> <span class="literal string">&quot;help&quot;</span><span class="punctuation">,</span> <span class="literal string">&quot;copyright&quot;</span><span class="punctuation">,</span> <span class="literal string">&quot;credits&quot;</span> <span class="operator word">or</span> <span class="literal string">&quot;license&quot;</span> <span class="keyword">for</span> <span class="name">more</span> <span class="name">information</span><span class="operator">.</span> <span class="operator">&gt;&gt;&gt;</span> <span class="keyword namespace">import</span> <span class="name namespace">sqlite3</span> <span class="operator">&gt;&gt;&gt;</span> </pre> <p><strong>Type CTRL+D to exit.</strong></p> <p>Next thing to do is installation of python-setup tools:</p> <pre class="code bash literal-block"> /usr/bin/python2.7 <span class="name builtin">cd</span> /usr/src/python2.7/ wget http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg sh setuptools-0.6c11-py2.7.egg --prefix<span class="operator">=</span>/opt/python2.7 </pre> <p>In this tutorial, I'm skipping the installation of mysql for python since you will have to install it per user basis in their virtual environments. All we need to do next is to install virtualenv to our python 2.7 so we can distribute them to our users</p> <pre class="literal-block"> cd /opt/python2.7/bin/ ./easy_install virtualenv </pre> </div> <div class="section" id="mod-wsgi-installation"> <h1>mod_wsgi installation</h1> <pre class="code bash literal-block"> <span class="name builtin">cd</span> /opt/python2.7/lib/python2.7/config/ ln -s ../../libpython2.7.so . <span class="name builtin">cd</span> /usr/src/python2.7/ wget http://modwsgi.googlecode.com/files/mod_wsgi-3.3.tar.gz tar zxvf mod_wsgi-3.3.tar.gz <span class="name builtin">cd </span>mod_wsgi-3.3 ./configure --with-python<span class="operator">=</span>/opt/python2.7/bin/python make make install </pre> <p>Now this will install mod_wsgi.so file in /usr/local/apache/modules folder, as I experienced during cPanel easy_apache rebuilds it will completely empty this folder, thus any virtualhost includes relying on mod_wsgi will fail and easy_apache will not be able to properly rebuild your httpd.conf file and whole apache or php upgrade you were running will fail and revert back to last good state.</p> <p>To avoid this we will now copy the mod_wsgi to different folder:</p> <pre class="literal-block"> mkdir /usr/local/apache/extramodules mv /usr/local/apache/modules/mod_wsgi.so /usr/local/apache/extramodules/ </pre> <p><strong>Note that this wsgi module is built with python 2.7 and will not work with any other version.</strong> If for some strange reason later on you try to run any given python application with system's default python2.4 it will not work.</p> <p>Anyways, all we need to do next is to include mod_wsgi into the apache configuration:</p> <pre class="literal-block"> nano /usr/local/apache/conf/includes/pre_virtualhost_global.conf </pre> <p>and paste:</p> <pre class="literal-block"> LoadModule wsgi_module /usr/local/apache/extramodules/mod_wsgi.so AddHandler wsgi-script .wsgi </pre> <p>Now do configtest:</p> <pre class="literal-block"> root&#64;toy2 [~]# service httpd configtest Syntax OK </pre> <p>If you get <strong>Syntax OK</strong> at the end, you can safely restart your apache:</p> <pre class="literal-block"> /scripts/restartsrv httpd </pre> <p>This is it, our apache can now serve python apps like Django, pylons, etc...</p> </div> <div class="section" id="setting-up-django-projects"> <h1>Setting up Django projects</h1> <p>Just like in my <a class="reference external" href="/blog/2010/django-on-cpanel-with-python2-6-virtualenv-and-mod_wsgi/">previous post</a> I'll use '<strong>[username]</strong>' for cpanel username reference and '<strong>[domain]</strong>' for that user's domain. To match your needs you will have to replace those where appropriate.</p> <p>We will quickly setup user's virtualenv, be aware though some of the python packages requires compile rights, and most of the cPanel setups I've seen so far are forbidding compiler access to their users. You can either make an permanent exception for this user, or you can instruct your user to contact you whenever he needs to install some package that requires compile rights (PIL, mysql...).</p> <p>To enable compile rights for that specific user you can find that user in WHM <strong>WHM-&gt;Compiler Access-&gt;Allow specific users to use the compilers</strong> or if you don't like to exit the shell you can do it like this:</p> <pre class="literal-block"> gpasswd -a [username] compiler </pre> <p>to remove it, use the same path in WHM and in shell:</p> <pre class="literal-block"> gpasswd -d [username] compiler </pre> <p>For now, lets add this user to the compiler group since we will be installing some python packages for him in the start. Also you will have to enable your user a normal login shell, let's first check what shell this user has.</p> <pre class="literal-block"> cat /etc/passwd |grep [username] </pre> <p>it should return something like this:</p> <pre class="literal-block"> [username]:x:928:923::/home/[username]:/usr/local/cpanel/bin/jailshell </pre> <p>This user has jailshell enabled to change that into normal shell do this:</p> <pre class="literal-block"> usermod -s /bin/bash [username] </pre> <p>If it's already bash, don't change it. You can also do this via <strong>WHM-&gt; Manage Shell Access.</strong></p> <p>So let's finaly create a virtualenv for our user:</p> <pre class="code bash literal-block"> <span class="name builtin">cd</span> /home/<span class="operator">[</span>username<span class="operator">]</span> /opt/python2.7/bin/virtualenv --no-site-packages --distribute virtualenv chown -R <span class="operator">[</span>username<span class="operator">]</span>.<span class="operator">[</span>username<span class="operator">]</span> virtualenv </pre> <p>this will create a new directory <strong>/home/[username]/virtualenv</strong> with our new virtual python environment for our user. From this point on we will do everything as the user and not root, until told otherwise.</p> <pre class="literal-block"> su [username] source virtualenv/bin/activate </pre> <p>And the promt will change to something like this:</p> <pre class="literal-block"> (virtualenv)[username]&#64;[domain] [~]# </pre> <p>that way we know we are loged in as user and we are using his virtualenv. Now let's install Django in this users environment, simply enter:</p> <pre class="literal-block"> easy_install django </pre> <p>Now is also a good time to install additional python site-packages like mysql and PIL.</p> <pre class="literal-block"> easy_install pil easy_install mysql-python </pre> <p>So we have the django installed inside the users virtualenv, all we have to do now is start one of the instance. It's probably the best practice to keep django sites outside of public_html folders on cpanel servers. So we will do (<strong>still as a user</strong>)</p> <pre class="code bash literal-block"> <span class="name builtin">cd </span>mkdir djangosites <span class="name builtin">cd </span>djangosites django-admin.py startproject <span class="operator">[</span>projectname<span class="operator">]</span> </pre> <p>So by now we should have a directory structure like this:</p> <pre class="literal-block"> /home/[username]/virtualenv &lt; - python virtual environment /home/[username]/djangosites &lt;- django sites folder /home/[username]/djangosites/[projectname] &lt;- django project </pre> <p>Create the wsgi script for that project, usualy <strong>[projectname].wsgi</strong> in django sites folder</p> <pre class="literal-block"> nano /home/[username]/djangosites/[projectname].wsgi </pre> <p>and paste the following code:</p> <pre class="code python literal-block"> <span class="keyword namespace">import</span> <span class="name namespace">sys</span> <span class="keyword namespace">import</span> <span class="name namespace">site</span> <span class="keyword namespace">import</span> <span class="name namespace">os</span> <span class="name">vepath</span> <span class="operator">=</span> <span class="literal string">'/home/[username]/virtualenv/lib/python2.7/site-packages'</span> <span class="name">prev_sys_path</span> <span class="operator">=</span> <span class="name builtin">list</span><span class="punctuation">(</span><span class="name">sys</span><span class="operator">.</span><span class="name">path</span><span class="punctuation">)</span> <span class="name">site</span><span class="operator">.</span><span class="name">addsitedir</span><span class="punctuation">(</span><span class="name">vepath</span><span class="punctuation">)</span> <span class="name">sys</span><span class="operator">.</span><span class="name">path</span><span class="operator">.</span><span class="name">append</span><span class="punctuation">(</span><span class="literal string">'/home/[username]/djangosites'</span><span class="punctuation">)</span> <span class="name">new_sys_path</span> <span class="operator">=</span> <span class="punctuation">[</span><span class="name">p</span> <span class="keyword">for</span> <span class="name">p</span> <span class="operator word">in</span> <span class="name">sys</span><span class="operator">.</span><span class="name">path</span> <span class="keyword">if</span> <span class="name">p</span> <span class="operator word">not</span> <span class="operator word">in</span> <span class="name">prev_sys_path</span><span class="punctuation">]</span> <span class="keyword">for</span> <span class="name">item</span> <span class="operator word">in</span> <span class="name">new_sys_path</span><span class="punctuation">:</span> <span class="name">sys</span><span class="operator">.</span><span class="name">path</span><span class="operator">.</span><span class="name">remove</span><span class="punctuation">(</span><span class="name">item</span><span class="punctuation">)</span> <span class="name">sys</span><span class="operator">.</span><span class="name">path</span><span class="punctuation">[:</span><span class="literal number integer">0</span><span class="punctuation">]</span> <span class="operator">=</span> <span class="name">new_sys_path</span> <span class="keyword namespace">from</span> <span class="name namespace">django.core.handlers.wsgi</span> <span class="keyword namespace">import</span> <span class="name">WSGIHandler</span> <span class="name">os</span><span class="operator">.</span><span class="name">environ</span><span class="punctuation">[</span><span class="literal string">'DJANGO_SETTINGS_MODULE'</span><span class="punctuation">]</span> <span class="operator">=</span> <span class="literal string">'[projectname].settings'</span> <span class="name">application</span> <span class="operator">=</span> <span class="name">WSGIHandler</span><span class="punctuation">()</span> </pre> <p>give the scripts execute permissions:</p> <pre class="literal-block"> chmod +x /home/[username]/djangosites/[projectname].wsgi </pre> <p>and exit user login with:</p> <pre class="literal-block"> exit </pre> <p>now you will have your <strong>root propmt</strong> at the shell:</p> <pre class="literal-block"> root&#64;servername [/home/username]# </pre> <p><strong>If you have decided so, now is the good time to disable the compiler access for this user.</strong></p> <p>create the apache include folder for that virtualhost:</p> <pre class="literal-block"> mkdir -p /usr/local/apache/conf/userdata/std/2/[username]/[domain]/ nano /usr/local/apache/conf/userdata/std/2/[username]/[domain]/django.conf </pre> <p>and add:</p> <pre class="code apacheconf literal-block"> <span class="name tag">&lt;ifmodule</span> <span class="literal string">mod_wsgi.c</span><span class="name tag">&gt;</span> <span class="name builtin">WSGIScriptAlias</span> / <span class="literal string other">/home/</span>[username]/djangosites/[projectname].wsgi <span class="name builtin">WSGIDaemonProcess</span> [projectname] <span class="keyword">user</span>=[username] <span class="keyword">group</span>=[username] processes=5 threads=15 display-name=%{GROUP} <span class="name builtin">WSGIProcessGroup</span> [projectname] <span class="name builtin">WSGIApplicationGroup</span> %{GLOBAL} <span class="name tag">&lt;/ifmodule&gt;</span> </pre> <p>Ok, so this will create a processgroup for this specific django projects with 5 processes each 15 threads with our user's uid and gid. This is great since we have isolated this user to his own application pool and his own home path, also we are enforcing user quotas since if we omit &quot;user=[username] group=[username]&quot; from WSGIDaemonProcess the deamon process will run ass apache user nobody, and all files made/uploaded via this application will not count into this user's quota. They will also make lot's of problems if the user don't grant global write permissions on some folders. Just like mod_php and php_suexec.</p> </div> <div class="section" id="shell-fork-bomb-exception"> <h1>Shell fork bomb exception</h1> <p>On the other hand, running 5 processes each with 15 threads (you can customize this ofcourse), will hit the defaults ulimits enforced by cPanels shell fork bomb protection (if enabled). You could either disable the protection, or you can make a little adjustments to its logic so that you can make exceptions per user allowing them a bit more freedom but still not unlimited.</p> <p>I have found this method somewhere on cPanel forums (I can't seem to find the exact post right now, but I will update this post when I find it), anyways with little patching of profile, limits and bashrc you can create a list of users that have a slightly higher limits, so our django users won't run into any memory or fork limits normally enforced by <strong>fork bomb protection</strong>.</p> <p>I've created a little tarball with installer scripts for patched fork bomb logic, I advise you to check the files prior to installing them. Script will make a copies of your original files, to the same destination with .orig suffixes.</p> <pre class="code bash literal-block"> <span class="name builtin">cd</span> /usr/src/ wget https://toic.org/files/django/forkbomb.tar.gz tar zxvf forkbomb.tar.gz <span class="name builtin">cd </span>forkbomb ./install.sh </pre> <p>Now all you have to do is put one username per line in <strong>/etc/profile.exclude</strong></p> <p>Each user in that line will have slightly bigger ulimits</p> </div> <div class="section" id="back-to-the-django"> <h1>Back to the Django</h1> <p>Let's also make admin media paths and media folder paths</p> <div class="system-message"> <p class="system-message-title">System Message: WARNING/2 (<tt class="docutils">&lt;string&gt;</tt>, line 323)</p> Literal block expected; none found.</div> <p>mkdir /home/[username]/djangosites/[projectname]/media nano /usr/local/apache/conf/userdata/std/2/[username]/[domain]/django-media.conf</p> <p>and add.</p> <pre class="code apacheconf literal-block"> <span class="name builtin">Alias</span> <span class="literal string other">/admin_media/</span> <span class="literal string other">/home/</span>[username]/virtualenv/lib/python2.7/site-packages/Django-1.2.4-py2.7.egg/django/contrib/admin/media/ <span class="name tag">&lt;Directory</span> <span class="literal string">/home/[username]/virtualenv/lib/python2.7/site-packages/Django-1.2.4-py2.7.egg/django/contrib/admin/media</span><span class="name tag">&gt;</span> <span class="name builtin">Order</span> deny,allow <span class="name builtin">Allow</span> from <span class="keyword">all</span> <span class="name tag">&lt;/Directory&gt;</span> <span class="name builtin">Alias</span> <span class="literal string other">/media/</span> <span class="literal string other">/home/</span>[username]/djangosites/[projectname]/media/ <span class="name tag">&lt;Directory</span> <span class="literal string">/home/[username]/djangosites/[projectname]/media</span><span class="name tag">&gt;</span> <span class="name builtin">Order</span> deny,allow <span class="name builtin">Allow</span> from <span class="keyword">all</span> <span class="name tag">&lt;/Directory&gt;</span> </pre> <p>Just replace your python version, django version and username and projectname in paths. This will create [domain]/media - for site media and [domain]/admin_media/ for admin media</p> <p>Now verify those apache include files and rebuild apache conf:</p> <pre class="literal-block"> /scripts/verify_vhost_includes /scripts/rebuildhttpdconf </pre> <p>As always you can make subdomains for your media files, you will just need to update your [projectname]/settings.py</p> <p>And that's it, hope you enjoy your django installation</p> </div> branko@toic.org (branko)Sun, 27 Mar 2011 20:01:02 +0000https://toic.org/blog/2011/wsgi-on-cpanel-improved/CodingLinuxTutorialsDjango on cpanel with python2.6, virtualenv and mod_wsgi https://toic.org/blog/2010/django-on-cpanel-with-python2-6-virtualenv-and-mod_wsgi/<p><img align='left' src="/media/filer_public/47/a4/47a4abec-d7e5-4beb-b61e-cb2688d0dc02/umltopython.png" alt="ssh" style="padding-top: 20px; margin-right:20px; margin-bottom: 20px;" /><br />Here is the simple to follow step by step tutorial on howto run latest Django on cpanel powered servers with python 2.6, virtual env and mod_wsgi.<br />If you are running cPanel on your server it's most probably a rhel or centos distro, and those will have by default python2.4 installed. You can't just overwrite your python2.4 as many of the system utilities are depended on the python2.4 So what we will do is install an alternate python 2.6 and setup our server to run Django instances with python2.6</p> <p>First of all let's make a folder where we will download all those source packages</p> <div class="section" id="installing-python-and-dependencies"> <h1>Installing python and dependencies</h1> <pre class="literal-block"> mkdir -p /usr/src/python26 &amp;&amp; cd /usr/src/python26 </pre> <p>Next, we have to download and install sqlite3 dependency. It's not needed if your Django instances will not run sqlite databases, but on a safe side it's better to install it than not.</p> <pre class="code bash literal-block"> wget http://www.sqlite.org/sqlite-amalgamation-3.6.4.tar.gz tar zxvf sqlite-amalgamation-3.6.4.tar.gz <span class="name builtin">cd </span>sqlite-3.6.4 ./configure make make install </pre> <p>If all went well it's time to install python 2.6 as alternate install.</p> <pre class="code bash literal-block"> <span class="name builtin">cd</span> /usr/src/python26 wget http://www.python.org/ftp/python/2.6.5/Python-2.6.5.tgz tar zxvf Python-2.6.5.tgz <span class="name builtin">cd </span>Python-2.6.5 ./configure --prefix<span class="operator">=</span>/opt/python2.6 --with-threads --enable-shared make make install </pre> <p>This will install python 2.6 in <strong>/opt/python2.6</strong> and will not interfere with system python install. Let's make a symbolic link to our newly installed python:</p> <pre class="literal-block"> ln -s /opt/python2.6/bin/python /usr/bin/python2.6&lt; </pre> <p>Also we must instruct our system where it should find the libs for new python.</p> <pre class="code bash literal-block"> <span class="name builtin">echo</span> <span class="literal string single">'/opt/python2.6/lib'</span> &gt;&gt; /etc/ld.so.conf.d/opt-python2.6.conf ldconfig </pre> <p>You shoud check if your new python install is working as it should:</p> <pre class="code python literal-block"> <span class="operator">/</span><span class="name">usr</span><span class="operator">/</span><span class="name builtin">bin</span><span class="operator">/</span><span class="name">python2</span><span class="operator">.</span><span class="literal number integer">6</span> <span class="name">Python</span> <span class="literal number float">2.6</span><span class="operator">.</span><span class="literal number integer">5</span> <span class="punctuation">(</span><span class="name">r265</span><span class="punctuation">:</span><span class="literal number integer">79063</span><span class="punctuation">,</span> <span class="name">Jul</span> <span class="literal number integer">15</span> <span class="literal number integer">2010</span><span class="punctuation">,</span> <span class="literal number integer">16</span><span class="punctuation">:</span><span class="literal number integer">55</span><span class="punctuation">:</span><span class="literal number integer">23</span><span class="punctuation">)</span> <span class="punctuation">[</span><span class="name">GCC</span> <span class="literal number float">4.1</span><span class="operator">.</span><span class="literal number integer">2</span> <span class="literal number integer">20080704</span> <span class="punctuation">(</span><span class="name">Red</span> <span class="name">Hat</span> <span class="literal number float">4.1</span><span class="operator">.</span><span class="literal number integer">2</span><span class="operator">-</span><span class="literal number integer">48</span><span class="punctuation">)]</span> <span class="name">on</span> <span class="name">linux2</span> <span class="name">Type</span> <span class="literal string">&quot;help&quot;</span><span class="punctuation">,</span> <span class="literal string">&quot;copyright&quot;</span><span class="punctuation">,</span> <span class="literal string">&quot;credits&quot;</span> <span class="operator word">or</span> <span class="literal string">&quot;license&quot;</span> <span class="keyword">for</span> <span class="name">more</span> <span class="name">information</span><span class="operator">.</span> <span class="operator">&gt;&gt;&gt;</span> </pre> <p>Type <strong>CTRL+D</strong> to exit. Next thing to do is installation of python-setup tools:</p> <pre class="code bash literal-block"> <span class="name builtin">cd</span> /usr/src/python26 wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg sh setuptools-0.6c11-py2.6.egg --prefix<span class="operator">=</span>/opt/python2.6 </pre> <p>Make sure to define the prefix on where your new python 2.6 installation is residing. We should make a temporary alias for new python as it will be needed for installation of python mysql package:</p> <pre class="literal-block"> alias python=&quot;/opt/python2.6/bin/python&quot; </pre> <p>Let's check if everything is working ok:</p> <pre class="code python literal-block"> <span class="name">python</span> <span class="name">Python</span> <span class="literal number float">2.6</span><span class="operator">.</span><span class="literal number integer">5</span> <span class="punctuation">(</span><span class="name">r265</span><span class="punctuation">:</span><span class="literal number integer">79063</span><span class="punctuation">,</span> <span class="name">Jul</span> <span class="literal number integer">15</span> <span class="literal number integer">2010</span><span class="punctuation">,</span> <span class="literal number integer">16</span><span class="punctuation">:</span><span class="literal number integer">55</span><span class="punctuation">:</span><span class="literal number integer">23</span><span class="punctuation">)</span> <span class="punctuation">[</span><span class="name">GCC</span> <span class="literal number float">4.1</span><span class="operator">.</span><span class="literal number integer">2</span> <span class="literal number integer">20080704</span> <span class="punctuation">(</span><span class="name">Red</span> <span class="name">Hat</span> <span class="literal number float">4.1</span><span class="operator">.</span><span class="literal number integer">2</span><span class="operator">-</span><span class="literal number integer">48</span><span class="punctuation">)]</span> <span class="name">on</span> <span class="name">linux2</span> <span class="name">Type</span> <span class="literal string">&quot;help&quot;</span><span class="punctuation">,</span> <span class="literal string">&quot;copyright&quot;</span><span class="punctuation">,</span> <span class="literal string">&quot;credits&quot;</span> <span class="operator word">or</span> <span class="literal string">&quot;license&quot;</span> <span class="keyword">for</span> <span class="name">more</span> <span class="name">information</span><span class="operator">.</span> <span class="operator">&gt;&gt;&gt;</span> <span class="name">CTRL</span> <span class="operator">+</span> <span class="name">D</span> </pre> <p>So next in line is mysql for python:</p> <pre class="code bash literal-block"> <span class="name builtin">cd</span> /usr/src/python26 wget http://netcologne.dl.sourceforge.net/project/mysql-python/mysql-python/1.2.3/MySQL-python-1.2.3.tar.gz tar zxvf MySQL-python-1.2.3.tar.gz <span class="name builtin">cd </span>MySQL-python-1.2.3 python setup.py build python setup.py install </pre> <p>Again let's verify the install:</p> <pre class="code python literal-block"> <span class="name">cd</span> <span class="operator">/</span><span class="name">usr</span><span class="operator">/</span><span class="name">src</span><span class="operator">/</span><span class="name">python26</span> <span class="name">python</span> <span class="name">Python</span> <span class="literal number float">2.6</span><span class="operator">.</span><span class="literal number integer">5</span> <span class="punctuation">(</span><span class="name">r265</span><span class="punctuation">:</span><span class="literal number integer">79063</span><span class="punctuation">,</span> <span class="name">Jul</span> <span class="literal number integer">15</span> <span class="literal number integer">2010</span><span class="punctuation">,</span> <span class="literal number integer">17</span><span class="punctuation">:</span><span class="literal number integer">57</span><span class="punctuation">:</span><span class="literal number integer">29</span><span class="punctuation">)</span> <span class="punctuation">[</span><span class="name">GCC</span> <span class="literal number float">4.1</span><span class="operator">.</span><span class="literal number integer">2</span> <span class="literal number integer">20080704</span> <span class="punctuation">(</span><span class="name">Red</span> <span class="name">Hat</span> <span class="literal number float">4.1</span><span class="operator">.</span><span class="literal number integer">2</span><span class="operator">-</span><span class="literal number integer">48</span><span class="punctuation">)]</span> <span class="name">on</span> <span class="name">linux2</span> <span class="name">Type</span> <span class="literal string">&quot;help&quot;</span><span class="punctuation">,</span> <span class="literal string">&quot;copyright&quot;</span><span class="punctuation">,</span> <span class="literal string">&quot;credits&quot;</span> <span class="operator word">or</span> <span class="literal string">&quot;license&quot;</span> <span class="keyword">for</span> <span class="name">more</span> <span class="name">information</span><span class="operator">.</span> <span class="operator">&gt;&gt;&gt;</span> <span class="keyword namespace">import</span> <span class="name namespace">sqlite3</span> <span class="operator">&gt;&gt;&gt;</span> <span class="keyword namespace">import</span> <span class="name namespace">MySQLdb</span> <span class="operator">&gt;&gt;&gt;</span> <span class="name">CTRL</span> <span class="operator">+</span> <span class="name">D</span> </pre> <p>It shouldn't return any errors while issuing import statements. Ok for the end let's install the virtualenv so we can create a virtual python enviroment for each user and his Django instances</p> <pre class="code bash literal-block"> <span class="name builtin">cd</span> /opt/python2.6/bin ./easy_install virtualenv </pre> <p>And here we conclude the python instalation. Next order of business is mod_wsgi</p> </div> <div class="section" id="mod-wsgi-installation"> <h1>mod_wsgi installation</h1> <pre class="code bash literal-block"> <span class="name builtin">cd</span> /opt/python2.6/lib/python2.6/config/ ln -s ../../libpython2.6.so . <span class="name builtin">cd</span> /usr/src/python26 wget http://modwsgi.googlecode.com/files/mod_wsgi-3.2.tar.gz tar zxvf mod_wsgi-3.2.tar.gz <span class="name builtin">cd </span>mod_wsgi-3.2 ./configure –with-python<span class="operator">=</span>/opt/python2.6/bin/python make make install </pre> <p>This will install mod_wsgi in <strong>/usr/local/apache/modules/mod_wsgi.so</strong> Note that this wsgi module is built with python 2.6 and will not work with any other version. All we have to do now is add the mod_wsgi in our apache conf. You can do that on cpanel servers via WHM. Log into <strong>WHM</strong> , click on <strong>Apache Configuration</strong> then <strong>Include Editor</strong> then <strong>Pre-Virtualhost</strong> Include and select <strong>All versions</strong> Paste this:</p> <pre class="literal-block"> LoadModule wsgi_module /usr/local/apache/modules/mod_wsgi.so AddHandler wsgi-script .wsgi </pre> <p>** Save and restart apache.** And that's it. Our apache is now ready to serve python scripts via mod_wsgi Now all we have to do is install some Django instances for our user.</p> </div> <div class="section" id="setting-up-django-projects"> <h1>Setting up Django projects</h1> <p>I'll use '<strong>[username]</strong> ' for cpanel username reference and '<strong>[domain]</strong> ' for that user's domain. To match your needs you will have to replace those where appropriate. First of all you will have to decide if you will do empty virtual environments so that the user can decide what python site-packages will he use or will you preinstall some packages so that user can only update his virtualenv with packages that are missing. If you are choosing the first method user will have to have compile rights under security setting of cpanel WHM otherwise he will not be able to install packages as python mysql, PIL and some others. If you choose the former method you can preinstall those packages systemwide for our alternate python install, but whenever you update those packages all virtual environments will be updated with them. It's a matter of personal preference. I for myself rather like the clean virtualenv, and before delivering it to the client you as root preinstall in his virtualenv those packages requiring gcc rights. So let's create a virtualenv for our user:</p> <pre class="code bash literal-block"> <span class="name builtin">cd</span> /home/<span class="operator">[</span>username<span class="operator">]</span> /opt/python2.6/bin/virtualenv --no-site-packages --distribute virtualenv chown -R <span class="operator">[</span>username<span class="operator">]</span>.<span class="operator">[</span>username<span class="operator">]</span> virtualenv </pre> <p>this will create a new directory __ /home/[username]/virtualenv __ with our new virtual python environment for our user. You will have to enable normal shell for this user, let's first check what shell this user has.</p> <pre class="literal-block"> cat /etc/passwd |grep [username] </pre> <p>it should return something like this:</p> <pre class="literal-block"> [username]:x:928:923::/home/[username]:/usr/local/cpanel/bin/jailshell </pre> <p>This user has jailshell enabled to change that into normal shell do this:</p> <pre class="literal-block"> usermod -s /bin/bash [username] </pre> <p>You can also do this via <strong>WHM-&gt; Manage Shell Access.</strong> From this point on we will do everything as the user and not root, until told otherwise.</p> <pre class="literal-block"> su [username] source virtualenv/bin/activate </pre> <p>And the promt will change to something like this:</p> <pre class="literal-block"> (virtualenv)[username]&#64;[domain] [~]# </pre> <p>that way we know we are loged in as user and we are using his virtualenv. Now let's install Django in this users environment, simply enter:</p> <pre class="literal-block"> easy_install django </pre> <p>Now is also a good time to install additional python site-packages like mysql and PIL. First temporarily enable compilers for this user via <strong>WHM-&gt;Compiler Access-&gt;Allow specific users to use the compilers</strong> do the modules install with:</p> <pre class="literal-block"> easy_install [module_name] </pre> <p><strong>disable the compiler access afterwards</strong></p> <p>So we have the django installed inside the users virtualenv, all we have to do now is start one of the instance. It's probably the best practice to keep django sites outside of public_html folders on cpanel servers. So we will do (<strong>still as a user</strong>)</p> <pre class="literal-block"> cd mkdir djangosites cd djangosites django-admin.py startproject [projectname] </pre> <p>So by now we should have a directory structure like this:</p> <pre class="literal-block"> /home/[username]/virtualenv &lt; - python virtual environment /home/[username]/djangosites &lt;- django sites folder /home/[username]/djangosites/[projectname] &lt;- django project </pre> <p>Create the wsgi script for that project, usualy <strong>[projectname].wsgi</strong> in django sites folder</p> <pre class="literal-block"> nano /home/[username]/djangosites/[projectname].wsgi </pre> <p>and paste the following code:</p> <pre class="code python literal-block"> <span class="keyword namespace">import</span> <span class="name namespace">sys</span> <span class="keyword namespace">import</span> <span class="name namespace">site</span> <span class="keyword namespace">import</span> <span class="name namespace">os</span> <span class="name">vepath</span> <span class="operator">=</span> <span class="literal string">'/home/[username]/virtualenv/lib/python2.6/site-packages'</span> <span class="name">prev_sys_path</span> <span class="operator">=</span> <span class="name builtin">list</span><span class="punctuation">(</span><span class="name">sys</span><span class="operator">.</span><span class="name">path</span><span class="punctuation">)</span> <span class="name">site</span><span class="operator">.</span><span class="name">addsitedir</span><span class="punctuation">(</span><span class="name">vepath</span><span class="punctuation">)</span> <span class="name">sys</span><span class="operator">.</span><span class="name">path</span><span class="operator">.</span><span class="name">append</span><span class="punctuation">(</span><span class="literal string">'/home/[username]/djangosites'</span><span class="punctuation">)</span> <span class="name">new_sys_path</span> <span class="operator">=</span> <span class="punctuation">[</span><span class="name">p</span> <span class="keyword">for</span> <span class="name">p</span> <span class="operator word">in</span> <span class="name">sys</span><span class="operator">.</span><span class="name">path</span> <span class="keyword">if</span> <span class="name">p</span> <span class="operator word">not</span> <span class="operator word">in</span> <span class="name">prev_sys_path</span><span class="punctuation">]</span> <span class="keyword">for</span> <span class="name">item</span> <span class="operator word">in</span> <span class="name">new_sys_path</span><span class="punctuation">:</span> <span class="name">sys</span><span class="operator">.</span><span class="name">path</span><span class="operator">.</span><span class="name">remove</span><span class="punctuation">(</span><span class="name">item</span><span class="punctuation">)</span> <span class="name">sys</span><span class="operator">.</span><span class="name">path</span><span class="punctuation">[:</span><span class="literal number integer">0</span><span class="punctuation">]</span> <span class="operator">=</span> <span class="name">new_sys_path</span> <span class="keyword namespace">from</span> <span class="name namespace">django.core.handlers.wsgi</span> <span class="keyword namespace">import</span> <span class="name">WSGIHandler</span> <span class="name">os</span><span class="operator">.</span><span class="name">environ</span><span class="punctuation">[</span><span class="literal string">'DJANGO_SETTINGS_MODULE'</span><span class="punctuation">]</span> <span class="operator">=</span> <span class="literal string">'[projectname].settings'</span> <span class="name">application</span> <span class="operator">=</span> <span class="name">WSGIHandler</span><span class="punctuation">()</span> </pre> <p>give the scripts execute permissions:</p> <pre class="literal-block"> chmod +x /home/[username]/djangosites/[projectname].wsgi </pre> <p>and exit user login with:</p> <pre class="literal-block"> exit </pre> <p>now you will have your <strong>root propmt</strong> at the shell:</p> <pre class="literal-block"> root&#64;servername [/home/username]# </pre> <p>create the apache include folder for that virtualhost:</p> <pre class="literal-block"> mkdir -p /usr/local/apache/conf/userdata/std/2/[username]/[domain]/ nano /usr/local/apache/conf/userdata/std/2/[username]/[domain]/django.conf </pre> <p>and add:</p> <pre class="code apacheconf literal-block"> <span class="name tag">&lt;ifmodule</span> <span class="literal string">mod_wsgi.c</span><span class="name tag">&gt;</span> <span class="name builtin">WSGIScriptAlias</span> / <span class="literal string other">/home/</span>[username]/djangosites/[projectname].wsgi <span class="name builtin">WSGIDaemonProcess</span> [username] processes=7 threads=1 display-name=%{GROUP} <span class="name builtin">WSGIProcessGroup</span> [username] <span class="name builtin">WSGIApplicationGroup</span> %{GLOBAL} <span class="name tag">&lt;/ifmodule&gt;</span> </pre> <p>edit the virtualhost entry for that domain:</p> <pre class="literal-block"> nano /usr/local/apache/conf/httpd.conf </pre> <p>and in that virtualhost add:</p> <pre class="literal-block"> Include &quot;/usr/local/apache/conf/userdata/std/2/[username]/[domain]/*.conf&quot; </pre> <p>save &amp; exit</p> <p><strong>test the apache configuration prior to reload:</strong></p> <pre class="literal-block"> service httpd configtest </pre> <p>Should say syntax OK at the end, if so reload the apache:</p> <pre class="literal-block"> /scripts/restartsrv_httpd </pre> <p>And voila, you have your django instance up&amp; running You can now add via User's cpanel media subdomain for django media files, and define the path to the subdomain root in setings.py media root can live inside public_html folder. You can also create additional subdomain for admin media and copy the admin media content to that subdoman. For example: admin-media.[domain]:</p> <pre class="literal-block"> cp -r /home/[username]/virtualenv/lib/python2.6/site-packages/Django-1.2.1-py2.6.egg/django/contrib/admin/media/* /home/[username]/public_html/admin-media/ </pre> </div> branko@toic.org (branko)Sat, 14 Aug 2010 19:59:44 +0000https://toic.org/blog/2010/django-on-cpanel-with-python2-6-virtualenv-and-mod_wsgi/CodingLinuxTutorialsTail -f in python, truncate aware https://toic.org/blog/2009/tail-f-in-python-truncate-aware/<p>While doing a little coding I tried to find some tail -f class in python that will recognize when file that we tailing is been truncated. All I found was some tail -f classes that brakes on file truncate or rotate. </p> <p></p> <p>Eventually I came up with this</p> <pre class="code python literal-block"> <span class="keyword namespace">import</span> <span class="name namespace">time</span> <span class="keyword namespace">from</span> <span class="name namespace">os</span> <span class="keyword namespace">import</span> <span class="name">stat</span> <span class="keyword namespace">from</span> <span class="name namespace">os.path</span> <span class="keyword namespace">import</span> <span class="name">abspath</span> <span class="keyword namespace">from</span> <span class="name namespace">stat</span> <span class="keyword namespace">import</span> <span class="name">ST_SIZE</span> <span class="keyword">class</span> <span class="name class">LogTail</span><span class="punctuation">:</span> <span class="keyword">def</span> <span class="name function">__init__</span><span class="punctuation">(</span><span class="name builtin pseudo">self</span><span class="punctuation">,</span> <span class="name">logfile</span><span class="punctuation">):</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">logfile</span> <span class="operator">=</span> <span class="name">abspath</span><span class="punctuation">(</span><span class="name">logfile</span><span class="punctuation">)</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">f</span> <span class="operator">=</span> <span class="name builtin">open</span><span class="punctuation">(</span><span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">logfile</span><span class="punctuation">,</span><span class="literal string">&quot;r&quot;</span><span class="punctuation">)</span> <span class="name">file_len</span> <span class="operator">=</span> <span class="name">stat</span><span class="punctuation">(</span><span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">logfile</span><span class="punctuation">)[</span><span class="name">ST_SIZE</span><span class="punctuation">]</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">f</span><span class="operator">.</span><span class="name">seek</span><span class="punctuation">(</span><span class="name">file_len</span><span class="punctuation">)</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">pos</span> <span class="operator">=</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">f</span><span class="operator">.</span><span class="name">tell</span><span class="punctuation">()</span> <span class="keyword">def</span> <span class="name function">_reset</span><span class="punctuation">(</span><span class="name builtin pseudo">self</span><span class="punctuation">):</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">f</span><span class="operator">.</span><span class="name">close</span><span class="punctuation">()</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">f</span> <span class="operator">=</span> <span class="name builtin">open</span><span class="punctuation">(</span><span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">logfile</span><span class="punctuation">,</span> <span class="literal string">&quot;r&quot;</span><span class="punctuation">)</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">pos</span> <span class="operator">=</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">f</span><span class="operator">.</span><span class="name">tell</span><span class="punctuation">()</span> <span class="keyword">def</span> <span class="name function">tail</span><span class="punctuation">(</span><span class="name builtin pseudo">self</span><span class="punctuation">):</span> <span class="keyword">while</span> <span class="literal number integer">1</span><span class="punctuation">:</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">pos</span> <span class="operator">=</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">f</span><span class="operator">.</span><span class="name">tell</span><span class="punctuation">()</span> <span class="name">line</span> <span class="operator">=</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">f</span><span class="operator">.</span><span class="name">readline</span><span class="punctuation">()</span> <span class="keyword">if</span> <span class="operator word">not</span> <span class="name">line</span><span class="punctuation">:</span> <span class="keyword">if</span> <span class="name">stat</span><span class="punctuation">(</span><span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">logfile</span><span class="punctuation">)[</span><span class="name">ST_SIZE</span><span class="punctuation">]</span> <span class="operator">&lt;</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">pos</span><span class="punctuation">:</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">_reset</span><span class="punctuation">()</span> <span class="keyword">else</span><span class="punctuation">:</span> <span class="name">time</span><span class="operator">.</span><span class="name">sleep</span><span class="punctuation">(</span><span class="literal number integer">1</span><span class="punctuation">)</span> <span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">f</span><span class="operator">.</span><span class="name">seek</span><span class="punctuation">(</span><span class="name builtin pseudo">self</span><span class="operator">.</span><span class="name">pos</span><span class="punctuation">)</span> <span class="keyword">else</span><span class="punctuation">:</span> <span class="literal string doc">&quot;&quot;&quot;print, return or otherwise manipulate the tailed line&quot;&quot;&quot;</span> <span class="keyword">print</span> <span class="name">line</span> </pre> <p>Maybe it's a too much overhead to check each time for file size, but you get the general idea.</p> <p>So anyways here's a usage example</p> <pre class="code python literal-block"> <span class="name">tail</span> <span class="operator">=</span> <span class="name">LogTail</span><span class="punctuation">(</span><span class="literal string">&quot;/var/log/messages&quot;</span><span class="punctuation">)</span> <span class="name">tail</span><span class="operator">.</span><span class="name">tail</span><span class="punctuation">()</span> </pre> <p>This will print out any new line appended to /var/log/messages file. If the file gets truncated or log rotated, class will detect it and will return to the start.</p> branko@toic.org (branko)Tue, 11 Aug 2009 19:55:19 +0000https://toic.org/blog/2009/tail-f-in-python-truncate-aware/Coding