toic.org - Entries for the category Monitoringhttps://toic.org/blog/categories/monitoring/To better understand your systems, one must observe it.en-usZinniaThu, 08 Oct 2015 18:41:24 +0000Apache monitoring tool ApTop beta released https://toic.org/blog/2012/apache-monitoring-tool-aptop-beta-released/<p>This is my first public publish to PyPi, without further delays, here is the full project description</p> <p><h1>ApTop</h1></p> <p>ApTop ia a top(1) alike monitoring utility for Apache.</p> <p>Being daily involved in monitoring system load in shared environment I find it lacking proper tool for monitoring Apache behavior in real time. Granted there are some tools for this job, they are usually of limited options for some monitoring needs.</p> <p>ApTop was written as a hobby project in python for debugging purposes while investigating several system load situations caused by web applications in shared hosting environment.</p> <p>ApTop is still an early beta release with some known bugs, nevertheless it still provides a great starting point in monitoring Apache. Every feedback and feature request at this point is more than welcomed.</p> <div class="section" id="description"> <h1>Description</h1> <p>ApTop will show near real time Apache active working proceses and virtual host connections. By default sleeping / waiting for connections slots will be filtered out.</p> <p>ApTop is also displaying several screens</p> <div class="section" id="dashboard-or-home-screen"> <h2>Dashboard or Home screen</h2> <p>This screen displays some relevant informations gathered from <a class="reference external" href="http://httpd.apache.org/docs/2.2/mod/mod_status.html">apache-status</a> page. It is by default sorted by shortest last apache access time &quot;SS&quot; and it displays only active connections. Displayed fields are the same as in server-status page, so you can find detailed explanation for their meaning at the bottom of server-status page.</p> <p>Displayed fields are:</p> <p>PID - OS process ID M - Mode of operation CPU - CPU usage, number of seconds SS - Seconds since beginning of most recent request&lt; Req - Milliseconds required to process most recent request Conn - Kilobytes transferred this connection Acc - Number of accesses this connection / this child / this slot Client - Remote client VHost - Local virtualhost Request - Request method and partial URL</p> </div> <div class="section" id="count-by-vhosts-screen"> <h2>Count by vhosts screen</h2> <p>This screen displays virtualhosts in order by highest count of active connections along with connection count.</p> <p>Very useful for detecting high traffic sites.</p> </div> <div class="section" id="count-by-clients-screen"> <h2>Count by clients screen</h2> <p>This screen displays clients in order by highest count of active connections along with connection count.</p> <p>Very useful for detecting resource exhausters.</p> </div> </div> <div class="section" id="requirements"> <h1>Requirements</h1> <p>ApTop should work just fine on any unix based system with python standard packages and lxml.</p> <p>Make sure you have libxml2 libxslt libxslt-devel libxml2-devel packages instaleld prior to building lxml:</p> <p>For RHEL based distro you can install those with</p> <pre class="literal-block"> yum install libxml2 libxslt libxslt-devel libxml2-devel </pre> <p>After that you can install lxml python package with <strong>easy_install lxml</strong> or <strong>pip install lxml</strong></p> <p>ApTop is using Apaches <strong>mod_status</strong> with <strong>ExtendedStatus On</strong> for gathering data. Please ensure that <strong>mod_status</strong> is enabled and proper status url is defined in <strong>aptop.conf</strong> There are some concernes regarding enabled ExtendedStatus On, but I think <a class="reference external" href="http://www.philchen.com/2008/06/02/apache-20-mod_status-effects-on-performance-server-resources">this links</a> covers this nicely.</p> </div> <div class="section" id="instalation"> <h1>Instalation</h1> <p>These installation options are for RHEL based distros, adjust accordingly.</p> <blockquote> yum install libxml2 libxslt libxslt-devel libxml2-devel python-setuptools</blockquote> <p>Now you can do</p> <blockquote> easy_install ApTop</blockquote> <p>or</p> <blockquote> easy_install <a class="reference external" href="https://bitbucket.org/btoic/aptop/get/master.tar.gz">https://bitbucket.org/btoic/aptop/get/master.tar.gz</a></blockquote> <p>You can now run aptop.py</p> </div> <div class="section" id="configuration"> <h1>Configuration</h1> <p>ApTop has some built in default configuration directives, but it will also look first in ~/.aptop.conf and than /etc/aptop.conf for overrides.</p> <p>ApTop uses .ini style configuration files and main cofniguration options should be placed after <strong>[aptop]</strong> section.</p> <p>By default ApTop will use <a class="reference external" href="http://localhost/server-status">http://localhost/server-status</a> url for grabbing data to display, and with default refresh rate of 5 seconds.</p> <div class="section" id="available-configuration-options"> <h2>Available configuration options:</h2> <blockquote> status_url = URL</blockquote> <p>This configuration option will require you to enter valid mod_status status url for server you wish to monitor. This defaults to <a class="reference external" href="http://localhost/server-status">http://localhost/server-status</a> Note that Apache should have ExtendedStatus conf option set to On</p> <blockquote> refresh = seconds</blockquote> <p>This option will control ApTop default refresh interval. It defaults to 5 seconds and can't be lower than 1 second.</p> <p>You can even monitor the remote systems if they granted you access rights to theirs server-status URL. In the future ApTop will provide options to save remote monitored servers to its configuration.</p> </div> </div> <div class="section" id="runtime-options"> <h1>Runtime options</h1> <p>While running ApTop responds to following options:</p> <p><strong>H</strong> - display main screen</p> <p><strong>V</strong> - display vhosts by conn. count</p> <p><strong>c</strong> - display clients by conn. count</p> <p><strong>R</strong> - reverse the current sort order</p> <p><strong>I</strong> - toggle active/inactive slots</p> </div> <div class="section" id="aditional-links"> <h1>Aditional links</h1> <p><a class="reference external" href="http://toic.org">http://toic.org</a></p> <p><a class="reference external" href="https://bitbucket.org/btoic/aptop">https://bitbucket.org/btoic/aptop</a></p> </div> branko@toic.org (branko)Wed, 12 Dec 2012 23:34:19 +0000https://toic.org/blog/2012/apache-monitoring-tool-aptop-beta-released/CodingMonitoringMunin centralized monitoring on Centos https://toic.org/blog/2009/munin-centralized-monitoring-on-centos/<p>Munin is a great tool for performance graphing your servers, by default it will graph resources on localhost, however if you wish to monitor multiple servers from single location, then you must deploy a central Munin server.</p> <p>Having central graphs mean you can have central hub of data about performance of your systems. This can later be better presented, viewed and analyzed, </p> <p>Let's begin setting up a central server</p> <p>To get things started we will need one server for centralized graphs. For start this can be a low budget dedicated server or a small vps, if you have large amount of monitored nodes, and large amout of metrics on them, then you will probably want to invest in better disks. Updating huge amounts of rrd files and regenerating html an graphs can be disk IO intensive with large instalations.</p> <div class="section" id="installing-munin"> <h1>Installing munin</h1> <p>This will be a minimal install for a central munin server. I'm using a small vps with minimal centos 5 install.</p> <p>First let's setup elrepo:</p> <pre class="literal-block"> rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm </pre> <p>Now we need to install munin and munin-node (if you wish to monitor this host as well):</p> <pre class="literal-block"> yum install -y munin munin-node </pre> <p>by default munin will put its html files into <strong>/var/www/html/munin</strong> folder If you wish to move that to another place, now is your time. For the sake of simplicity I'll just leave it where it is. Of course we will need apache to access munin html files, so if you don't have apache installed do:</p> <pre class="literal-block"> yum install -y httpd </pre> <p>now start the apache:</p> <pre class="literal-block"> service httpd start </pre> <p>If you left everything as it is munin html should be available at: <a class="reference external" href="http://yourhostname.com/munin/">http://yourhostname.com/munin/</a> You may notice that there is nothing there yet, just wait until we configure all other hosts. start the munin-node on this host (if you installed it):</p> <pre class="literal-block"> service munin-node start </pre> <p>Make sure your cron is runing:</p> <pre class="literal-block"> service crond status </pre> <p>and let's go configure those other hosts.</p> </div> <div class="section" id="installing-munin-node"> <h1>Installing munin-node</h1> <div class="section" id="installing-on-cpanel"> <h2>Installing on cPanel</h2> <p>Since lot's of my servers to monitor are with cPanel installed there is an easy way to install munin.</p> <p>Login to your <strong>whm</strong> go to: <strong>Manage plugins</strong>, now find <strong>Munin</strong>, click a check box, scroll down and click save. After the munin is installed it should appear in your <strong>whm</strong> at the bottom of the navigation. Go and check up if the munin is installed correctly.</p> <p>Installing trough cPanel will install munin-node and munin, you can disable the munin graphing later if you like.</p> </div> <div class="section" id="installing-on-non-cpanel"> <h2>Installing on non cPanel</h2> <p>We can install munin on Centos trough Elrepo. first we will setup elrepo:</p> <pre class="literal-block"> rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm </pre> <p>and then install a munin-node:</p> <pre class="literal-block"> yum install -y munin-node </pre> <p>voila... let's configure nodes on remote servers now.</p> </div> </div> <div class="section" id="configuring-munin-node-on-remote-hosts"> <h1>Configuring munin-node on remote hosts</h1> <p>For both cPanel and non-cPanel servers all we need to do is add allowed host in munin-node.conf:</p> <pre class="literal-block"> nano /etc/munin/munin-node.conf </pre> <p>add at the end of the file:</p> <pre class="literal-block"> allow ^192\.168\.0\.20$ </pre> <p>where 192.168.0.20 is the IP address of you central munin server. restart the munin-node:</p> <pre class="literal-block"> service munin-node restart </pre> <p>If you have firewall installed on that host (and I hope you do), allow the incoming tcp port 4949 for the IP of the central node.</p> <p>In <a class="reference external" href="http://www.configserver.com/cp/csf.html">csf</a> add the following line:</p> <pre class="literal-block"> tcp:in:d=4949:s=192.168.0.20 </pre> <p>to your <strong>/etc/csf/csf.allow</strong> file</p> <p>or just run:</p> <pre class="literal-block"> iptables -A INPUT -p tcp -s 192.168.0.2 --dport 4949 -m state --state NEW,ESTABLISHED -j ACCEPT </pre> <p>Modify this to your firewall, and don't forget to replace 192.168.0.20 with your munin server IP. Now everything should be ready for data collection from central server</p> </div> <div class="section" id="configuring-munin-server"> <h1>Configuring munin server</h1> <p>We need to configure munin conf file on our central server to collect data from remote servers:</p> <pre class="literal-block"> nano /etc/munin/munin.conf </pre> <p>If you didn't change any locations of html files and munin datastore you realy don't need to change that in the conf file.</p> <p>What we are interested with are the host sections. You will notice there is configuration for our localhost You can change its name now, leave the address field as it is.</p> <p>To add up a new host just add:</p> <pre class="literal-block"> [myhost.mydomain.com] address 192.168.0.10 use_host_name yes </pre> <p>change the 192.168.0.10 with the IP of the server you wish to monitor. you can now add as many host you like. Make sure that you have enabled outgoing connections on tcp port 4949 on your central munin server. After a while the first results should start to appear.</p> </div> <div class="section" id="configuring-multi-host-display-graphs"> <h1>Configuring multi host display graphs</h1> <p>The real benefit of having all the host graphs and data on one place is you can easily make multi host graphs and compare the loads on the servers. This could help you grasp a bigger picture of individual server workloads and give you an idea what to improve and how to load balance between the machines.</p> <p>Here is one of the example graphs, showing apache request per second. If the machines were the same hardware configuration that would give indications that some of the machines have higher hit rate and we would need to rewrite our load balancing.</p> <p>We could do the same thing with load graphs and see which servers have the spikes, and distribute the workload on some less loaded servers.</p> <img alt="/media/filer_public/00/62/00627fe5-be95-4a9f-bb98-fc5979a0b320/munin1.png" src="/media/filer_public/00/62/00627fe5-be95-4a9f-bb98-fc5979a0b320/munin1.png" /> <img alt="/media/filer_public/37/24/37244bfb-ece6-4bb3-b152-b2d4bc0d6d6d/munin2.png" src="/media/filer_public/37/24/37244bfb-ece6-4bb3-b152-b2d4bc0d6d6d/munin2.png" /> <p><strong>So how do we configure this?</strong></p> <p>First you need to find out rrd's name of the data you wish to put on the graph.</p> <p>for example apache accesses per second:</p> <pre class="literal-block"> cd /var/lib/munin/yourdomain.com </pre> <p><strong>ls -lh</strong> in the directory and you will find out what data is available to munin. in case of the apache accesses data we will have few files named:</p> <blockquote>hostname.domainname.com-<span style="color: red;">apache_accesses-accesses80</span>-d.rrd hostname2.domainname.com-<span style="color: red;">apache_accesses-accesses80</span>-d.rrd</blockquote><p>what we are interested with are those fields (marked in red) after the domain name separated by dash. Ok let's write a conf in munin.conf for this two hosts:</p> <pre class="literal-block"> nano /etc/munin/munin.conf </pre> <p>Go under the host definitions in your conf file and add:</p> <blockquote><strong>[domainname.com;Totals] <span style="color: green;">update no</span> <span style="color: blue;">apacheaccess</span>.graph_title Apache access side by side <span style="color: blue;">apacheaccess</span>.graph_order hostname=hostname.domainname.com:<span style="color: red;">apache_accesses.accesses80</span> hostname1=hostname1.domainname.com.com:<span style="color: red;">apache_accesses.accesses80</span></strong></blockquote><p>Notice the red lines, they are the same as rrd filenames red parts we saw earlier, just replace dash with dot. Green text is to disable updates for this domain declaration since updates are already done at the host declaration in the conf file. Blue is the graph representation name, followed by title in the first line and data in second.</p> <p>This way you can make all the side by side graphs for all the data munin collected in rrd files. After the changes wait for a next munin update and enjoy the graphs</p> </div> branko@toic.org (branko)Fri, 09 Oct 2009 19:56:24 +0000https://toic.org/blog/2009/munin-centralized-monitoring-on-centos/Monitoring