MRTG-Graph-hack

[b4rt]

some stuff glued together to integrate mrtg into torrentflux. i tried to stay as simple as possible to make integration easy. (and it is complete non-destructive to tf-code in its current form).  the default-config/setup uses mrtgs rateup (no rrd). (my fluxpoller.pl-data-gatherer also supports cacti output-format for ppl with running rrd+cacti systems)
the implementations are smart enough that it is possible to add/delete mrtg-targets without changing a line of code only by configuring the targets in the mrtg-target-config-file (mrtg.flux.cfg).

requires:

• mrtg http://oss.oetiker.ch/mrtg/
• torrentflux 2.1 (any version should do)
• linux (possibly.. actually i dont know ~)
• perl
• cron

included:

• fluxpoller.pl : data-gatherer (see this thread)
• flux-mrtg-update.sh : shell-script that is invoked from cron
• flux-mrtg.conf : config-file for mrtg *changes needed*
• mrtg.flux.cfg : file with mrtg-target-definitions *changes needed*
• mrtg.php : "frontend" for the graphs
• mrtg.sql : sql-file to enable the mrtg-frontend

defined graphs:

• traffic : graph of incoming + outgoing flux-traffic
• connections : graph of flux-tcp-connections (sum)

setup:

1. pre-requisites

1.1 install mrtg http://oss.oetiker.ch/mrtg/

2. set up mrtg

2.1 create required dirs
create a directory with the name mrtg in your flux-docroot-dir.
(on a default 2.1 that would be /var/www/mrtg)
ensure the user running your webserver is able to write to this dir (create files in it).

2.2 setup my scripts
copy my scripts to a place you like. in the default-config the locations are:
fluxpoller.pl : /usr/local/bin/fluxpoller.pl
flux-mrtg-update.sh : /usr/local/bin/flux-mrtg-update.sh
ensure these scripts are executable by the user running your webserver.

2.3 setup config(-files)
copy the config files to a place you like.
in the default-config the locations are:
mrtg.flux.cfg : /etc/mrtg/mrtg.flux.cfg
flux-mrtg.conf : /etc/mrtg/flux-mrtg.conf
ensure these files are readable by the user running your webserver.
ensure the user running your webserver can create a file in that dir. (mrtg creates a lock-file besides its config-files)
edit the two files and adjust it to your installation.
all places that (maybe) have to be adjusted are flagged with the string "CHANGEME".
exactly : every line after a CHANGEME may need adjustments.
tip: for "lists" do "grep -A 1 CHANGEME flux-mrtg.conf" and "grep -A 1 CHANGEME mrtg.flux.cfg"

2.4 init+test mrtg
at this stage your mrtg is (should be~) ready to go. to test it and because mrtg will complain about its logs the first 3 invocations my advice is to get things started by hand.
call as the user running your webserver (adjust paths if needed):

Code:

/usr/local/bin/flux-mrtg-update.sh /etc/mrtg/flux-mrtg.conf

if the only errors/warnings are about logfiles then it looks good.
do the call until you dont get any output anymore.
to control that things are right at this point check these dirs (referencing default-config):
/var/www/mrtg : should contain 2 .inc files and some pngs
/usr/local/torrent/.mrtg : should contain mrtg-logs and rateups "flat-file-db"

2.5 setup cron-job
whereever you define your custom-cronjobs.. add this one (paths = default-config, webserver-user/user doin mrtg-stuff is www-data) adjust if needed):

Code:

# invoke mrtg-update every 5 mins
*/5 * * * *    www-data /usr/local/bin/flux-mrtg-update.sh /etc/mrtg/flux-mrtg.conf &> /dev/null

3. integrate into torrentflux

3.1 sql
use the included mrtg.sql

3.2 setup "frontend"
copy the file mrtg.php to flux-document-root (default : /var/www)

3.3 changes in file config.php
add this line:

Code:

define('_ID_MRTG','Graph');

3.4 changes in file index.php
its up to you where you link to mrtg.php. what i use is :
look for this:

Code:

<a href="who.php">

and add above:

Code:

<?php 
  if ($cfg['enable_mrtg'] == 1) {
    echo '<a href="mrtg.php">';
    echo '<img src="images/all.gif" width="16" height="16" title="" border="0" align="absmiddle">';
    echo _ID_MRTG;
    echo '</a>|';
  }
?>

4. done
you made it, time to lean back and enjoy your graphs



EDIT:
i attached new screenies after my setup was running for a while. still unimpressive because my torrent is kinda idle atm.

EDIT-2:
5. Examples for additional Graphs (targets) :
maybe some of you want to graph more than just traffic + connections so here are some examples to start with.
i didnt test any of these !
for that kind of stuff (stuff not really being flux but being system itself) i use a much more sophisticated rrd-based solution.
(if interested : rrd: http://oss.oetiker.ch/rrdtool/ cacti: http://www.cacti.net/ )

• of course you need a tool to gather and output the data in a format mrtg can read. fluxpoller.pl only supports traffic+connections so thats up to you. just do a google-search, there exist tons of these tools. the ones i use in the examples actually really exist and might be a start to search for.
• of course you have to adjust these examples for your system.
• quite sure you have to adjust the graph-settings to get "nice-looking" graphs.
• you dont have to change any code when adding graphs (targets) if you use my hack as suggested. just add the new graphs (targets) in mrtg.flux.cfg (mrtg.php is smart enough to recognize new graphs and flux-mrtg-update.sh allways updates all anyway)

to add a graph (=target) add it at the end of mrtg.flux.cfg .

Example : Graphing the Load Average

Code:

# ------------------------------------------------------------------------------
# target : loadavg
# ------------------------------------------------------------------------------

# gather data for this target
# CHANGEME
Target[loadavg]: `/usr/bin/mrtg-load -m 1000`

# MaxBytes
MaxBytes[loadavg]: 50000
# 1 kilo def
kilo[loadavg]: 1000

# title
Title[loadavg]: loadavg
# title in the graph
PNGTitle[loadavg]: loadavg

# options.
Options[loadavg]: gauge,integer,growright,nobanner,nopercent,pngdate,transparent

# peaks
WithPeak[loadavg]: ym

# legend
YLegend[loadavg]: loadavg
ShortLegend[loadavg]: &nbsp;
# ....

Example : Graphing apache stats

Code:

# ------------------------------------------------------------------------------
# target : apachestats
# ------------------------------------------------------------------------------

# gather data for this target
# CHANGEME
Target[apachestats]: `/usr/bin/mrtg-apache -m 100 localhost`

# MaxBytes
MaxBytes[apachestats]: 20000
# 1 kilo def
kilo[apachestats]: 1000

# title
Title[apachestats]: apachestats
# title in the graph
PNGTitle[apachestats]: apachestats

# options.
Options[apachestats]: gauge,integer,growright,nobanner,nopercent,pngdate,transparent

# peaks
WithPeak[apachestats]: ym

# legend
YLegend[apachestats]: loadavg
ShortLegend[apachestats]: &nbsp;
# ....

Example : Graphing CPU-Temperature (needs sensors !)

Code:

# ------------------------------------------------------------------------------
# target : sensors_temperature
# ------------------------------------------------------------------------------

# gather data for this target
# CHANGEME
Target[sensors_temperature]: `mrtg-sensors -m 800 dummy dummy w83781d-i2c-0-2d Temp`

# MaxBytes
MaxBytes[sensors_temperature]: 10000
# 1 kilo def
kilo[sensors_temperature]: 1000

# title
Title[sensors_temperature]: sensors_temperature
# title in the graph
PNGTitle[sensors_temperature]: temperature

# options.
Options[sensors_temperature]: gauge,integer,growright,nobanner,nopercent,pngdate,transparent,bits

# peaks
WithPeak[sensors_temperature]: ym

# legend
YLegend[sensors_temperature]: temperature
ShortLegend[sensors_temperature]: &nbsp;
# ....

Example : Graphing CPU-Fans (needs sensors !)

Code:

# ------------------------------------------------------------------------------
# target : sensors_fans
# ------------------------------------------------------------------------------

# gather data for this target
# CHANGEME
Target[sensors_fans]: `mrtg-sensors -m 400 w83781d-i2c-0-2d FanCPU1 w83781d-i2c-0-2d FanCPU2`

# MaxBytes
MaxBytes[sensors_fans]: 100000
# 1 kilo def
kilo[sensors_fans]: 1000

# title
Title[sensors_fans]: sensors_fans
# title in the graph
PNGTitle[sensors_fans]: fans

# options.
Options[sensors_fans]: gauge,integer,growright,nobanner,nopercent,pngdate,transparent,bits

# peaks
WithPeak[sensors_fans]: ym

# legend
YLegend[sensors_fans]: fans
ShortLegend[sensors_fans]: &nbsp;
# ....

EDIT: new example

Example : Graphing disk-usage
didnt even think of this so far because i do disk-usage-graphing with cacti. (so i dont even know a tool like "mrtg-du" but writing that if none can be googled should be no big deal.)
so if you want to graph the disk-usage of the torrent-dir a config to start with may be:

Code:

# ------------------------------------------------------------------------------
# target : disk_usage
# ------------------------------------------------------------------------------

# gather data for this target
# CHANGEME
Target[disk_usage]: `/usr/local/bin/foo.bar /usr/local/torrent`

# flux torrent dir capacity
MaxBytes[disk_usage]: 100000000000
# 1024 bytes are 1 kilo
kilo[disk_usage]: 1024

# title
Title[disk_usage]: disk_usage
# title in the graph
PNGTitle[disk_usage]: disk_usage

# options.
Options[disk_usage]: gauge,growright,nobanner,pngdate,transparent

# peaks
WithPeak[disk_usage]: ym

# legend
YLegend[disk_usage]: Disk-Usage
ShortLegend[disk_usage]: &nbsp;
# ....

happy graphing,
b4rt

[PiForever]

wow!! what a fantastic job, and great explanation....many thanx friend. will test soon....

[wobbles]

Great mod b4rt, I wanted a mod like this. Will intergrate it into my tf server later and let you know how it goes

[gringo]

awesome hack, i will try it asap.  just started a new job so not much time right now for this

[kanchke]

thumbs up !

got to try this one out !

// update: it works ... more or less

I keep getting this error:

Code:

WARNING: Problem with External get '/usr/local/bin/fluxpoller.pl connections mrtg':
   Expected a Number for 'in' but got ''

ERROR: Target[connections][_IN_] ' $target->[1]{$mode} ' did not eval into defined data

Obviously I get no stats concerning the connections.
Another small thing I had to change is crontab:

Code:

# invoke mrtg-update every 5 mins
*/5 * * * *    www-data /usr/local/bin/flux-mrtg-update.sh /etc/mrtg/flux-mrtg.conf &> /dev/null

where www-data is the default user apache is running, I had to change it to www because I'm using Lighttpd

Thanx for the great hack and clear howto 

k.

[b4rt]

WARNING: Problem with External get '/usr/local/bin/fluxpoller.pl connections mrtg':
   Expected a Number for 'in' but got ''

ERROR: Target[connections][_IN_] ' $target->[1]{$mode} ' did not eval into defined data

pls try : '/usr/local/bin/fluxpoller.pl connections mrtg' and see what it prints. the first line (in mrtg-format) is "IN"/down (which i used for connections)
if that produces error/bad output pls post that and ill take a look into it

b4rt

[kanchke]

I get/got :

Code:

/usr/local/bin/fluxpoller.pl connections mrtg
/bin/grep: not found
/bin/grep: not found
/bin/grep: not found

0
up 16
host.domain.com

As I am using FreeBSD grep & netstat are located in /usr/bin and not in /bin, I changed the according lines in fluxpoller.pl. 

Code:

/usr/local/bin/fluxpoller.pl connections mrtg
0
0
up 16

Still no connection results are displayed and I do have torrents running.  No worries though, for me personally the most important/usefull feature are the traffic graphs 

k.

[b4rt]

I get/got :

Code:

/usr/local/bin/fluxpoller.pl connections mrtg
/bin/grep: not found
/bin/grep: not found
/bin/grep: not found

0
up 16
host.domain.com

As I am using FreeBSD grep & netstat are located in /usr/bin and not in /bin, I changed the according lines in fluxpoller.pl. 

Code:

/usr/local/bin/fluxpoller.pl connections mrtg
0
0
up 16

Still no connection results are displayed and I do have torrents running.  No worries though, for me personally the most important/usefull feature are the traffic graphs 

k.

the problem sure comes from the parsing of the netstat-output on your system.
(i assume the result "0" from fluxpoller.pl is not correct and you _have_ open connections)
if you want me to try to fix that i would need some snip of you :
execute this (as the user running your flux-setup, change path to executables if needed):

Code:

netstat -e -p --tcp -n 2> /dev/null | grep -v root | grep -v 127.0.0.1

btw blame me for using "grep" in a perl-script... i use it in the connection-sub because processing that in one sys-call-go (with greps -c switch) is sooo convenient ~. hey at least this part is done in native code (just kiddin )

maybe others run into a problem like this so :
howto solve "/foo/bar: not found"

solution 1 - the easy and bad way
i discourage usage of this because its dumb and has serious impact on the scripts runtime.
(if it even works... it only uses "whereis"+"awk")
open fluxpoller.pl and:
find this:

Code:

my $autoFindBinaries = 0;

change to:

Code:

my $autoFindBinaries = 1;

solution 2 - the clean and safe way
define the correct path(s) to all used executables in fluxpoller.pl
eg if fluxpoller.pl has trouble executing grep (/bin/grep: not found) do this:
1. find correct path to the grep-executable on your system (egrep,fgrep,etc should do as well)
    to let find help you doin the job you might try this:
   

Code:

    find / -name "*grep" -printf "%u\t%m\t%p\n" | less
   

2. change the var in fluxpoller.pl to have that path as its value:
open fluxpoller.pl and:
find this:

Code:

my $BIN_GREP = "/bin/grep";

change to:

Code:

my $BIN_GREP = "/the/path/you/just/found/grep";

regards,
b4rt

[PiForever]

brilliant....thanx for sharing

[ryaner]

solution 2 - the clean and safe way
define the correct path(s) to all used executables in fluxpoller.pl
eg if fluxpoller.pl has trouble executing grep (/bin/grep: not found) do this:
1. find correct path to the grep-executable on your system (egrep,fgrep,etc should do as well)
    to let find help you doin the job you might try this:
   

Code:

    find / -name "*grep" -printf "%u\t%m\t%p\n" | less
   

So much overkill there. whereis grep or which grep would do and it'd be much quicker.

[b4rt]

solution 2 - the clean and safe way
define the correct path(s) to all used executables in fluxpoller.pl
eg if fluxpoller.pl has trouble executing grep (/bin/grep: not found) do this:
1. find correct path to the grep-executable on your system (egrep,fgrep,etc should do as well)
    to let find help you doin the job you might try this:
   

Code:

    find / -name "*grep" -printf "%u\t%m\t%p\n" | less
   

So much overkill there. whereis grep or which grep would do and it'd be much quicker.

actually my perl-script uses whereis to locate binaries if needed.
overkill ? no if you dont have that bin on your path which / whereis prolly wont find it.
my find-line does.

regards,
b4rt

[ryaner]

Ah right sorry about that. I don't actually run your script, just saw the find line and thought you were using it.

[b4rt]

Ah right sorry about that. I don't actually run your script, just saw the find line and thought you were using it.

of course i dont use that find-call in-script, was only meant as safe way for ppl who dont know how to find a executable "somewhere" (wo locate-db)

[b4rt]

Update : new Example ~

[HackMan]

Hi, first thank u for the great hack. The images are built, but i have the problem thet there is no data on th images. (look atattched image)

does anybody have an idea whats wrong in my configuration?

Greetz HackMan