Posts tagged ‘nms’

8 June, 2010

Using rrdgraph’s –right-axis options

by gorthx

Update 16 Jan 2013: I just noticed this post is one of my most-viewed. Unfortunately, when I moved it to wordpress, I lost the images that go with it. I’m planning to create new ones, but will have to create some data for it, as I’m no longer working as a network admin. The examples should still work, though, and if you have access to .rrd files, you can try them out for yourself – just adjust the DSes yourself.


(Note: This is a very simplified (but real-life!) example. Usually we’ll include the “in” data on the same graph, and Errors and QueueDrops etc, but that clutters up the example.)

So, say we have an interface that’s dropping packets. Not too many, but the ideal number is zero, so we’d like to see them in the graphs in our NMS (which is based on rrdtool as all decent NMSes are). We use rrdgraph to show packets out in black and packets that should have gone out, but were discarded instead, in purple:

rrdtool graph images/router-errors-unscaled.png
--title "unscaled"
--vertical-label 'Pkts/Second'
--start end-2day
--end -1hr
--width 800
--height 250
--imgformat PNG
--interlace
DEF:ifOutUcastPkts=router.rrd:ifOutUcastPkts:AVERAGE
DEF:ifOutNUcastPkts=router.rrd:ifOutNUcastPkts:AVERAGE
DEF:ifOutDiscards=router.rrd:ifOutDiscards:AVERAGE
CDEF:ifOutPkts=ifOutUcastPkts,ifOutNUcastPkts,+
LINE1:ifOutPkts#003300:ifOutPkts/sec
LINE1:ifOutDiscards#990099:ifOutDiscards/sec\n
GPRINT:ifOutPkts:AVERAGE:"Avg ifOutPkts %1.2lf\n"
GPRINT:ifOutDiscards:MAX:"Max ifOutDiscards %1.2lf"

That produces a graph like this:

unscaled example graph

(Click on the thumbnails to get the full graphs.)

We can read the average rate of discarded packets in the graph key at the bottom, and there are tiny little blips in the purple line that represents discards , but we don’t have a strong visual cue that something is off.

One possible solution is to scale up the discard values relative to the total packets. A factor of 100 ought to do it. Then we’ll use the –right-axis options to rrdgraph to label the right-hand y-axis accordingly.

We add this CDEF to provide the scaling (the LINE1 etc commands will need to be altered accordingly; you’ll see those in the final snippet):

CDEF:scaled_ifOutDiscards=ifOutDiscards,100,*

That gives us a graph that looks like this:

scaled example graph

Note that it now looks like we’re dropping up to 70 packets/second – we still have to read the stats in the key at the bottom of the graph. So let’s get the secondary y-axis correctly labeled & scaled, with the following commands:

--right-axis-label 'Discards/Second'
--right-axis 0.01:0

–right-axis-label prints the specified text along the right-hand axis.
–right-axis [scale:shift] scales and/or shifts the tickmarks on the right axis relative to the left axis. In this case, the new values we’re displaying are 100X the original values, so we need to scale our axis accordingly: 0.01. More simply: left/right = 1/100. We don’t need to start at a value other than 0, so we set the shift value to 0.

example graph with second y-axis

Hmmm…rrdtool has automatically converted our values to milli-units. (Note the lower-case m in the labels.) Let’s fix that with the –right-axis-format command:

--right-axis-format %1.1lf

example graph with second y-axis, formatted

And that’s all there is to it!

The final rrdgraph command looks like this:
rrdtool graph images/router-right-axis-format.png
--title "right-axis-format"
--vertical-label 'Pkts/Second'
--right-axis-label 'Discards/Second'
--right-axis 0.01:0
--right-axis-format %1.1lf
--start end-2day
--end -1hr
--width 800
--height 250
--imgformat PNG
--interlace
DEF:ifOutUcastPkts=router.rrd:ifOutUcastPkts:AVERAGE
DEF:ifOutNUcastPkts=router.rrd:ifOutNUcastPkts:AVERAGE
DEF:ifOutDiscards=router.rrd:ifOutDiscards:AVERAGE
CDEF:scaled_ifOutDiscards=ifOutDiscards,100,*
CDEF:ifOutPkts=ifOutUcastPkts,ifOutNUcastPkts,+
LINE1:ifOutPkts#003300:ifOutPkts/sec
LINE1:scaled_ifOutDiscards#990099:ifOutDiscards/sec\n
GPRINT:ifOutPkts:AVERAGE:"Avg ifOutPkts %1.2lf\n"
GPRINT:ifOutDiscards:MAX:"Max ifOutDiscards %1.2lf"

20 June, 2009

OSBridge Recap

by gorthx

This week I attended Open Source Bridge here in Portland.

Typically, I managed to miss the keynotes both days. There is something about conferences which makes me sleep through my alarm.

It was really hard choosing which talks to attend. The results of the coin toss:

Wednesday:
Tcl/Tk: Grandpa might be old, but he can still kick your ass! I went to this primarily because I use Expect so much. (Well, I use Expect.pm, but I remember my roots.) Webb gave a good intro to Tcl/Tk (“Tickle-Tea-Kay!”) despite some initial technical difficulties. I finally figured out the brackets vs braces variable expansion.

Then I gave my talk. Thankfully, Impress did not surprise me. I now have my unicorn badge.

Spindle, Mutilate, & Metaprogram: This was really cool, although it seemed similar to things that came out of the Perl community a few years back. I’d like to see a throwdown between Markus Roberts & Damian Conway.

Assholes are killing your project: I only managed about 20 minutes of this talk before I got too depressed & had to leave. Sorry, Donnie! We’ll talk about this later.

I spent some time in the hall track & then hit the yoga session. This was an excellent pick-me-up after a day of talking and brain-filling, and set me up for my BoF and then some time at the pub.

Thursday:
Arrived too late for chromatic’s Intro to Parrot so hung out in the speaker lounge and watched Andy and Irving’s run-through of Virtualize vs Containerize: Fight! I love the mashups.

Next up was Emma McGratten’s Ask Forgiveness not Permission, which had a lot of excellent reasons (financial & otherwise) for using open source, but not many tips on how to subversively bring it into your organization. I’m sure I know someone who could give a talk about that. :cough:

Lunch today was the excellent KOiFusionPDX Food Cart! They came to the conference site & provided excellent korean tacos. (Yeah, I know, sounds weird – but TRUST ME.)

Speaking of trust…Trust the Vote sounds like an excellent project. Unfortunately the question period started devolving into political discussion, and I didn’t want to just dive right in there and ask them why the hell they’re using MySQL instead of PostgreSQL.

Maria Webster got her unicorn badge for Faking it Till I Make It. Check out her blog to see what geeky women are up to.

bzr vs git smackdown with Selena & Emma. I’ve already made up my mind (git all the way!), but it’s good to listen to alternatives.

The Meditiation for Geeks session didn’t go too well for me, because I was so tired that any time I got close to The Zone, I almost fell over onto a fellow PostgreSQL Smurf. Still, the yoga & meditation sessions are a great way to unwind prior to the post-con socializing & I’d like to see more of this.

Pg took over the room & had our PostgreSQL BoF, which replaced the regular PDXPUG meeting.

Josh Berkus was riding a bicycle around town, which made me inordinately happy. I want to see if we can provide more bikes for attendees next year.

Friday: The Unconference rocked my socks:
1. Emma Jane’s “Playing with yourself” about Open Source documentation teams. I am even sadder that I missed WOSCON. This got me totally excited to contribute to docs. (Especially for certain Perl modules – but that’s a discussion for another post.) Highlights: the conference team is working on a style guide, and a library of personas (which isn’t public yet)

2. I signed up with DayOn, a local volunteer effort. This will be fantastic once we can get people trained in what’s actually reasonable to ask for.

3. I did a Network Management Basics talk (“FCAPS: What the hell?!?”) with Ua and Adam. We talked about the FCAPS model & where various tools we use fit. A very high percentage of them are rrdtool-based, so we talked about that a bit as well. Adam showed us his munin install. I keep trying to find other people in town who are as into Net Management as I am…I sort of feel like I need a 12-step program sometimes. On the way out, Ua proposed a Super-Sekrit project which we’ll start working on in September. (Excitement!)


I saw up-close what it took to put on this conference and I’d like to congratulate the organizers on their success! Great job, and can’t wait until next year!

20 June, 2008

Cisco::Reconfig

by gorthx

Poking around on CPAN a week or so ago, I stumbled across Cisco::Reconfig.

It looked pretty interesting, and turned out to be the work of David Muir Sharnoff. He’s responsible for Net::Netmask, which is one of my five favorites, so I had to check it out.

Sample router config (not all lines shown):

interface Loopback0
 ip address 10.254.254.1 255.255.255.255
!
interface Ethernet0/0
 description Admin LAN
 ip address 10.10.1.1 255.255.255.0
 duplex auto
!
interface Serial0/0
 description to Internet ID W065432
 ip address 1.1.1.1 255.255.255.252
!
interface Ethernet0/1
 no ip address
 shutdown
!
interface Ethernet1/0
 description User LAN
 ip address 10.10.2.1 255.255.254.0
 duplex auto
!
interface Ethernet1/1
 ip address 10.10.4.1 255.255.254.0
 duplex auto
!

Let’s write a quick & dirty script^W program to look for blank interface descriptions, because that’s something I find really annoying:

#!/usr/local/bin/perl

use strict;
use Cisco::Reconfig;

my $host        = "gabrielle";
my $host_config = "$host.confg";
print ("Checking: $hostn");
my $config  = readconfig("$host_config");

for my $intf ( $config->get('interface') ) {
    next if ( $intf->get('shutdown') );	#ignore, don't care
    next if ( $intf =~ /Loopback/ );	#idc

    my $descr   = $intf->get('description');
    chomp ($intf, $descr);  	# hm, kinda feel like I shouldn't have to do this?
    print ("$intf: $descrn");	# just to be chatty
    if (! $descr) {
        print ("$host: $intf: Description is blank!n");
    }
}

exit 0;

:::–>./sample.pl
Checking: gabrielle
interface Ethernet0/0: description Admin LAN
interface Serial0/0: description to Internet ID W065432
interface Ethernet1/0: description User LAN
interface Ethernet1/1:
gabrielle: interface Ethernet1/1: Description is blank!

I’m primarily interested in it for checking compliance of our configurations with our business standards. No need to chunk through a config line-by-line with regexps; in a lot of cases, a simple ‘get’ will tell me if something’s configured or not (eg “snmp server-location”.)

Specific features that look really intriguing:
– you can generate the commands you need to “fix” your config
– the “context” method allows you to draw out the surrounding lines

I can’t wait to mess around with this some more.

Tags: , ,
31 May, 2008

Survey of Perl Modules I Can’t Live Without

by gorthx

Specifically for the management of IP networks & equipment – routers, switches, etc

1. Connection automation:

Because it’s really handy to update, say, the passwords on all 1000 of your network devices in a couple of hours by kicking off a single script. It’s faster and more reliable. Of course, if you fat-finger the password in the script, you’ve just fat-fingered it on your entire network, so please test it first.

Net::Telnet::Cisco – this is really cool, but doesn’t work on older equipment, due to Cisco’s lack of a standardized interface. So, we roll our own with Expect.pm.

—–
2. IP Addressing:

Socket.pm of course, though not being a C programmer I have a hard time remembering the syntax off the top of my head so I keep a couple of examples around.

my $ip = $ARGV[0];
my @addr = split(/./, $ip);
my $addr = pack(‘C4’, @addr);
my $name = gethostbyaddr($addr, AF_INET);
print (“Name: $namen”);

my $host = $ARGV[0];
my $ip = scalar gethostbyname($host);
my @ip = unpack(“C4”,$ip);
my $ip = join(“.”,@ip);
print (“IP: $ipn”);
—–
3. Subnetting:

Net::Netmask, baby! For figuring out network & bcast addresses, accept no substitutes.

my $ip = “10.1.1.1”;
my $netmask = “255.255.255.0”;
my $block = new Net::Netmask($ip, $netmask);
my $bcast_addr = $block->broadcast();
print (“Broadcast address: $bcast_addrn”);

—–
4. SNMP. Y’all knew I would have to talk about this.

Several options, two I have actual experience with.

A lot of people use Net::SNMP I use SNMP.pm (which comes with Net-SNMP – confused?* :) ) It’s probably the most complex, but also (to me) the most useful. I can get pretty much any data I want with this module.

SNMP-Simple – much more user-friendly than SNMP.pm. Lighter, faster, but you can only get *values* back from this, you can not get the OIDs, so it requires some pre-knowledge of what you want to monitor, which isn’t always possible.

—–
5. RRD[ps].pm – perl module for Tobi Oetiker’s RRDTool. Many network performance tools are based on RRDTool: mrtg, cacti, orca, NMIS. It’s indispensable if you’re going to write your own monitoring app, or tinker with one of the aforementioned tools.

—–
6. Date::Format and Date::Manip. I use Date::Format because it’s so easy to create timestamps for log files & reports – it follows the Unix strftime format. Date::Manip is huge and slow, but it’s the only thing I found that could handle some complex user time reporting I was doing a while back, and I love it.

—–
7. Net::MAC. I haven’t had time to experiment with this to the extent I’d like, but I have a feeling it’s going to be as useful as Net::Netmask. I’ll let you know in a couple of weeks.

*One of my favorite jokes, just for me.

Tags: ,