Posts tagged ‘rrdtool’

20 April, 2012

This is what counter rollover looks like.

by gorthx

A friend came to me about a performance monitoring tool he was using for his routers; he was concerned that his traffic in/out graphs for his 10G linkes weren’t matching up nicely with the values he found on the interface counters. The difference was fairly large and not equal across all interfaces (which would indicate a math problem). Without actually seeing his graphs & stats, my initial reaction [1] was “counter rollover!” and pointed him here for an explanation.

Basically, if you have a link that is 100Mb or faster, you want to use ifHCInOctets and ifHCOutOctets instead of ifInOctets and ifOutOctets [2]. (The caveat, of course, is that not all interface types support the ifHC counters, not even those that should.) With reasonably steady traffic, graphs depicting counter rollover have a characteristic jagged pattern. And of course, completely incorrect data.

read more »

Advertisements
Tags: ,
30 March, 2012

Monitoring Cisco IP SLAs: rrdtool

by gorthx

Last week I set up an SLA to monitor jitter & looked at the stats available on the command line. This week, I’ll graph those stats.

Like I mentioned, this is quick-and-dirty monitoring, just meant to give me an idea of what I’m working with. This isn’t necessarily something I’d deploy to production, especially since there are products available that already do this. (Which I hope to have time to review in the near future.)

These OIDs from CISCO-RTTMON-MIB look promising:

read more »

8 June, 2010

Using rrdgraph’s –right-axis options

by gorthx

Update 16 Jan 2013: I just noticed this post is one of my most-viewed. Unfortunately, when I moved it to wordpress, I lost the images that go with it. I’m planning to create new ones, but will have to create some data for it, as I’m no longer working as a network admin. The examples should still work, though, and if you have access to .rrd files, you can try them out for yourself – just adjust the DSes yourself.


(Note: This is a very simplified (but real-life!) example. Usually we’ll include the “in” data on the same graph, and Errors and QueueDrops etc, but that clutters up the example.)

So, say we have an interface that’s dropping packets. Not too many, but the ideal number is zero, so we’d like to see them in the graphs in our NMS (which is based on rrdtool as all decent NMSes are). We use rrdgraph to show packets out in black and packets that should have gone out, but were discarded instead, in purple:

rrdtool graph images/router-errors-unscaled.png
--title "unscaled"
--vertical-label 'Pkts/Second'
--start end-2day
--end -1hr
--width 800
--height 250
--imgformat PNG
--interlace
DEF:ifOutUcastPkts=router.rrd:ifOutUcastPkts:AVERAGE
DEF:ifOutNUcastPkts=router.rrd:ifOutNUcastPkts:AVERAGE
DEF:ifOutDiscards=router.rrd:ifOutDiscards:AVERAGE
CDEF:ifOutPkts=ifOutUcastPkts,ifOutNUcastPkts,+
LINE1:ifOutPkts#003300:ifOutPkts/sec
LINE1:ifOutDiscards#990099:ifOutDiscards/sec\n
GPRINT:ifOutPkts:AVERAGE:"Avg ifOutPkts %1.2lf\n"
GPRINT:ifOutDiscards:MAX:"Max ifOutDiscards %1.2lf"

That produces a graph like this:

unscaled example graph

(Click on the thumbnails to get the full graphs.)

We can read the average rate of discarded packets in the graph key at the bottom, and there are tiny little blips in the purple line that represents discards , but we don’t have a strong visual cue that something is off.

One possible solution is to scale up the discard values relative to the total packets. A factor of 100 ought to do it. Then we’ll use the –right-axis options to rrdgraph to label the right-hand y-axis accordingly.

We add this CDEF to provide the scaling (the LINE1 etc commands will need to be altered accordingly; you’ll see those in the final snippet):

CDEF:scaled_ifOutDiscards=ifOutDiscards,100,*

That gives us a graph that looks like this:

scaled example graph

Note that it now looks like we’re dropping up to 70 packets/second – we still have to read the stats in the key at the bottom of the graph. So let’s get the secondary y-axis correctly labeled & scaled, with the following commands:

--right-axis-label 'Discards/Second'
--right-axis 0.01:0

–right-axis-label prints the specified text along the right-hand axis.
–right-axis [scale:shift] scales and/or shifts the tickmarks on the right axis relative to the left axis. In this case, the new values we’re displaying are 100X the original values, so we need to scale our axis accordingly: 0.01. More simply: left/right = 1/100. We don’t need to start at a value other than 0, so we set the shift value to 0.

example graph with second y-axis

Hmmm…rrdtool has automatically converted our values to milli-units. (Note the lower-case m in the labels.) Let’s fix that with the –right-axis-format command:

--right-axis-format %1.1lf

example graph with second y-axis, formatted

And that’s all there is to it!

The final rrdgraph command looks like this:
rrdtool graph images/router-right-axis-format.png
--title "right-axis-format"
--vertical-label 'Pkts/Second'
--right-axis-label 'Discards/Second'
--right-axis 0.01:0
--right-axis-format %1.1lf
--start end-2day
--end -1hr
--width 800
--height 250
--imgformat PNG
--interlace
DEF:ifOutUcastPkts=router.rrd:ifOutUcastPkts:AVERAGE
DEF:ifOutNUcastPkts=router.rrd:ifOutNUcastPkts:AVERAGE
DEF:ifOutDiscards=router.rrd:ifOutDiscards:AVERAGE
CDEF:scaled_ifOutDiscards=ifOutDiscards,100,*
CDEF:ifOutPkts=ifOutUcastPkts,ifOutNUcastPkts,+
LINE1:ifOutPkts#003300:ifOutPkts/sec
LINE1:scaled_ifOutDiscards#990099:ifOutDiscards/sec\n
GPRINT:ifOutPkts:AVERAGE:"Avg ifOutPkts %1.2lf\n"
GPRINT:ifOutDiscards:MAX:"Max ifOutDiscards %1.2lf"