Archive for ‘network management’

4 May, 2012

CVOICE+QoS

by gorthx

When I first started diving into QoS, I really hated it. There are a lot of moving parts, and whenever I poked at it I felt like I was sticking my hand in a moving engine. I had the opportunity to take some Cisco training this year, but found out that Cisco got rid of their 5-day QOS class. (At the time, I was relieved, because 5 days solid of QoS sounded like something only a masochist would do.) The only QOS training Cisco offers now comes as part of the CVOICE. I signed up for that, figuring I’d take what I could get, with the added bonus that I’d be able to communicate better with my voice team [1].

read more »

Advertisements
Tags:
20 April, 2012

This is what counter rollover looks like.

by gorthx

A friend came to me about a performance monitoring tool he was using for his routers; he was concerned that his traffic in/out graphs for his 10G linkes weren’t matching up nicely with the values he found on the interface counters. The difference was fairly large and not equal across all interfaces (which would indicate a math problem). Without actually seeing his graphs & stats, my initial reaction [1] was “counter rollover!” and pointed him here for an explanation.

Basically, if you have a link that is 100Mb or faster, you want to use ifHCInOctets and ifHCOutOctets instead of ifInOctets and ifOutOctets [2]. (The caveat, of course, is that not all interface types support the ifHC counters, not even those that should.) With reasonably steady traffic, graphs depicting counter rollover have a characteristic jagged pattern. And of course, completely incorrect data.

read more »

Tags: ,
13 April, 2012

Monitoring Cisco IP SLAs: syslog messages

by gorthx

Now that I have an SLA in place, and have some baseline data, I want some notifications in case my SLAs drop. I’ll cover basic syslog messages here, and SNMP traps in another post.

Cisco’s docs have a neat graph that shows how the reaction-config interprets the thresholds. There’s also a handy-dandy chart that tells us which reactions are available for each type of SLA.

I configured a udp-jitter SLA, so everything in the ‘UDP Jitter’ column that’s marked with a Y is something I can configure a reaction for. I’ll check RTT [1], jitter, MOS, and timeout for starters. Initially, I tried this out on just a few routers, with some numbers very close to my collected stats (see last week’s graphs), so I could make sure it was working. Here, I have adjusted them to some more realistic numbers; YMMV.

read more »

30 March, 2012

Monitoring Cisco IP SLAs: rrdtool

by gorthx

Last week I set up an SLA to monitor jitter & looked at the stats available on the command line. This week, I’ll graph those stats.

Like I mentioned, this is quick-and-dirty monitoring, just meant to give me an idea of what I’m working with. This isn’t necessarily something I’d deploy to production, especially since there are products available that already do this. (Which I hope to have time to review in the near future.)

These OIDs from CISCO-RTTMON-MIB look promising:

read more »

23 March, 2012

Configuring Cisco IP SLA for jitter

by gorthx

We’ve been playing around with IP SLAs at work lately (here’s a quick overview of SLAs and what they can do for you). Right now we’re mainly interested in monitoring VOIP services, so we set up an SLA to monitor jitter.

First, we need to configure a responder. This is the machine that will field all of the SLA requests. For our tests, we used a tiny lab router (a stock 2921) – we want to find out how much extra load SLAs would add, so we used something small in order to maximize opportunities for breakage.

read more »

10 February, 2012

Get Cisco Serial Numbers with SNMP

by gorthx

A friend asked me how to go about doing this, and I figured I’d post it here so he can find it again if he needs it.

You’ve already discovered that snmp-server chassis-id is a user-maintained field and therefore not reliable [1], so you can skip trying to use chassisId (1.3.6.1.4.1.9.3.6.3) from OLD-CISCO-CHASSIS-MIB. It’s supposed to be depracated anyway.

Newer equipment supports the ENTITY-MIB. (For certain definitions of “support” … it’s not pretty.)

There are a number of ways to do this. The most straightforward, if you’re starting from scratch, is to walk entPhysicalClass and look for items of type 3, chassis.

It’ll look like this:
:::-->snmpwalk -v 2c -M ~/.snmp/mibs -m ENTITY-MIB -c public -O s myswitch entPhysicalClass | grep chassis
entPhysicalClass.1001 = INTEGER: chassis(3)

Then, use entPhysicalSerialNum and the iid you just found (the 1001 in the previous example) to find the serial number:
:::-->snmpget -v 2c -M ~/.snmp/mibs -m ENTITY-MIB -c public -O s myswitch entPhysicalSerialNum.1001
entPhysicalSerialNum.1001 = STRING: FOC14475A35

In case you don’t have those mibs installed & don’t want to bother with it, here are the numerical equivalents:
entPhysicalClass .1.3.6.1.2.1.47.1.1.1.1.5
entPhysicalSerialNum .1.3.6.1.2.1.47.1.1.1.1.11

That method would look like this:
:::-->snmpwalk -v 2c -c public -O s myswitch .1.3.6.1.2.1.47.1.1.1.1.5 | grep "INTEGER: 3"
mib-2.47.1.1.1.1.5.1001 = INTEGER: 3
:::-->snmpget -v 2c -c public -O s myswitch .1.3.6.1.2.1.47.1.1.1.1.11.1001
mib-2.47.1.1.1.1.11.1001 = STRING: "FOC14475A35"

Now, if you want to get fancy and maybe find out the model number as well, you can then check any of the following (in order of how useful they’ve been to me personally):
entPhysicalModelName .1.3.6.1.2.1.47.1.1.1.1.11.13
entPhysicalDescr .1.3.6.1.2.1.47.1.1.1.1.11.2
entPhysicalName .1.3.6.1.2.1.47.1.1.1.1.11.7

…which would look like this:
:::-->snmpget -v 2c -M ~/.snmp/mibs -m ENTITY-MIB -c public -O s myswitch \
entPhysicalModelName.1001 \
entPhysicalDescr.1001 \
entPhysicalName.1001
entPhysicalModelName.1001 = STRING: SM-ES3G-16-P
entPhysicalDescr.1001 = STRING: SM-ES3G-16-P
entPhysicalName.1001 = STRING: 1



1 – These things tend to migrate off the equipment they were originally configured on, on to other machines, via a copy & paste vector. Pretty soon you have five or six different boxes that supposedly have the same serial number.

9 December, 2011

Friday Snark – Cisco’s VTP MIB

by gorthx

OK, Cisco, what.is.up. with the CISCO-VTP-MIB? Let’s look at the vtpVersion OID, for starters. One would expect that it would contain, maybe, the vtp version running on the host, right?

Check this out, from the definition for that OID in the mib (available here)

"The version of VTP in use on the local system."
…so it is the VTP version…

"A device will report its version capability and not any particular version in use on the device."

…only not really. (Yes, the documentation contradicts itself. This is one of the reasons network management is so frustrating.)

Turns out you need to read further down in the MIB and use managementDomainVersionInUse (“The current version of the VTP that is in use by the designated management domain.”) to get the VTP version currently running. Right-O.

(The rest of the definition for vtpVersion states: “If the device does not support vtp, the version is none(3).” This wasn’t my experience – most of my switches (3500 and 6500 series) reported ‘none’, and were actually running v1 or v2.)

17 September, 2010

Altering [Custom] Syslogs on a Cisco ASA 5550

by gorthx

Since last time, I’ve made a couple of additions to my event list, and my custom logging filter now looks like this:

vpn# sh run | include logg
logging enable
logging timestamp
logging list gabs_test level notifications
logging list gabs_test message 713228
logging list gabs_test message 113006
logging list gabs_test message 302010
logging list gabs_test message 113005
logging trap gabs_test
logging asdm informational
logging facility 22
logging host internal [ip]

I’d also like to capture the user’s client, which is specified in this message:
%ASA-6-713184: Group = [group], Username = [user], IP = [internal_ip], Client Type: Linux Client Application Version: whatever

I need to disable the filter anyway while I add the new message, so I’ll just re-write the event list with the messages in numerical order.* I’m also going to do this from the command line, because it is so much faster than going through the ASDM gui:
vpn# config t
vpn(config)# ! disable the logging filter by setting the level
vpn(config)# ! back to notifications while we work
vpn(config)# logging trap notifications
vpn(config)# ! remove the logging filter entirely
vpn(config)# no logging list gabs_test
vpn(config)# ! re-create filter with additional messages
vpn(config)# logging list gabs_test level notifications
vpn(config)# logging list gabs_test message 113005
vpn(config)# logging list gabs_test message 113006
vpn(config)# logging list gabs_test message 302010
vpn(config)# logging list gabs_test message 713184
vpn(config)# logging list gabs_test message 713228
vpn(config)# ! re-enable the filter
vpn(config)# logging trap gabs_test
vpn(config)# exit

make sure it looks right with ‘sh run | include logg’ and save it with ‘write’ or ‘copy run start’.


*Neatness counts, people!

3 September, 2010

Configuring [Custom] Syslogs on a Cisco ASA 5550

by gorthx

Configuring logging on a Cisco ASA 5550 isn’t too tricky, but I found the docs lacking in examples, so figured I’d post mine.

Using the Cisco ASDM, the parameters you need to modify are in the Configuration -> Device Management -> Logging hierarchy. (I’m sorry there aren’t any pictures to go with this; hopefully you can follow along via the breadcrumbs in the ASDM.)

1. First, enable logging:
Configuration -> Device Management -> Logging -> Logging Setup
Make sure the “Enable Logging” box is checked.

2. Specify parameters for the destination server:
Configuration -> Device Management -> Logging -> Syslog Servers
Click “Add”.
Select the source interface (probably “internal”).
Enter the IP address of the server.
Select an appropriate protocol & port (UDP and 514 are the defaults).

3. Configure your logging facility:
Configuration -> Device Management -> Logging -> Syslog Setup
Choose your facility (0-7). Because I have one central server that handles all my logging, I usually set this to something special to allow me to split out the messages into different log files. (This is a configuration parameter in /etc/syslog.conf and is covered in another post.)

4. Configure the severity level you want to log:
Configuration -> Device Management -> Logging -> Logging Filters
Select “Syslog Servers” and click “Edit”.
Filter on Server -> Notifications (logging level 5 – you’ll get every message between Emergency and Notification levels, inclusive.)

Verify that you are receiving messages on the syslog server. You can stop here if this is all you want.

We, however, decided that we wanted to include these messages:
%ASA-6-713228: Group = [group], Username = [user], IP = [external_ip], Assigned private IP address [internal_ip] to remote user
…but these are “informational” messages (note the “6” in the second message field). So we tried bumping up our logging level, and were promptly overwhelmed with a lot of other stuff we didn’t want to read. (Informational level is chatty.) What we really want is to log at the Notification level, but also include messages with ID 713228.

Tip: don’t rely on the syslog message reference (or on this post, for that matter) to tell you which messages you might want to log – look at your own logs. The message texts in the logs won’t necessarily match those in the docs.

There are a couple of ways to configure custom logging of this type:

1. Simplest:
Configuration -> Device Management -> Logging -> Syslog Setup
Find the message ID (713228 in our case) in the “Syslog ID Setup” window.
Select the message ID, click “Edit” and change the level to “Notifications”.
Click “OK” to finish.

Apply and save your changes.

Here’s what Option #1 looks like from the cli:
vpn# sh run | include logging
logging enable
logging timestamp
logging trap notifications
logging asdm informational
logging facility 22
logging host internal [ip]
logging message 713228 level notifications

(Logging facility 22 corresponds to local6; see http://en.wikipedia.org/wiki/Syslog)

OR

2. More complex:
Configuration -> Device Management -> Logging -> Event Lists
Click “Add”.
Give it a name (no spaces), eg gabs_test.

Configure the base event class and/or severity:
Under “Event Class/Severity Filters”, click “Add”.
Leave “Event Class” set to “All”*; select “Notifications” under “Severity”; click “OK”.

*Apparently the first three numbers of the message are associated with the event class, but I was unable to find a reference doc that specified which were which. That doesn’t matter here because we want all Notifications, regardless of event class.

Then add the additional specific message ID we’re interested in:
Under “Message ID Filters”, click “Add”, then enter the message ID (again, 713228); click “OK”.

Click “OK” one more time to save your new Event List.

Now apply the Event List you just created to the appropriate destination (syslogs, in our case):
Configuration -> Device Management -> Logging -> Logging Filters
Select “Syslog Servers” and click “Edit”.
Check the “Use Event List” button, select your list, and click “OK”. Notice that the event list now appears in the “All Event Classes” field.

Apply and save your changes.

Here’s what Option #2 looks like from the cli:
vpn# sh run | include logging
logging enable
logging timestamp
logging list gabs_test level notifications
logging list gabs_test message 713228
logging trap gabs_test
logging asdm informational
logging facility 22
logging host internal [ip]

Why would you select one method over the other? Method #1 is certainly easier, but it does require you to remember that you’ve altered the default behavior. Method #2 is a bit more complicated to set up, but it’s more obvious that you’re doing something special due to the “Event list” keyword in the Logging Filters page, so that’s the method I ultimately chose.

Tip: You can’t rename or copy a filter (boo!) so choose your name carefully.

Tip: You also can’t edit a filter while it’s in use – you need to remove it from the destination under “Logging Filters”, apply/save, edit the filter, apply/save, re-enable it for the destination, and apply/save.

Obviously, there is a lot more you could do with this, by creating different custom filters and then applying them to different email destinations. I’ll save that for the next rainy weekend.

Reference: http://www.cisco.com/en/US/docs/security/asa/asa83/asdm63/configuration_guide/monitor_syslog.html

Tags: ,
8 June, 2010

Using rrdgraph’s –right-axis options

by gorthx

Update 16 Jan 2013: I just noticed this post is one of my most-viewed. Unfortunately, when I moved it to wordpress, I lost the images that go with it. I’m planning to create new ones, but will have to create some data for it, as I’m no longer working as a network admin. The examples should still work, though, and if you have access to .rrd files, you can try them out for yourself – just adjust the DSes yourself.


(Note: This is a very simplified (but real-life!) example. Usually we’ll include the “in” data on the same graph, and Errors and QueueDrops etc, but that clutters up the example.)

So, say we have an interface that’s dropping packets. Not too many, but the ideal number is zero, so we’d like to see them in the graphs in our NMS (which is based on rrdtool as all decent NMSes are). We use rrdgraph to show packets out in black and packets that should have gone out, but were discarded instead, in purple:

rrdtool graph images/router-errors-unscaled.png
--title "unscaled"
--vertical-label 'Pkts/Second'
--start end-2day
--end -1hr
--width 800
--height 250
--imgformat PNG
--interlace
DEF:ifOutUcastPkts=router.rrd:ifOutUcastPkts:AVERAGE
DEF:ifOutNUcastPkts=router.rrd:ifOutNUcastPkts:AVERAGE
DEF:ifOutDiscards=router.rrd:ifOutDiscards:AVERAGE
CDEF:ifOutPkts=ifOutUcastPkts,ifOutNUcastPkts,+
LINE1:ifOutPkts#003300:ifOutPkts/sec
LINE1:ifOutDiscards#990099:ifOutDiscards/sec\n
GPRINT:ifOutPkts:AVERAGE:"Avg ifOutPkts %1.2lf\n"
GPRINT:ifOutDiscards:MAX:"Max ifOutDiscards %1.2lf"

That produces a graph like this:

unscaled example graph

(Click on the thumbnails to get the full graphs.)

We can read the average rate of discarded packets in the graph key at the bottom, and there are tiny little blips in the purple line that represents discards , but we don’t have a strong visual cue that something is off.

One possible solution is to scale up the discard values relative to the total packets. A factor of 100 ought to do it. Then we’ll use the –right-axis options to rrdgraph to label the right-hand y-axis accordingly.

We add this CDEF to provide the scaling (the LINE1 etc commands will need to be altered accordingly; you’ll see those in the final snippet):

CDEF:scaled_ifOutDiscards=ifOutDiscards,100,*

That gives us a graph that looks like this:

scaled example graph

Note that it now looks like we’re dropping up to 70 packets/second – we still have to read the stats in the key at the bottom of the graph. So let’s get the secondary y-axis correctly labeled & scaled, with the following commands:

--right-axis-label 'Discards/Second'
--right-axis 0.01:0

–right-axis-label prints the specified text along the right-hand axis.
–right-axis [scale:shift] scales and/or shifts the tickmarks on the right axis relative to the left axis. In this case, the new values we’re displaying are 100X the original values, so we need to scale our axis accordingly: 0.01. More simply: left/right = 1/100. We don’t need to start at a value other than 0, so we set the shift value to 0.

example graph with second y-axis

Hmmm…rrdtool has automatically converted our values to milli-units. (Note the lower-case m in the labels.) Let’s fix that with the –right-axis-format command:

--right-axis-format %1.1lf

example graph with second y-axis, formatted

And that’s all there is to it!

The final rrdgraph command looks like this:
rrdtool graph images/router-right-axis-format.png
--title "right-axis-format"
--vertical-label 'Pkts/Second'
--right-axis-label 'Discards/Second'
--right-axis 0.01:0
--right-axis-format %1.1lf
--start end-2day
--end -1hr
--width 800
--height 250
--imgformat PNG
--interlace
DEF:ifOutUcastPkts=router.rrd:ifOutUcastPkts:AVERAGE
DEF:ifOutNUcastPkts=router.rrd:ifOutNUcastPkts:AVERAGE
DEF:ifOutDiscards=router.rrd:ifOutDiscards:AVERAGE
CDEF:scaled_ifOutDiscards=ifOutDiscards,100,*
CDEF:ifOutPkts=ifOutUcastPkts,ifOutNUcastPkts,+
LINE1:ifOutPkts#003300:ifOutPkts/sec
LINE1:scaled_ifOutDiscards#990099:ifOutDiscards/sec\n
GPRINT:ifOutPkts:AVERAGE:"Avg ifOutPkts %1.2lf\n"
GPRINT:ifOutDiscards:MAX:"Max ifOutDiscards %1.2lf"