Mistakes I make with Perl hashes

I’ve been doing a bit of Perl programming the past couple months. I felt pretty rusty at first (it’s been over a year since I’ve written anything serious with it) but am getting back into the swing of things. My main use of Perl is manipulating delimited text data (from databases or flat files) for reports or loading into databases and the like. For these types of tasks, I really prefer hashes (and HoH…oH) to arrays because I can can give my variables appropriately-named keys, such as $switch{‘card’}{‘port’}. It’s a lot more obvious what that’s doing than $switch[12][2]. Obvious is good, especially 6 months later when I’ve come back to a project and am saying “what the hell is this” (as we all do).

I always seem to make the same three mistakes with hashes. The first two feature the fun symptom “I’m getting data that’s different from what I expect, and it’s randomly different every time I run my program”:

1. Oops, I need my data in a certain order. This is the first thing I forget if I haven’t written any Perl code for a while: Perl hands hash data back in whatever order it feels like. When I’m writing my initial tests, I’m using smaller data sets (like maybe one or two hash “rows”) and I get lucky and my data’s in the sort order I want. Then I get some real data, and … “oh right!” I fix this by using an AoH instead, or storing (some of) the keys in a separate array (feels kludgy), and I suppose some day I will get around to trying one of the permutations of Tie::Hash.

2. I fail to provide a unique key for the hash. This is another one that doesn’t become apparent until I’m working with real data: a “random” small data sample has two unique identifiers, but when get a bigger data set I find out there’s actually five, and so on, and pretty soon I’ve got a HoH…oH that’s 17 levels deep. (I kid.) (Maybe.)

3. The bane of my existence: typos in my hash keys. ‘use strict’ doesn’t protect from this. Writing tests helps^[1], but they still occasionally slip through^[2]. I troubleshoot with rigorous use of Data::Dumper::Simple and the debugger.

—

1 – Thank you, PDX.pm.
2 – My test for variable names: 3 months (6 months, a year) later, is it still obvious what this variable holds, and would I name it the same thing?

Net::Ping permissions

While Net::Ping::External neatly maneuvers around Net::Ping‘s root access requirement, sometimes you want to use a feature of Net::Ping that Net::Ping::External doesn’t provide, such as getting ping response time. (Net::Ping::External is a bit slower; timing it doesn’t provide adequately accurate data for me.)

You can ask your sysadmin to allow you sudo access to a script that uses Net::Ping, but that sometimes leads to file ownership issues.

There is another option available, provided:
1. You are using Solaris10
2. You have a pretty good relationship with your sysadmin.

Request that your sysadmin add net_icmpaccess to your entry in /etc/user_attr, as described here.

Log out & log back in to activate your new perms, and voila.

PDX.pm meeting notes

Jeff (aka @duckyd) gave a presentation about CPAN Awesomeness at last night’s pdx.pm.

Slides are on github!  Your homework is to find them. :)  Here are some highlights.

There are over 16K modules on the CPAN as of 11 Oct 2009.  Wow.

Jeff’s recommended changes from the default cpan shell configuration:
– auto_commit 1
– prerequisites_policy follow
– build_requires_install_policy yes
– prefer_installer MB (Module::Build)
– change your make_install_make_command and mbuild_install_mbuild_command to include your sudo command.

—

Spend 10 minutes & give something back to CPAN every time you install a module:  Simply set up CPAN::Reporter!

Recommendations:
– make sure that you set cc_author to ‘no’.  (In the latest version, that’s the default.)
– set it to prompt you to edit/send the report if the tests fail.  This way you can judge if the failure is due to your own
boneheadedness.
– you can set the transport value to a file to run reports without sending them.
– for help:  perldoc CPAN::Reporter::Config

For automatic continuous testing, set up CPAN::Reporter::Smoker.  (Doesn’t actually install anything, justs runs tests.)

Recommendations:
– don’t run it as root;  you are the canary in the coal mine.
– create a dedicated user that has essentially no privs on your machine.
– run it on a separate Perl install (core modules only).
– this is a cool place to use that RAND option for prefer_installer in CPAN.

—

Another cool tip that I REALLY DIG because I have systems with multiple perls & users associated with them:
Set your shebang line to:

#!/usr/bin/env perl

pdx.pm code sprint #1

Hacking on 5.10.1 was the plan…that happened for a couple of us. :)

Duke proposed a PDX.pm sprint to work on 5.10.1:
“I think if everyone learns how to

a) get a copy of the perl git repo
b) keep it in sync
c) run the perl test suite, including running a single test at a time
d) submit a small documentation patch

then we will have a great start.”

This is perfect for a first sprint: “small, solve-able tasks” that will get everyone up & running, plus have the potential to actually be productive.

Your first code sprint with a new group of people is like the first day on the job…except nobody realizes that we’re each FNGs*. It can take some time to figure out how to work together.

It’s really great if people get the stuff that’s going to take time from the code sprint out of the way beforehand. For example, cloning the perl5 git repo (step a):

:::=>git clone git://perl5.git.perl.org/perl.git

(Took me 13 minutes.)

Duke suggested some advance reading as well – how to use the repo (cd perl; perldoc pod/perlrepository.pod) (you will need to install perl-doc if you don’t have it; it was not installed on ubuntu.)

OSBridge Recap

This week I attended Open Source Bridge here in Portland.

Typically, I managed to miss the keynotes both days. There is something about conferences which makes me sleep through my alarm.

It was really hard choosing which talks to attend. The results of the coin toss:

Wednesday:
Tcl/Tk: Grandpa might be old, but he can still kick your ass! I went to this primarily because I use Expect so much. (Well, I use Expect.pm, but I remember my roots.) Webb gave a good intro to Tcl/Tk (“Tickle-Tea-Kay!”) despite some initial technical difficulties. I finally figured out the brackets vs braces variable expansion.

Then I gave my talk. Thankfully, Impress did not surprise me. I now have my unicorn badge.

Spindle, Mutilate, & Metaprogram: This was really cool, although it seemed similar to things that came out of the Perl community a few years back. I’d like to see a throwdown between Markus Roberts & Damian Conway.

Assholes are killing your project: I only managed about 20 minutes of this talk before I got too depressed & had to leave. Sorry, Donnie! We’ll talk about this later.

I spent some time in the hall track & then hit the yoga session. This was an excellent pick-me-up after a day of talking and brain-filling, and set me up for my BoF and then some time at the pub.

Thursday:
Arrived too late for chromatic’s Intro to Parrot so hung out in the speaker lounge and watched Andy and Irving’s run-through of Virtualize vs Containerize: Fight! I love the mashups.

Next up was Emma McGratten’s Ask Forgiveness not Permission, which had a lot of excellent reasons (financial & otherwise) for using open source, but not many tips on how to subversively bring it into your organization. I’m sure I know someone who could give a talk about that. :cough:

Lunch today was the excellent KOiFusionPDX Food Cart! They came to the conference site & provided excellent korean tacos. (Yeah, I know, sounds weird – but TRUST ME.)

Speaking of trust…Trust the Vote sounds like an excellent project. Unfortunately the question period started devolving into political discussion, and I didn’t want to just dive right in there and ask them why the hell they’re using MySQL instead of PostgreSQL.

Maria Webster got her unicorn badge for Faking it Till I Make It. Check out her blog to see what geeky women are up to.

bzr vs git smackdown with Selena & Emma. I’ve already made up my mind (git all the way!), but it’s good to listen to alternatives.

The Meditiation for Geeks session didn’t go too well for me, because I was so tired that any time I got close to The Zone, I almost fell over onto a fellow PostgreSQL Smurf. Still, the yoga & meditation sessions are a great way to unwind prior to the post-con socializing & I’d like to see more of this.

Pg took over the room & had our PostgreSQL BoF, which replaced the regular PDXPUG meeting.

Josh Berkus was riding a bicycle around town, which made me inordinately happy. I want to see if we can provide more bikes for attendees next year.

Friday: The Unconference rocked my socks:
1. Emma Jane’s “Playing with yourself” about Open Source documentation teams. I am even sadder that I missed WOSCON. This got me totally excited to contribute to docs. (Especially for certain Perl modules – but that’s a discussion for another post.) Highlights: the conference team is working on a style guide, and a library of personas (which isn’t public yet)

2. I signed up with DayOn, a local volunteer effort. This will be fantastic once we can get people trained in what’s actually reasonable to ask for.

3. I did a Network Management Basics talk (“FCAPS: What the hell?!?”) with Ua and Adam. We talked about the FCAPS model & where various tools we use fit. A very high percentage of them are rrdtool-based, so we talked about that a bit as well. Adam showed us his munin install. I keep trying to find other people in town who are as into Net Management as I am…I sort of feel like I need a 12-step program sometimes. On the way out, Ua proposed a Super-Sekrit project which we’ll start working on in September. (Excitement!)

—
I saw up-close what it took to put on this conference and I’d like to congratulate the organizers on their success! Great job, and can’t wait until next year!

When I get a TUIT…

I use Perl’s split function a lot more than I use join. Every time I use join I go through an iteration like this:

my @things = split(/-/, $value);
[do cool stuff to @things]
my $new_value = join(/:/, @things); #D'oh! Should be join(':', @things);

I understand why (split can take a regexp, join must be on a specific value) but that doesn’t mean it still doesn’t trip me up. When I get time, I’ll write something that will let me put my arrays back together the same way I took them apart.

Adventures in QA & Testing, Part I.

A couple of weeks ago at PDX.pm we had a Quality Assurance Tools panel discussion. I was invited to be on the panel for the “beginning tester” perspective.

QA concepts are already familiar to me from my time as a microbiologist/immunologist with the FDA. We had a QA division that evaluated us quarterly in the following areas*:
Safety:
– appropriate gear (lab goats, goggles, gloves appropriate for what you’re handling)
– OSHA regs (aisle spacing, fire extinguishers, etc)
– drills (fire, acid spill, etc)

Keeping things neat:
– do we have expired chemicals hanging around
– are we keeping our documentation up-to-date and readable
– record-keeping (temperature records for fridges & incubators)

Calibration:
– solution & culture standardization
– instrument calibration (making sure all the lasers point the right way!)

Here’s how I relate this to software testing:

Safety equipment:
Backups & version control. If you have these, you can get yourself out of anything. Remember to practice restoring your backups.

Keeping things neat:
perltidy & perlcritic are your friends. (I still say perlcritic needs to have a drinking game that goes along with it.) Keep your code & documentation fresh.

Calibration:
Testing. Making sure that, given a certain input/environment, your code will produce certain output. For a while I confused testing with error handling – but error handling only deals with a certain set of inputs/$ENV. You want to include error handling in your testing – make sure that something that should throw an error actually does.

Once I got a grip on what I wanted to do, I had to figure out how to accomplish it. Learning how to use the tools was the hard part. Hard enough, in fact, that it took me a year of sporadic false starts before I actually did anything productive. I’m not blessed with a separate QA team for my programming tasks; I have to do it myself, but that is no excuse for having crappy code.

My largest project is my own fork of NMIS, which has no existing tests. (It may now, I forked it a while ago.) I went for the low-hanging fruit & started by testing a simple subroutine that altered text input:

my $ifName = "Serial1/0/0.0";
is (convertIfName($ifName),
'serial1-0-0-0',
'convertIfName should replace non-alphanumeric chars with hyphens and lowercase the whole schmear'
);

Over the course of 3 days, I wrote something like 200 tests.(eta actually I think I mean assertions. I’m still learning the lingo.) These were all simple unit tests (basically, does this one little block of code do what it’s supposed to do). I haven’t started yet with integration testing (does it play well with others).

The Payoff:
– I gained a much better understanding of how my code works. And found some interesting glitches – edge cases that (in theory, anyway) would never be executed in the existing production environment, but should probably be tested for anyway in case I decide I want to use them somewhere else.
– I found a lot of unused code & duplicated code, and places I could use now-standard Perl modules (like I said, my fork is old).
– Best of all, I can change my code (at least the parts I have tests for) at will and not worry that I’m going to screw something else up.

Glitches I hit:
– I already have been bitten in the ass by an edge case.
– Haven’t experienced any time savings yet, due to the learning curve.
– I’m about even with aggravation savings at this point – I am taking the next steps (mocking objects, getting ready to test an actual script^Wprogram instead of a module) and it’s like starting all over.

Places I’ve found answers to my burning testing questions:
– perlmonks archives
– stackoverflow
– Perl-QA wiki
– my local .pm IRC

—

* artificial categories which made it easier to draw parallels with software testing; thanks to Peter Eschright for the great idea from his talk at the recent BarCamp Portland.

Survey of Perl Modules I Can’t Live Without, Part II

Part I covered modules specific to the network management part of my job. These are my favorite general-purpose modules.

1. Viewing data structures:
Data::Dumper::Simple
with:
$Data::Dumper::Indent = 1; #JMO
I learned a lot about references using this module, too.

2. Saving myself from the tyranny of Microsoft:
Spreadsheet::ParseExcel automates what would be a daily, very tedious task. (Don’t ask unless you are willing to buy me a beer in order to hear the story behind this.)

3. Automating version control:
CVS::Simple, which I’m in the process of replacing with Git::Wrapper. I’m learning git at the same time, so it’s quite a wild ride.

4. I’m writing tests, try not to faint:
Test::Most and Test::Mockobject

5. Enforcing coding standards:
Perl::Critic

6. Having fun at my co-workers’ expense:
Lingua::Bork. Pass the daily reports through this, and see who’s actually reading them.

7. And of course, DBI. Don’t leave home without it.

Update on some others I mentioned:

Cisco::Reconfig is still intriguing. I’ve encountered a couple of quirks and am trying to figure out if It’s Just Me ™ or they’re actual bugs.

Last time I worked with Net::MAC was v1.2, and I encountered what I thought might be a bug when iterating over an array of mac addresses. I didn’t need it for any heavy lifting (just converting macs to cisco format), so instead of filing a bug report, I stuck with my hand-rolled solution. The problem has been fixed in 1.5.

Fun with ctags

A while back I’d created a Perl module to hold two “odds & ends” subroutines that I used in a lot of my programs. I gave it the unfortunate name of “Misc.pm”. Of course, it lived up to its name & grew over time to contain many more functions; I should have just named it JunkDrawer.pm and been done with it. I decided it was time for a cleanup, and I split everything out into more appropriately-named modules. But then I had the problem of what to do with existing programs that referenced Misc.pm. I could have just loaded all the new modules, but that seemed messy. I had to have a way to figure out which functions each program used & thus track them back to their shiny new module. jshirley suggested I try ctags.

Cisco Syslog Parser – slides

Here are the slides from my PDX.pm talk this week. A link to the accompanying podcast will be along soon.

Other fun things we discussed at the meeting:
Cisco::Reconfig
trapgen
logger
logwatch

Thanks for the lively discussion!

[edit] podcast!

[edit] Clarification of two items from the podcast:
– multiline messages do indeed come in multiple packets. There is a message counter that increments for each message, so you could use the host name + message counter to match up multi-line messages. For what I’m doing, the important part is in that first line, so the payoff isn’t worth the investment.
– re hypens in the mnemonic field of the system message: I went back through and wasn’t able to find any examples of this, so I retract my statement. (I do have examples of system messages with hyphens in the facility field.)