Archive for ‘Programming’

25 April, 2014

Mistakes I make with Perl hashes

by gorthx

I’ve been doing a bit of Perl programming the past couple months. I felt pretty rusty at first (it’s been over a year since I’ve written anything serious with it) but am getting back into the swing of things. My main use of Perl is manipulating delimited text data (from databases or flat files) for reports or loading into databases and the like. For these types of tasks, I really prefer hashes (and HoH…oH) to arrays because I can can give my variables appropriately-named keys, such as $switch{‘card’}{‘port’}. It’s a lot more obvious what that’s doing than $switch[12][2]. Obvious is good, especially 6 months later when I’ve come back to a project and am saying “what the hell is this” (as we all do).

I always seem to make the same three mistakes with hashes. The first two feature the fun symptom “I’m getting data that’s different from what I expect, and it’s randomly different every time I run my program”:

1. Oops, I need my data in a certain order. This is the first thing I forget if I haven’t written any Perl code for a while: Perl hands hash data back in whatever order it feels like. When I’m writing my initial tests, I’m using smaller data sets (like maybe one or two hash “rows”) and I get lucky and my data’s in the sort order I want. Then I get some real data, and … “oh right!” I fix this by using an AoH instead, or storing (some of) the keys in a separate array (feels kludgy), and I suppose some day I will get around to trying one of the permutations of Tie::Hash.

2. I fail to provide a unique key for the hash. This is another one that doesn’t become apparent until I’m working with real data: a “random” small data sample has two unique identifiers, but when get a bigger data set I find out there’s actually five, and so on, and pretty soon I’ve got a HoH…oH that’s 17 levels deep. (I kid.) (Maybe.)

3. The bane of my existence: typos in my hash keys. ‘use strict’ doesn’t protect from this. Writing tests helps[1], but they still occasionally slip through[2]. I troubleshoot with rigorous use of Data::Dumper::Simple and the debugger.



1 – Thank you, PDX.pm.
2 – My test for variable names: 3 months (6 months, a year) later, is it still obvious what this variable holds, and would I name it the same thing?

Advertisements
3 August, 2012

First Foray into R

by gorthx

I was talking with [name redacted] about a side project the other day and he said “OMG you’re still using *gnuplot*?” So I figured I’d better get with the program and learn some R.

Luckily for me, Portland has an R users’ group, and they held a hackathon/workshop last week, newbies welcome. They’re a good group of people & I heartily recommend the workshop night. Special thanks to Homer for the personalized help and suggestions.

read more »

Tags: , ,
25 August, 2010

Net::Ping permissions

by gorthx

While Net::Ping::External neatly maneuvers around Net::Ping‘s root access requirement, sometimes you want to use a feature of Net::Ping that Net::Ping::External doesn’t provide, such as getting ping response time. (Net::Ping::External is a bit slower; timing it doesn’t provide adequately accurate data for me.)

You can ask your sysadmin to allow you sudo access to a script that uses Net::Ping, but that sometimes leads to file ownership issues.

There is another option available, provided:
1. You are using Solaris10
2. You have a pretty good relationship with your sysadmin.

Request that your sysadmin add net_icmpaccess to your entry in /etc/user_attr, as described here.

Log out & log back in to activate your new perms, and voila.

2 July, 2009

pdx.pm code sprint #1

by gorthx

Hacking on 5.10.1 was the plan…that happened for a couple of us. :)

Duke proposed a PDX.pm sprint to work on 5.10.1:
“I think if everyone learns how to

a) get a copy of the perl git repo
b) keep it in sync
c) run the perl test suite, including running a single test at a time
d) submit a small documentation patch

then we will have a great start.”

This is perfect for a first sprint: “small, solve-able tasks” that will get everyone up & running, plus have the potential to actually be productive.

Your first code sprint with a new group of people is like the first day on the job…except nobody realizes that we’re each FNGs*. It can take some time to figure out how to work together.

It’s really great if people get the stuff that’s going to take time from the code sprint out of the way beforehand. For example, cloning the perl5 git repo (step a):

:::=>git clone git://perl5.git.perl.org/perl.git

(Took me 13 minutes.)

Duke suggested some advance reading as well – how to use the repo (cd perl; perldoc pod/perlrepository.pod) (you will need to install perl-doc if you don’t have it; it was not installed on ubuntu.)

read more »

6 June, 2009

When I get a TUIT…

by gorthx

I use Perl’s split function a lot more than I use join. Every time I use join I go through an iteration like this:

my @things = split(/-/, $value);
[do cool stuff to @things]
my $new_value = join(/:/, @things); #D'oh! Should be join(':', @things);

I understand why (split can take a regexp, join must be on a specific value) but that doesn’t mean it still doesn’t trip me up. When I get time, I’ll write something that will let me put my arrays back together the same way I took them apart.

Tags:
1 June, 2009

Adventures in QA & Testing, Part I.

by gorthx

A couple of weeks ago at PDX.pm we had a Quality Assurance Tools panel discussion. I was invited to be on the panel for the “beginning tester” perspective.

QA concepts are already familiar to me from my time as a microbiologist/immunologist with the FDA. We had a QA division that evaluated us quarterly in the following areas*:
Safety:
– appropriate gear (lab goats, goggles, gloves appropriate for what you’re handling)
– OSHA regs (aisle spacing, fire extinguishers, etc)
– drills (fire, acid spill, etc)

Keeping things neat:
– do we have expired chemicals hanging around
– are we keeping our documentation up-to-date and readable
– record-keeping (temperature records for fridges & incubators)

Calibration:
– solution & culture standardization
– instrument calibration (making sure all the lasers point the right way!)

Here’s how I relate this to software testing:

Safety equipment:
Backups & version control. If you have these, you can get yourself out of anything. Remember to practice restoring your backups.

Keeping things neat:
perltidy & perlcritic are your friends. (I still say perlcritic needs to have a drinking game that goes along with it.) Keep your code & documentation fresh.

Calibration:
Testing. Making sure that, given a certain input/environment, your code will produce certain output. For a while I confused testing with error handling – but error handling only deals with a certain set of inputs/$ENV. You want to include error handling in your testing – make sure that something that should throw an error actually does.

Once I got a grip on what I wanted to do, I had to figure out how to accomplish it. Learning how to use the tools was the hard part. Hard enough, in fact, that it took me a year of sporadic false starts before I actually did anything productive. I’m not blessed with a separate QA team for my programming tasks; I have to do it myself, but that is no excuse for having crappy code.

My largest project is my own fork of NMIS, which has no existing tests. (It may now, I forked it a while ago.) I went for the low-hanging fruit & started by testing a simple subroutine that altered text input:

my $ifName = "Serial1/0/0.0";
is (convertIfName($ifName),
'serial1-0-0-0',
'convertIfName should replace non-alphanumeric chars with hyphens and lowercase the whole schmear'
);

Over the course of 3 days, I wrote something like 200 tests.(eta actually I think I mean assertions. I’m still learning the lingo.) These were all simple unit tests (basically, does this one little block of code do what it’s supposed to do). I haven’t started yet with integration testing (does it play well with others).

The Payoff:
– I gained a much better understanding of how my code works. And found some interesting glitches – edge cases that (in theory, anyway) would never be executed in the existing production environment, but should probably be tested for anyway in case I decide I want to use them somewhere else.
– I found a lot of unused code & duplicated code, and places I could use now-standard Perl modules (like I said, my fork is old).
– Best of all, I can change my code (at least the parts I have tests for) at will and not worry that I’m going to screw something else up.

Glitches I hit:
– I already have been bitten in the ass by an edge case.
– Haven’t experienced any time savings yet, due to the learning curve.
– I’m about even with aggravation savings at this point – I am taking the next steps (mocking objects, getting ready to test an actual script^Wprogram instead of a module) and it’s like starting all over.

Places I’ve found answers to my burning testing questions:
perlmonks archives
stackoverflow
Perl-QA wiki
– my local .pm IRC

* artificial categories which made it easier to draw parallels with software testing; thanks to Peter Eschright for the great idea from his talk at the recent BarCamp Portland.

Tags:
28 May, 2009

Survey of Perl Modules I Can’t Live Without, Part II

by gorthx

Part I covered modules specific to the network management part of my job. These are my favorite general-purpose modules.

1. Viewing data structures:
Data::Dumper::Simple
with:
$Data::Dumper::Indent = 1; #JMO
I learned a lot about references using this module, too.

2. Saving myself from the tyranny of Microsoft:
Spreadsheet::ParseExcel automates what would be a daily, very tedious task. (Don’t ask unless you are willing to buy me a beer in order to hear the story behind this.)

3. Automating version control:
CVS::Simple, which I’m in the process of replacing with Git::Wrapper. I’m learning git at the same time, so it’s quite a wild ride.

4. I’m writing tests, try not to faint:
Test::Most and Test::Mockobject

5. Enforcing coding standards:
Perl::Critic

6. Having fun at my co-workers’ expense:
Lingua::Bork. Pass the daily reports through this, and see who’s actually reading them.

7. And of course, DBI. Don’t leave home without it.

Update on some others I mentioned:

Cisco::Reconfig is still intriguing. I’ve encountered a couple of quirks and am trying to figure out if It’s Just Me ™ or they’re actual bugs.

Last time I worked with Net::MAC was v1.2, and I encountered what I thought might be a bug when iterating over an array of mac addresses. I didn’t need it for any heavy lifting (just converting macs to cisco format), so instead of filing a bug report, I stuck with my hand-rolled solution. The problem has been fixed in 1.5.

Tags:
22 May, 2009

Replicating cvs’s -I option in git.

by gorthx

I’m a fairly recent convert to git, and have been moving a bunch of my coding & doc projects to it. It’s been mostly seamless, but I had one kinda tricky piece left: the daily commit (automated, of course) of any notable changes to my Cisco router & switch configs. The tricky part is handling certain lines in the configs that change each time, but aren’t necessarily of interest (for example, ntp clock-period), and I don’t want to kick off a commit if that’s all that’s changed.

cvs has this nifty -I cli option, similar to the -I option to gnu diff – it allows you to specify a regexp and the cvs diff will ignore all lines that match that regexp.

Here’s a sampling of what I had:
cvs diff
-I 'clock-period'
-I '#time'
-I 'set.spantree.port.*cost'
[filename]

(Note that you can’t have spaces in the regexp you pass to cvs.)

git doesn’t have a cli switch for this; I was having a tough time figuring out how to make it use gnu diff. This gave me the tip I needed.

So, here we go!

0. If you don’t have gnu diff on your machine, install it. (I got mine from sunfreeware.com.) You can just run diff without any args to see the options – if you’re missing “I”, you don’t have the right diff.

1. Set up a wrapper script that uses gnu diff:
:::-->cat /home/gabrielle/bin/ciscodiff.sh
#!/usr/bin/bash
#make git use gnu diff and ignore certain lines
/path/to/gnu/diff
-I 'clock-period'
-I '#time'
-I 'set spantree port.*cost'
$2 $5 | cat

Notes:
– I don’t need the . instead of the space, like I did in the cvs regexp – so this is a more restrictive match. (Which I like.)
– $2 and $5 specify which of the parameters for git diff actually are passed through to this diff. See the “git Diffs” section of the manual.

Make sure to make this executable. :)

2. Then, back in my git repo, I added the following to .git/config:
[diff "ciscoconf"]
command = /home/gabrielle/bin/ciscodiff.sh

3. Then I created .gitattributes, like so:
*-confg diff=ciscoconf

git will now use my special diff wrapper *only* on files with names that match the *-confg glob pattern.

Voila.

Tags:
26 February, 2009

Fun with ctags

by gorthx

A while back I’d created a Perl module to hold two “odds & ends” subroutines that I used in a lot of my programs. I gave it the unfortunate name of “Misc.pm”. Of course, it lived up to its name & grew over time to contain many more functions; I should have just named it JunkDrawer.pm and been done with it. I decided it was time for a cleanup, and I split everything out into more appropriately-named modules. But then I had the problem of what to do with existing programs that referenced Misc.pm. I could have just loaded all the new modules, but that seemed messy. I had to have a way to figure out which functions each program used & thus track them back to their shiny new module. jshirley suggested I try ctags.

read more »

Tags: ,
25 February, 2009

git on Solaris

by gorthx

Boy howdy, was this a trial. Sheesh. (I am in no way pointing fingers at git for this mess…really, it’s this server [that I don’t have admin rights on] that doesn’t behave the way I expect.)

Solaris 10 on x86. (uname -a: SunOS princess 5.10 Generic_125101-10 i86pc i386 i86pc)
git 1.5.6.2

Here’s what I had to do, gathered from various places around the web (the Makefile editing was gleaned mainly from http://discuss.joyent.com/viewtopic.php?pid=175313).

First, I had to install my own gmake, openssl, curl, and a couple of other required libraries.

Then:
./configure --prefix=/home/gabrielle/usr/local
--with-openssl=/home/gabrielle/usr/local/include
--without-tcltk
--without-expat

Edit the makefile:
:::-->diff Makefile Makefile.orig
167,168c167,168
< CFLAGS = -g -O2 -Wall -I/home/gabrielle/usr/local/include
CFLAGS = -g -O2 -Wall
> LDFLAGS =
172d171
< CURLDIR=/home/gabrielle/usr/local
891d889
< NO_ICONV=1

Make sure we’re using gnu make:
:::-->/home/gabrielle/usr/local/bin/make -v
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-pc-solaris2.10

Then install like so:
/home/gabrielle/usr/local/bin/make INSTALL=/usr/ucb/install
/home/gabrielle/usr/local/bin/make INSTALL=/usr/ucb/install install

Voila doesn’t really seem the appropriate thing to say here.

Tags: , ,