PDXPUGDay Recap

Last weekend we held the biggest PDXPUGDay we’ve had in a while! 5 speakers + a few lightning talks added up to a fun lineup. About 1/3 of the ~50 attendees were in town for FOSS4G; I think the guy from New Zealand will be holding the “visitor farthest from PDXPUG” for a good long while. Some folks from SEAPUG daytripped down (hi!) and we made plans for PDXPUG to road trip up there, probably for next year’s LinuxFestNW.

My highlights:
HSTORE, XML, JSON, and JSONB – David Wheeler
– Pg’s XML features are pretty neat, but I still think XML needs to DIAF. Perhaps that’s just my previous experience speaking.
– We renamed the HSTORE containment operator (@>) to “ice cream cone operator”, courtesy Mark Wong.
– Operations on JSON are slower than on HSTORE. That’s interesting.
– The storage overhead for JSONB is higher than for regular JSON, because it doesn’t compress very well. Josh B took an audience vote on improving compression at the expense of slowing down operations, and it was pretty evenly split.
– As usual, David included benchmarks and gave good overviews of when to use which data type.

Snapshotted Data Versioning – Eric Hanson
Eric gave a talk about this at PDXPUG last year and was showing an updated version of what Aquameta’s up to. Eric’s philosophy is “make everything data, and then make a UI for it”.
– Implemented FUSE for Pg, bidirectional, so you can change your data by making updates directly in the database or by editing a text file on the filesystem. I believe this was described as “perverse” by a certain audience member.

Data Near Here – Veronika Megler
– Another update to a previous PDXPUG talk
– Scientists report that they spend up to 80% of their time just finding data relevant to their research. Not collecting – locating previously saved data. What a time sink.
– Parsers for each data format have to be custom coded.

Portal Update – Kristin Tufte
– Another example of pulling data from many different sources in many “unique” formats!
– Current research on pedestrian counts uses the crosswalk buttons as a potential method to count pedestrians.
– I’d like to get ahold of the traffic light data, to see if the light at 32nd and Powell really is the longest light in Portland, or if that’s just my imagination.

AWS Faceoff (Cloud Shootout!) – Josh Berkus
I don’t care too much about Postgres on AWS – if I’m going to go that route, I’ll buy my own hardware, TYVM.
– RDS has a limited number of extensions installed, and PL/R isn’t one of them.* They did just add pg_stat_statements, which is cool. The Amazon support people are taking requests, and are attentive to the community, according to Josh. (I don’t have enough experience with that to have an opinion.)
– performance on RDS just isn’t that great; Josh got 325 TPS read/write, and 1430 TPS read-only.
– Then there was the cost comparison; RDS and Heroku don’t look that great compared to hosting it yourself, but you’d need to factor in the cost of support staff there.

Thanks for a great event!

—
* I decided to see for myself what extensions were available. Mark warned me “don’t shed too many tears for what they don’t have”. To my surprise, many of my favorites are available – pgperl, plpgsql, postgis, and tablefunc! (SO EXCITE MUCH PIVOT)

Check what’s available on your instance with this command:
SHOW rds.extensions;

Note that “SELECT * FROM pg_available_extensions ORDER BY name;” will show you a bunch of stuff that’s not necessarily available on RDS. (Something I wish they’d fix.)

This Week in Geekville: Barcamp!

This was my first Barcamp. I’m sad I couldn’t make it to all of Saturday’s sessions – they looked great!

I only made it to two talks. First, Peter Eschright’s “Rat Salad” talk, which, I admit, I was attracted to by the possibility of gross stories. (I have some, being a former employee of CFSAN myself*.) We had an interesting discussion on what software development industry can learn from food safety initiatives.

Next up, my session on munin. (Which is pronounced “moonin”, like the thing you do on the Barfly bus.) I was hoping to find other users to discuss it, but that seemed to be what everyone came to the session looking for too. Next time I will just do my standard network management intro talk & review of tools. This was too specific for Barcamp, I think.

Igals’ TrainPorn session was next on my list, but I got sucked into Audrey Eschright’s “Creating Awkwardness” on my way through the forum. Lots of discussion about circles of friends vs circles of trust, how to protect your information, and of course some tales from the trenches (my favorite part.)

The beer ran out 30 minutes before my “How to change a flat” session. One should never attempt bike maintenance without beer, but we did anyway. We had a bigger crowd than I expected. Thanks to @robotadam for being a spokesmodel**, and @shawnzyoo for the backup!

I finished up with an intro yo-yo session from @pdxyoyo, and only hit myself in the face once.

It was fun meeting people I’ve only been hearing about. :) Thanks to Cubespace for hosting & providing yummy yummy food.

Coming up next week, apparently I’m participating in the QA talk at PDX.pm. I’ll be the one wearing a red shirt.

* My copy of the “Food Defect Action Levels” publication is a big hit at parties.
** Pun intentional.

OSBridge: Call for Proposals

Read about it here:

http://opensourcebridge.org/2009/02/call-for-proposals-we-welcome-your-ideas-until-march-31/

And then submit your proposal!

We’re speaking at PLUG tonight!

Selena & I will be giving a modified version of our OSCON talk at PLUG tonight.  Time & location on the PLUG website.  See you there!

This Week in Geekville.

This week at PDX.pm, chromatic gave an excellent talk about “how not to fail”.  There was quite a bit of discussion of Godzilla attacks and the calculus thereof, which I don’t think I can adequately capture.  “You had to be there”.  Anyway, on to my notes!

The keys:
– Recognize points of failure before the actual failure occurs.  The idea being to prevent the failure.
– Perform root cause analysis.  Your biggest problem may not be the most annoying, obvious, or easiest to solve.
– Do the simplest, fastest thing you can do to ease the problem.  You’re conducting an experiment to see if it has any effect on the problem.

Some examples of places you can get hung up:
– Waiting for a decision from someone else.  That someone else could be you.  Self-awareness is critical.
– Waiting for someone else to do something.  This is usually a sign of a technical bottleneck, such as a complicated process.  “The only person who can ____ is too _____.” (The blanks on this slide had my hopes up for madlibs.)
– Becoming demotivated (which is a word I thought only I used) by tedious or annoying tasks.  (Or people.)  Get rid of the annoying thing.  (Please obey all local laws.)  Motivational tip:  leave something to do tomorrow!  That way you start the day on a high note.  [I do this all the time – in fact, I keep a list on hiveminder of “instant gratification projects” – things I can do in 15 or 30 minutes – and any time I’m frustrated & need to feel like I’ve accomplished something, I do one of those.]
– Lacking direction.  Try creating a one-sentence problem statement.

Time-based releases force you to get your release process under control – and maybe even automated!  It gets you in to a rhythm.  (My rhythm, since it’s just me and I don’t really have “releases”, is to have “Refactoring Fridays”.)

Tips that especially apply to FOSS projects, where just about everyone is a volunteer:
– Make it easy for people to get involved
– Encourage discussion
– If someone new has contributed something, but it’s not quite up to snuff, tell them (nicely) how they could improve next time

PDXPUG Meeting recap – Logic and Databases with Jeff Davis

[ Jeff’s slides are available at: http://www.pgcon.org/2008/schedule/events/83.en.html ]

Last night’s meeting was Jeff Davis’ last presentation as a Portlander (for a while at least!). He is leaving his job at Laika to pursue streaming database nirvana at Truviso, a company whose database product is based on PostgreSQL. There’s a bunch of whip-smart people working there, and Jeff will fit right in. If you’ve got other questions about what Truviso does, feel free to ask Jeff.

Our “question of the day” was – What are your plans for the summer? We had some great answers — including that Len’s son Ari is hosting the morning NPR news show for the next two weeks! Go Ari! It sounds like James *might* be leaving us for Redmond — stay tuned for more news on that. Gab will be riding her bike, and other people are going to just enjoy the fact that it finally seems like it *is* actually summertime.

Mark Wong is working with Joshua Drake (and Gab?) to install our new Performance Lab downtown on Saturday. Look for updates and announcements about what those servers will be used for from Mark and JD next week. HP donated two of the machines, and I believe that one other came from Sun.

We also had cocktails served by Gabrielle. THANK YOU, GAB!

Jeff’s presentation started with using logic to help express in English basic SQL concepts. For example, JOIN can be expressed with AND, and UNION can be expressed with OR. Thinking about what questions you’ll be asking about the data, and also forming those questions and statements of fact in ways that allow the assignment of clear ‘true’ or ‘false’ values will help enormously in ensuring that your data structures and SQL queries work the way that you and your customers expect.

At the risk of butchering the next discussion, here’s my take on the second part: Jeff spent quite a bit of time talking about the merits of using statements of fact when describing states, and how focusing on the truth values helps you test your assumptions about results from queries. His closing words on this topic: “Use an iterative process to clean up data before migrating it to a new system: make assumptions, test the assumptions, revise the assumptions, and repeat.”

He used the example of “pending” versus “approved by purchasing but not yet received”. This sparked a lively debate about the terminology – is it accurate to say that “pending” is Context Sensitive, and “approved by purchasing…” is Context Insensitive? Most of us seemed to agree that the second example was more useful — although Len Shapiro made the case that “pending” wasn’t so bad. He made the case that we just needed a Data Dictionary to explain it in the application. James brought up that programmers and businesses are lazy, and are unlikely to keep a piece of documentation like that up-to-date. There was a lot more discussion about this.. but at that point, I think I asked Gab for another drink, and then got Len a second drink.

In the third segment, Jeff went over two ways that the SQL standard treats NULLs – as UNKNOWN or as NO VALUE. In particular he shared these two statements:

test=# select sum(column1) FROM (values (1), (NULL)) t;
sum
—–
1
(1 row)

test=# select (1 + NULL) as plus;
plus
——

(1 row)

We chewed on that for a bit. In the first case, NULL is being treated as “no value”, and just ignored by SUM(). In the second case, NULL is being treated as UNKNOWN – as the NULL is not treated as the same data type (or domain) as ‘1’, and so the answer is NULL.

(oh boy, i hope i got that right, or there’s going to be a long thread to explain this :D)

So, then James brought up the 17 possible meanings for NULL, and how our puny human brains could not compute. Several people made exploding brain noises, we continued to talk about NULL and aggregate functions and how COUNT() seems to violate ALL the proper rules, and… whew.

—

It was almost 9pm, and we all decided it was time to retire to the Lucky Lab.

Len brought me up to speed on Tom’s latest GSOC work, and I said that I would help test out his graphical Planner analyzer tool. It’s something that helps you understand why it is that the Planner chose the plan it did. I’m sure that I’ll be able to speak more articulately about it after I play with it :)

This week in geekville.

PACUG‘s 2nd meeting. (Well, probably not their second meeting *ever*, but the second meeting since they started back up recently.) Phillip James gave a talk about Network Access Control, which I expected to be about things like 802.1x. Turns out I wasn’t reading the mail list closely enough – his talk was specifically about Cisco’s NAC appliance, including very detailed instructions on how to deploy it. He’s going to sanitize his slides a bit and pass them by the appropriate authorities & they should be publicly available next week.

Cisco’s NAC appliance (which is made up of several parts: a manager, a server, agents) sits in-line (usually; there are OOB options coming up) at various points on your network and controls who’s allowed to access which corporate resources. You can allow the same user different access permissions based on what type of system they’re on (eg Mac/Windows, *nix), who owns it (e.g., user’s personal equipment vs corporate), how they’re connecting (e.g., LAN, VPN), and the status of various applications on the machine (e.g., is their AV up-to-date). Licensing, of course, is dependent on the number of users & sites.

Four actions the NAC performs:
1. Identify the device & user
2. Enforce policies in a consistent manner (HR departments probably require consistent enforcement across all users in order for necessary disciplinary action to be taken)
3. Quarantine & Remediate non-compliant equipment
4. Configure & manage access policies

Steps to deployment (this is actually a useful checklist for *any* application deployment, IMO):
– gather your list of contacts. Phillip included an exhaustive list of all the contacts you need to have within the organization
– technical requirements analysis
– ops requirements analysis (eg, training, how many licenses)
– design phase
– lab testing (you have a lab, right?)
– field testing at select sites; rule of thumb is 10-15% of your final deployment
– production deployment

A good test plan:
– deploy it first as audit only. This gives you a baseline of compliance before you start enforcement. (And allows you to troubleshoot piece-by-piece instead of just dropping it all in there at once.)
– next: checks without enforcement. User is given a popup notification that they are out of compliance with whatever policy, and they have the option to correct the issue now or bypass it.
– once compliance is at an agreed-upon level (say, 85-90%), enable checks with enforcement. From my experience, if you wait for a level of compliance from users (esp if they’re given the option to bypass it), you’ll be waiting a loooooooong time. My preference would be to give them a cutoff date instead.
– make sure the test plan has an explicit definition of a “successful” test.

Phillip finished up with my favorite part of any presentation: War Stories. (He called them “Tips from the Field”.) One gotcha to consider is that tcpdump will only show traffic destined to the NAC appliance, not through it.

I would have liked to have heard a bit more about how the NAC actually operates – it sounds like the connection gets transferred around between different VLANS – one for authentication, then to a different one depending on the permissions granted to the user, which is pretty intriguing.

After the presentation, we had a short “general networking Q&A”. I should have come prepared with questions. (Maybe about monitoring Metro Ethernet connections.) A few Cisco SEs were present to answer questions.
Some items of interest to me:
Apparently there is a bug/feature in IPv6 reflexive ACLs. Good to know in advance.
VPN Tunnel issues: sometimes the tunnel will show as up, but is not passing traffic, and you have to bounce it (“clear crypto sa” IIRC). It would be nice to have a warning about this situation. Supposedly the interface should be flapping if this is going on, which should generate an SNMP trap. I need to look into that.

The next meeting is about Virtual Switching System (VSS), and I’m sad I’m going to miss it. (Due to the holiday weekend, the meeting’s been moved to July 9th, which conflicts with PDX.pm.)

This week in geekville.

PDX.pm! Held at FreeGeek. 16 attendees this week. A couple of new people.
—
Announcements:
– Call for shirt designs – must be in sometime in the next month.
– Barcamp is coming up.
– Something about Summer of Code? ;)

The “Say Your Name Game” topic tonight was to answer one of two questions:
– Where are the younger perl users?
– What can *you* do in 20 minutes a day (or two hours a week) to get Perl 6 out the door?

The general consensus on the younger users question is that Perl needs to be “cooler”. Programming games was a popular suggestion on how to change Perl’s image a bit; Wil suggested “more sex”.
Naturally the “20 minutes a day” question resulted in a lot of riffs on “rock hard abs”. Which eventually became “rock hard Parrot”.
—
Meeting:
Serious projector issues, as seems to happen every 3rd or 4th meeting. Selena upped the ante from “whiteboard with a non-dry-erase marker” to “VNC with all laptop-bearing attendees.”

IM IN UR DATABASE, Selena Deckelmann (a talk about procedural languages in PostgreSQL)

(This will be short because I want to be lazy & just link to her slides.)

A procedural language (such as PL/PgSQL, PL/Perl, PL/Python) is basically the expansion set for the SQL toy – you can create functions that do fun things like date calculations (what I usually use it for) or loop over a set of data. Essentially, you are letting the database do what it’s good at (handle data).

The first example we looked at was PL/LOLCODE. Mainly because it’s (currently) a very simple example, and fun. (Fun is the theme here.) You can only return one row at a time with PL/LOLCODE right now though. Selena suggests reading the docs; Joshua Tolley is doing an excellent job of documenting what he’s doing and this would be a valuable manual for anyone else who wants to write a procedural language for PostgreSQL.

Then we moved on to PL/PgSQL. Same basic format (CREATE function, RETURN, double-dollar quoting etc), but much more robust, of course.

This week in geekville.

PDX.pm = 21 attendees, SRO.

Announcements:
– Call for shirt designs – we didn’t have shirts last year. I hope there’s enough interest for Josh to get women’s sized shirts again.
– We need speakers for upcoming meetings. (I almost got pegged for March. This got me to thinking of what could I possibly talk about at the Perl meeting…)
– PDXPUG‘s next meeting will be Extreme Database Makeover featuring RT.
– This week’s lottery numbers: 6 15 21 32 7

Schwern‘s “Skimmable Code” talk started out with defining “lexical encapsulation”, a term I was curious about because I’d googled it before the meeting and the 3rd result was the meeting announcement; there wasn’t a really clear definition available. So: lexical scope refers to the scope of a block of code, and encapsulation means nothing leaks out (or in). Lexical encapsulation means that you can read all the code on the screen (a single screen, we’ll get to that in a moment) & know what it does, without having to look at the rest of the code to figure out what & where some variables are/do/come from.

This brings us to the concept of “skimmability” – how much of the code do you need to understand to do what you need to do (refactor, tweak, add new functionality). Can you get in, fix it, & get out without mucking up something else? You want small, self-contained blocks of code that will fit on a single screen.

Example of something bad: a global var in a subroutine – you have no idea where else it is used or what it does without reading the entire code. (Note to – I have the mother lode of “Pathological Code Base.” Let me know next time you need some examples. :) )

Tips:
– Andy Lester’s ‘ack’ is very useful for searches through multiple modules.
– Fun way to check your code blocks: pull the block out, run the compiler against it. That will tell you what needs to be passed in & out.
– version control comments: why you did it. What you did will be visible in the diff.
– enhance skimmability by using vertical whitespace; different things should be separated, similar things grouped.

My notes end here, as I had to leave early.

eta: Here’s something else I found that is relevant to this discussion:
Andy Lester – The world’s two worst variable names

Things I need to look into:
– Devel::Cover
– can I set cvs to ignore whitespace