Posts tagged ‘LUGs’

7 February, 2014

PDXPUG labs – Streaming Rep Saturday

by gorthx

Several years back, a new fellow on a ride with my bike club had a rather serious crash. I’ll spare you the gory details here1, other than to say that he was very lucky we had two Wilderness First Responders, a nurse, and an Army medic with us. The experience made quite an impression on me, and I got a Wilderness First Aid certification a few months after that.

The way the NOLS courses work is you have lectures about eg “how to splint a broken arm with found materials”, then you divvy up into pairs or small groups for practice. One person is the “victim” and the other has to fix them. The first time I took the class, I “killed” most of my patients. I did much better the next few times, and have had several occasions to be grateful for these skills.

Over the past year, I’ve been thinking a lot about how training like this applies to our everday work as IT specialists. A lot of us inherit systems that may not be documented, or maybe the backup’s not actually running, etc. And then disaster strikes, and we think, “well, I know *in theory* what I’m supposed to do”. Good luck!

My point is, we need to practice this stuff. Preferably in a low-pressure environment where it’s ok to make mistakes. This is right up there with the concept that “you don’t have a backup unless you’ve successfully restored it recently”. Do many places encourage (or even allow) you to practice restores, though? I know of a few that do, but for the most part people are pretty silent about this and I suspect it’s because nobody’s doing “what’s right”.

This was my reasoning behind starting a lab series with PDXPUG. Streaming Rep seemed like a good candidate for the first lab; several PDXPUG members expressed interest, and I personally was feeling a bit rusty because I haven’t had to actually configure it since 9.1.

My goals were:
– create a checklist of steps to follow to set it up. This is one of those things you probably won’t have to do very often, but why not make it as streamlined as possible?
– figure out if it’s actually working correctly
– practice a fail over, practice restoring a standby, etc

The group recap is here.

The checklist is up on github; the group found a few mistakes in my original.

Additional cool tidbits I learned:
wal_level has to be hot_standby on the master, or the standby won’t start up. (Postgres is pretty good about letting you know exactly what the problem is, though.) I was unable to get around this without completely reimaging the master; I would like to dig into that further and see if there’s a shortcut.

SELECT * FROM pg_stat_replication; shows you the current rep status. Combine this with \watch for extra entertainment. Available on the master, and on standbys if you’re cascading. (Hmm, setting up cascading rep might be another fun lab…)

pg_xlogfile_name translates ID -> WAL name, also only available on the master.

\df *log* …somehow, all these years I have missed that you can pass globs to the \ commands.

You can actually ship and stream WALs, but be prepared for 2X the network load.

CREATE USER applies login privs; CREATE ROLE does not.

Items that we’ll be looking at during Streaming Rep part 22:
– what is the “right” # for max_wal_senders – is it just the number of standbys? Any reason you’d want to have more?
– what causes that “FATAL: database is starting up” problem, and how do you fix it?
– how the heck do we monitor this thing, and what is reason for alarm
– the “human-readable” WAL transform

Yes, there will be more labs. Watch the PDXPUG blog for announcements. If you’re in the Portland area, please come join us.

1 – I will tell them over beer, though.
2 – Thursday Feb 27, sign up here

Tags: , ,
15 October, 2009 meeting notes

by gorthx

Jeff (aka @duckyd) gave a presentation about CPAN Awesomeness at last night’s

Slides are on github!  Your homework is to find them. :)  Here are some highlights.

There are over 16K modules on the CPAN as of 11 Oct 2009.  Wow.

Jeff’s recommended changes from the default cpan shell configuration:
– auto_commit 1
– prerequisites_policy follow
– build_requires_install_policy yes
– prefer_installer MB (Module::Build)
– change your make_install_make_command and mbuild_install_mbuild_command to include your sudo command.

Spend 10 minutes & give something back to CPAN every time you install a module:  Simply set up CPAN::Reporter!

– make sure that you set cc_author to ‘no’.  (In the latest version, that’s the default.)
– set it to prompt you to edit/send the report if the tests fail.  This way you can judge if the failure is due to your own
– you can set the transport value to a file to run reports without sending them.
– for help:  perldoc CPAN::Reporter::Config

For automatic continuous testing, set up CPAN::Reporter::Smoker.  (Doesn’t actually install anything, justs runs tests.)

– don’t run it as root;  you are the canary in the coal mine.
– create a dedicated user that has essentially no privs on your machine.
– run it on a separate Perl install (core modules only).
– this is a cool place to use that RAND option for prefer_installer in CPAN.

Another cool tip that I REALLY DIG because I have systems with multiple perls & users associated with them:
Set your shebang line to:

#!/usr/bin/env perl

4 September, 2008

We’re speaking at PLUG tonight!

by gorthx

Selena & I will be giving a modified version of our OSCON talk at PLUG tonight.  Time & location on the PLUG website.  See you there!

Tags: ,
11 July, 2008

This Week in Geekville.

by gorthx

This week at, chromatic gave an excellent talk about “how not to fail”.  There was quite a bit of discussion of Godzilla attacks and the calculus thereof, which I don’t think I can adequately capture.  “You had to be there”.  Anyway, on to my notes!

The keys:
– Recognize points of failure before the actual failure occurs.  The idea being to prevent the failure.
– Perform root cause analysis.  Your biggest problem may not be the most annoying, obvious, or easiest to solve.
– Do the simplest, fastest thing you can do to ease the problem.  You’re conducting an experiment to see if it has any effect on the problem.

Some examples of places you can get hung up:
– Waiting for a decision from someone else.  That someone else could be you.  Self-awareness is critical.
– Waiting for someone else to do something.  This is usually a sign of a technical bottleneck, such as a complicated process.  “The only person who can ____ is too _____.” (The blanks on this slide had my hopes up for madlibs.)
– Becoming demotivated (which is a word I thought only I used) by tedious or annoying tasks.  (Or people.)  Get rid of the annoying thing.  (Please obey all local laws.)  Motivational tip:  leave something to do tomorrow!  That way you start the day on a high note.  [I do this all the time – in fact, I keep a list on hiveminder of “instant gratification projects” – things I can do in 15 or 30 minutes – and any time I’m frustrated & need to feel like I’ve accomplished something, I do one of those.]
– Lacking direction.  Try creating a one-sentence problem statement.

Time-based releases force you to get your release process under control – and maybe even automated!  It gets you in to a rhythm.  (My rhythm, since it’s just me and I don’t really have “releases”, is to have “Refactoring Fridays”.)

Tips that especially apply to FOSS projects, where just about everyone is a volunteer:
– Make it easy for people to get involved
– Encourage discussion
– If someone new has contributed something, but it’s not quite up to snuff, tell them (nicely) how they could improve next time

8 July, 2008

We’re speaking at OSCON

by gorthx

Come hear what we have to say about Building a User Group!

Lesson 0

Tags: ,
20 June, 2008

PDXPUG Meeting recap – Logic and Databases with Jeff Davis

by gorthx

[ Jeff’s slides are available at: ]

Last night’s meeting was Jeff Davis’ last presentation as a Portlander (for a while at least!). He is leaving his job at Laika to pursue streaming database nirvana at Truviso, a company whose database product is based on PostgreSQL. There’s a bunch of whip-smart people working there, and Jeff will fit right in. If you’ve got other questions about what Truviso does, feel free to ask Jeff.

Our “question of the day” was – What are your plans for the summer? We had some great answers — including that Len’s son Ari is hosting the morning NPR news show for the next two weeks! Go Ari! It sounds like James *might* be leaving us for Redmond — stay tuned for more news on that. Gab will be riding her bike, and other people are going to just enjoy the fact that it finally seems like it *is* actually summertime.

Mark Wong is working with Joshua Drake (and Gab?) to install our new Performance Lab downtown on Saturday. Look for updates and announcements about what those servers will be used for from Mark and JD next week. HP donated two of the machines, and I believe that one other came from Sun.

We also had cocktails served by Gabrielle. THANK YOU, GAB!

Jeff’s presentation started with using logic to help express in English basic SQL concepts. For example, JOIN can be expressed with AND, and UNION can be expressed with OR. Thinking about what questions you’ll be asking about the data, and also forming those questions and statements of fact in ways that allow the assignment of clear ‘true’ or ‘false’ values will help enormously in ensuring that your data structures and SQL queries work the way that you and your customers expect.

At the risk of butchering the next discussion, here’s my take on the second part: Jeff spent quite a bit of time talking about the merits of using statements of fact when describing states, and how focusing on the truth values helps you test your assumptions about results from queries. His closing words on this topic: “Use an iterative process to clean up data before migrating it to a new system: make assumptions, test the assumptions, revise the assumptions, and repeat.”

He used the example of “pending” versus “approved by purchasing but not yet received”. This sparked a lively debate about the terminology – is it accurate to say that “pending” is Context Sensitive, and “approved by purchasing…” is Context Insensitive? Most of us seemed to agree that the second example was more useful — although Len Shapiro made the case that “pending” wasn’t so bad. He made the case that we just needed a Data Dictionary to explain it in the application. James brought up that programmers and businesses are lazy, and are unlikely to keep a piece of documentation like that up-to-date. There was a lot more discussion about this.. but at that point, I think I asked Gab for another drink, and then got Len a second drink.

In the third segment, Jeff went over two ways that the SQL standard treats NULLs – as UNKNOWN or as NO VALUE. In particular he shared these two statements:

test=# select sum(column1) FROM (values (1), (NULL)) t;
(1 row)

test=# select (1 + NULL) as plus;

(1 row)

We chewed on that for a bit. In the first case, NULL is being treated as “no value”, and just ignored by SUM(). In the second case, NULL is being treated as UNKNOWN – as the NULL is not treated as the same data type (or domain) as ‘1’, and so the answer is NULL.

(oh boy, i hope i got that right, or there’s going to be a long thread to explain this :D)

So, then James brought up the 17 possible meanings for NULL, and how our puny human brains could not compute. Several people made exploding brain noises, we continued to talk about NULL and aggregate functions and how COUNT() seems to violate ALL the proper rules, and… whew.

It was almost 9pm, and we all decided it was time to retire to the Lucky Lab.

Len brought me up to speed on Tom’s latest GSOC work, and I said that I would help test out his graphical Planner analyzer tool. It’s something that helps you understand why it is that the Planner chose the plan it did. I’m sure that I’ll be able to speak more articulately about it after I play with it :)

Tags: , ,
6 June, 2008

This week in geekville.

by gorthx

PACUG‘s 2nd meeting. (Well, probably not their second meeting *ever*, but the second meeting since they started back up recently.) Phillip James gave a talk about Network Access Control, which I expected to be about things like 802.1x. Turns out I wasn’t reading the mail list closely enough – his talk was specifically about Cisco’s NAC appliance, including very detailed instructions on how to deploy it. He’s going to sanitize his slides a bit and pass them by the appropriate authorities & they should be publicly available next week.

Cisco’s NAC appliance (which is made up of several parts: a manager, a server, agents) sits in-line (usually; there are OOB options coming up) at various points on your network and controls who’s allowed to access which corporate resources. You can allow the same user different access permissions based on what type of system they’re on (eg Mac/Windows, *nix), who owns it (e.g., user’s personal equipment vs corporate), how they’re connecting (e.g., LAN, VPN), and the status of various applications on the machine (e.g., is their AV up-to-date). Licensing, of course, is dependent on the number of users & sites.

Four actions the NAC performs:
1. Identify the device & user
2. Enforce policies in a consistent manner (HR departments probably require consistent enforcement across all users in order for necessary disciplinary action to be taken)
3. Quarantine & Remediate non-compliant equipment
4. Configure & manage access policies

Steps to deployment (this is actually a useful checklist for *any* application deployment, IMO):
– gather your list of contacts. Phillip included an exhaustive list of all the contacts you need to have within the organization
– technical requirements analysis
– ops requirements analysis (eg, training, how many licenses)
– design phase
– lab testing (you have a lab, right?)
– field testing at select sites; rule of thumb is 10-15% of your final deployment
– production deployment

A good test plan:
– deploy it first as audit only. This gives you a baseline of compliance before you start enforcement. (And allows you to troubleshoot piece-by-piece instead of just dropping it all in there at once.)
– next: checks without enforcement. User is given a popup notification that they are out of compliance with whatever policy, and they have the option to correct the issue now or bypass it.
– once compliance is at an agreed-upon level (say, 85-90%), enable checks with enforcement. From my experience, if you wait for a level of compliance from users (esp if they’re given the option to bypass it), you’ll be waiting a loooooooong time. My preference would be to give them a cutoff date instead.
– make sure the test plan has an explicit definition of a “successful” test.

Phillip finished up with my favorite part of any presentation: War Stories. (He called them “Tips from the Field”.) One gotcha to consider is that tcpdump will only show traffic destined to the NAC appliance, not through it.

I would have liked to have heard a bit more about how the NAC actually operates – it sounds like the connection gets transferred around between different VLANS – one for authentication, then to a different one depending on the permissions granted to the user, which is pretty intriguing.

After the presentation, we had a short “general networking Q&A”. I should have come prepared with questions. (Maybe about monitoring Metro Ethernet connections.) A few Cisco SEs were present to answer questions.
Some items of interest to me:
Apparently there is a bug/feature in IPv6 reflexive ACLs. Good to know in advance.
VPN Tunnel issues: sometimes the tunnel will show as up, but is not passing traffic, and you have to bounce it (“clear crypto sa” IIRC). It would be nice to have a warning about this situation. Supposedly the interface should be flapping if this is going on, which should generate an SNMP trap. I need to look into that.

The next meeting is about Virtual Switching System (VSS), and I’m sad I’m going to miss it. (Due to the holiday weekend, the meeting’s been moved to July 9th, which conflicts with

Tags: , ,
9 April, 2008

This week in geekville.

by gorthx! Held at FreeGeek. 16 attendees this week. A couple of new people.

– Call for shirt designs – must be in sometime in the next month.
Barcamp is coming up.
– Something about Summer of Code? ;)

The “Say Your Name Game” topic tonight was to answer one of two questions:
– Where are the younger perl users?
– What can *you* do in 20 minutes a day (or two hours a week) to get Perl 6 out the door?

The general consensus on the younger users question is that Perl needs to be “cooler”. Programming games was a popular suggestion on how to change Perl’s image a bit; Wil suggested “more sex”.
Naturally the “20 minutes a day” question resulted in a lot of riffs on “rock hard abs”. Which eventually became “rock hard Parrot”.

Serious projector issues, as seems to happen every 3rd or 4th meeting. Selena upped the ante from “whiteboard with a non-dry-erase marker” to “VNC with all laptop-bearing attendees.”

IM IN UR DATABASE, Selena Deckelmann (a talk about procedural languages in PostgreSQL)

(This will be short because I want to be lazy & just link to her slides.)

A procedural language (such as PL/PgSQL, PL/Perl, PL/Python) is basically the expansion set for the SQL toy – you can create functions that do fun things like date calculations (what I usually use it for) or loop over a set of data. Essentially, you are letting the database do what it’s good at (handle data).

The first example we looked at was PL/LOLCODE. Mainly because it’s (currently) a very simple example, and fun. (Fun is the theme here.) You can only return one row at a time with PL/LOLCODE right now though. Selena suggests reading the docs; Joshua Tolley is doing an excellent job of documenting what he’s doing and this would be a valuable manual for anyone else who wants to write a procedural language for PostgreSQL.

Then we moved on to PL/PgSQL. Same basic format (CREATE function, RETURN, double-dollar quoting etc), but much more robust, of course.

18 February, 2008

This week in geekville.

by gorthx = 21 attendees, SRO.

– Call for shirt designs – we didn’t have shirts last year. I hope there’s enough interest for Josh to get women’s sized shirts again.
– We need speakers for upcoming meetings. (I almost got pegged for March. This got me to thinking of what could I possibly talk about at the Perl meeting…)
PDXPUG‘s next meeting will be Extreme Database Makeover featuring RT.
– This week’s lottery numbers: 6 15 21 32 7

Schwern‘s “Skimmable Code” talk started out with defining “lexical encapsulation”, a term I was curious about because I’d googled it before the meeting and the 3rd result was the meeting announcement; there wasn’t a really clear definition available. So: lexical scope refers to the scope of a block of code, and encapsulation means nothing leaks out (or in). Lexical encapsulation means that you can read all the code on the screen (a single screen, we’ll get to that in a moment) & know what it does, without having to look at the rest of the code to figure out what & where some variables are/do/come from.

This brings us to the concept of “skimmability” – how much of the code do you need to understand to do what you need to do (refactor, tweak, add new functionality). Can you get in, fix it, & get out without mucking up something else? You want small, self-contained blocks of code that will fit on a single screen.

Example of something bad: a global var in a subroutine – you have no idea where else it is used or what it does without reading the entire code. (Note to – I have the mother lode of “Pathological Code Base.” Let me know next time you need some examples. :) )

– Andy Lester’s ‘ack’ is very useful for searches through multiple modules.
– Fun way to check your code blocks: pull the block out, run the compiler against it. That will tell you what needs to be passed in & out.
– version control comments: why you did it. What you did will be visible in the diff.
– enhance skimmability by using vertical whitespace; different things should be separated, similar things grouped.

My notes end here, as I had to leave early.

eta: Here’s something else I found that is relevant to this discussion:
Andy Lester – The world’s two worst variable names

Things I need to look into:
– Devel::Cover
– can I set cvs to ignore whitespace