28 February, 2014

SCALE12x Report

by gorthx

Last week I headed to sunny LA for the SoCal Linux Expo, aka SCALE. This was my first time at SCALE & I’ll definitely be going back (it reminded me of my first OSCON several years ago).

I’ve wanted to go to this conference for a few years now & finally had a reason to go: I gave a talk at LAPgDay (aka the Postgres track) on Friday. The Pg talks were mostly SRO throughout the day! That’s pretty exciting. I had a lot of fun giving my talk about Pg’s vacuum & autovacuum processes & got some good feedback about it. Which reminds me: if you haven’t already, please fill out the speaker survey; there’s a link on the back of your badge.

On Saturday, I spent most of my time in the Postgres booth (of course) with friends old and new. I’m just about finished following up with folks who had questions for me, so if you haven’t heard from me yet, hang tight. :) I managed to take one quick spin through the Expo Hall, and have to say I am very intrigued with the DIYBio movement, and hope the PDX group gets something going soon. I also picked up yet another copy of The Manga Guide to Databases at the NoStarch booth, and gave it away the day I got home. I can’t seem to keep a copy of that book for myself!

Two Pg tips I picked up this weekend (thanks Steven & Joe):
1. “TABLE my_table;” is a handy alias for “SELECT * FROM my_table;”

2. The ‘AS’ keyword is not required to specify a field name for an alias. Apparently it’s been this way for a while, and I just managed to avoid being bitten by it until last weekend when I misplaced a comma.

21 February, 2014

Ideas for future PDXPUG workshops

by gorthx

The recent Streaming Rep Lab at PDXPUG was such an excellent learning experience. I really want to continue having these sessions.

Our definition of a “workshop” is pretty loose. There’s a basic list of topics we want to cover, but no real agenda or leader; it is truly a group effort.

Here are some ideas I’m toying with for future workshops.

- Choosing a High Availability plan
- Different ways to take backups
- Troubleshooting slow queries
- Monitoring (this could easily be a series on its own)
- Disaster Recovery
- Oh no, somebody deleted pg_xlog
- Transaction wraparound
- Postgres on zfs
- Postgres packet captures
- Tour of contrib modules
- Foreign Data Wrappers
- Benchmarking changes to GUCs, e.g. maintenance_work_mem.

Tags: , ,
14 February, 2014

I’m speaking at SCALE next week

by gorthx

I’m giving my Autovacuum talk at SCALE next week. There’s a whole track of Postgres talks on Friday.

I’ll also be hanging out at the Pg booth on Saturday – come by & say hello!

7 February, 2014

PDXPUG labs – Streaming Rep Saturday

by gorthx

Several years back, a new fellow on a ride with my bike club had a rather serious crash. I’ll spare you the gory details here1, other than to say that he was very lucky we had two Wilderness First Responders, a nurse, and an Army medic with us. The experience made quite an impression on me, and I got a Wilderness First Aid certification a few months after that.

The way the NOLS courses work is you have lectures about eg “how to splint a broken arm with found materials”, then you divvy up into pairs or small groups for practice. One person is the “victim” and the other has to fix them. The first time I took the class, I “killed” most of my patients. I did much better the next few times, and have had several occasions to be grateful for these skills.

Over the past year, I’ve been thinking a lot about how training like this applies to our everday work as IT specialists. A lot of us inherit systems that may not be documented, or maybe the backup’s not actually running, etc. And then disaster strikes, and we think, “well, I know *in theory* what I’m supposed to do”. Good luck!

My point is, we need to practice this stuff. Preferably in a low-pressure environment where it’s ok to make mistakes. This is right up there with the concept that “you don’t have a backup unless you’ve successfully restored it recently”. Do many places encourage (or even allow) you to practice restores, though? I know of a few that do, but for the most part people are pretty silent about this and I suspect it’s because nobody’s doing “what’s right”.

This was my reasoning behind starting a lab series with PDXPUG. Streaming Rep seemed like a good candidate for the first lab; several PDXPUG members expressed interest, and I personally was feeling a bit rusty because I haven’t had to actually configure it since 9.1.

My goals were:
- create a checklist of steps to follow to set it up. This is one of those things you probably won’t have to do very often, but why not make it as streamlined as possible?
- figure out if it’s actually working correctly
- practice a fail over, practice restoring a standby, etc

The group recap is here.

The checklist is up on github; the group found a few mistakes in my original.

Additional cool tidbits I learned:
wal_level has to be hot_standby on the master, or the standby won’t start up. (Postgres is pretty good about letting you know exactly what the problem is, though.) I was unable to get around this without completely reimaging the master; I would like to dig into that further and see if there’s a shortcut.

SELECT * FROM pg_stat_replication; shows you the current rep status. Combine this with \watch for extra entertainment. Available on the master, and on standbys if you’re cascading. (Hmm, setting up cascading rep might be another fun lab…)

pg_xlogfile_name translates ID -> WAL name, also only available on the master.

\df *log* …somehow, all these years I have missed that you can pass globs to the \ commands.

You can actually ship and stream WALs, but be prepared for 2X the network load.

CREATE USER applies login privs; CREATE ROLE does not.

Items that we’ll be looking at during Streaming Rep part 22:
- what is the “right” # for max_wal_senders – is it just the number of standbys? Any reason you’d want to have more?
- what causes that “FATAL: database is starting up” problem, and how do you fix it?
- how the heck do we monitor this thing, and what is reason for alarm
- the “human-readable” WAL transform

Yes, there will be more labs. Watch the PDXPUG blog for announcements. If you’re in the Portland area, please come join us.

1 – I will tell them over beer, though.
2 – Thursday Feb 27, sign up here

Tags: , ,
17 January, 2014

Model 66 – treadle

by gorthx

Part 1.
Part 2.


The treadle
Again, disassemble this into its component parts so it’s easier to work on. This was the perfect excuse I needed to replace the Honkin Huge Screwdriver that disappeared with my ex-husband. A lot of these bolts were stuck pretty firmly; PB Blaster wasn’t helping, so I used the old trick of using a crescent wrench to turn the screwdriver. Voila.

My original plan was to scrub the rust and paint off of “the irons” (as they are called) and then spraypaint it black. I spent a couple of hours just on the screws, and when I discovered how much rust there was under the paint that was covering the pedal, well… when my sweetie suggested (again) “You know, you could just take this to the powder coaters that is right around the corner“, I agreed. Best $100 I spent this year.

It’s a simple machine, but again, take lots of photos when you’re taking this thing apart. There is a pretty good text guide here.

Order of reassembly:
- attach belt control to dress guard, to minimize the number of pieces you need to keep track of.
- attach the belt guide to the center brace
- attach center brace to legs. It’s easier to do this if it’s upside down. Put the bottom bolts in first (they’re at the top if you’re doing it upside down), then the top bolts. The side pieces are interchangeable so there are two sets of holes for the top. You want the center brace angled toward the back. (This is where photos of the disassembly come in handy, especially if it’s been 3 weeks since you’ve had it apart & have just gotten the big pieces back from being powder coated.)
- like any other project involving multiple fasteners, get each bolt threaded just a bit before you start tightening them.
- connect the pitman to the drive arm.
- attach drive arm to center brace
- attach dress guard. I don’t have the belt thrower reattached correctly yet; the spring has lost a lot of its spring over the past century.
- attach pedal to center brace & adjust the cone bearings (put those bike maintenance skills to use!)
- connect pitman to pedal

Connecting the pitman to the drive arm was the hardest part; the bearing housing + bearings had to be reassembled in place around the drive arm. This requires very sticky grease and/or an extra set of hands to help chase bearings around the workbench.
bearings in pitman housing

So! This machine is up & running new, and I sewed actual fabric (a quilt block) with it this week. I spent a good amount of time learning to work the treadle without thread in the machine, and then stitching on paper, before I tried to work on a project. It is a lot like learning how to drive :).



10 January, 2014

Model 66 – head

by gorthx

Part 1.


As you can probably guess, I got this baby home and hooked up a belt and discovered it wouldn’t sew. The treadle wouldn’t power the head, indicating it was gummed up somehow. The handwheel wasn’t completely stuck, so I figured “how hard could it be?”

Tools and tips:
The best advice I read, besides the obvious “take pictures from every angle, and take more than you think you could possibly need”, is to keep screws in their taps as much as possible. Some of these babies are tiny, and doing this helped me keep track of them.

This project mat from ifixit was invaluable. It helped me keep track of parts while I was working, and because I took pictures, I have a labeled record of all the parts.

There are a lot of good manuals and instructions online, so I won’t give a blow-by-blow here other than specific trouble spots I encountered. (If you use the manuals from Tools for Self Reliance, please consider making a donation.)

Tension mechanism: this just didn’t look right to me1, and once I found a manual online with images I could actually pick up details from2, I realized it *wasn’t* my imagination, the spring really wasn’t in the right spot. And thus began the extraction of the tension housing: Every few hours I’d drip a little more PB Blaster in the set screw hole, and tap it gently with a mallet and block of wood. Finally, on day 3 of this, I got it to move. A few more cycles of PB Blaster-tapping-waiting, and I smacked it right on out. To get it completely loose, I pushed it almost completely inside the machine by hitting it with the mallet & wood block. Then used the handle of a plastic toothbrush to pop it out from the inside.

Bobbin winder: this was coated with enough grease and dirt that none of the parts would spin and the spring-loaded stop latch wouldn’t even move. This required a complete teardown, cleaning, and relubing. A camera is the most essential tool here. The rubber bobbin wheel is much easier to remove/replace when the bobbin winder is not attached to the machine. It’s so much easier that it’s worth taking off the entire bobbin winder just to replace that little wheel, even though it is a major pain in the kiester to reattach the winder.

Bobbin and hook area: I tried for quite a while to remove the bobbin latch, because I’d read “absolutely do not remove this screw”, then found out that there are two different styles of bobbin latch – one needs to be unscrewed, the other doesn’t. They don’t actually look all that different. Use the manual from Tools for Self Reliance to figure out which one you have; you could break the bobbin latch if you do this wrong.

Underneath: I took this as far apart as I dared; I didn’t want to mess with the timing, because the machine would stitch when cranked by hand. PB Blaster and sewing machine oil were the key here. This was probably the easiest part of the whole refurb. When this was back together, I could give the handwheel a little flick and it spun freely for at a few full rotations.

Next: the treadle!

1 – it helps that I know my way around a sewing machine.
2 – a lot of the manuals available now are scans of photocopies of originals. The TFSR are the best I’ve found, as they are not reproductions.

3 January, 2014

Singer model 66 treadle

by gorthx

And now for something completely different…several months back I picked up a 1912 Singer model 66 “Redeye” off of Craigslist. I was looking for a completely different machine, but I’ve wanted a treadle for a long time, and from the photos it looked in ok shape. One thing led to another and I found myself driving a ways east of town on a reasonably nice day that I otherwise would have spent biking, to check out this machine.


The good:
- the treadle pedal & drive wheel moved freely
- it has a metal pitman; the wood ones are prone to drying out, cracking, and often end up missing
- the (oak) cabinet’s ornamentation is intact; sometimes the wood carvings come off, or drawers are missing, etc
- the handwheel on the machine head1 moved
- the decals on the head were in pretty good shape overall; they tend to get rubbed off with use. This machine is over 100 years old, after all.

The bad:
- layer of grime on everything
- paint splatters all over the cabinet; it was going to need stripping completely.
- no belt so I couldn’t actually test it

I’ve since decided that one should examine these things in daylight and not in a warehouse. As an alternative, flash photos will highlight problem areas better than you can see them with your eyeballs. And next time, I’ll bring a belt with me so I can determine whether the machine runs or not. That said, I’m glad I bought it because fixing it up was a good experience and I now have a machine that sews better than my formerly TOTL European machine :koff:Bernina2:koff: for much less money.

And yes, these things are “heavier than a dead preacher”3 so come prepared with additional muscle or tools to tear it down so you can take it home easily. Or both.

There are 4 projects here:
- cabinet refurb
- treadle refurb
- head external refurb
- head internal refurb

All of these took significantly more time than I expected; I really didn’t know what I was getting into. To keep posts to a reasonable length, I’ll talk about the cabinet & head outside first, and the treadle and head internals separately.

A full photo set is on flickr.

The head – outside
As I mentioned, this machine’s decals were in pretty nice shape. I was interested in keeping them that way, so cleaning the outside of the head basically just involved rubbing them very very gently with oil in order to remove the grime. My BF actually volunteered to do this, but then, he used to detail cars, and has the patience for this kind of thing. This is probably anathema to some people, but I didn’t want to look at the “patina” (read: yucky old brown finish), so off it came with some rubbing alcohol. A coat of carnauba wax made it shine again. Metal polish took most of the rust off of the chrome pieces, and that was that.

The cabinet
In addition to the layer of grime, this machine appeared to have been left uncovered while someone nearby spraypainted. The treadle pedal was almost solid white and the cabinet was speckled all over; there was even a handprint on the back.

I broke the cabinet down into its component pieces and used Citristrip to remove the old finish. Citristrip does a much better job indoors at temps over 60*F than in a 40*F garage. This was my first time stripping carved wood, and I don’t intend to repeat the experience.

My method:
- glop on the citristrip & wait until it’s ready. If it’s starting to get clear & a little dry, like it’s “setting up”, it’s getting close.
- with a plastic scraper, remove as much of the stripper & old finish as you possibly can. Old credit cards work great for this, if you are too cheap to go buy something you are just going to throw away anyway.
- around the carved wood, I used a brass brush to get rid of the stripper. This is not recommended for soft woods, and probably wasn’t advisable for oak – IOW, don’t do this. A stiff nylon brush would have been easier on the wood, but didn’t do jack about removing the crud.
- remove remaining stripper with 00 steel wool soaked in mineral spirits (odorless will work fine for this.)
- remove more remaining stripper with a rag soaked in mineral spirits
- remove still more remaining stripper with another rag soaked in mineral spirits (this stuff was hard to get rid of)

The PNW is a very humid environment, and the veneer had buckled and pulled away from the core over the years. Just few small chips were missing, so I opted to re-glue the loose veneer. I only own a couple of clamps, so this step took several days.

Once the wood was dry, I filled the screw holes, sanded that down, and then all that was left was a quick buff with 0000 steel wool (which I’m told is too fine for oak, but it made a palpable difference), a swipe with tack cloth, and it was ready for finish. I use Tried & True Danish oil on almost all of my wood furniture; it’s not as protective as polyurethane, but it’s a lot easier to apply and I love the way it looks. And smells. I put two coats on for now, and then reassembled the cabinet.

Next up: head internals.

1 – The head is the part that actually does the sewing.
2 – Alternate title: “Nothing sews like a Bernina… except this Singer I have from 1912.”
3 – http://www.granny-miller.com/treadle-sewing-machine-advice/

29 November, 2013

OSX, VMWare, CentOS, and postfix

by gorthx

(Original title: “mail server arrrgh”)

I’m running VMWare Fusion on my Mac. I often run multiple VMs for testing Postgres on various OSes, and decided it would be fun if I could get system emails from the guests on the mac host. Yeah, “it would be fun”. Even though I was working with Postfix, this task gave me flashbacks to my sendmail experiences many years ago.

This is what I had to do to get it working. This isn’t in the order I did all the steps; there was a ton of trial, error, and wtfery that went on here. Also, this is on a machine that’s behind a few firewalls; probably not something you want to configure on a server that actually has a port open on the Internet. Although it seems that most of the mailserver-related footguns are no longer enabled by default. Caveat Emptor, or something.

Host: Mac OSX Lion 10.7.5
Guests: CentOS 6.4
VMWare: Fusion 4.1.3

Set up postfix on the mac host; starting with the instructions here: http://www.phase2technology.com/blog/how-to-enable-local-smtp-server-postfix-on-os-x-leopard/

This “worked” in that the simple “telnet to port 25″ test worked (see “Useful tools and commands” below), but I had to make some additional changes to receive mail from my guest OSes.

On the Mac host, I made some changes to /etc/postfix/main.cf:
myhostname = princess
mydomain = localdomain
mydestination = $myhostname, $myhostname.$mydomain, localhost.$mydomain, localhost, mailhost.$mydomain
inet_interfaces =,
mynetworks =,

The value for inet_interfaces is the IP of my vmware interface, obtained from ifconfig. You want the vmware8 interface and it should be a 192.168 address. I just added the whole vmware subnet to mynetworks parameter. You can read more about these parameters in main.cf; it is surprisingly well-commented.

I also commented out imap_submit_cred_file line as discussed here https://discussions.apple.com/thread/3247974. I’m not going to bother with this for local use.

‘postfix reload’ did not apply these changes for me; I had to explicitly stop and restart it1:
postfix stop
postfix start

On the guest:
First, I added the vmware host IP to /etc/hosts: princess princess.localdomain mailhost mailhost.localdomain

This worked via the ‘telnet to port 25′ test method, but regular email wasn’t getting delivered; in fact, it was bouncing. Looking into the mail queue, I discovered it was trying to use a completely different IP for its mailhost.

I checked out the ‘hosts’ value in /etc/nsswitch.conf (just grep hosts /etc/nsswitch.conf) and determined that it was set to ‘files dns’, which ordinarily indicates that /etc/hosts should override dns from the vmware dns server. That wasn’t what was happening. A bit of investigation revealed that postfix was already running2, so I had to dink with it on the guests as well.

Fixed by editing /etc/postfix/main.cf (on the guest) to include this:
relayhost = mailhost.localdomain


Thanks to mjm for keeping me sane during this.

Useful tools/commands:
‘postfix status’ # see what postfix thinks it’s doing and get its pid, since the processes aren’t named postfix and therefor ‘ps -ef | grep post’ won’t show them :)

netstat -an | grep ‘\.25 ‘ # see if anything’s listening on port 25:
(master *) :::-->netstat -an | grep '\.25 '
tcp4 0 0 *.* LISTEN
tcp4 0 0 *.* LISTEN

mailq # see status of queued mail messages

[tail|vi|whatever] /var/log/mail.log # see connections, postfix startup/reload, etc

SMTP test program, something like this http://www.port25.com/how-to-check-an-smtp-connection-with-a-manual-telnet-session-2

You may want this as well: http://topicdesk.com/faqs/os-x-server-mail-services-faq/69-how-do-i-completely-disable-postfix-and-cyrus

1 – I’ve since been told it’s better to use launchctl to control processes.
2 – This surprised me, given the other things that are locked down/not available on the CentOS default.

22 November, 2013

My Current “Most Useful” Android Apps

by gorthx

I’ve only been using CamScanner for a month or so, but it’s already saved me at least an hour of dealing with a flatbed scanner. It’s very easy to photograph, crop, convert to B&W, convert to a .pdf and send them off to someone (or share via DropBox or some other method). I recommend doing the short tutorial to get a feel for what the different icons mean.

The MultcoLib app was really slow when it was first released, but has improved dramatically. Search, hold, renew – all from your phone!

Much-awaited by Portlanders, the TriMet Ticketer still has some quirks, not the least of which is that some bus drivers don’t seem to know what to do with electronic tickets. Still, it’s super-convenient if you’re stranded somewhere. As long as it wasn’t Tri-Met that stranded you there.

I started using CardioTrainer to record my running workouts after my iPod nano died1. It really grew on me – I get a map of my route, it works for a variety of activities (running, mt biking, hiking, house cleaning2), it interfaces with a HRM, etc. Unfortunately, the only way to get access to your raw data (if you haven’t rooted your phone) is to upload it to the Noom website and use their export tool. This doesn’t work with my phone (HTC One), and the company has no plans for further development. So I’m looking for an alternative, because I really like being able to fold, spindle, and mutilate my data myself. etc Strava is completely unacceptable, for reasons I won’t go into here.

The Android Ravelry app Ravulous is the only paid app I have right now; I think it’s a dollar or two, and totally worth it for those spur-of-the-moment stops into the LYS. It does have some quirks:
- you can’t access your favorites
- you can’t update projects with * in the notes (known issue, see the Ravelry forums)
If you like to take a lot of WIP photos, I recommend getting the Ravelry Photo Uploader as a companion app.

1 – More about that in a later post.
2 – No, seriously.

8 November, 2013

nook Simple Touch 18-month review

by gorthx

I’ve had my simple touch for well over a year now, and I’m still pretty happy with it, especially since I’m traveling a bunch.

Initial review
6 month review

Updates on specific issues:
- I haven’t had any further problems with mangled print; I suspect that was a feature of the particular material I was reading.
- I never did get ebrary to work correctly, but my library doesn’t offer that anymore anyway, so it’s moot.
- I said I would purchase one again (and they’re going for about $70 right now), but I’d rather get a kobo just for philosophical reasons. I like the “open ebook format” idea.

Speaking of kobo, yes you can read kobo books on your nook. You just need to download them in the Adobe epub format, and use ADE to load them. (This works just fine with Wine on Ubuntu.)

My library offers two e-lending options: OverDrive (the new incarnation of library2go) and 3M Cloud Library. They’re both now searchable from my library’s catalog and from the lender’s website, which makes finding stuff to read much simpler. I prefer OverDrive; downloads from the 3M Cloud Library are quirky and very slow (sometimes taking multiple hours), and the app crashes frequently. I haven’t tried running it on Windows yet; my problems may be due to using it with Wine.

Library2go used to provide a way to “return” ebooks before the lending period was up; no more. 3M doesn’t allow that either. If you are particularly voracious you will bump up against the 5 item limit in short order. Checked-out materials become unreadable on your device once the lending period is up. (Other e-lending solutions I’ve tried allow you to keep reading the book until you re-sync your device.) This left me stranded 30 pages from the end of a book I was really enjoying. Boo.

The nook itself has started getting a bit flaky: the touchscreen randomly doesn’t work, and sometimes the hard buttons cause it to skip ahead multiple pages. I cleaned it with a soft toothbrush & eyeglasses cloth, as discussed here, and that seems to have cleared it up[1]. If not, there’s always the option to take it apart and put it back together again.

One strange thing I’ve noticed is ADE shows a lot of “missing” books that don’t show up on a regular directory listing of the nook. They seem to be duplicate file listings; even if I delete them through ADE, they show back up next time I connect the nook. Still trying to figure that piece out.

1 – This is what happens when you take electronics camping.


Get every new post delivered to your Inbox.