Saturday, February 20, 2016

The Vernal Pool Monkey Flower

The vernal pool monkey flower, Mimulus tricolor, was thought to be extinct. It reappeared in 1999 after a flood stripped the non-native sod from the big field West of our farm. Thousands of this tiny annual sprouted from a heretofore unknown seedbank hidden in the exposed soil.

It's a tiny plant, barely standing 3cm tall. It is not uncommon to see individuals where the bloom is significantly larger that the plant. They seem to prefer an open meadow habitat where the soil is mostly bare and having spent much of the winter under a shallow pool of water.

Seventeen years later, the sod has reasserted itself and the vernal pool monkey flower has vanished again. Hopefully, it waits in the soil for another flood to strip away the competition.

My rendition of this plant is a maze. There are two entrances and two exits marked with red and light blue arrows. However, there is exactly one path between any two points, so you can decide for yourself what your origin and destination are.

Thursday, February 18, 2016

Style and Code Quality

Way back in the early 80s, I worked for a company in East Glacier, Montana, owned by a refugee of the original IBM team that created the language APL.  When I started working for him, I asked him about style standards, how did he want me to code?  He said he wanted no standards of style. I should code in a manner that I thought best and was the most readable to me.

He had a copy of a code quality study from IBM written in the 70s.  It was about the rates of errors caught by code reviewers correlated with style standards.  It found the opposite of what is assumed to be true today.  It essentially concluded that a uniform style made reviewers lazy and they would gloss over stylistically familiar code and miss seeing errors.  However, if the coding style were unfamiliar, the reviewer had to study the code to understand it.  With that study, more errors were detected and code quality went up. 

I've never seen this study in the public domain.  As it pre-dated the internet, perhaps it was never published online.  The man that owned the company died many years ago in a motorcycle accident, his company is long gone.

I really wish I could find  a copy of that study.  I want to go back in time to reread this paper.  Was it a good study?  Is it even relevant to the languages and software that we write today? After all, this study was lauded by a man directly involved in one of the most cryptic and difficult to read programming languages ever conceived.

Sunday, November 08, 2015

Our Wedding Vows

Beatrice: For all this long time you have stuck together;
Proven beyond doubt, you're birds of a feather;
Eighteen years is an awfully long wait;
Quick answer my questions, before it's too late;

I turn now to Paul, for he's a bit older;
will you answer the query, I read from my folder?

Yes, I will answer, when you read from the folder;
And really, I am only a tiny bit older.

and Lars I will have some to ask of you, too
will you speak out and say what is really true?

Lars: yes, of course, I'll say what is true
I look forward to those questions from you.

Beatrice: Paul, do you take this man Lars as your mate?
with full knowledge of the rumors that he's really not straight?

Yes, I know he's not really straight,
and yes, by all means, he should be my mate,

Lars, Paul over there, should he be your groom?
is it who you'll marry? be specific, we'd best not assume.

yes, it is Paul I'll marry as my groom,
and being specific it's not "who", but "whom".

Paul will you stand by Lars in both sickness and health?
and stay by his side in either poorness or wealth?

Paul: I will stand with Lars in poorness of health,
I will stay by him during the sickness of wealth,

Beatrice: Lars is it your desire to care for your Paul?
to be with him and hold him through anything at all?

Lars: through anything at all, I'll care for my Paul,
to be with him and hold him, I give him my all.

we've covered both the good and the bad,
We're now to the point of the happy and glad.
Now is time to be a little bit bolder,
I want you to stand shoulder to shoulder
Through the power given me by Oregon State
It is time you speak together and seal your fate,

There once were a couple men I knew
a wedding it seemed we'd never view
eighteen years they waited
their dreams quite deflated
but now all they must say is....

Paul & Lars:
I do!

Tuesday, May 05, 2015

History as a Birthday Present

On my 55th birthday, I received an unusual gift: an armchair history adventure similar to a PBS episode of the History Detectives. It started with the gift of three old photographs acquired from a local thrift store. 

Written on the back of one of the images was a cryptic “T+T RR ACME MINE 1915”. I wondered if I could find some context for this crash that would explain the strange ordering of the train: two box cars, the engine and then two more box cars. 

I started my research by first identifying the name of the railroad, “T+T RR” stands for Tonopah and Tidewater Railroad, a short line railroad built in 1905 by the Pacific Coast Borax Company to transport borax from the local mines. It was roughly two hundred miles long, stretching from Ludow, CA to Beatty, NV. The rail line was abandoned in 1940. 

Wanting to see the location of the crash on Google Earth, I then searched for “ACME Mine” in Google, only to find that ACME is a rather common name. However, since I knew that the Tonopah and Tidewater Railroad ran through the Mojave desert, I examined links in the Google search results that also mentioned California. I focused closely at an entry for the “Amargosa Mine (ACME Mine)” on Looking at some of the icons of nearby locations, I found reference to a place called China Ranch. In Google Maps, I was able to find “China Ranch Date Farm and Bakery”.

That was the key to finding the location in Google Earth.  I entered search for the bakery and then scrolled down the canyon and found remains of an old railroad grade:

 Reorienting the perspective so as to look at the hillside: 

I see direct and obvious connections with the landscape in the original photographs.  I found the the location of the crash.  

The question that I wanted answered was how did the crash happen.  Looking over the terrain, I can see nothing the helps explain the ordering of the cars in the train.  

Armed with some new search terms, I entered “China Ranch” “T&T” and found the Google scan of a book called “Railroads of Nevada and Eastern California: The southern roads”. I found this paragraph:
... some new traffic was generated through construction of the so-called ACME spur from Morrison (later ACME) to the main line in Amargosa Canyon ... The tracks ran past China Ranch and through a picturesque canyon to a gypsum deposit at Acme. No particular importance attaches to the line, unless it be recalled that it was on this branch that two cars got away from an engineer and coasted all the way to the junction, resulting in a bad wreck
Further perusal of the scanned book, I found unattributed reproductions of two of my three photos. Captioning linked the photos to the derailment in the quoted paragraph above.  So I found a reference to the derailment, but no details. 

Browsing lead to a cache of images of the Tonopah and Tidewater Railroad called the “Henrick Collection”, I noticed a flaw in my search strategy. I kept using the term “derailment” and it seemed the term “wreck” was more common in that era. The simple search “T&T 1915 Wreck” yielded the gold that I needed: a newspaper clipping from the “Tonapah Daily Bonanza” with an account from a participant in the rescue and clean up.

It turned out that this was a pretty dramatic crash: two runaway box cars were being chased by an engine towing two other box cars. On finally catching and coupling, they were going too fast for the curve and rolled. It was, perhaps, a good thing they rolled, as the story implies, a passenger rail car with a complement of passengers waited on the track nearby or below.  One person, the fireman, died from injuries suffered from bailing out at the wrong place. The engineer was “frightfully maimed and burned”, but recovered within a year and returned to work.  The locomotive, #9 survived, was righted and labored in the heat on the Mojave rails for another twenty-five years.

That explains the odd ordering of the cars with engine in the middle.  My original question had been answered.

Of course, this spawns more questions.  Are these photos of the era or later generation reproductions? How did they end up in a Humane Society Thrift Store in Corvallis, OR?  These questions are for a future history expedition.

I've got to say, Paul, this was a brilliant birthday present.  It wasn't just a thing, it was a wonderful mystery to solve.  It demonstrates the power of the open Web. I'm grateful to live in a time where such a research project is achievable from my home desk in mere hours.  

Saturday, January 17, 2015

The Smoothest Migration

I must say that it was the smoothest migration that I have ever witnessed. The Socorro system data has left our data center and taken up residence at Amazon.

Since 2010, HBase has been our primary storage for Firefox crash data.  Spread across something like 70 machines, we maintained a constant cache of at least six months of crash data.  It was never a pain free system.  Thrift, the system through which Socorro communicated with HBase, seemed to develop a dislike for us from the beginning.  We fought it and it fought back.

Through the adversity that embodied our relationship with Thrift/HBase, Socorro evolved fault tolerance and self healing.  All connections to external resources in Socorro are wrapped with our TransactionExecutor.  It's a class that recognizes certain types of failures and executes a backing off retry when a connection fails.  It's quite generic, as it wraps our connections to HBase, PostgreSQL, RabbitMQ, ElasticSearch and now AmazonEC2.  It ensures that if an external resource fails with a temporary problem, Socorro doesn't fail, too.

Periodically, HBase would become unavailable. The Socorro system, detecting the problem, would back down, biding its time while waiting for the failed resource to recover.  Eventually, after probing the failed resource, Socorro detects recovery and picks up where it left off.

Over the years, we realized that one of the major features that originally attracted us to HBase was not giving us the payoff that we had hoped.  We just weren't using the MapReduce capabilities and found the HBase maintenance costs were not worth the expense.

Thus came the decision that we were to migrate away.  Initially, we considered moving to Ceph and began a Ceph implementation of what we call our CrashStorage API.

Every external resource in Socorro lives encapsulated in a class that implements the Crash Storage API.  Using the Python package Configman, crash storage classes can be loaded at run time, giving us a plugin interface.  Ceph turned out to be a bust when the winds of change directed us to move to AmazonS3. Because we implemented the CrashStorage API using the Boto library, we were able to reuse the code.

Then began the migration.  Rather than just flipping a switch, our migration was gradual.  We started 2014 with HBase as primary storage:

Then, in December, we started running HBase and AmazonS3 together.   We added the new AmazonS3 CrashStorage classes to the Configman managed Socorro INI files.  While we likely restarted the Socorro services, we could have just sent SIGHUP, prompting them to reread their config files, load the new Crash Storage modules and continue running as if nothing had happened.

After most of a month, and completing a migration of old data from HBase to  Amazon, we were ready to cut HBase loose.

I was amused by the non-event of the severing of Thrift from Socorro.  Again, it was a matter of editing HBase out of the configuration, sending a SIGHUP, causing HBase to fall silent. Socorro didn't care.  Announced several hours later on the Socorro mailing list, it seem more like a footnote than an announcement: "oh, by the way, HBase is gone".

Oh, the migration wasn't completely perfect, there were some glitches.  Most of those were from minor cron jobs that were used for special purposes and inadvertently neglected.

The primary datastore migration is not the end of the road.  We still have to move the server processes themselves to Amazon system.  Because everything is captured in the Socorro configuration, however, we do not anticipate that this will an onerous process.

I am quite proud of the success of Socorro's modular design.  I think we programmers only ever really just shuffle complexity around from one place to another.  In my design of Socorro's crash storage system, I have swung a pendulum far to one side, moving the complexity into the configuration.  That has disadvantages.  However, in a system that has to rapidly evolve to changing demands and changing environments, we've just demonstrated a spectacular success.

Credit where credit is due: Rob Helmer spearheaded this migration as the DevOp lead. He pressed the buttons and reworked the configuration files.  Credit also goes to Selena Deckelmann, who lead the way to Boto for Ceph that gave us Boto for Amazon.  Her contribution in writing the Boto CrashStorage class was invaluable.  Me?  While I wrote most of the Boto CrashStorage class and I'm responsible for the overall design, I was able to mainly just be a witness to this migration.  Kind of like watching my children earn great success, I'm proud of the Socorro team and look forward to the next evolutionary steps for Socorro.

Thursday, November 13, 2014

the World's One Door

Last evening, just before I retired for the night, a coworker, Edna Piranha (not her real name), tweeted something that intrigued me:

the WORLD… their WORLD… the WORLD.. world world, world? world! world. wow that word looks funny. world.
Suddenly my brain shifted into what I can only call poetry mode.  Words and phrases similar to the word, "world" began surfacing and swimming around in my mind. After about twenty minutes, I replied to her tweet with:
 @ednapiranha may your weird wonder ward our wired world for we were old and wandered and whirled from the word of the one door.
I immediately went to bed and began a night woven with those words.  They were in my dreams.  I'd wake up with chants in my head, "world - we're old" and "wonder - one door". It haunted me all night long and now continues into the next day.

Ten years ago, a dear friend, since deceased, came up with a new word, ospid, that perfectly describes what I was experiencing.
n, an object which, for a brief period after its creation, intensely fascinates its creator.  Once the fascination is over, the object is no longer an ospid.
I look forward to the moment, hopefully later today, when this is no longer an ospid. 

Wednesday, October 29, 2014

Judge the Project, Not the Contributors

I recently read a blog posting titled, The 8 Essential Traits of a Great Open Source Contributor I am disturbed by this posting. While clearly not the intended effect, I feel the posting just told a huge swath of people that they are neither qualified nor welcome to contribute to Open Source. The intent of the posting was to say that there is a wide range of skills needed in Open Source. Even if a potential contributor feels they lack an essential technical skill, here's an enumeration of other skills that are helpful.
Over the years, I’ve talked to many people who have wanted to contribute to open source projects, but think that they don’t have what it takes to make a contribution. If you’re in that situation, I hope this post helps you get out of that mindset and start contributing to the projects that matter to you.
See? The author has completely good intentions. My fear is that the posting has the opposite effect. It raises a bar as if it is an ad for a paid technical position. He uses superlatives that say to me, “we are looking for the top people as contributors, not common people”.

Unfortunately, my interpretation of this blog posting is not the need for a wide range of skills, it communicates that if you contribute, you'd better be great at doing so. In fact, if you do not have all these skills, you cannot be considered great. So where is the incentive to participate? It makes Open Source sound as if it an invitation to be judged as either great or inadequate.

Ok, I know this interpretation is through my own jaundiced eyes. So to see if my interpretation was just a reflection of my own bad day, I shared the blog posting with a couple colleagues.  Both colleagues are women that judge their own skills unnecessarily harshly, but, in my judgement are really quite good. I chose these two specifically, because I knew both suffer “imposter syndrome”, a largely unshakable feeling of inadequacy that is quite common among technical people.   Both reacted badly to the posting, one saying that it sounded like a job posting for a position for which there would be no hope of ever landing.

I want to turn this around. Let's not judge the contributors, let's judge the projects instead. In fact, we can take these eight traits and boil them down to one: essential trait of a great open source project:
Essential trait of a great open source project:
Leaders & processes that can advance the project while marshalling imperfect contributors gracefully.
That's a really tall order. By that standard, my own Open Source projects are not great. However, I feel much more comfortable saying that the project is not great, rather than sorting the contributors.

If I were paying people to work on my project, I'd have no qualms about judging their performance any where along a continuum of “great” to “inadequate”. Contributors are NOT employees subject to performance review.  In my projects, if someone contributes, I consider both the contribution and the contributor to be “great”. The contribution may not make it into the project, but it was given to me for free, so it is naturally great by that aspect alone.

Contribution: Voluntary Gift

Perhaps if the original posting had said, "these are the eight gifts we need" rather than saying the the gifts are traits of people we consider "great", I would not have been so uncomfortable.

A great Open Source project is one that produces a successful product and is inclusive. An Open Source project that produces a successful product, but is not inclusive, is merely successful.