I’ve spoken twice recently this year on the instrumentation of complex systems, at SF Bay Area Erlang Factory in February and at Portland, OR Write the Docs in May. The Erlang Factory talk is a bit more engineer focused while the Write the Docs talk is for a more general, technically inclined group.
This is an incomprehensive bibliography for the books and articles that have influenced my thinking around Instrumentation by Default for complex systems. As this is an area of ongoing personal research this bibliography is necessarily a vague snapshot.
There are two kinds of associations common in software:
A fixed mapping from a set of pre-defined names to values.
A dynamic mapping from a type domain to another.
As an example of the first, consider C structs or Haskell data declarations with named fields. You have a finite set of names–known at compile time–which map–at runtime–to some values. Languages like Haskell with their fancy Hindley–Milner type systems can make more compile-time guarantees about the types of values but all the languages I’m aware of which have this sort of static association at least disallow–at compile-time–referencing a name which doesn’t exist in the mapping. In Erlang, this sort of static association is called a ‘record’.
I’m one of the Lucky Few (thousand?) that get to work with Erlang professionally. Lots of people I meet are interested in the language–or Elixir, increasingly, though I happen to think LFE should be the new hotness–but have only used the language for small hobby projects or just read some books. “What do you use Erlang for?” I get asked.
There’s something I was told that’s been flitting around my mind, paraphrased: “Functional testing, done repeatedly, is pretty much monitoring.” I’m not entirely convinced of the truth of this but it’s got me thinking about how to make a hypermedia-ish REST API cooperate with clients that want to assert properties about it. Specifically, today, I wanted to check that the representations of all resources in a staging API conformed to a schema and iterate over all possible objects in the staging system.
I had two thoughts, but let’s go over the problem domain first just so we’re clear.
Infrastructure services: it’s what I do now. The primary service I work on is FireEngine, previously mentioned in this blog. It’s a network device abstraction service, all bristling with Erlang goodness and RESTful delights. My team actively supports and develops three other major projects in addition, two of which are co-eternal with FireEngine. One of these–which will go unnamed as there’s been no official public comment on it–is a business process serving client of FireEngine. It’s a studiously object-oriented Rails Prime, rails-api kind of application, the brain-child of my colleague Josh Schairbaum (@jschairb).
I got a chance, finally, to play with Yammer’s Dropwizard a bit this weekend. It’s an interesting library/mini-framework that does so very much to showcase the best of RESTful API development in Java, sans all the Object Oriented Dark Ages ride-alongs: not much XML beyond pom.xml, very clear documentation and example code for ready study.
What I came away most impressed with were the nods to the needs of Operations in the base library: specifically, health checks and metrics.
In my work at Rackspace with the FDaAT git features heavily. My group’s rate of development is fairly rapid: it is not atypical for our projects to receive multiple pull requests in a day, in addition to the work I’m doing. How I keep my changes separated from an evolving codebase without falling behind involves a particular bit of bookkeeping.
The last few months I’ve not done a terrible amount hobby programming; I spent much of the time interviewing (successfully!) at Rackspace and, since arriving in San Antonio, have been busy learning the team code-base. There were a few hobby projects I finished during February-April period, the one I’m most pleased with is a ternary search trie, a data-structure described by Bentley and Sedgewick in their 1996 paper Fast Algorithms for Searching and Sorting Strings, among other places.
I find great joy in programming. Mostly, I make tools, to solve perceived problems or, my favorite, to ease burdens seen otherwise as only natural. It’s a joy informed by a stubborn implicit reductionism in my thinking: any system can be analyzed, faults isolated and then repaired or mitigated without inherently changing the system. Trivial counterexamples abound for almost any system you might imagine: laws that carry perverse incentives and reduce, rather that lift up, the behaviors of a society; empirical observations that become a myopic end unto themselves, driving innovation toward the numbers rather than toward the initial problem; a state management daemon that does not guarantee deterministic ordering of events, injecting the unexpected into what was previously perceived as a well-ordered environment.
This is the second in a series of articles that walk-through setting up and maintaining a kit sufficient to run a tech business on. Please make yourself familiar with the first article, in which a base image box was setup and a puppet central master box was derived from the base.
In this article we’ll make puppet self-hosting, properly version-control our puppet configuration and put together a push-style deployment system for every manner of source code.
As the deployment system is best put in place in the context of a reasonable puppet setup we’ll be chicken-egging it here for a short bit, but, don’t worry, there’s much less work in this second piece than the first.
When I was brought on as CarePilot as Systems Administrator / Operations Developer we ran on a single(!) box in Amazon’s EC2, no redundancy and nothing in the way of configuration control. I kept nothing of that box–even moving away from EC2 to Rackspace. CarePilot runs on kit built up from scratch, all the way up and down from the DB replication and backups, to the application deployment process to high-level monitoring and notifications. It was my goal at CarePilot to automate, within a reasonable degree, the maintenance and repair of the machines. Some things I’ve invented, others I’ve picked up and made use of.
While the CarePilot kit was custom crafted, I believe that it’s component pieces–released under commercial-venture friendly open-source licenses–could be used as a base for most any tech-startup. This is the first article in a series that will document the kit I’ve produced, introducing some of the open-source bits of tech I’ve created and server as a bit of a tutorial for those I merely make use of.
If at the end of these articles you can’t piece together a stable base for a new business let me know: I wrote something wrong.