West of the Mississippi

A Theory of Failure

systems thinking

A Theory of Failure

I think about failure a lot. Why do things go wrong and what can we do about it? Professionally I’m a software engineer, presently the Infrastructure Lead for Goodwater Capital. How does software fail and what can we do about it? How do human processes fail and what can

Safe Bad

The last several years of my career have been spent in specialized teams doing specialized things. In a purely technical sense, that's been delightful and I've really learned a lot. My work at Dropbox has me working in a split experience team and mentorship is part of my work responsibility

Writing "Hands-On Concurrency with Rust"

Writing "Hands-On Concurrency with Rust"

Hey, I wrote a book! Let me tell you all about the work.

Hopper Rework: multiple senders

Hopper Rework: multiple senders

I wrapped up my last post [https://blog.troutwine.us/2017/12/09/reliably-transmitting/] by saying > I think, next steps, I'll try and get multiple senders going at once. That should be fun. and that's what I've done! It turns out, it also was pretty fun. You can see the

Rust, Looking Forward in 2018

Rust, Looking Forward in 2018

This past year I was pretty well a full-time Rust developer working on a handful of key projects: * postmates/cernan [https://github.com/postmates/cernan] * postmates/hopper [https://github.com/postmates/hopper/] * postmates/quantiles [https://github.com/postmates/quantiles/] Each of these saw a non-trivial amount of work poured into

Hopper Rework: reliably transmitting from sender to receiver

Hopper Rework: reliably transmitting from sender to receiver

In the previous Hopper Rework post [https://blog.troutwine.us/2017/12/03/mmap-prototyping/] I introduced my ambition to get disk-backed MPSC Sender / Receiver going on top of * fine-grained, atomic synchronization * mmap'ing queue files instead of the giant-struct-in-a-mutex nightmare we have now. Well! I did some fiddling on the mmap_

Hopper Rework: mmap prototyping

Hopper Rework: mmap prototyping

A while back I spoke about cernan [https://github.com/postmates/cernan/] to the Rust Meetup in San Francisco [https://air.mozilla.org/bay-area-rust-meetup-november-2017/]. I kindly mentioned in my spiel that hopper [https://github.com/postmates/hopper/] – the disk-backed MPSC that allows cernan to cope with overload and keep a

Encheapening Cernan Internal Metrics

Encheapening Cernan Internal Metrics

In the new 0.7.x series of cernan [https://github.com/postmates/cernan] we stumbled on a neat, cheap approach for making internal metrics available inside a rust codebase, an approach that has legs in other projects, I'd say. This is going to be a quick note describing what

Build Good Software: Of Politics and Methods

software development

Build Good Software: Of Politics and Methods

Thank you to Hope Waggoner and Mike Sassak for their kind review of this essay. It wouldn't be half what it is without their help. -------------------------------------------------------------------------------- I’d like to speak a word for good software systems. I would like here to discover the meaning behind “good” and put forward

Announcing Cernan

Announcing Cernan

Originally published Jan. 13, 2017 on Medium [https://medium.com/@bltroutwine/announcing-cernan-28c245a12f91#.xejcwwugd] for Postmates [http://postmates.com/]. Today I'm excited to announce that Postmates is open-sourcing cernan [https://github.com/postmates/cernan/]! Cernan is a telemetry and logging aggregation server that sits quietly on yoru system and ships whatever

software development

American Fuzzy Lop’ing Rust

In the last six months I’ve been all in on Rust. The work I’m doing at Postmates is low-level and finicky, which is my kind of thing. In particular, the software I’m constructing: * must have high throughput and tolerable latency (the system is high-scale but is not

Moonconf Papers

The Moonconf version of my new talk "Fetching Moths from the Works: Correctness Methods in Software" went by really quickly, 114 slides in thirty minutes. Fast! Here's the papers I reference in the talk, in order of appearance: * "The Case of the Three Engineers vs. BART" :: Gordan Friedlander - 1974

Project Apollo: Reading List

Project Apollo: Reading List

Time to talk books! > @bltroutwine [https://twitter.com/bltroutwine] by the way, do you have something like an Apollo recommended reading list? Because I would be *so* into that. — Daniel D. Beck (@ddbeck) August 27, 2015 [https://twitter.com/ddbeck/status/636961208726355969] Aside from embedded and real-time systems, my main

Pressure Suits and a Suitcase?

Pressure Suits and a Suitcase?

The X-15 is a fascinating project in NASA and aeronautic engineering history. Running from 1958 to 1968 this plane couldn't take off: it was dropped from under the wing of a modified B-52 bomber. You can see it below, the black craft under the far wing. A single occupant craft,

Peculiar Books Reviewed: Grigori Medvedev's "The Truth About Chernobyl"

Peculiar Books Reviewed: Grigori Medvedev's "The Truth About Chernobyl"

Let me tell you a joke. > Days after the Chernobyl plant melted down General Tarakanov, aware of the extreme importance of clearing the reactor roof of radioactive graphite ahead of the weather, began accepting offers of robots from other nations to do the job. The West Germans, very confident, delivered

Peculiar Books Reviewed: Henry S. F. Cooper Jr.'s "Thirteen: The Apollo Flight that Failed"

Peculiar Books Reviewed: Henry S. F. Cooper Jr.'s "Thirteen: The Apollo Flight that Failed"

In the first Peculiar Books Reviewed we discussed David A. Mindell's delightful book "Digital Apollo" and, in particular, took from the book this lesson: a technical system which puts the human component in a position of supremacy over the machine is more capable of achieving the aim of the system

Peculiar Books Reviewed: Francis Spufford's "Backroom Boys"

Peculiar Books Reviewed: Francis Spufford's "Backroom Boys"

What is the soul of software engineering as a discipline? That is, who is it that the software engineer can esteem? What characteristics are laudable and worthy of emulation? Physicists have their heroes: Niels Bohr, say, careful to the point of being paralyzed by indecision but brilliant nonetheless, or Richard

Peculiar Books Reviewed: Alain de Botton's "Status Anxiety"

Peculiar Books Reviewed: Alain de Botton's "Status Anxiety"

I went to a conference earlier this month where everyone was uniformly lovely and brilliant and interesting and everyone agreed that it was an excellent conference and damn near everyone felt like surely, soon, all the other uniformly lovely and brilliant and interesting people would realize that they, and they

Instrumentation Talks, To Date

Instrumentation Talks, To Date

I've spoken twice recently this year on the instrumentation of complex systems, at SF Bay Area Erlang Factory in February and at Portland, OR Write the Docs in May. The Erlang Factory talk is a bit more engineer focused while the Write the Docs talk is for a more general,

An Incomprehensive Bibliography

An Incomprehensive Bibliography

This is an incomprehensive bibliography for the books and articles that have influenced my thinking around Instrumentation by Default for complex systems. As this is an area of ongoing personal research this bibliography is necessarily a vague snapshot. Books The following texts are sorted--roughly--in order of importance: * Charles Perrow, "Normal

Peculiar Books Reviewed: Charles Perrow's "Normal Accidents"

Peculiar Books Reviewed: Charles Perrow's "Normal Accidents"

Wheels roll over feet, kitchen knives slice into fingers, heaters catch houses on fire, software crashes losing work and chemical plants blow up. Each of these things is man-made and each is performing actions they were not intended to perform. As Charles Perrow terms it in Normal Accidents: Living with

Peculiar Books Reviewed: David A. Mindell's "Digital Apollo"

peculiar books reviewed

Peculiar Books Reviewed: David A. Mindell's "Digital Apollo"

This is the first of a series of monthly book reviews intended to make the case for expanding the canon of Software Engineering texts. Don't get me wrong, books like Code Complete or the Mythical Man Month are venerable and valuable, but I contend that the corpus should be more

The Utility of Erlang Records: Where They Should and Probably Shouldn't be Replaced by Maps

The Utility of Erlang Records: Where They Should and Probably Shouldn't be Replaced by Maps

There are two kinds of associations common in software: 0. A fixed mapping from a set of pre-defined names to values. 1. A dynamic mapping from a type domain to another. As an example of the first, consider C structs or Haskell data declarations with named fields. You have a

Making Trivial Erlang/OTP Releases With Relx

Thanks to Tristan Sloughter [https://twitter.com/t_sloughter] and his recent work on relx [https://github.com/erlware/relx/pull/29] making an Erlang/OTP no-downtime release is now super, super trivial. Like, so trivial you should just go ahead and plan to do it. In this article I'm

When would you choose Erlang?

When would you choose Erlang?

I'm one of the Lucky Few (thousand?) that get to work with Erlang professionally. Lots of people I meet are interested in the language--or Elixir, increasingly, though I happen to think LFE [http://lfe.github.io/] should be the new hotness--but have only used the language for small hobby projects