Brian L. Troutwine: A Fairly Okay Fellow

Ternary Search Trie in Erlang

2012-04-29T00:00:00-05:00

The last few months I’ve not done a terrible amount hobby programming; I spent much of the time interviewing (successfully!) at Rackspace and, since arriving in San Antonio, have been busy learning the team code-base. There were a few hobby projects I finished during February-April period, the one I’m most pleased with is a ternary search trie, a data-structure described by Bentley and Sedgewick in their 1996 paper Fast Algorithms for Searching and Sorting Strings, among other places.

I wrote an implementation from the paper with the intention of writing a Scrabble bot around it, but I’ll probably not finish that project. I’m releasing the code, extracted from the Scrabble bot under the MIT license. Find it on Github. Novel features:

extensively typed
uses proper to test algorithm properties

Problems:

works only on strings, as typed
never stress tested nor optimized for performance
the work of idle hands, not necessarily careful

Patches welcome!

LOVE Maze

2012-02-26T00:00:00-06:00

I find great joy in programming. Mostly, I make tools, to solve perceived problems or, my favorite, to ease burdens seen otherwise as only natural. It’s a joy informed by a stubborn implicit reductionism in my thinking: any system can be analyzed, faults isolated and then repaired or mitigated without inherently changing the system. Trivial counterexamples abound for almost any system you might imagine: laws that carry perverse incentives and reduce, rather that lift up, the behaviors of a society; empirical observations that become a myopic end unto themselves, driving innovation toward the numbers rather than toward the initial problem; a state management daemon that does not guarantee deterministic ordering of events, injecting the unexpected into what was previously perceived as a well-ordered environment.

Emergent phenomena can be real tough to combat; crafting tools that create problems even as they solve others can be a heartbreaking experience.

Still, there’s the joy in programming–even with the heartbreak–and I occasionally find it necessary to strike off on a new path, usually one of complete frivolity, to have delight without grief. To that end, I made a maze running game in LOVE in a kind of a mad rush on Saturday. I knew neither lua nor love2d at the start, all the better as I’d been meaning to at least become familiar with both.

I’m impressed with lua’s minimal design, but it goes rather too far. The most standout issue is not nearly the worst sticks in my mind nonetheless:

string.format cannot format all of lua’s provided types, by default

How do you throw a boolean value into string.format? You have to redefine string.format, that’s how. It seems to me that, at a minimum, a language should have the facility of string formatting its base types. I found the lack of bitwise operations frustrating–though there are various C libraries available and lua 5.2 ships with one natively–and tables could use more operations by default. Note ‘rand_key’ is pretty ugly as lua can’t tally up non-numerical keys in a table wall[keys] = nil exists in place of a table.remove for non-numerical keys.

I am looking forward to using lua from C projects as glue. Lua’s C API is a far sight better than CPython’s–which I’ve traditionally used a a glue language–and all of the non-optimal implementation choices made in Lua can easily be addressed in C. I’ll need to figure out how to compile native code alongside a LOVE project.

Speaking of which, I have nothing but exuberant enthusiasm for LOVE2d. It was nothing other than an absolute pleasure to work with and the community behind it answered my 0.7.1 bug-related questions with what turns out to be a characteristic irreverence and sincere warmth to folks that just happen to wander in.

You can find the source code for ‘maze’ at github or just below this sentence. Patches welcome!


-- A maze runner.
--
-- This program is a maze running game. There's not death, no time limits and
-- nothing in the way of story. Go from the green square to the red, little
-- golden square!
--
-- Inspired by https://love2d.org/wiki/Tutorial:Gridlocked_Player
--
-- Developed with love 0.8.0, straight out of bitbucket tip.


-- Pre-define some colors. Love doesn't have any built in and it's rather nice
-- to refer to color names.
white = { 255, 255, 255, 255 }
grey = { 128, 128, 128, 255 }
red = { 255, 0, 0, 255 }
green = { 0, 255, 0, 255 }
blue = { 0, 0, 255, 255 }
gold = { 255, 215, 0, 255 }

-- Defines the dimensions of the world. When possible, we refer only to grid
-- numbers, but love2d's calls require pixel values. base_size refers to the
-- width and height of one grid cell.
x_grid_max = 130
y_grid_max = 99
base_size = 8
width  = base_size*(x_grid_max+1)
height = base_size*(y_grid_max+1)
love.graphics.setMode( width, height )

-- All cells are one of three types, only OPEN is passable by the player and the
-- maze carving algorithm. The maze is surrounded by a 'moat' of water, which is
-- really just a hack to make the mathematics of this simplistic. No edge cases.
WATER = 2
WALL = 1
OPEN = 0

--
-- Framework Functions
--

-- love.load, well, loads all of the preliminary data for the program. 'player'
-- is what you might expect, 'maze' the object (is that the right lua term) that
-- holds the position of the 'start' and 'exit' squares and 'map' which is the
-- world in which the player will move.
--
-- I also set the random seed based on current time. Bit of a gripe: os.time
-- returns in millisecond range, meaning the random seed isn't going to be that
-- great.
--
function love.load()
   player = {
      grid_x = 1,
      grid_y = 1,
   }
   maze = {
      ["exit"] = {
         grid_x = x_grid_max - 1,
         grid_y = y_grid_max - 1
      },
      ["start"] = {
         grid_x = 1,
         grid_y = 1
      }
   }

   time = os.time()
   math.randomseed( time )

   map = generate_maze()
end

-- love.draw updates the screen for every tick. We first layer in the maze
-- itself from 'map', then drop in the exit, start and player squares. The
-- player is a tasteful gold, but not dangerous like Midas.
function love.draw()
   -- the maze
   for x=0, x_grid_max do
      for y=0, y_grid_max do
         if map[y][x] == OPEN then
            love.graphics.setColor( white )
            love.graphics.rectangle("fill", x * base_size, y * base_size, base_size, base_size)
         elseif map[y][x] == WALL then
            love.graphics.setColor( grey )
            love.graphics.rectangle("line", x * base_size, y * base_size, base_size, base_size)
         elseif map[y][x] == WATER then
            love.graphics.setColor( blue )
            love.graphics.rectangle("fill", x * base_size, y * base_size, base_size, base_size)
         end
      end
   end

   -- the exit
   love.graphics.setColor( red )
   love.graphics.rectangle("fill", maze.exit.grid_x*base_size, maze.exit.grid_y*base_size, base_size, base_size)

   -- the start
   love.graphics.setColor( green )
   love.graphics.rectangle("fill", maze.start.grid_x*base_size, maze.start.grid_y*base_size, base_size, base_size)

   -- the player
   love.graphics.setColor( gold )
   love.graphics.rectangle("fill", player.grid_x*base_size, player.grid_y*base_size, base_size, base_size)
end

-- love.keypressed handles inputs per tick; I handle only movement and
-- escaping. That lua doesn't have a switch statement is somewhat irking to me,
-- but I suppose love2d is meant for prototypes? I'm certainly inexperienced
-- with both the language and the library. I _think_ love.draw consumes a
-- powerful amount of CPU in re-drawing the maze per tick.
function love.keypressed(key)
   if key == "up" then
      if collide(-1, 0) then
         player.grid_y = player.grid_y - 1
      end
   elseif key == "down" then
      if collide(1, 0) then
         player.grid_y = player.grid_y + 1
      end
   elseif key == "left" then
      if collide(0, -1) then
         player.grid_x = player.grid_x - 1
      end
   elseif key == "right" then
      if collide(0, 1) then
         player.grid_x = player.grid_x + 1
      end
   elseif key == 'escape' then
      love.event.push('quit')
   end
end

--
-- Internal Functions
--

-- generate_maze does what you might think. The algorithm is something like
-- Prim's.
function generate_maze()
   -- fill map entirely
   map = {}
   for i=0, y_grid_max do
      map[i] = {}
      for j=0, x_grid_max do
         map[i][j] = WALL
      end
   end

   -- build the moat
   for i=0, y_grid_max do
      for j=0, x_grid_max do
         map[0][j] = WATER
         map[y_grid_max][j] = WATER
      end
      map[i][x_grid_max] = WATER
      map[i][0] = WATER
   end

   -- craft the maze
   map[maze.start.grid_y][maze.start.grid_x] = OPEN --mark the entrance

   ---- walls contains those positions known to be walls. The function index is
   ---- a hash of the (y,x) coordinates. The algorithm will strip one wall
   ---- randomly out of walls, possibly mark it as open space and, possibly, add
   ---- the neighbor cells into 'walls'.
   walls = {
      ["0102"] = { y=1, x=2 },
      ["0201"] = { y=2, x=1 }
   }
   seen =  { ["0101"] = { x=1, y=1 } }
   while next(walls) ~= nil do
      key = rand_key(walls)

      wall = walls[key]
      walls[key] = nil
      seen[key] = wall

      y = wall.y
      x = wall.x


      north      = is_open(map, y-1, x)
      south      = is_open(map, y+1, x)
      west       = is_open(map, y,   x-1)
      east       = is_open(map, y,   x+1)

      -- Directions are named in terms of the cardinal directions. Up and down
      -- the Y-axis is N/S, left and right on the X-axis is W/E. is_center
      -- asserts that any new OPEN space cannot join some tunnels, although it
      -- does not disallow diagonal OPEN cells, which I dislike the look of.
      is_center = (north and south) or (north and west) or (north and east) or
                  (south and west) or (south and east) or (east and west)

      if not is_center then
         map[y][x] = OPEN
         add_wall(walls, seen, map, y-1, x) -- north
         add_wall(walls, seen, map, y+1, x) -- south
         add_wall(walls, seen, map, y, x-1) -- east
         add_wall(walls, seen, map, y, x+1) -- west
      end
   end

   -- search for the exit
   for i=1, y_grid_max-1 do
      if map[i][x_grid_max-1] == OPEN then
         maze.exit.grid_y = i
      end
   end
   return map
end

-- Possibly add the wall at (y,x) into 'walls', unless it is not a WALL or is in
-- 'seen', meaning we've already ruled it out as a candidate to go OPEN.
function add_wall(walls, seen, map, y, x)
   key = string.format("%.2d%.2d", y, x)

   if (map[y][x] == WALL) and (seen[key] == nil) then
      walls[key] = { y=y, x=x }
   end
end

-- This seems like a bitterly ugly hack: pull a random key out of a given table.
function rand_key(hash)
   ks = {}
   for k,v in pairs(hash) do table.insert(ks, k) end
   return ks[math.random(1, #ks)]
end

-- Determines if a given cell is, indeed, OPEN.
function is_open(map, y, x)
   if map[y][x] == OPEN then
      return true
   else
      return false
   end
end

-- Has the player hit somthing? This function performs the check.
function collide(y, x)
   if map[player.grid_y + y][player.grid_x + x] ~= OPEN then
      return false
   end
   return true
end

Wrangling Servers -- A Proper Foundation

2012-02-22T00:00:00-06:00

This is the second in a series of articles that walk-through setting up and maintaining a kit sufficient to run a tech business on. Please make yourself familiar with the first article, in which a base image box was setup and a puppet central master box was derived from the base.

In this article we’ll make puppet self-hosting, properly version-control our puppet configuration and put together a push-style deployment system for every manner of source code.

As the deployment system is best put in place in the context of a reasonable puppet setup we’ll be chicken-egging it here for a short bit, but, don’t worry, there’s much less work in this second piece than the first.

Puppet hosting Puppet

You’ll recall that in our last article our two files in /etc/puppet were not version controlled. Make sure you aren’t working on the puppet master and create a new git repository:

$ mkdir -p ~/projects/us/troutwine/ops/etcpuppet
$ cd ~/projects/us/troutwine/ops/etcpuppet

I like to keep my projects namespaced by domain, then by departmental affiliation–even if I’m the only one working on that domain’s IP. The exact path you supply is probably going to differ. Once in etcpuppet/

$ git init
Initialized empty Git repository in ~/projects/us/troutwine/ops/etcpuppet/.git/

Create in this directory three files. The first file is config.ru and serves as the instruction set for the thin web-server that powers puppet master.

$ cat config.ru
# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
# $:.unshift('/opt/puppet/lib')

$0 = "master"

# if you want debugging:
# ARGV << "--debug"

ARGV << "--rack"
require 'puppet/application/master'
# we're usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Application[:master].run

Our second file acts as configuration for the puppet master itself.

$ cat puppet.conf
[main]
ssldir=$vardir/ssl

[master]
certname=puppet

These files you should recall from the first article.

With no deployment rig in place, we’ll begin by muddling through and relay rsyncing code into place by hand, gradually elaborating on the process up to the final method. I’m aware of some folks that rsync their configuration directly into place–from developer machine to production system–but I resist doing that for a few reasons. Firstly, if /etc/puppet is to be the location of puppet configuration and user/group puppet is to own this directory the user puppet must be granted remote login access. The puppet user should not be granted any manner of login access because the data it owns is used, without suspicion, to manipulate all the systems in your cluster: any possible breach of the puppet user account by a remote party would be disastrous. Secondly, to mitigate any possible breeches of the puppet user, our puppet configuration will be deployed on a read-only filesystem. Any attacker that might gain access to the puppet user will be unable to alter data owned by that user: the R/O filesystem will be mounted by root and unalterable except by the root user. (Other attack vectors exist; we’ll address these later through network design.)

First, rsync the configuration codebase to your puppet box. Recall that I’m running a virtual box instance on a host-only network; I have the localhost puppet IP alias as localpuppet in /etc/hosts and I’ve an ssh-key accessible account on the puppet master box.

$ rsync -vzr --delete --exclude='.git' -e ssh . localpuppet:etcpuppet
sending incremental file list
created directory /home/blt/etcpuppet
./
config.ru
puppet.conf
manifests/
manifests/.gitkeepme

sent 595 bytes  received 76 bytes  1342.00 bytes/sec
total size is 479  speedup is 0.71

On the puppet host:

puppet:~$ sudo rsync -vzr --delete --exclude='.git' etcpuppet/ /etc/puppet/
sending incremental file list

sent 599 bytes  received 76 bytes  1350.00 bytes/sec
total size is 479  speedup is 0.71

Nothing should be synced into /etc/puppet as there are no files modified from the previous article. What we’re going to need now are puppet modules to host puppet itself. This very task is the over-saturated ‘Hello World’ of puppet, to the exclusion of other, more interesting, works. Here’s what I’m going to do: in this article we’ll plug submodules into the repository we’ve created and edit a file or two. Bam: self-hosting puppet. If you’re interested in the details or if you need the tutorial, dive into the source of the submodules introduced. Those I’ve written are documented sufficiently to act as a tutorial for the determined.

Before we begin adding modules we’re going to need some boilerplating. The manifests/site.pp is puppet’s main method, so to speak: everything included in this file is all that is available to a puppet agent.

import 'nodes.pp'

Exec {
  path => "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
}

The first line pulls in all of our node definitions. In puppet, a ‘node’ is a machine type. Exact demarcation is left up to the user but determined by machine hostname. We’ll use a mixture of ‘type’ and ‘typeNumber’, ‘puppet’ and ‘db0’ being exemplars of both methods, respectively. More of that shortly. The second block sets the shell path for all command invocations. It’s not strictly necessary to set this–puppet can sometimes figure out your path from the host system–but it is better to be explicit when possible, else all subsequent Execs will require a path line. That’s annoying and bug prone. Read more here.

Our next plate for the boiler is manifests/nodes.pp; the import at the top of site.pp is a relative one. You’ll find that puppet’s inclusion rules are a little strange. There’s a difference, oh yes, between the keywword ‘import’, which you should almost never use, and ‘include’, which you’ll use quite a bit. Please read the two documents I’ve just linked and make sure you understand them; while the puppet language has some warts they are, at least, well documented warts. manifests/nodes.pp will act as a base of and import all specific node definitions.

import 'nodes/*.pp'

# The base class acts as a repository for configuration common to all the more
# specific node definintions imported above.
class base {
  include supervisor, puppet

  # Ensure machine time is always synced to external reference. All system
  # defaults are acceptable.
  package { ntp: ensure => present, }
  # Include for libssl-dev for native rubygems that need SSL (EventMachine).
  package { 'libssl-dev': ensure => present, }
  # Require all machines to have rsync installed for various purposes.
  package { rsync: ensure => present, }

}

The only node definition we’ve got at the moment is for the puppet node, manifests/nodes/puppet.pp:

node 'puppet' {
  include base, puppet::master
}

That’s everything. All that’s needed now is to install the submodules referenced and their dependencies. From the root of your repository:

$ git submodule add git://github.com/blt/puppet-module-supervisor.git modules/supervisor
$ git submodule add git://github.com/blt/puppet-apt.git modules/apt
$ git submodule add git://github.com/blt/puppet-nginx.git modules/nginx
$ git submodule add git://github.com/blt/puppet-puppet.git modules/puppet

Deploy as above, execute puppet agent --test and commit. That’s it. I urge you to have a look through the modules’ source; I’ve worked to comment the modules well and you really do need to have a handle on everything. Note especially the nginx module: you’ll find many that attempt to use puppet’s limited language to compile vhost definitions into nginx’s more, hmm, vigorous configuration language. I believe this is an exercise in:

complication
frustration

neither of which you want when there’s something needs fixing, the site’s offline and you’re caught in the lurch between the dainty needs of puppet and unhappy folks. The vhost resource in the nginx module will take care of the hidden details of the nginx module but require that you pass in a template as content for the vhost. That way you get the full acrobatics of nginx without any of the heartache of mimicking that in puppet.

Now that puppet is self-hosting, you should commit all of this to version control, finally.

$ git add * && git commit -m "Initial commit"

Congratulations: your puppet configuration is now checked into version control. Add any remote repositories you might care to and push your code up and away to Github or some other reasonably disaster-proof repository host. I tend to use the remote branch name ‘backup’ for this need and reserve ‘production’ for the fork that will be deployed to live puppet master.

Deployment

Using rsync is passable when we’re developing but it’s not going to cut it long-term. (You can get pretty elaborate with rsync, though. I’m fond of daisy-chaining rsyncs across machines, spinning up as needed with inotifywait.) The deployment strategy we’re going to build requires:

A central git repository for ‘blessed’ production / staging / testing repos.
A message bus.
A message to command the invocation server.

The two options that I’d like to make you aware of for this last point are mcollective and my traut. Mcollective has company backing, scales way up and is a pseudo-interactive shell to all nodes in the collective. You can do very cool stuff with mcollective. Alternatively, traut is a small open-source project, has no company backing and is really just a cron-like for messaging. You write your commands in any language that will take stdin, unlike mcollective which is a ruby only party. Traut is also stupid simple to get running and keep that way.

If you know cron, you know traut. Mcollective is much larger project that you should probably transition to when your business takes off, your ops team is funded or you have a bit of time on your hands. Once you have more than twenty computers in your cluster–or if you find yourself ssh’ing into your machines constantly–put some time into learning mcollective.

Till then, deployment works like this:

A user pushes code into a blessed code repository, the post-receive hook is invoked and sends a specially formed message through the bus.
The post-receive hook’s message is caught by a traut script which builds a deployable slug, placing it in a remotely accessible directory on the filesystem. This slug building script then fires off a notification message through the bus.
Traut clients, pre-configured to invoke a command for the notification message, download the new slug and deploy it.

The slug building hook that I’ll introduce here is primitive: it simple-mindedly strips out git metadata and bundles all the source-code into a single squashfs file. This is great for puppet configurations, less so for complex applications. Adding heuristic running of a project’s Makefile, Rakefile, ant or another script is not at all difficult; I just didn’t get around to it. If this becomes a problem for you and I’ve not fixed it up between writing this and your involvement, take out an issue?

A central git repository

Go ahead and fire up another machine, call it ‘git’. This machine will host the ‘blessed’ repositories using gitolite to provide access control. A word of warning, the manner in which gitolite is configured makes it difficult to fully automate: doing commits into gitolites’ gitolite-admin repository is pretty slick, but I’m not clever enough to use puppet with that. (If you are, email me!)

Once the machine is ready, the first thing you’re going to need to do is get the git box’s puppet agent keyed into puppet master. To this point I’ve ignored networking. Perhaps you’re in an environment with DNS pre-baked. If not, I’ll cover setting up DNS in a later article. For now, and this is not a bad solution, consider setting IPs statically in /etc/hosts. Until I actually write the DNS article, that’s what I’ll be doing. Be sure that the hostname puppet points to your puppet master box.

git:~# puppet agent --test  --waitforcert 30 --server puppet

Now, switch back to the puppet master

puppet:~# puppet cert list
  git.troutwine.us (8E:A8:C0:03:9D:53:1A:CA:FC:25:16:82:88:F8:3F:B4)

The FQDN will be different for you (most likely) but you should see the git box waiting to have its certificate signed.

puppet:~# puppet cert sign git.troutwine.us
notice: Signed certificate request for git.troutwine.us
notice: Removing file Puppet::SSL::CertificateRequest git.troutwine.us at '/var/lib/puppet/ssl/ca/requests/git.troutwine.us.pem'

Substituting, of course, for your domain.

We’re going to use a preseed to automate the package configuration of gitolite for two reasons:

the default user in the package is ‘gitolite’ rather than ‘git’ (I hate that) and
an admin ssh key must be supplied for the package to finish its installation.

The second point is the most pressing but the first is important too: there is no shame in being meticulous if such behaviors simplify setup or remove surprises for your users.

From the root of your puppet configuration, install the gitolite module:

$ git submodule add git://github.com/blt/puppet-gitolite.git modules/gitolite

The node definition for ‘git’ is very short. In manifests/nodes/git.pp:

node 'git.troutwine.us' {
  include base

  # Install the gitolite daemon and provide the admin key.
  class { 'gitolite':
    gituser => 'git',
    admin_key => 'ssh-rsa AAAAB3NSKIPAFEW',
    path => '/var/lib/git',
  }
}

The installation of the gitolite daemons and user is interfaced through a class: you may not, therefore, install multiple copies of gitolite on a single system. This is counter to the full capability of gitolite–multiple installed copies, one per user–but I find it much less error-prone to divide gitolite installations per domain.

Installing traut and the RabbitMQ message bus

Before we fuss about with post-receive hooks we’ll get the remaining kit for deployment humming. Spin up a new machine to host the message bus, call it ‘mq0’. Introduce the node mq0 to puppet and add the RabbitMQ configuration module to your root configuration:

$ git submodule add git://github.com/blt/puppet-rabbitmq.git modules/rabbitmq

Unless you have a certificate authority in place–which I’m assuming you don’t, at this point–add my openssl module as well:

$ git submodule add git://github.com/blt/puppet-openssl.git modules/openssl

A better name for the openssl module might be ‘poor-mans-ca’, but I thought that a bit long. The module, inspired by ssh-auth, will build, sign and distribute certificates sufficient to run an encrypted internal network. It is not a secure certificate authority for the wider internet–don’t issue these things to hosts that have to rove or, God forbid, to a client. However, so long as you’re able to keep the machine hosting the private keys secure–and we’re going to install the keys to the puppet master, so you should–this will be just dandy. To be clear: if an attacker gets access to the filesystem of your puppet master, they will be able to decrypt any communication over channels signed with the keys generated by this module.

Why go to all the trouble of generating certificates for all clients and servers? Hopefully the benefit of running communications to a server daemon over an encrypted channel is obvious to you, but generating keys for a client, as well? Control, simply. The message bus will be used to cause sensitive tasks to kick off; mere encrypted channels do not deny unknown parties from establishing connections and pumping messages through. The RabbitMQ setup advocated here will:

require clients to supply a known username and password and
supply a certificate co-signed with the server’s own.

In manifests/nodes.pp add the following:

# Until there's a pressing need to construct a more traditional CA for
# internal services, the puppet-openssl module can construct a primitive
# one. The master will be placed on puppet, making that box _extremely_
# sensitive to tampering. It was, of course, already _extremely_ sensitive to
# tampering.
if $hostname == 'puppet' {
  class { 'openssl::certmaster':
    ca_name => 'rabbitmq',
    ensure => present,
  }
}
Openssl::Server {
  ca_name => 'rabbitmq',
}
openssl::server {
  'mq0' : ensure => present;
}
Openssl::Client {
  ca_name => 'rabbitmq',
}
openssl::client {
  'puppet': ensure => present;
  'git'   : ensure => present;
  'mq0'   : ensure => present;
}

All three nodes will be issued client certificates for RabbitMQ, placed in /etc/rabbitmq/ssl/client. The location is configurable, more nodes can be added as can more services.

Installing the RabbitMQ daemon is a relatively simple matter. Create an mq node definition, manifests/nodes/mq.pp:

node /^mq\d+/ {
  include base, rabbitmq
}

The installation of the traut daemon should be unsurprising

$ git submodule add git://github.com/blt/puppet-traut.git modules/traut

except that, instead of creating a new node, we’ll add ‘include traut’ to the base node class in manifests/nodes.pp. The traut module comes with a few optional goodies, one of which is hare. In manifests/nodes.pp add:

# The traut daemon which allows cron-like action in response to AMQP
# messages. Here we install cron on all systems and enable a 'puppet agent
# --test' event.
$traut_vhost = '/traut'
$traut_user  = 'traut'
$traut_pass  = '264l8uSlCeZSGZiCQHns'
$traut_key   = '/etc/rabbitmq/ssl/client/key.pem'
$traut_chain = '/etc/rabbitmq/ssl/client/cert.pem'

class { 'traut':
  ensure => present,
  vhost => $traut_vhost,
  host => 'mq0',
  username => $traut_user,
  password => $traut_pass,
  exchange => 'traut',
  debugging => true,
  version => '1.0.1',
  private_key => $traut_key,
  cert_chain => $traut_chain,
  require => File[$traut_key, $traut_chain],
  subscribe => File[$traut_key, $traut_chain],
}
include traut::hare

This is a bit long and I urge you to read the documentation provided with the puppet-traut module. Suffice it to say that all nodes will have traut installed, traut will connect to the RabbitMQ daemon on mq0 over SSL (with a client certificate) and using the supplied password and username. Be sure to change, at least, the value of $traut_pass in your setup.

To enable specially coded messages on every push for the ‘puppet repository, in manifests/nodes/git.pp add:

# Ensure that on pushes into the 'puppet' git repository traut notifications
# are sent out via the post-hook.
gitolite::resource::posthook { 'puppet':
  mqpass => "${base::gitolite_posthook_password}",
}

With that in mind, go ahead and setup the ‘puppet’ repository by cloning gitolite-admin and adding the appropriate entries. Be sure to commit and push back your changes.

That done, in manifests/nodes/mq.pp add:

# The traut vhost RabbitMQ user and /traut vhost are are used by the traut
# system daemon. /traut should be shared among multiple RabbitMQ users, where
# the traut user _must_ be exclusive to the similarly named daemon.
rabbitmq::resource::vhost { "${base::traut_vhost}":
  ensure => present,
}
rabbitmq::resource::user { "${base::traut_user}":
  require => Rabbitmq::Resource::Vhost["${base::traut_vhost}"],
  password => "${base::traut_pass}",
  ensure => present,
}
rabbitmq::resource::user::permissions { "${base::traut_user}":
  require => Rabbitmq::Resource::User["${base::traut_user}"],
  vhost => "${base::traut_vhost}",
  ensure => present,
}

This creates the RabbitMQ traut user with the password and the traut vhost, all specified in nodes.pp.

Redeploy and re-run puppet agent on all hosts. You should see traut installed to each system, as well as RabbitMQ on mq0. Depending on the order that puppet agent fires on your nodes you may need to run it several times so that ssl keys distribute properly. See the puppet-openssl documentation for more details.

Before moving on, ensure that all nodes have ‘mq0’ resolves in DNS or are set in /etc/hosts; you can manage that with puppet if you’d like. Add the following to manifests/nodes.pp:

# Without an internal DNS system, nor a pressing need for one until the
# cluster grows substantially, set hostnames manually through /etc/hosts.
host {
  'puppet.troutwine.us':
    ensure => present,
    ip => '192.168.56.11',
    host_aliases => 'puppet';
  'git.troutwine.us':
    ensure => present,
    ip => '192.168.56.12',
    host_aliases => 'git';
  'mq0.troutwine.us':
    ensure => present,
    ip => '192.168.56.13',
    host_aliases => 'mq0';
}

substituting, of course, for your setup’s IP addresses and domain name.

At last: deploying

The final piece to this article are a series of shell-scripts that turn traut events into deployable code. I’ve rolled this into a puppet module as well:

$ git submodule add git://github.com/blt/puppet-slugbuild.git modules/slugbuild

In manifests/git.pp add the following:

class { 'slugbuild':
  ensure => present,
  githosts => 'git.troutwine.us,github.com',
  gitcentral => 'git.troutwine.us',
  mqpass => "${base::gitolite_posthook_password}",
}
slugbuild::resource::traut { 'puppet':
  ensure => present,
}
slugbuild::resource::authorized_key {
  'puppet root':
    ensure => present,
    key => 'AAAABSKIPAFEW';
}

the exact happenings are documented in the module. Hopefully you noticed that slugbuild::resource::authorized_key requires you specify an ssh public key. This key, specifically, is for the root puppet master user to the ‘slugbuild’ user on the git node. Using this access, the puppet node’s root will sync slugs. In manifests/nodes/puppet.pp add:

# This key is generated so that the root user can pull source slugs from the
# slugbuilder. Note, however, that the public key is _not_ automatically
# placed into the slugbuilder's authorized_keys and must be done so manually
# with slugbuild::resource::authorized_key on the slugbuild node.
user { 'root':
  ensure => present,
}
ssh::resource::key { 'id_rsa':
  root => '/root/.ssh/',
  ensure => present,
  user => 'root',
}
ssh::resource::known_hosts { 'root':
  root => '/root/.ssh/',
  hosts => 'git.troutwine.us',
  user => 'root',
}

# The slugbuild sync will, post-build, sync all available slugs to the local
# machine using the credentials generated above.
class { 'slugbuild::slugclient':
  ensure => present,
  mqpass => "${base::gitolite_posthook_password}",
  slughost => 'git.troutwine.us',
}
slugbuild::resource::sync { 'puppet-sync':
  project => 'puppet',
  ensure => present,
}

# Ensure that newly available puppet configuration slugs are mounted and
# linked as /etc/puppet
class { 'puppet::resource::redeploy':
  ensure => present,
}

The first half constructs ssh keys for the root user of the puppet master, the latter half sets up puppet master as a slugbuild client to git.troutwine.us. Be sure to use the contents of /root/.ssh/id_rsa.pub as data to the key parameter of slubuild::resource::authorized_key.

Redeploy your puppet configuration through the rsync daisy chains. Run puppet-agent on all your machines. Commit all of your changes. With everything in place remove /etc/puppet on the puppet master and push to the gitolite repository you’ve created. After a few moments, you should find that slugbuild on the git node has created slugs, these have been transferred over to the puppet node and the latest has been mounted as /etc/puppet. Subsequent commits will likewise be handled, with the mount point being swapped atomically.

Where to next?

There ain't nothing more to write about, and I am rotten glad of it, because if I'd a knowed what a trouble it was to make a book I wouldn't a tackled it, and ain't a-going to no more.
Huckleberry Finn

I figured that this article would take a week, tops, to write. Instead, it took nearly a month! See my Github for all the puppet-* activity. Find the repository I’ve created using the above instructions here.

That said, there’s still plenty to do but I feel remiss in not taking more time to document the modules already used. You should, at this point, have a pretty decent base on which to build. I’ll keep fleshing this out, but go ahead and email me or get in touch with the Puppet Users mailing list if you’re eager to get moving. I’m also available for hire, as it happens.

These last few articles have been similar in scope to the puppet books I’m aware of on the market–a fact which didn’t occur to me at the outset. Turnbull’s books were an excellent reference when I started out:

The Puppet project also has very decent documentation. I keep the following bookmarked:

Next time we’ll start digging back through some modules and examining them in detail, examining several simple modules in a bundle and, later, a single module per article. The puppet-openssl will certainly be a long article just to itself.

Wrangling Servers -- Introduction and Preliminaries

2012-01-20T00:00:00-06:00

When I was brought on as CarePilot as Systems Administrator / Operations Developer we ran on a single(!) box in Amazon’s EC2, no redundancy and nothing in the way of configuration control. I kept nothing of that box–even moving away from EC2 to Rackspace. CarePilot runs on kit built up from scratch, all the way up and down from the DB replication and backups, to the application deployment process to high-level monitoring and notifications. It was my goal at CarePilot to automate, within a reasonable degree, the maintenance and repair of the machines. Some things I’ve invented, others I’ve picked up and made use of.

While the CarePilot kit was custom crafted, I believe that it’s component pieces–released under commercial-venture friendly open-source licenses–could be used as a base for most any tech-startup. This is the first article in a series that will document the kit I’ve produced, introducing some of the open-source bits of tech I’ve created and server as a bit of a tutorial for those I merely make use of.

If at the end of these articles you can’t piece together a stable base for a new business let me know: I wrote something wrong.

What are you getting yourself into?

Kindly note, I don't speak for CarePilot. The opinions and preferences I express here are not necessarily representative of the views of CarePilot. I'm just a guy talking on his own.

Imagine that you’re employee #3 of a startup and are getting pulled into more Ops work as the load on your few EC2 boxes gets high: all day you fiddle with this and that, then bam one of the servers goes offline and you have to piece a new image up by hand. The site is offline, meanwhile, and employees #1 and #2 can’t help but give you the stink-eye. Or, imagine that you’re the kind of person to release small profit generating web-apps every few months and one has finally taken off. Success! But the load on the $36 Heroku instance you’re running on is too high and the site’s suffering. You can crank up the Heroku toggles to meet the load, but you’re going to lose profitability that way. Time to move to virtual hosting, keeping in mind that you have to keep your Ops work to a minimum.

What do you do, in either case? Use Puppet. Puppet is a relatively easy to use state management tool–that makes an excellent sideline into acting as a configuration tool–is well documented and backed by a company of nice folks and nice users. Puppet is not an easy thing to bootstrap, however. This article, and the one that follows it, will walk you through getting a production-ready puppet setup bootstrapped, along with a few extra goodies that I think you’ll find very helpful.

Should take a few hours. In this article I’ll walk you through:

setting up a base box image,
version controlling puppet configuration right off and
bootstrapping a central puppet master from a base box.

The lack of configuration management makes this a much larger task than it would be with configuration management: take heart, this is the hard part.

Bootstrapping Puppet

The absolute heart of a managed server cluster, as conceived here, is Puppet: without it configuration is a one-off affair, down boxes are not easily replaced and there exists no central repository for understanding the arrangement of the whole system. There are alternatives–Chef being the most common–but the kit I’m going to outline here uses Puppet for two reasons:

I went to University and did some projects with a fellow completely enamored of it–I believe the good IT department makes heavy use of it.
I lurked in the Chef and Puppet communities for a time and found the conversation in Puppet’s more helpful to the almost-hopelessly ignorant. Posting a puppet related question to ServerFault and having a detailed answer within the half-hour is a fine thing.

The base OS used for this kit is Debian Squeeze. With Puppet being written in ruby and Debian support for ruby being somewhat crummy–the Debian Ruby Team, as I understand it, is understaffed and the ruby community has values somewhat hostile to Debian’s own–there’s a hassle here. Namely: do I install puppet from rubygems or from Debian’s packages? Considerations:

Debian’s puppet requires ruby 1.8.7 where mainstream ruby development largely targets 1.9.
Puppet’s central master has had memory-leak issues with ruby 1.8.7.
We’ll be installing system utilities through rubygems later in this series.

Rather than suffer with ruby 1.8.7 puppet will be installed as a gem, the process of which will prime us for quite a bit of work later on.

Before we go further, I suggest you get a virtual-machine setup going--or rent some servers in a cloud--and follow along. I've used Virtualbox on my personal machine to spot-check the writing of this series, though I won't provide instructions on its use. To mimic standard VPS system setup, configure your systems to have two network interfaces, one for NAT the other for host-only networking. Make `eth0` the NAT interface. Make sure that `eth1` has a static address.

I like to have a template base box which forms the basis of all other box types. By that I mean most cloud VPS companies allow you to store an ‘image’, a pre-configured OS alongside their offered fresh-install OS images. I like to keep a single base image–carepilot-base, say–which all newly spun-up boxes are elaborated from, through the use of Puppet. This causes a cyclical dependency issue which must be resolved by hand: namely, the puppet-master machine will necessarily require a human to coax it into being. As the puppet-master is not a production critical piece of infrastructure–its being offline does not stop customers interacting with the website or other product–I find this acceptable. If you do not and do come up with an alternative to breaking the dependency cycle drop me a line!

On the base box we’ll install:

our desired ruby version,
the puppet gem and
supervisord to run the puppet client.

Yes, supervisord. Writing init scripts is a bummer–not to mention largely non-portable–and adding daemonizing code to any system tools you might create is, likewise, a bummer. In the perfect Unix spirit, why not delegate daemonization to a special-purpose tool?

Installing Ruby

It’s the ill-named ruby1.9.1 and rubygems1.9.1 we want. In actuality, these install, as of this writing, the 1.9.2 series of interpreter. I’m sure there’s an interesting story there.

base:~# aptitude install ruby1.9.1 rubygems1.9.1

Some gems we’ll need to install have so-called ‘native extensions’–C code–so we’re going to need a compiler on our production systems. Possibly a bummer, depending on your environment. Being certified to handle credit card processing or to handle some gambling related matters–I’m vague on gambling, sorry–puts a C compiler or make facility on a production system right out. In general, sure, it’s important to make your default system as secure as possible. Consider, though, that:

the modern Debian system has, in its base install, several Turing-complete interpreters able to produce machine code,
large portions of GCC’s support libraries and tools are already installed in the base system, an interpreted language could be made to use those tools,
once a Turing-complete language with FFI ability is already present on a box you’ve installed the equivalent of a C compiler and make system onto a production system and
real system security is about restricting remote access, user access controls and segregation.

You can build a box to host your own apt repository for gems and other odds and ends that need a compiler–keeping those production systems perfectly clear of this one machine-code production vector–but I’m not convinced that the effort involved in that is rational given the slight nature of the hazard. It’s unpleasant work and you’ll spend a fair bit of time on it, continuously, but it can be done. I won’t do it here and in this series I’ll assume that you’ll have installed the following on the base system:

base:~# aptitude install ruby1.9.1-dev build-essential libssl-dev

To my knowledge there’s no automatic alternatives system for ruby. Fun thing about Debian, though, is that it’s:

always got some tool that will meet your needs and
isn’t as thoroughly documented as you might hope, commensurate to your possible needs.

Folks on the various mailing lists are super helpful, however.

base:~# update-alternatives --install /usr/bin/ruby ruby /usr/bin/ruby1.9.1 400 \
  --slave /usr/share/man/man1/ruby.1.gz ruby.1.gz /usr/share/man/man1/ruby1.9.1.1.gz \
  --slave /usr/bin/irb irb /usr/bin/irb1.9.1 \
  --slave /usr/bin/gem gem /usr/bin/gem1.9.1

You could do something similar for 1.8 as well, but I’m not going to bother. You will need to have git and openssl present on all of your systems:

base:~# aptitude install git openssl

Installing Puppet

Compared to getting ruby installed, this is going to be a breeze:

base:~# gem install facter --version '1.6.4' --no-ri --no-rdoc
Successfully installed facter-1.6.4
1 gem installed

base:~# gem install puppet --version '2.7.9' --no-ri --no-rdoc
Successfully installed puppet-2.7.9
1 gem installed

(I hate wasting disk space and don’t reference the installed documentation on my production servers ever, so I’ve skipped building rdocs and the ruby index.) Note those version numbers; they’ll be enshrined in puppet configuration later, but for now I’d like to make sure, if you’re following along, that we’re on the same footing.

One fun problem with Debian and rubygems is that gems’ binaries are not installed in the default system $PATH. “Wait a second,” I can hear my year-ago self saying to a chilly room and two napping dogs, “they meant to dump gem binaries into /var/lib?” Yep: so the argument went, /var/lib is state data and the state for rubygems is all the code plus the executables, never mind the fact that it’s not uncommon to mount /var on it’s own partition and set it noexec in fstab. Anyway, I’ve heard the Debian team’s going to fix this in a later version of the OS. For now, it’s up to us:

make sure that your /var partition is mounted exec (it probably will be) and
add [ -d /var/lib/gems/1.9.1/bin/ ] && export PATH="/var/lib/gems/1.9.1/bin/:$PATH" to the top of /etc/bash.bashrc

There must be a puppet group and user available on the box:

base:~# adduser --group --system --home /etc/puppet --disabled-password puppet

Installing Supervisord

Finally, supervisord:

base:~# aptitude install supervisor

What we don’t want right now is to start the puppet client, either in daemon form or foreground. It’s a right pain dealing with puppet if it mis-creates an SSL key and on the base box we’ll be forced to destroy that key for each new box derived from the base. It’s okay, though: we’ll write puppet configuration such that the first, necessarily manual, run of each puppet client will secure the box in its intended role.

Setting up the Puppetd Box

Clone the base image and make a box with hostname puppet. This machine will host the daemonized puppet-master daemon from which all puppet clients will pull configuration directives. By default, puppet master uses webrick to serve content–a tremendously slow and wasteful web-server fit only for development purposes. In my experience, even past a few clients, webrick is woefully under-powered. Happily, it’s easy enough to get a production-ready puppet master installation going.

First step, Nginx: a proper web-server.

The nginx in Squeeze’s default package list is far too old: just over two years at this point. Enable the backports repository by creating /etc/apt/sources.list.d/backports.list with the content:

deb http://backports.debian.org/debian-backports squeeze-backports main

Then:

# aptitude update && aptitude install -t squeeze-backports nginx

This will ensure that the backports package list is available in your cached apt index and that nginx is installed from the backports repository. By default nginx-full is installed, though since we’re installing the dummy you can opt to use nginx-light by installing that package after installing the dummy; the dummy will remain installed and update nginx-light as needed.

Second step, Thin: a ruby-rack web-server

If you’re not so familiar with ruby-land, ‘rack’ is a ruby glue layer between many web-servers and, well, whatever you want to build on top of that layer. Rails3 is a rack library and puppet, too. What we’re going to do is use the rackup file–‘.ru’ extension, by convention, which contains instructions for a rack aware web-server, like thin–shipped with Puppet. Then we’ll create several thin / puppet master processes, load-balancing over them with nginx and managed by supervisord.

Installing thin is a breeze:

puppet:~# gem install thin --no-ri --no-rdoc
Successfully installed eventmachine-0.12.10
Successfully installed daemons-1.1.6
Successfully installed thin-1.3.1
3 gems installed

Again, I’ve elected to not install documentation. You should notice that there is now a thin executable in your $PATH, but not in a really pleasing place to type out. The Debian alternatives system again:

puppet:~# update-alternatives --install /usr/bin/thin thin /var/lib/gems/1.9.1/bin/thin 400

The rackup file for puppet is /var/lib/gems/1.9.1/gems/puppet-2.7.9/ext/rack/files/config.ru and should be copied into /etc/puppet:

puppet:~# cp /var/lib/gems/1.9.1/gems/puppet-2.7.9/ext/rack/files/config.ru /etc/puppet/

In /etc/supervisor/conf.d/puppetmaster.conf create a file with this content:

[program:puppetmaster]
numprocs=3
command=/usr/bin/thin start -e development --socket /var/run/puppet/master.%(process_num)02d.sock --user puppet --group puppet --chdir /etc/puppet -R /etc/puppet/config.ru
process_name=%(program_name)s_%(process_num)02d
startsecs=5

Thin is run in development mode so that it will not daemonize. The supervisord option numprocs is used to cluster instances, at small cost of RAM and initial complexity. Be sure not to spin these processes up as we do not yet have puppet configuration in place.

Third step, Nginx+Thin: servicing puppet clients

By default, puppet master stores its SSL certificates in /etc/puppet. This is less than ideal: we’re going to version the manifests and configuration in this directory with git–and we don’t want to check-in private certificates, just in case. Not to mention that there are better places on disk for SSL certs. Create /etc/puppet/puppet.conf like so:

[main]
ssldir=$vardir/ssl

[master]
certname=puppet

Hopefully the ‘ssldir’ is sensible enough. That last line certname=puppet changes the default name of the puppet master pem from the FQDN to, well, just ‘puppet’. The FQDN pem makes it difficult to have a generic and relatively simple bootstrapping process because we’d be forced to tweaking our actions slightly for every new domain; boot-strapping should be as quick and as simple as possible. Else, you’ll probably forget a step and spend forty-five frustrating minutes wondering why nothing works when I’ve been so blase in declaring the straight-forward nature of this work.

You might wonder why there have been two files placed in /etc/puppet but no versioning having been applied. The deployment process will assume that your puppet configuration is held in a git repository. I'll walk through this in the next article, introducing an interim deployment strategy as we work to get the kit online sufficient to support the final process.There won't be any more edits to files in /etc/puppet for the remainder of this article.

You can execute the puppetmasterd script in non-daemon, debug mode and determine if your installation has gone well. You should see a ‘Finishing transaction’ message as the final output of this command:

puppet:~# puppetmasterd --no-daemonize --debug

Send the process SIGINT to return your terminal; you won’t hurt anything. For restricted, privileged writing of pidfiles, domain sockets and logs:

puppet:~# mkdir /var/run/puppet && chown puppet:puppet /var/run/puppet
puppet:~# mkdir /var/log/puppet && chown puppet:puppet /var/log/puppet

With thin running all that remains is getting nginx to front for it. Edit /etc/nginx/sites-enabled/default to read

server {
  # I'm assuming that 'puppet' resolves to a private IP. Don't run Puppet on the public internet!
  listen puppet:8140;

  ssl on;
  ssl_certificate /var/lib/puppet/ssl/certs/puppet.pem;
  ssl_certificate_key /var/lib/puppet/ssl/private_keys/puppet.pem;
  ssl_ciphers ALL:-ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;
  ssl_client_certificate  /var/lib/puppet/ssl/ca/ca_crt.pem;
  ssl_verify_client optional;

  # Force all filetypes to be sent raw. (Ouch!)
  types { }
  default_type application/x-raw;

  # serving static files from mount point
  location /production/file_content/files/ {
    ## problem here is that puppet factor doesn't give compact CIDR for network
    # allow   192.168.56.0/16;
    # deny    all;

    alias /etc/puppet/files/;
  }

  # serving static files from modules mounts
  location ~ /production/file_content/[^/]+/files/ {
    ## see above
    # allow   192.168.56.0/16;
    # deny    all;

    root /etc/puppet/modules;

    # rewrite /production/file_content/module/files/file.txt
    # to /module/file.text
    rewrite ^/production/file_content/([^/]+)/files/(.+)$  $1/$2 break;
  }

  location / {
    proxy_pass http://puppet-production;
    proxy_redirect   off;
    proxy_set_header Host             $host;
    proxy_set_header X-Real-IP        $remote_addr;
    proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header X-Client-Verify  $ssl_client_verify;
    proxy_set_header X-Client-Verify  SUCCESS;
    proxy_set_header X-Client-DN      $ssl_client_s_dn;
    proxy_set_header X-SSL-Subject    $ssl_client_s_dn;
    proxy_set_header X-SSL-Issuer     $ssl_client_i_dn;
  }
}

and /etc/nginx/conf.d/puppet-production-upstream.conf to read:

upstream puppet-production {
  server unix:/var/run/puppet/master.00.sock;
  server unix:/var/run/puppet/master.01.sock;
  server unix:/var/run/puppet/master.02.sock;
}

We’re causing the local nginx server to respond to hostname ‘puppet’ requests by passing them on to the puppet master backends responding over the domain sockets we defined in the last section. Nginx handles the SSL mongering for puppet and sets proxy values, as defined in the ‘default’ vhost.

Note that, thanks to our use of certname in puppet configuration we can refer to the generic ‘puppet.pem’ in the ssl configuration. I won’t elaborate on the hows and the whys–mostly because they’re uninteresting–but if you’d like to read up on the Nginx bits do so here and search this document. If I’ve skipped over something unrealistically fast drop me a line. You’ll find my email address in the footer of this page. Search for ‘Contact me.’

You should now be able to start the puppet master. First, restart nginx:

puppet:~# /etc/init.d/nginx restart

Now cause supervisord to re-read its configuration and start up the thin process hosting puppet master.

puppet:~# supervisorctl reread
puppet:~# supervisorctl update
puppet:~# supervisorctl start puppetmaster:*

Before you attempt to run the puppet agent, be aware that puppet master running on ruby 1.9.2 has an interesting issue, which, since we’re using a base system, is going to be easy enough to correct. Do all of the following, taking careful note of the systems the commands are run on.

puppet:~# ln -s /var/lib/puppet/ssl/certs/ca.pem $(openssl version -d|cut -d\" -f2)/certs/$(openssl x509 -hash -noout -in /var/lib/puppet/ssl/certs/ca.pem).0

You should be able to restart nginx and issue

puppet:~# puppet agent --test --noop

with no problems to report. The final message output will be:

Exiting; no certificate found and waitforcert is disabled

This means that the puppet agent did make a connection to the puppet master server but rejected sending any catalogs down as the key presented by the client was unknown to the server: you must sign the certificate.

puppet:~# puppet cert --sign puppet.troutwine.us
notice: Signed certificate request for puppet.troutwine.us
notice: Removing file Puppet::SSL::CertificateRequest puppet.troutwine.us at '/var/lib/puppet/ssl/ca/requests/puppet.troutwine.us.pem'

(Your fully qualified domain will vary. See the help text of puppet cert for more details.) To bring the base system up to speed, note the output of

puppet:~# cat /var/lib/puppet/ssl/certs/ca.pem

then, on the base system,

base:~# mkdir -p /var/lib/puppet/ssl/certs
base:~# chown puppet:puppet /var/lib/puppet/ssl/certs/
base:~# cat << EOF >> /var/lib/puppet/ssl/certs/ca.pem
YOUR CERTIFICATE KEY TEXT GOES HERE
EOF
base:~# chmod 644 /var/lib/puppet/ssl/certs/ca.pem
base:~# ln -s /var/lib/puppet/ssl/certs/ca.pem $(openssl version -d|cut -d\" -f2)/certs/$(openssl x509 -hash -noout -in /var/lib/puppet/ssl/certs/ca.pem).0

Every system cloned from the base from this point forward will be keyed to the created puppet master key. That means, should the master key change you’ll need to manually swap them on every live host and update the base image.

What have we got and where to next?

Venerable Gon'yo asked Joshu,
  "How is it when a person does not have a single thing?"
Joshu said,
  "Throw it away."
Gon'yo said,
  "I say I don't have a single thing. What could I ever throw away?"
Joshu said,
  "If so, carry it around with you."

Gon'yo's One "Thing"

We have a little more than nothing to carry around, happily. What we do have is a pre-configured base system coupled to a central puppet master, though with no version control of puppet manifests, no puppet manifests and nothing to ensure that puppet is actually doing the job we set for it to do. That’s all fixable.

In the next article we’ll accomplish three things:

version controlled manifests,
the self-hosting of puppet and
the tech needed to get a more convenient deployment model going.

Articles thereafter will cover the inclusion of ops management tools, including:

Puppet Dashboard and
Redmine.

Happy hacking!