Hosting with Puppet - Design
Two years ago I was a small
time Cfengine user moving
to Puppet on a large
installation, and more specifically introducing it to a managed
hosting provider (which is an important factor driving my whole design
and decision making process later). I knew how important it's going to
be to get the base design right, and I did a lot of research on Puppet
infrastructure design guidelines but with superficial results. I was
disappointed, the DevOps crowd was producing tons of material
on configuration management, couldn't at least a small part be
applicable to large installations? I didn't see it that way then, but
maybe that knowledge was being reserved for consulting gigs. After
criticizing it is only fair that I write something of my own on the
subject.
First of all, a lot has happened since. Wikimedia decided
to release
all their Puppet code to the public. I learned a lot, even if most of
it was what not to do - but that was the true knowledge to be
gained. One of the most
prominent Puppet Forge
contributors, example42 labs,
released
the next
generation of their Puppet modules, and the quality has increased
immensely. The level of abstraction is high, and for the first time I
felt the Forge can possibly become a provider for me. Then 8
months ago the annual PuppetConf conference hosted engineers
from Mozilla
and Nokia
talking about design and scaling challenges they faced running Puppet
in a big enterprise. Someone with >2,000 servers sharing their
experiences with you, soak it up.
* Puppet design principles
Running Puppet in a hosting operation is a very specific use
case. Most resources available to you will concern running one or two
web applications, on a hopefully standardized software stack across a
dozen servers all managed by Puppet. But here you are a level above
that, running thousands of such apps and sites, across hundreds of
development teams that have nothing in common. If they are developing
web-apps in
Lisp you are there to facilitate it, not to tell
stories about
Python.
Some teams are heavily involved with their infrastructure, others
depend entirely on you. Finally, there are "non-managed" teams which
only need you to manage hardware for them but you still want to
provide them with a
hosted Puppet service. All this
influences my design heavily, but must not define it. If it works for
a 100 apps it must work for 1 just the same, so the design principles
below are universal.
- Object oriented
Do not treat manifests like recipes. Stop writing node
manifests. Write modules.
Huge manifests with endless instructions,
if conditionals,
and node (server) logic are a trap. They introduce an endless cycle of
"squeezing in just one more hack" until the day you throw it all away
and re-factor from scratch. This is one of the lessons I learned from
Wikimedia.
Write modules (see Modular services and Module levels) that are
abstracted. Within modules write small abstracted classes with
inheritance in mind (see Inheritance), and write defined types
(defines) for resources that have to be instantiated many times. Write
and distribute templates where possible, not static files, to reduce
chances of human error, to reduce number of files to be maintained by
your team, and finally number of files compiled into catalogs (which
concerns scaling).
Here's a stripped down module sample to clarify this topic, and those
discussed below:
# - modules/nfs/manifests/init.pp
class nfs (
$args = 'UNSET'
){
# Abstract package and service names, Arch, Debian, RedHat...
package { 'portmap': ensure => 'installed', }
service { 'portmap': ensure => 'running', }
}
# - modules/nfs/manifests/disable.pp
class nfs::disable inherits nfs {
Service['portmap'] { ensure => 'stopped', }
}
# - modules/nfs/manifests/server.pp
class nfs::server (
$args = 'UNSET'
){
package { 'nfs-kernel-server': ensure => 'installed', }
@service { 'nfs-kernel-server': ensure => 'running', }
}
# - modules/nfs/manifests/mount.pp
define nfs::mount (
$arg = 'UNSET',
$args = 'UNSET'
){
mount { $arg: device => $args['foo'], }
}
# - modules/nfs/config.pp
define nfs::config (
$args = 'UNSET'
){
# configure idmapd, configure exports...
)
- Modular services
Maintain clear roles and responsibilities between modules. Do not
allow overlap.
Maybe it's true that a server will never run
PHP without an
accompanying web server, but it's not a good reason to bundle PHP
management into the
apache2 module. Same principle is here to
prevent combining
mod_php and
PHP-FPM management
into a single module. Write
php5, phpcgi, phpfpm modules, and
use them for
Apache2, Lighttpd, Nginx web servers interchangeably.
- Module levels
Exploit
modulepath support. Multiple module paths are
supported, they can greatly improve your design.
Reserve default
/etc/puppet/modules path for modules exposing
the top level API (for lack of a better acronym). These modules should
define
your policy for all the software you
standardize on, how a software distribution is installed and how it's
managed:
iptables, sudo, logrotate, dcron, syslog-ng, sysklogd,
rsyslog, nginx, apache2, lighttpd, php5, phpcgi, phpfpm, varnish,
haproxy, tomcat, fms, darwin, mysql, postgres, redis, memcached,
mongodb, cassandra, supervisor, postfix, qmail, puppet it
self,
puppetmaster, pepuppet (enterprise edition),
pepuppetmaster...
Use the lower level modules for defining actual policy and
configuration for development teams in organizations (or customers in
the enterprise), and their servers. Here's an example:
- /etc/puppet/teams/t1000/
|_ /etc/puppet/teams/t1000/files/
|_ php5/
|_ apc.ini
|_ /etc/puppet/teams/t1000/manifests/
|_ init.pp
|_ services.pp
|_ services/
|_ encoder.pp
|_ webserver.pp
|_ webserver/
|_ production.pp
|_ users/
|_ virtual.pp
|_ /etc/puppet/teams/t1000/templates/
|_ apache2/
|_ virtualhost.conf.erb
For heavily involved teams the "
services" classes are here to
enable them to manage their own software, code deployments and
simillar tasks.
- Inheritance
Understand class inheritance, and use it to abstract your code to
allow for black-sheep servers.
These servers are
always present - that one server in
20 which does things "just a little differently".
# - teams/t1000/manifests/init.pp
class t1000 {
include ::iptables
class { '::localtime': timezone => 'Etc/UTC', }
include t1000::users::virtual
}
# - teams/t1000/manifests/webserver.pp
class t1000::webserver inherits t1000 {
include ::apache2
::apache2::config { 't1000-webcluster':
keep_alive_timeout => 10,
keep_alive_requests => 300,
name_virtual_hosts => [ "${ipaddress_eth1}:80", ],
}
}
# - teams/t1000/manifests/webserver/production.pp
class t1000::webserver::production inherits t1000::webserver {
include t1000::services::encoder
::apache2::vhost { 'foobar.com':
content => 't1000/apache2/virtualhost.conf.erb',
options => {
'listen' => "${ipaddress_eth1}:80",
'aliases' => [ 'prod.foobar.com', ],
},
}
}
Understand how resources are inherited across classes. This will not
work:
# - teams/t1000/manifests/webserver/legacy.pp
class t1000::webserver::legacy inherits t1000::webserver {
include ::nginx
# No, you won't get away with it
Service['apache2'] { ensure => 'stopped', }
}
Only a sub-class inheriting its parent class can override resources of
that parent class. But this is not a deal breaker, once you understand
it. Remember our "
nfs::disable" class from an earlier
example, which inherited its parent class "
nfs" and proceeded
to override a service resource?
# - teams/t1000/manifests/webserver/legacy.pp
class t1000::webserver::legacy inherits t1000::webserver {
include ::nginx
include ::apache2::disable
}
This was the simplest scenario. Consider these as well: legacy server
needs to run MySQL v5.1 in a cluster of v5.5 nodes, server needs
Nginx
h264 streaming support compiled into nginx binary and
its provider is a special package, server needs PHP 5.2 to run a
legacy e-commerce system...
- Function-based classifiers
Export only bottom level classes of bottom level modules to the
business,
as node classifiers:
# - manifests/site.pp (or External Node Classifier)
node 'man0001' { include t1000::webserver::production }
This leaves system engineers to define system policy with a 100%
flexibility, and allows them to handle complex infrastructure. They in
turn must ensure the business is never lacking, a server either
functions as a production webserver or not, it must never include top
level API classes.
- Dynamic arguments
Do not limit your templates to a fixed number of features.
Use hashes to add support for optional arbitrary settings that can be
passed onto resources in defines. When a developer asks for a new
feature there is nothing to modify, nothing to re-factor, options hash
(in earlier "
apache2::vhost" example) is extended and the
template is expanded as needed with new conditionals.
- Convergence
Embrace continuous repair.
Design for it.
Is it to my benefit to go all wild on class relationships to squeeze
everything into a single puppet run? But if just one thing changes
whole policy breaks apart. Micro manage class dependencies and
resource requirements. If a webserver refused to start because a
Syslog-ng
FIFO was missing we know it will succeed on the next
run. Within a few runs we can deploy whole clusters across
continents.
There is however a specific here which is not universal, a hosting
operation needs to keep agent run intervals frequent to keep up with
an endless stream of support requests. Different types of operations
can get away with 45-60 minute intervals, and sometimes use them for
one reason or another (ie. scaling issues). I followed the work of
Mark Burgees (author of Cfengine) for years and agree with Cfengine's
5 minutes intervals for just about any purpose.
- Configuration abstraction
Know how much to abstract, and where to draw the line.
Services like
Memcache and
MongoDB have a small set
of run-time parameters. Their respective "
*::config" defines
can easily abstract their whole configuration files into a dozen
arguments expanded into variables of a single template. Others like
Redis support hundreds of run-time parameters, but if you
consider that >80% of Redis servers run in production with default
parameters even a 100 arguments accepted by "
redis::config"
is not too much. For any given server you will provide 3-4 arguments,
the rest will be filled from default values, and yet when you truly
need to deploy an odd-ball Redis server the flexibility to do so is
there without the need to maintain a hundred
redis.conf
copies.
Services like
MySQL and
Apache2 can exist in an
endless number of states, which can not be abstracted. Or to be honest
they can, but you make your team miserable when you set out to make
their jobs better.
This is where you draw the
line. For the most complex software distributions abstract
only the fundamentals and commonalities needed to deploy the
service. Handle everything else through "
*::dotconf",
"
*::vhost", "
*::mods" etc. defines.
- Includes
Make use of includes in services which support them,
and those
that don't.
Includes allow us to maintain small fundamental configuration files,
which include site specific configuration from small configuration
snippets dropped into their
conf.d directories. This is a
useful feature when trying to abstract and bring together complex
infrastructures.
Services which do not support includes by default can fake them. Have
the "
*::dotconf" define install configuration snippets and
then call an
exec resource to assemble primary configuration
file from individual snippets in the improvised conf.d directory
(alternative approach is provided
by
puppet-concat). This
functionality also allows you to manage shared services across shared
servers, where every team provides a custom snippet in their own
repository. They all end up on the shared server (after review)
without the need to manage a single file across many teams (opening
all kind of access-control questions).
- Service controls
Do not allow Puppet to become the
enemy of the junior
sysadmin.
Every defined type managing a service resource should include 3
mandatory arguments, let's call them:
onboot, autorestart,
and
autoreload. On clustered setups it is not considered
useful to bring back broken or outdated members into the pool on boot,
it is also not considered useful to automatically restart such service
if detected as "crashed" while it's actually down for maintenance, and
often times it is not useful to restart such a service when a
configuration change is detected (and in the process flush 120GB of
data from memory).
Balance these arguments and provide sane defaults for every single
service on its own merits. If you do not downtime will occur. You will
also have sysadmins stopping Puppet agents the moment they login,
naturally forgetting to start it again, and 2 weeks later you realize
half of your cluster is not in compliance (Puppet monitoring is
important, but is an implementation detail).
- API documentation
Document everything. Use
RDoc markup and auto-generate HTML
with
puppet doc.
At the top of every manifest: document every single class, every
single define, every single of their arguments, every single variable
they search for or declare, and provide multiple usage examples for
each class and define. Finally include contact information, bug
tracker link and any copyright notices.
Puppet includes a tool to auto generate documentation from these
headers and comments in your code. Have it run periodically refreshing
your API documentation, and export it to your operations and
development teams. It's not just a nice thing to do for them, it is
going to save you from re-inventing the wheel on your Wiki
system. Your Wiki now only needs the theory documented; what is
Puppet, what is revision control, how to commit a change... and these
bring me to the topics of implementation and change management, which
are beyond the scope of design.