In managed hosting you're not often present in design stages of new
applications and sometimes you end up supporting strange
infrastructure. Or at least that was my experience in the past. So
little by little I found my self supporting huge
(persistent) Redis databases, against
my better judgment.
Someone sent me a link to
the Redis Sentinel
beta announcement
last month. It may even make it into the 2.6 release... but
all of this I had to implement on my own long ago. A lot of developers
I supported didn't even want to use the 2.4 branch (in my
opinion just the memory fragmentation improvements are more than
enough reason to ditch 2.2 forever). Another highly
anticipated Redis feature,
the Redis Cluster,
may not even make it into the 2.6 release. That's too bad,
there's too much features with Redis that are always "just around the
corner", yet I have a feeling I'll be supporting Redis 2.4 for at
least another 3 years, with all its flaws and shortcomings (I
scratched the surface in my last article with AOF
corruption, and not-so-cheap hardware needed for reliable persistent
storage).
Typically I would split members of a Redis cluster across 2 or more
power sources and switches. But that's just common sense for any HA
setup, as is not keeping all your masters together. Redis doesn't have
multi-master replication so a backup 'master' is always passive, and
is just another slave of the primary with slaves of its own. If the
primary master fails only half of the slaves have to be failed-over to
the backup master. This has its problems (ie. double complexity of
replication monitoring by the fail-over agents), but the benefits
outweight failing-over a whole cluster to the new master. That could
take half a day, as fail-over is an expensive operation (it is a full
re-sync from the master). You can find replication implementation
details here.
If you can't allow slaves to serve stale data (tunable
in redis.conf) you need enough redundancy in the pool to be
able to run at half capacity for at least a few hours, until at least
one of the outdated slaves is fully re-synced to its new master. And
that finally brings me to knowing when is the right time to
fail-over.
Any half decent load balancer can export the status of a backend pool
through an API, or just a HTTP page (if yours can't it's time to use
the open source HAproxy). That
information is ripe for exploiting to our advantage, but we need to be
weary of false positives. I can't share my own solutions, but you will
want all N slaves confirming that the master pool is truly degraded,
and initiate fail-overs one by one to avoid harmonics if you are
serving stale data, or all at once if you aren't. For all that you
will need them to communicate with each other, and
a simple message broker can do
the job well.
As I am writing these last notes I realized I haven't mentioned
another fundamental part of any Redis deployment I do - backups. This
article documents
the persistence
implementation in Redis, and explains that
the RDB engine snapshots are good for taking
backups. RDB is never modified directly, and snapshots are renamed
into their final destination atomically only when they are
complete. From here it's trivial to write a script that initiates a
background save, waits until it's done and transfers the fresh
snapshot off site.