Unknown and partly hidden

Pulling strings

anrxc — 2011-11-06T00:17:09+01:00

After one year of managing a network of 10 servers with Cfengine I'm currently building two clusters of 50 servers with Puppet (which I'm using for the first time), and have various notes to share. With my experience I had a feeling Cfengine just isn't right for this project, and didn't consider it seriously. These servers are all running Debian GNU/Linux and Puppet felt natural because of the good Debian integration, and the number of users whom also produced a lot of resources. Chef was out of the picture soon because of the scary architecture; CouchDB, Solr and RabbitMQ... coming from Cfengine this seemed like a bad joke. You probably need to hire a Ruby developer when it breaks. Puppet is somewhat better in this regard.

Puppet master needs Ruby, and has a built-in file server using WEBrick. My first disappointment with Puppet was WEBrick. Though PuppetLabs claim you can scale it up to 20 servers, that proved way off, the built-in server has problems serving as little as 5 agents/servers, and you get to see many dropped connections and failed catalog transfers. I was forced to switch to Mongrel and Nginx as frontend very early in the project, on both clusters. This method works much better (even though Apache+Passenger is the recommended method now from PuppetLabs), and it's not a huge complication compared to WEBrick (and Cfengine which doesn't make you jump through any hoops). Part of the reason for this failure is my pull interval, which is 5 minutes with a random sleep time of up to 3 minutes to avoid harmonics (which is still a high occurrence with these intervals and WEBrick fails miserably). In production a customer can not wait on 30/45 minute pull intervals to get his IP address whitelisted for a service, or some other mundane task, it must happen within 10 minutes... but I'll come to these kind of unrealistic ideas a little later.

Unlike the Cfengine article I have no bootstrapping notes, and no code/modules to share. By default the fresh started puppet agent will look for a host called "puppet" and pull in what ever you defined to bootstrap servers in your manifests. As for modules, I wrote a ton of code and though I'd like to share it, my employer owns it. But unlike Cfengine v3 there's a lot of resources out there for Puppet which can teach you everything you need to know, so I don't feel obligated to even ask.

Interesting enough, published modules would not help you get your job done. You will have to write your own, and your team members will have to learn how to use your modules, which also means writing a lot of documentation. Maybe my biggest disappointment is getting disillusioned by most Puppet advocates and DevOps prophets. I found articles and modules most of them write, and experiences they share have nothing to do with the real world. It's like they host servers in a magical land where everything is done in one way and all servers are identical. Hosting big websites and their apps is a much, much different affair.

Every customer does things differently, and I had to write custom modules for each of them. Just between these two clusters a module managing Apache is different, and you can abstract your code a lot but you reach a point where you simply can't push it any more. Or if you can, you create a mess that is unusable by your team members, and I'm trying to make their jobs better not make them miserable. One customer uses an Isilon NAS, the other has a content distribution network, one uses Nginx as a frontend, other has chrooted web servers, one writes logs to a NFS, other to a Syslog cluster... Now imagine this on a scale with 2,000 customers and 3 times the servers and most of the published infrastructure design guidelines become laughable. Instead you find your self implementing custom solutions, and inventing your own rules, best that you can...

I'm ultimately here to tell you that the projects are in a better state then they would be with the usual cluster management policy. My best moment was an e-mail from a team member saying "I read the code, I now understand it [Puppet]. This is fucking awesome!". I knew at that moment I managed to build something good (or good enough), despite the shortcomings I found, and with nothing more than using PuppetLabs resources. Actually, that is not completely honest. Because I did buy and read the book Pro Puppet which contains an excellent chapter on using Git for collaboration on modules between sysadmins and developers, with proper implementation of development, testing and production (Puppet)environments.

Reamde

anrxc — 2011-09-29T03:45:09+01:00

I received a copy of REAMDE by Neal Stepehenson, his latest book, two weeks before publication date and slowly worked my way through it. Having finished it I can say I really liked it. A lot of reviewers are disappointed that it wasn't nearly as complex as his earlier work, like Cryptonomicon and Anathem. But it was still a 1200 pages volume of Stephenson awesomeness. This is a thriller set in modern day, not an SF book.

Reason I enjoyed it a lot are the characters, a lot of them were (black or whitehat) hackers, sysadmins, programmers and gamers. Perhaps I felt closer to them than some readers did, who might have expected an epic historical novel or SF from Stephenson.

Book opens up with Richard, a CEO of Corporation 9592 that produces a massive online role-playing game called T'Rain. Somewhat like WoW but with a twist, the idea behind the world and how it was built is one of the best chapters in the book. Some hackers found a way to exploit a vulnerability shared by T'Rain players and wrote malware that encrypts their data with PGP and holds them for ransom. Payment is done in gold in the T'Rain world. At some point they ransom data of some carders, and their handlers who are very bad people and all hell breaks loose. Mobsters and terrorist cells get involved and story jumps to multiple continents. Neal Stephenson once said he likes making a good yarn, and he delivered on it again.

Building a VDR with Arch Linux

anrxc — 2011-06-26T01:10:29+01:00

Early in the last decade I had a so called SatDSL subscription, it was very expensive but the only way to get decent bandwidth. With the subscription I received a DVB-S PC tuner, a Technisat SkyStar2. A few years later when the subscription ran out I was left with the tuner and got interested in other ways of using the hardware. Those were some dark times for content providers, encryption systems were getting hacked left and right. The Wired magazine ran a great piece a few years ago about this period. Personally I was more interested in network traffic at the time, as a lot of it was not encrypted.

I used VDR software back then to build an approximation of a commercial SAT receiver. A friend put together an infra-red receiver for use with LIRC and I was set. With the rising popularity of HDTV and the DVB-S2 standard my current 'receiver' was getting outdated. It had a Duron 1200 CPU, with 300MB of SDRAM and a 320GB of storage. I was putting money on the side for several years to build a replacement (and get an appropriate HD TV set), and finally did it this year. These past few months I used Arch Linux with VDR, and built it piece by piece as a hobby. On the hardware side I used:

MBO: Asrock M3AUCC
CPU: AthlonII X3 455
RAM: 4GB DDR3
HDD: 1TB disks (RAID1)
GFX: GeForce GT240 (VDPAU)
DVB: SkyStar HD2 (DVB-S2 tuners)
LCD: Philips 43" HDTV

I didn't go with one of the Ion systems, because I don't watch that much TV, and the box will be put to many other uses (backups, TOR gateway, media streaming, distributed compiling, etc.). Every piece was carefully selected, and folks on the VDR mailing list helped me choose a suitable graphics card for HDTV, with good VDPAU support. As usual with Linux even that careful selection wasn't enough to ensure a smooth ride.

The sound-card drivers were deadlocking the machine, and it took a few days of experimentation to resolve it. The SkyStar HD2 cards work great with the mantis driver, but support for their infra-red receiver is missing. However experimental support is available through a patch published on the Linux-media mailing list, earlier this year.

All my previous VDRs were powered by Slackware Linux, and I developed some habits and preferences in running VDR. So I didn't use the ArchVDR project, or VDR packages, although they are good projects. I prefer to keep every VDR installation under a single directory tree, with the one currently in use always being symlinked to /opt/VDR. I swap them a lot, always patching (liemikuutio, ttxtsubs...), and testing a lot of plugins.

Instead of using runvdr and other popular scripts for controlling VDR I wrote my own script long ago. I call it vdrctl, it's much simpler but provides extensive control over loaded plugins and their own options, and has support for passing commands directly to a running VDR with svdrpsend. As the window manager I used awesome this time, opposed to FVWM that can be seen in this old gallery. Awesome runs in a fullscreen layout, and with some useful widgets.

For playing other media my long time favorite is Oxine, a Xine frontend well suited for TV sets. But I don't use its VDR support as one might expect (I use the vdr-xine plugin with regular xine-ui), instead I connect VDR with Oxine through the externalplayer-plugin. It's special in that it breaks the VDR connection to LIRC prior to launching an external player allowing to seamlessly control both applications.

Arch Linux makes it easy to build your own packages of all the media software to include VDPAU support. Official mplayer already has it, Xine is available in the AUR as xine-lib-vdpau and so on. But even more important, the good packaging infrastructure and management makes it easy to build and maintain custom kernels (for IR support ie.). It truly is the perfect VDR platform.

org-mode and the cycle

anrxc — 2011-06-22T03:57:09+01:00

I go through 50 or so support tickets, and Nagios alerts that must be acted on, in a given shift. There's also long term projects to manage. That's a lot of work, and a lot of distractions for a single work day. To keep it all under control I turned to the book "Time Management for Sysadmins" after enjoying some other books of the author. The book promotes a "cycle" system, with one TODO list/sheet per day, where all unfinished items from the previous day are transferred onto the next, and a few minutes invested in prioritizing and planning the day. There's much more to the method, but that's the gist of it, avoiding the "never ending TODO list of DOOM".

Tools of choice for the author are pen and paper, or a PDA is suggested as an alternative. I wouldn't carry a journal and don't own a PDA, but I already depended on org-mode heavily for my personal projects management and nothing could replace that, really. So while I was planning how to apply this new found wisdom and pair it with org-mode I came up with a system I wish to describe here. I wanted to do that for a long time, even though it's very specific to my needs, so most likely nobody can make use of it as is. But perhaps it gives you ideas, and even that would be great

Org-mode is a top notch project management tool, inside a good editor. Great tools for the job. Yet it's very easy to fall into the "list of doom" trap. In the context of org-mode its gurus actually recommend keeping big lists with a small number of files, because of good agenda overview and unlimited possibilities of org-mode... but I decided to avoid it and imitate the system the book promotes with one list per day, and project in my case (author splits them into sub-tasks, for day lists). If you're familliar with the cycle system and thinking it doesn't make sense read on, org-mode agenda view is the key.

I start my day with a terminal emulator, and I realized it would be best to have a dedicated tool that can be used from the command line. So I wrote mkscratch, a simple tool that sets me up with a scratchpad for the day, helps managing the old pads, but also pads of long term projects. It's not actually that specific to org-mode, yet it was written with it in mind so headers and file extensions reflect that. But it is easily a general purpose editor wrapper for creating scratchpads or project pads.

This is how I tie it all together, day-to-day pads are in ~/work/pads, and once invoked mkscratch creates a pad for the day, if one doesn't already exist, and links ~/pad.org to ~/work/pads/MM-DD.org. Then it starts Emacs if not already running, and opens the pad for the day. I might have scheduled a task days or months in advance and created a pad for that day, then only linking of ~/pad.org is performed and editor is opened. Long term projects get their own directories below ~/work/projects (because projects have other files as baggage), with some project identifier as the directory name, while the main pad for every project is called project.org. This is of some importance (see the .emacs snippet below).

A good part of the author's cycle system is maintaining a calendar with meetings and project deadlines meticulously marked. From the calendar items are copied onto the list for the day when their time is up (during those few minutes of planning, before the day starts or at the end of the previous day). While contemplating how to translate that to org-mode I realized using the Agenda View as is will do just fine, if only I could gather all scheduled items and approaching deadlines from all the files (pads and projects) into this single view. Even though it's not a very conventional usage pattern for org-mode (remember, we have up to 300 files near the years end), it can be done easily in .emacs:

;; Files that are included in org-mode agenda
(setq org-agenda-files
  (cons "~/work/pads/"
    (file-expand-wildcards 
     "~/work/projects/*/project.org"))
)

One of my past articles described connecting my window manager with org-mode for taking quick notes. I use the same method now for appending to pad files, not always, but it's very useful to quickly store a task when you're in the middle of something.

A few years ago I wrote another short article on org-mode, using it for project management while freelancing, it describes a different method but a part of it focused on using GPG to encrypt project files. The same can be applied here. EasyPG is now a part of Emacs, and every pad can be encrypted on the fly, and decrypted without any overhead in usage (even more so when using the GPG agent).

LILO to syslinux migration

anrxc — 2011-04-05T03:12:38+01:00

I've been using LILO for a long time, from my first GNU/Linux installation. Some people have the idea that it's a dead project, but it's not, development is still active. It never failed to boot for me. Until last night, when it failed to use an XZ compressed Linux v2.6.38.2. I would never give up LILO for Grub, but I was contemplating a switch to syslinux after watching the Fosdem talk by Dieter Plaetinck last month.

I had no more excuses to delay it any longer, so I switched. All my machines have /boot as the first partition on the disk, they are flagged bootable, and use the ext2 file-system. Migrating to syslinux was simple:

# pacman -Syu syslinux
# /usr/sbin/syslinux-install_update -i -a -m

# cp -arp /etc/lilo.conf /etc/lilo.conf.bak
# pacman -Rns lilo

The syslinux-install_update options stand for installing it to the MBR, where it replaces LILO. Then I wrote a configuration file: /boot/syslinux/syslinux.cfg. Actually, I wrote one in advance, because even though you might find it funny, it was a big deal for me to continue using my boot prompt as is. I've been using the LILO fingerprint menu for years. Not being able to port it would be a deal breaker. But it wasn't complex, actually.

If you want to use the Fingerprint theme, grab it from KDE-Look, convert the BMP to the PNG format and save it as splash.png. I'll include my whole syslinux.cfg for reference, but the important bits for the theme are menu and graphics sections. I didn't bother to tweak the ANSI settings as I don't intend to use them, so I copied them from the Arch Linux menu. Settings below will give you a pretty much identical look and feel (shadows could use a bit more work), and also provide a nice bonus over LILO when editing the boot options (by pressing Tab, as in LILO):

#
# /boot/syslinux/syslinux.cfg
#
# General settings
UI vesamenu.c32
DEFAULT linux
PROMPT 0
TIMEOUT 250

# Menu settings
MENU WIDTH 28
MENU MARGIN 4
MENU ROWS 3
MENU VSHIFT 16
MENU HSHIFT 11
MENU TIMEOUTROW 13
MENU TABMSGROW 11
MENU CMDLINEROW 11
MENU HELPMSGROW 16
MENU HELPMSGENDROW 29

# Graphical boot menu
#
# Fingerprint menu
#  http://kde-look.org/content/show.php/Fingerprint+Lilo-splash?content=29675
MENU BACKGROUND splash.png
#
#          element      ansi    f/ground  b/ground  shadow
MENU COLOR sel          7;37;40 #ffffffff #90000000 std
MENU COLOR unsel        37;44   #ff000000 #80000000 std
MENU COLOR timeout_msg  37;40   #00000000 #00000000 none
MENU COLOR timeout      1;37;40 #00000000 #00000000 none
MENU COLOR border       30;44   #00000000 #00000000 none
MENU COLOR title        1;36;44 #00000000 #00000000 none
MENU COLOR help         37;40   #00000000 #00000000 none
MENU COLOR msg07        37;40   #00000000 #00000000 none
MENU COLOR tabmsg       31;40   #00000000 #00000000 none

# Boot menu settings
LABEL linux
        MENU LABEL GNU/Linux
        LINUX ../vmlinuz26
        APPEND root=/dev/sda2 ro rootflags=data=ordered i915.modeset=1 video=VGA-1:1152x864 drm_kms_helper.poll=0
        INITRD ../kernel26.img

LABEL recovery
        MENU LABEL Recovery
        LINUX ../vmlinuz26
        APPEND root=/dev/sda2 ro rootflags=data=ordered
        INITRD ../kernel26-fallback.img

You can find the documentation, and menu options explained on the syslinux wiki. If you achieve an even more faithful copy send me an e-mail with the new values. Thank you.

Introducing play, a fork of cplay

anrxc — 2011-04-02T20:59:36+01:00

I've been using cplay for close to ten years now. It is a curses front-end to various audio players/decoders, and written in Python. Sure, I've been an Amarok fan for half that time, but when I just want to hear some music I find my self opening a terminal and starting cplay. I manage my music collection in Amarok, I grab and listen new podcasts in Amarok. Sometimes I even use it to play music, but not nearly as much as I do with cplay. I have 4 workstations at home, and they all do the same. Same thing with the server connected to the best set of speakers. Sure, I have a remote and Oxine there, but when I just want to hear some music I don't want to spend 5 minutes messing with the remote.

Through the years I added various small patches to my copy of cplay. They accumulated over time, and except for my color-support patch I didn't plan on sharing them. But in 2009 I found that the project page of cplay disappeared. I spent a year thinking it will pop-up, but it didn't. Then I noticed the Arch Linux package for cplay pulls the source from the Debian repositories, and realized it's not coming back.

I decided to publish my copy of cplay, so there exists yet another place where it's preserved. But as I'm not acting in any official role, nor do I consider my self a worthy coder to maintain cplay I decided to fork it and publish under a new name. That also gives me the excuse to drop anything I don't personally use, like gettext support. My project is called play, just play and the Git repository is now public, on git.sysphere.org. The first commit is an import of cplay-1.50pre7 from Ulf Betlehem, so if you're looking for that original copy you can grab it there.

Beside various bug fixes some of the more interesting new features are: color support, mplayer support, curses v5.8 support and pyid3lib support. Someone on IRC told me this week that they could never get cplay to work for them on Arch Linux, and they expressed interest in play. I decided to package it on AUR, and it's now available as play-git.

Securing Debian repositories

anrxc — 2011-03-29T05:07:09+01:00

I had to build some Debian packages recently after a long time. The experience really made me appreciate the simple approach taken by Arch Linux. When I finally built something up to Debian standards I had to distribute it on the network. Debian's APT can work over HTTP, FTP... but also SSH, which is pretty good for creating a more secure repository. There are a lot of articles covering this approach, but I didn't find any information on how to make it work in a chroot environment, and got stuck there for a while.

But to go back to the beginning, if you're setting up a repository the reprepro tool is pretty good for repository management. Having SSH in mind for later, some decisions have to be made where you'll store the repository and who will own it. The "reprepro" user is as good as any, so we can start by adding the user (and group) to the system and making its home /srv/reprepro. Then we can setup the bare repository layout, in this example the repository will reside in the debs directory:

# su reprepro
$ cd ~
$ mkdir -p debs/conf

You'll need two files in the conf sub-directory, those being "distributions" and "options". They are simple to write, and all the other articles explain them. You don't have to worry about the rest of the tree, once you import your first package, or .changes file, reprepro will take care of it then. If you intend to use GPG, and sign your Release file, it's a good time to create your signing key. Then in the distributions file configure the Sign option, and in the conf file add ask-passphrase.

Now we can add another user, one for APT clients, its home can be the same as our reprepro user has. User should be allowed to connect in the SSH daemon configuration file, and properly chrooted using the ChrootDirectory option. Here are the (only) binaries you will need in /srv/reprepro/bin: bash, find and dd. I mentioned getting stuck, well this was it, APT is using find and dd binaries internally, which strace revealed.

You can now publish your repository in /etc/apt/sources.list, for example:

deb ssh://apt@192.168.50.1:/debs/ squeeze main
deb-src ssh://apt@192.168.50.1:/debs/ squeeze main

That's the gist of it. User apt gets read-only access to the repository, if coming from an approved host. You can control host access with TCPWrappers, Iptables and even OpenSSH's own whitelist. Each APT client should have its key white-listed for access as well, but give some thought to key management, they don't have to be keys without a passphrase. You can setup the SSH agent on a machine you trust, and unlock a key. Using agent forwarding provided by the OpenSSH client you could login to a machine and install packages without being prompted for the passphrase, and without leaving your keys laying around. This alone would not scale in production, but is a good start.

Cfengine3 bootstrap

anrxc — 2010-11-17T00:47:52+01:00

Cfengine has been around for a very long time. Its dependencies are minimal, its resources needs minimal, its performance excellent. Version 2 of cfengine is well documented, thoroughly discussed and described in numerous tutorials. But v2 is also deprecated in favor of cfengine3. It so happens v3 was a rewrite, and many things changed. There are a few tutorials on v3 worth mentioning, but they all describe deploying the policy host, and little else.

I've been reading "Automating Linux and Unix System Administration" recently and the book covers cfengine2 extensively. Authors describe a complete, extremely modular, cfengine setup. I had problems wrapping my brain around methods for doing the same in v3. After spending some time with the cfengine reference manual I finally had a solid basic policy for deploying the policy host and clients. Which I want to share here: cf3-bootstrap.tar.bz2.

After installing cfengine with its workdir defined, let's say, as /srv/cfengine, we can bootstrap the policy server:

# /usr/sbin/cf-key
# cd /srv/cfengine/
# tar -xvjf cf3-bootstrap.tar.bz2 ; rm -i $_
# cd masterfiles/
# cp /usr/share/doc/cfengine/cfengine_stdlib.cf cf-stdlib.cf
# emacs update.cf
# /usr/sbin/cf-agent --bootstrap
# /usr/sbin/cf-execd -v -F

The cf-key will create the basic cfengine directory tree, and will generate the hosts public and private keys (cfengine functions somewhat like SSH in this regard). We edit the update.cf. to define our policy server, and its hostname. Other files will require more edits for customization, but update bundle is all that is needed for bootstrapping the policy host (and later pulling changes in policy).

Method for bootstrapping a client is similar. We run cf-key, and we copy the failsafe.cf and update.cf files from the policy host to /srv/cfengine/inputs. Then run cf-agent from that directory to bootstrap, and finally cf-execd's initial run (which BTW configures a cronjob for cf-execd to be executed periodically).

A few words about basic cfengine operation and policy. It is kept, and modified in the masterfiles directory. Remember, the policy host also manages it self, which means it too copies the policy to its inputs directory, and it runs from there. Clients pull the policy from the masterfiles directory on your policy host to their inputs directory, from where they are acted upon. The policy defines that our policy server runs the cf-serverd daemon, which acts as a file server for our clients. First time the client contacts the server their keys will be exchanged.

Example bundles I provided propose to keep the service configuration files, and any other data for distribution in the clientfiles directory. I am a long way from templating in cfengine3, but the authors of the book I mention are willing to argue distributing complete files is the right way to do things.

Now a few words about the policy I shared. I wanted a modular policy, one that will easily scale from 3 servers to 23. Cfengine configuration, and basic policy is in the masterfiles directory, I make use of the cfengine community library (distributed with cfengine package), and I also maintain a site specific library.cf file. The promises.cf file is the main cfengine policy file, it imports all needed policy files, and defines policies that should run on all hosts. All site specific policies are in the site/ sub-directory. The main file site.cf defines policies that should run per host (or groups of hosts), and does nothing else. Policies for daemons or anything else get their own file. I included a few basic ones with examples. One with some example Apache controls, one for editing resolv.conf and one generic that pretends to run actions on system users.

Where to take it from bootstrapping. First and foremost, policy should be kept in revision control. Git will do nicely. With several cfengine policy hosts, controlling clusters or groups of machines, we can think about masters of master servers. In that scenario the policy hosts can pull their complete policies with Git from the master(s). Developing complex policies will take some time, but it's worth investing your time. It will make your job better, but also be a good prospect for the future. I don't remember seeing an ad for a bigger shop that didn't require cfengine or puppet experience.

Mobile broadband and OpenVPN on GNU/Linux

anrxc — 2010-09-16T15:18:09+01:00

I wrote a short article on GPRS more than a year ago. Now I finally moved on to HSPA, after GPRS failed me in an emergency. I was unable to get OpenVPN working over it, and to make matters worse I found my (extremely expensive) accumulated bandwidth disappeared from the T-Com account. I decided to switch the provider, and technology, and bought a Huawei E180 mobile broadband modem from TELE2.

With a fairly recent Linux kernel the modem will be recognized correctly as storage (it has a MicroSD card slot), CD-ROM (read-only part of the stick with MS Windows software/driver) and a modem. So, usb_modeswitch is not needed. Modem can be plugged in and usbserial and ppp-generic modules should be loaded. As a PPP dialer I continue to use wvdial, configured like so:

; wvdial configuration
;   /etc/wvdial.conf
;
[Dialer tele2pin]
Modem = /dev/ttyUSB0
baud = 460800
Init1 = AT+CPIN=XYZW

[Dialer tele2]
Modem = /dev/ttyUSB0
baud = 460800
Stupid Mode = 1
Init2 = ATZ
Init3 = AT&F E1 V1 X1 &D2 &C1 S0=0
Init4 = AT+CGDCONT=1,"IP","data.tele2.hr"
Phone = *99#
Username = none
Password = none

Under the tele2pin dialer a PIN must be provided in place of "XYZW". Username and password in the tele2 dialer are not used, but must be provided. The APN "mobileinternet.tele2.hr" always produced a "No carrier detected" message so I switched to "data.tele2.hr". The Stupid mode is enabled so there's no time wasted waiting on the prompt, and that's about all you need to know.

To initiate a connection a PIN must be provided first, that's why there's a special dial section for it. Afterwords a connection can be established:

# wvdial tele2pin
# wvdial tele2

Unfortunately OpenVPN failed to work with this setup as well. Some mobile providers block ports, some do double NAT or otherwise mess with VPN connections... but not TELE2, and I eventually got it to work. I found it was a routing problem, after seeing the following pppd message in the syslog:

pppd: Could not determine remote IP address: defaulting to 10.64.64.64

The routing table would look like this:

Kernel IP routing table
Destination     Gateway Genmask         Flags Metric Ref Use Iface
10.64.64.64     *       255.255.255.255 UH    0      0     0 ppp0
default         *       0.0.0.0         U     0      0     0 ppp0

I determined the gateway by doing a simple traceroute to google.com, and a quick fix was:

# route del default gw 0.0.0.0
# route add -host 130.244.219.90 dev ppp0
# route add default gw 130.244.219.90
# route del -host 10.64.64.64
# /etc/rc.d/openvpn start

The packets will be properly routed now, but the DNS servers of the mobile provider can no longer be reached. Change your DNS servers to the ones provided by your (virtual)private network, or otherwise any DNS servers you would normally use.

Books on systems administration

anrxc — 2010-08-04T18:16:59+01:00

I had some money to spare recently, and I decided to buy some good technical books. Actually I went out to find some great books, and not so much with an agenda to learn from them but more to be used as reference for years to come.

First book I settled on was the Principles of Network and System Administration by Mark Burgess. Last edition is from 2003 but this book is still excellent and unique in many ways. It is one of the only (if not the only) book approaching the topic of systems administration from a scientific standpoint, treating it as a branch of engineering. Book on principles, best practice, ethics, users and management of the human-computer systems. At times it seems perhaps too academic in its approach in respect to business environments where something is always going wrong (Murphy is out to get you) and is often hard to maintain levels of operation this book advocates. Still it's something every good sysadmin should aspire to, I enjoyed re-reading it and it's just the kind of book I had in mind.

It felt kind of karmic that the fourth, 20th year anniversary, edition of the Unix and Linux System Administration Handbook was released these days. This series of books (previously UNIX and Linux treated separately) by Evi Nemeth (and others) is well known, and I just had to buy this edition. It is very much different than the book above, a 1300 pages volume filled with practical matters including real world experiences of its authors. The foreword says it is not one of those "administration" books targeting users who run a UNIX system in their garage. It's rather oriented towards running UNIX in the enterprise. This revised edition covers AIX, HP-UX, Solaris, RedHat, SuSE and Ubuntu, and as usual dedicates a large part of the book to networking. It could be worth to you to know that it also includes chapters on topics like DNSSEC and virtualization (being that the 3rd edition was released back in 2000).