Unknown and partly hidden

Often it's hard to beat a few lines of shell script to perform a basic monitoring task on a personal system. A system-load, or a file-system monitoring script running from cron is extremely easy to write and setup, while being invaluable in emergencies.

I wrote plenty of those during the years (load, fs, pacct, daemons... monitors), which I use in combination with network and intrusion monitoring. And here's how it works; e-mail alerts from all personal systems are sent out to a third-party system where they are immediately relayed to my local mailbox, and then archived, but not marked as read. That way a backup remains, but that archive mailbox is also available through IMAP. While having e-mail alerts is good, a live alert is better. Live in my case means IM, and if I'm on the road: SMS - which is easily done through a service like email2sms offered by the mobile service provider. Although the IM part is more to the point of this article.

Years ago while learning Python I wrote two bots, an IRC one and a Jabber one. Dealing with a protocol like IRC is great for learning, it involves many different libraries and problems. Text processing, sockets, databases, accessing web resources... When I lost interest (don't we all) I decided maintaining and running the Jabber version is a better choice, and besides, I love that protocol. Sometime later hooking the Jabber bot into the existing alert system was easy thanks to the imaplib library. Bot connects to the IMAP server and relays, to Jabber, all alerts which have the Unseen flag. Once read an alert is flagged as Seen, and that's all there is to it. Live alerts 24/7, over my favorite protocol, in my favorite messaging client.

If you are interested in writing your own bots you can checkout the xmpppy library. Handling the connection, presence and subscriptions can be done in as little as 20 lines of code. To complete the cycle I should also mention some of my favorites for local monitoring. For process monitoring the htop project provides a great interactive replacement for top in procps. Finally, no article would be complete without mentioning nmon - an amazing AIX and Linux performance monitor, developed (and unofficially supported) by an IBM employee.

One of the earliest articles here was about my personal backup solution, rsync based, with rotating backup-snapshots. Implemented as a simple shell script. It served me well, but I was never fully satisfied with it, it had a few Todo items attached to it which I never got around to address. I couldn't motivate my self to write BASH, and the script was working the way it was designed, running on a dozen of my machines.

After reading "Python for Unix System Administration" last week I decided it's time to rewrite it, in Python. Result is available in a git repository; rybackup.git. The script is designed to backup chosen directories, and files to an NFS backup server, or removable storage. Maintaining an arbitrary number of backup-snapshots going back hourly/daily/weekly/monthly as long as you need, rotating them as it goes. All of this is controlled by a few settings at the top of the script. In addition it has a few functions making it suitable to use for backup of eCryptfs encrypted home directories. Script will exit with EAGAIN if the eCryptfs mount is active, and relies on Dillon's cron to retry once in a while, after receiving the signal. To give you an idea of how it works, here is a directory tree after a few months of rotating snapshots:

2010-05-30 16:04 daily.1/
2010-05-29 00:04 daily.2/
2010-05-28 20:04 daily.3/
2010-05-31 20:03 hourly.1/
2010-05-31 16:04 hourly.2/
2010-05-31 00:05 hourly.3/
2010-05-30 20:03 hourly.4/
2010-04-04 00:05 monthly.1/
2010-03-10 00:04 monthly.2/
2010-05-28 00:02 weekly.1/
2010-05-13 00:04 weekly.2/

$ du -hs ; du -hs hourly.1
1.5G    .
778M    hourly.1

While I was at it, I wrote a simple restore script, ryrestore. The script makes it easy to restore a file, or a directory, so I don't have to dig through the backup server. Here is an example of how it works:

# ~/.backup/ryrestore.py /home/anrxc/.xinitrc

0: monthly.1 [2010-04-04 00:05:33]
1: monthly.2 [2010-03-10 00:04:10]
2: daily.2   [2010-05-29 00:04:21]
3: hourly.3  [2010-05-31 00:05:26]
4: hourly.2  [2010-05-31 16:04:16]
5: hourly.1  [2010-05-31 20:03:57]
6: hourly.4  [2010-05-30 20:03:56]
7: daily.3   [2010-05-28 20:04:01]
8: weekly.2  [2010-05-13 00:04:04]
9: daily.1   [2010-05-30 16:04:28]
10: weekly.1 [2010-05-28 00:02:52]

Snapshot: 5

sending incremental file list
.xinitrc

sent 2634 bytes  received 31 bytes  5330.00 bytes/sec
total size is 2543  speedup is 0.95

First time I read Virtual Light by William Gibson I liked it, but I didn't appreciate it as much as it deserved. Only after Pattern Recognition and the current series I realized just how good the whole Bridge Trilogy really is. To me the path towards Pattern Recognition is evident in every chapter, hinted in every reference. While waiting for the Zero History release I decided to buy the whole bridge series and read it again. I'm half way through now, and I'm loving every page of it. I was born too late to really appreciate Neuromancer anyway, it never influenced me, not like it did the previous generation. It wasn't even what introduced me to cyberpunk, it was Neal Stephenson's Snow Crash instead. Maybe that's the reason I love Gibson's recent work so much, his amazing portrayal of the present, his ability to spot patterns and nodal points where nobody else does... bits of the literal future right here, right now.

Virtual Light is set in the year 2005. Tokyo is recovering from a huge earthquake, and the society as a whole from AIDS. This is where we meet Chevette Washington, a bike courier. She lives on the San Francisco Bay Bridge, where squatters have built settlements. Visiting San Francisco, the bridge and Chevette's roommate (Skinner, a bridge veteran) is Yamazaki, a student of sociology from Japan. Another character is Rydell, a security guy and former policemen, who is brought in to help investigate a theft of VR glasses. Which just so happen were stolen by Chevette, on an impulse.

The second book, Idoru, finds Rydell working for hotel security where he befriends a guest, Colin Laney, a data analyst. Laney has a singular gift - he can intuitively spot trends developing within masses of seemingly unrelated data. Through Yamazaki, who is now in Tokyo, Rydell finds a new job for Laney. The assignment is for Lo/Rez - the hottest rock band on earth. The lead signer has just announced that he intends to marry Rei Toi, a software agent and Japanese idoru. Chia, a member of a Lo/Rez fan club from Seattle, travels to Tokyo to visit the local chapter of the fan-club and find out if rumors about the wedding are true.

The third book sees the culmination of all these events, and although All Tomorrow's Parties includes many of the same characters, it's not a direct sequel to either. The book offers its own story line, and is perhaps the best of the three. Laney can now see significant "nodal points" in the vast streams of data in the worldwide computer network, and he owes this gift to an experimental drug he received during his youth. Such nodal points are rare but significant events in history that forever change society, even though they might not be recognizable as such when they occur. Laney isn't quite sure what's going to happen when society reaches this latest nodal point, but he knows it's going to be big, and he knows it's going to occur in San Francisco. On the Bay Bridge.

What happens when we reach the nodal point? Finding out is a perfect prelude to Zero History coming in September.

The arrival of my new workstation saw the culmination of a 2 year quest to (drastically) improve my desktop environment. Machine came pre-installed with Ubuntu and LUKS which I disposed of, and installed the best desktop/workstation OS available at the moment, Arch Linux. Its solid UNIX fundations, its philosophy and package management, deserve an article in its own right so that is all I'll say about it now. Once the OS was installed I cloned my dotfiles.git repository and was ready to go. During these past few years I wrote about various software I use every day, but to see these components work in unison, to see the interaction and the big picture, is what matters to me most.

Following the order of my awesome tag layout is a good path through my workspace. But first to mention Zenburn, a color scheme I discovered a few years ago, which now plays a very important role. Just about everything on my desktop follows the schemes guidelines, everything but GTK and QT widgets. Zenburn is easy on my eyes and saved me a lot of headaches.

First tag is "term" where my terminals reside, Zenburn themed urxvt and screen connecting me to the outside world. An SSH client and Irssi are often found there. Long lasting sessions are always on that tag, but for quick terminal jobs the scratch module provides me with disposable terminals that slide-in or pop-up. While working, awesome's fair layout ensures each terminal gets an equal part of the screen, and one that requires my total attention I often maximize.

Next tag is "emacs", probably the most important tag, where I code, write and take notes. The Emacs org-mode plays a crucial role, I use its format for notes, documentation, keeping track of projects and working hours, auth credentials, personal agenda and much more. I do use eCryptfs, but every sensitive file is also GPG encrypted with some help of Emacs epa-mode. Which brings me to the GPG agent which I mentioned in many previous articles. Every time Emacs needs my key a PIN entry dialog will appear, every time I open a new SSH session a dialog will appear to unlock that key. I have dozens of crypto keys but it's easy to keep track of them in this manner.

Next tag is "web" with Firefox and vimperator that changed my browsing drastically. Once I wrote about connecting awesome with org-mode, and this is the tag where I utilize that connection the most. The Mod4+q key-binding spawns a little remember frame for taking a note, or automatically pasting the clipboard selection. I store huge amounts of web data in this way. Another very important connection is passing text field contents to Emacs, for editing. I use it almost exclusively for managing support tickets, once the ticket is opened in Emacs the post-mode is invoked.

Speaking of e-mail, the "mail" tag comes next. The realm of Alpine and awesome's magnifier layout. Most of the time there are two instances running, one personal and one connected to the company's IMAP server. By the way Alpine handles a 500k mailbox with ease, and only days ago I heard a Thunderbird user complain it couldn't handle just 60. Where would I be without it, I can't imagine. Every time a new mail comes in the tag turns red, because of the urgent flag, one key press and the client which triggered the event is automatically focused. Since I use Topal this tag too spawns a lot of PIN entry dialogs.

My fifth tag is reserved for IM, where Gajim was used almost exclusively until I needed OTR encryption on a daily basis. Now I run Pidgin, and I was very surprised that it took very little effort to make it look and behave exactly like Gajim. I spend a lot of time on this tag and it was very important to have zenburn in pidgin, otherwise all other efforts would be useless. The following tag, "rss", was very important while I was freelancing. Akregator would fetch the new projects feed every 5 minutes and often that responsiveness alone would land jobs. Last tag is "media", a floating layout tag with smplayer, utorrent, ROX, Okular... mostly for multimedia, and for reading.

The next stable release of the awesome window manager will introduce some new widget properties. When graphs and progress-bars were ported to Lua, in the 3.3 to 3.4 transition, some of the properties were lost. Most notably the progress-bar ticks, and the graphs ability to draw multiple values at once. Well, they are back, and will be included in awesome v3.4.5! To tell the truth they are not as nice as the old properties, because I tried to keep them as simple as possible (by design and implementation).

The progress-bar ticks introduce two new methods: "set_ticks_gap" and "set_ticks_size". Default gap size is 1, and tick size 4, in respect to the default progress-bar width of 100px. That's what the above picture shows, defaults. But if you use a lot of custom properties, and change the progress-bar size, it's up to you to pick the perfect gap and tick size for that progress-bar.

The graph stacking (also called multigraph by some) introduces these new methods: "set_stack" (false by default) and "set_stack_colors" (i.e. {"red", "white", "blue"}). The order of colors matters, because the "add_value" method now accepts an (optional) last argument, an index of a color from your stack color group. With these properties you can draw graphs similar to those found in Gnome, feed them multiple values and by specifying a color index they will all be drawn on the graph.

Remaining two are smaller properties, but could be as important as the others to some people. First of them found its way into awesome in the current 3.4.4 release. The progress-bar "max_value" property allows you to feed your progress-bars with any value without having to scale it to the 0-1 range. Graph widgets already supported this. The last property is the progress-bar "offset", which may not be included after all, but some future user might want it so I'll link to the mailing list patch. With offset the progress-bar will be drawn distanced from the border by as many pixels as the offset argument.

I got a new workstation last month, a laptop from the ThinkPad SL series. The TuxMobil article about installing Arch Linux on it is here. Overall it works good, but I soon regretted the decision to go with Lenovo. The ACPI support is almost non-existent, none of the function keys work, there's no bluetooth rfkill so it constantly draws power, and the machine can't wake up from suspend.

It is my workstation, but still what use is a laptop without any power management features? It's 2010, and I can barely comprehend the suspend/hibernate situation in Linux. Last two years with my TravelMate have been a constant battle, 3 months of suspend working, followed by periods when it was broken. Last of which is especially ugly, it breaks hibernation for people with Intel graphics. Worst of all, in periods when it was working you still couldn't suspend because you couldn't trust it.

These machines actually have the IdeaPad firmware, which rules out using thinkpad_acpi. Next up was lenovo-sl-laptop, a third party module which provides support for SL models, but only up to SL500. Then I turned to asus-laptop which provides official in-kernel support for ThinkPad SL. Unfortunately after inspecting the DSDT developers concluded SL510 support is not possible. These machines expose a wmi interface, but it's not handled by any current module. Developing one will not be easy.

I don't want to write to kernel mailing lists or Lenovo until I find more owners of SL510, or some other model with the same interface. Individually we could be ignored, together maybe we get the ball rolling towards "lenovo-sl-wmi".

The iPad was released and sales are sky high, software wise it is terrible but the fact makes me happy anyway. We are getting closer to some of the ideas laid down in 1994 by Neal Stephenson in his book The Diamond Age. Even though the age of nanoscience is only just beginning, there are some fundamental similarities between his Illustrated Primer, today's eBook readers, OLPC and the iPad.

These are the primers of the early 21st century. Beautiful devices that we read from, learn from and play with. We could consider our laptops as primers, but I can't wait to get my hands on one of those devices. At this point most likely the Sony PRS-600. Even though much different than the iPad it is still my first choice, because of the software limitations but also practicality. I would use a pad mostly for reading anyway, and here the E-Ink has the advantage, regarding contrast and battery life. Multiple new devices, by just about every big player on the market, were already announced. Some of them will run GNU/Linux and in the long run that will probably prove to be the best choice.

The mock-ups of next generation OLPC, the XO-2 are probably the closest, especially considering their role, to truly serve in education of children. The now classic article, Sic Transit Gloria Laptopi, by Ivan Krstic addresses some problems, and reminds me once more I shouldn't get carried away. There's still a long way to go. Gillian 'gus' Andrews gave an interesting talk on the subject, at "The Last HOPE" conference. The audio is still available: Hacking the Young Lady's Illustrated Primer.

Almost 9 months ago I wrote an article about Alpine and GnuPG. That article covers basics of integrating Topal and Alpine, I'd like to add more notes and talk about some usage scenarios. But first I should mention that if you are an Arch Linux user the topal package is now available in the AUR. I'm the maintainer and I would appreciate any feedback, so far it got only one vote and I expect more.

I remember trying to switch to GPG2 back when I was still using the PinePG filter and it was not as easy as I hoped for. Official support was not there, and I'm sorry I failed to mention in my last article that Topal works fine with gpg2. You can set "gpg-binary=gpg2" in your ~/.topal/config file to switch. Note that the old gpg option "--no-use-agent/--use-agent" makes no difference, gpg2 always requires the agent. What this means for you is that gpg2 will try to start the PIN entry dialog every time it needs the passphrase (even when you don't use the agent normally), so if you don't run X11 set "pinentry-program" to /usr/bin/pinentry-curses in your ~/.gnupg/gpg-agent.conf file. If you would like to start using the GnuPG Agent instruct Topal to always connect to it with "use-agent=3", and read my article on the GnuPG Passphrase Agent.

Do you remember the big SHA1 scare last year? Many people generated new keys, and many more dumped SHA1 for good. My article on GnuPG basics also covered personal cipher options, and forcing stronger digests. However Topal "gpg-options" setting by default uses --no-options which instructs gpg2 not to read your ~/.gnupg/gpg.conf file, which then fall-backs to SHA1 for signing. You should remove it in order to use personal digest and cipher algorithms, and it's also useful because of other options (like auto-key-retrieve if you want to fetch missing keys). You will find my own GnuPG configuration files in my dotfiles repository.

In the past year I wrote several articles on GnuPG and they bring a lot of visitors here. Some recurring Google searches are (more or less) "sharing private keys" and "splitting gpg keys". There's already a very popular (although outdated) article on the subject so head on there. But if you are an Alpine user read on. General scenario is this: you read your mail on a remote server, which you can't trust as much as your workstation (or removable storage). Reading mail on one host, verifying it and sending from another, maintaining multiple key-rings, or even placing the primary key on both machines... is tedious and risky. Using subkeys as explained in that article is one way of solving the problem, but with Topal you don't need to because of its remote and server mode of operation.

Each time Topal is invoked you can select the remote mode of operation. In remote mode Topal with connect to your trusted machine with SSH, transfer any files necessary with SCP and then perform the wanted GnuPG action. To make use of the remote mode on the secure machine you need the SSH daemon running and Topal started in the server mode ($ topal -server). To make the whole procedure as transparent as possible you can employ SSH public key authentication.

One year ago I wrote a small article about my usage of the awesome window manager. Specifics of awesome and benefits of using it (or tiling window managers in general) I would rather leave for another article. In this one I'd like to revisit some of my projects and code related to awesome. Last year I just upgraded from v2 to v3 which introduced the Lua programming language as a way of configuring and extending awesome. My goal then was to create an identical environment as I had with v2. One of the major obstacles was replacing the Ruby widget library called Amazing with a Lua one called Wicked. I still remember my first widgets, knowing very little Lua I had to resort to Awk to grab battery or mbox information. At the time I just started visiting the #awesome IRC channel on OFTC. I clearly remember someone saying "it would be nicer if it was in Lua"...

A year has passed, so what has changed? Previously I wrote about vicious, a modular widget library written in Lua which builds on the foundations laid down by Wicked. I had certain ideas about widgets that were not shared by a lot of people, so I had to do it for myself. Making Wicked modular would have been a big design change, and on top of that I wasn't confident enough in my Lua so I decided not to contribute back, but to create a new project. Now I am very satisfied how it turned out, I'm satisfied with the code and with contributions of other users. Result is a series of Lua modules that gather data about your system, basically system monitors like those provided by Conky... at the moment we use them to feed awesome widgets but they could be used in other places just as easy. For example one could use them for populating the Ion window manager status-bar. I made the project public sometime in June, it now counts 25+ widget types and gets 10+ downloads daily. It's hard to make an estimate about the number of actual users, but the code was downloaded well over 700 times.

Since I published the vicious git repository I wanted to use the git web interface for more than just those few files, so I put my awesome configuration in git and started pushing the changes. This easy access, a lot of custom (and well commented) code and my solutions to various usability problems quickly made my awesome-configs repo into a very popular starting point for new awesomers. It gets almost as much clone requests and downloads as the vicious repository. My Zenburn theme also became very popular, in fact so popular that from v3.4 it is a part of the awesome distribution. That's not all I contributed to the awesome tree, in recent months I started sending more and more code contributions... I contributed to other open source projects but I'm very proud of being a part of this one. It has a lot of users, most of which are experienced Unix users with an interest in improving their productivity and desktop usability. As someone said on IRC just yesterday "awesome is the ultimate sysadmin console".

One of my modules that is just gaining some attention is the Scratchpad manager. It brings back functionality that was present in v2, but also expands on it by providing a drop-down applications manager, contributed by the author of Wicked. Former Ion users will also be familiar with the scratch.pad functionality, while the scratch.drop module allows users to have their favorite terminal emulator, or application launcher like gmrun, slide in from a screen edge. Another useful module that can be found in my repo is the On-Screen Keyboard, initially written by another awesome user, which I ported to v3.4. You can see it in action in this screenshot.

Finally let's see what other users have been up to. The author of Wicked wrote Eminent - a dynamic tagging library (its functionality will be familiar to WMII users) and Rodentbane - which allows for rapid control of the mouse pointer using only the keyboard. Other notable modules are Revelation (implementing OSX like expose), Shifty (dynamic tagging with advanced client matching) and Obvious (another widget library). With this I conclude my little tour of planet awesome.

I have seen the Sherlock Holmes reboot. It is a movie that provides good entertainment; a dark mystery, solid action scenes, fantastic London scenery and an excellent director. The plot and characters are no different than anything else coming out of Hollywood these days, so it is bearable... but when I combine what I just said with the fact that the story revolves around one of my favorite characters ever I can't but be disappointed, utterly.

Reinventing, or better said rewiring, the character is acceptable... a darker Holmes, filthy, unshaven and manic is legitimate. But they had to poison it with American macho bullshit, which is especially hard to stand. Holmes even has a women in this story, he who is "not a whole-souled admirer of womankind", that is just preposterous. The movie ends by hinting there will be a sequel (they will milk it dry, no doubt) and I hope it will be better than this. But now I am certain there will never be a better Holmes than the role played by Jeremy Brett in the 1984 Granada TV series. He is, and will remain without any doubt, The Holmes.

I love that adaptation, and enjoy watching those episodes, he adds so much to the character; the short bursts of laughter, the mood swings and his eccentric hand gestures... Robert Downey Jr. is not even close. Watching Jeremy Brett play is rewarding, he was Holmes, truly living the role and completely absorbed in it. The Wikipedia article explains it all, and sheds more light on what became of Mr. Brett. Even though Granada filmed a lot of episodes they did not cover all the stories. Fortunately we can get them all for free today, on a site like Project Gutenberg.