Unknown and partly hidden

One year ago I wrote a small article about my usage of the awesome window manager. Specifics of awesome and benefits of using it (or tiling window managers in general) I would rather leave for another article. In this one I'd like to revisit some of my projects and code related to awesome. Last year I just upgraded from v2 to v3 which introduced the Lua programming language as a way of configuring and extending awesome. My goal then was to create an identical environment as I had with v2. One of the major obstacles was replacing the Ruby widget library called Amazing with a Lua one called Wicked. I still remember my first widgets, knowing very little Lua I had to resort to Awk to grab battery or mbox information. At the time I just started visiting the #awesome IRC channel on OFTC. I clearly remember someone saying "it would be nicer if it was in Lua"...

A year has passed, so what has changed? Previously I wrote about vicious, a modular widget library written in Lua which builds on the foundations laid down by Wicked. I had certain ideas about widgets that were not shared by a lot of people, so I had to do it for myself. Making Wicked modular would have been a big design change, and on top of that I wasn't confident enough in my Lua so I decided not to contribute back, but to create a new project. Now I am very satisfied how it turned out, I'm satisfied with the code and with contributions of other users. Result is a series of Lua modules that gather data about your system, basically system monitors like those provided by Conky... at the moment we use them to feed awesome widgets but they could be used in other places just as easy. For example one could use them for populating the Ion window manager status-bar. I made the project public sometime in June, it now counts 25+ widget types and gets 10+ downloads daily. It's hard to make an estimate about the number of actual users, but the code was downloaded well over 700 times.

Since I published the vicious git repository I wanted to use the git web interface for more than just those few files, so I put my awesome configuration in git and started pushing the changes. This easy access, a lot of custom (and well commented) code and my solutions to various usability problems quickly made my awesome-configs repo into a very popular starting point for new awesomers. It gets almost as much clone requests and downloads as the vicious repository. My Zenburn theme also became very popular, in fact so popular that from v3.4 it is a part of the awesome distribution. That's not all I contributed to the awesome tree, in recent months I started sending more and more code contributions... I contributed to other open source projects but I'm very proud of being a part of this one. It has a lot of users, most of which are experienced Unix users with an interest in improving their productivity and desktop usability. As someone said on IRC just yesterday "awesome is the ultimate sysadmin console".

One of my modules that is just gaining some attention is the Scratchpad manager. It brings back functionality that was present in v2, but also expands on it by providing a drop-down applications manager, contributed by the author of Wicked. Former Ion users will also be familiar with the scratch.pad functionality, while the scratch.drop module allows users to have their favorite terminal emulator, or application launcher like gmrun, slide in from a screen edge. Another useful module that can be found in my repo is the On-Screen Keyboard, initially written by another awesome user, which I ported to v3.4. You can see it in action in this screenshot.

Finally let's see what other users have been up to. The author of Wicked wrote Eminent - a dynamic tagging library (its functionality will be familiar to WMII users) and Rodentbane - which allows for rapid control of the mouse pointer using only the keyboard. Other notable modules are Revelation (implementing OSX like expose), Shifty (dynamic tagging with advanced client matching) and Obvious (another widget library). With this I conclude my little tour of planet awesome.

I have seen the Sherlock Holmes reboot. It is a movie that provides good entertainment; a dark mystery, solid action scenes, fantastic London scenery and an excellent director. The plot and characters are no different than anything else coming out of Hollywood these days, so it is bearable... but when I combine what I just said with the fact that the story revolves around one of my favorite characters ever I can't but be disappointed, utterly.

Reinventing, or better said rewiring, the character is acceptable... a darker Holmes, filthy, unshaven and manic is legitimate. But they had to poison it with American macho bullshit, which is especially hard to stand. Holmes even has a women in this story, he who is "not a whole-souled admirer of womankind", that is just preposterous. The movie ends by hinting there will be a sequel (they will milk it dry, no doubt) and I hope it will be better than this. But now I am certain there will never be a better Holmes than the role played by Jeremy Brett in the 1984 Granada TV series. He is, and will remain without any doubt, The Holmes.

I love that adaptation, and enjoy watching those episodes, he adds so much to the character; the short bursts of laughter, the mood swings and his eccentric hand gestures... Robert Downey Jr. is not even close. Watching Jeremy Brett play is rewarding, he was Holmes, truly living the role and completely absorbed in it. The Wikipedia article explains it all, and sheds more light on what became of Mr. Brett. Even though Granada filmed a lot of episodes they did not cover all the stories. Fortunately we can get them all for free today, on a site like Project Gutenberg.

Last year I noticed an interesting cover picture in a bookshop. Girl sporting a dragon tattoo, on a book that was called exactly the same; "The Girl with the Dragon Tattoo", by the late Swedish journalist Stieg Larsson. I have a thing for dragon tattoos and despite this cheesy title I got interested. I researched a little on-line and found that it's a story about a journalist and a female hacker (the tattooed girl, imagine that), hired to write a biography for an old Swedish industrialist. Cheesy title and tattooed female hackers sounded too cheap and I didn't buy in.

In October I noticed that they made a movie based on the book, called "Men who hate women", which also turned out to be the original title of the book (that dragon thing was a result of the English translation). Movie was good, a thriller with dark northern atmosphere, a detective story and a mystery all in one. I immediately knew the book is excellent and that I made a mistake. Best selling book of the decade in Europe, and the whole trilogy (which it is a part of) an international phenomenon.

The actual story is much more deeper than it can be depicted in just 2 hours of film. It works on so many levels; corporate world and journalism, serial murders and a dark mystery. Interestingly all that is only an introduction to a deeper story about the girl with the dragon tattoo. The second book took a surprising turn for me, I had no idea the girl will become the center of attention. But let's stay with the first book a bit more. As much as I liked the book and considered it a good story there were some small things that almost ruined it for me. Larsson (being a reputable journalist) paid so much attention to detail that I can't understand some (what I consider) huge errors in judgment. To start with the obvious, he nailed the basic tech but when it came to hacking he had to go out and invent stuff. Some things he wrote about are much more complicated than what is already out there, completely unnecessary. In fact I've read an article in a Swedish newspaper where they asked a security consultant about hacks from all three books. Guy laughed on some counts. One telephone call and you could have it all worked out in detail, for the whole series. Then there is Dragan Armansky: born in Croatia by an Armenian father and a Bosnian-Muslim mother, somehow ending up with a Serbian nationality (as registered by the immigration office). What is going on here, a Korean-Mexican with a French passport would sound more convincing.

I was glad that things started revolving around the girl in the second book, titled "Girl who played with fire". When I least expected it some Russian spies turned up and again added some (more than) fantastic elements to the story, but by then I was more or less immune to these mass-market elements. I kept going because of the girl, which was a fascinating character. You could object to a lot of things regarding her, but along with the (sometimes) unconvincing hacking only thing that bothered me was her photographic memory. It is a myth. Why couldn't she have "extraordinary memory", which would be plausible for a person with asperger (or something like the syndrome, which is suggested she has)? Second book tells us much more about the girl. She is wanted by the police, and while she hacks their networks to stay one step ahead we get to learn a lot about her past. Again Larsson's story works on many levels. The journalist from book one is also back, this time working on a story about sex trafficking. The end of the book leaves things unresolved and leaves you desperately wanting more.

The air castle that blew up is the third and final book of the series (which is also known as the Millennium trilogy). It ties up all the loose ends from previous books and gives a nice closure, I consider it a good ending. The girl was a great character and it would be nice to see here once again, but unfortunately the author died just as he completed the third book. Last week I watched an interview with the Croatian publisher of the series. He just got back from Sweden and said that there are rumors that Larsson wanted to write a bigger series, some 6 or 7 books. In fact he said that Larsson already had an outline for the fifth book on his computer. I guess we will never know, but these books we do have definitely deserve your attention.

Some 6 months ago I wrote about file-system encryption on GNU/Linux and my software of choice, eCryptfs. The entry was followed by an article describing my setup, which I updated today. I learned from my mistakes and those of others, and the article now introduces a pretty different setup. Besides these changes I also did minor updates from time to time, like adding information on two-factor authentication and tmpfs with the purpose of further securing your setup. If you by any chance followed my initial article converting your current setup can be done in minutes. If you haven't already found your own path to a better practice.

Ubuntu stays the only distribution with a completely integrated eCryptfs setup. Integration in their last release is excellent. I spent a lot of time with eCryptfs and Ubuntu developers and learning from their experience was invaluable. I also saw a lot of Ubuntu users that loosed their data (by their own fault). Which brings me to stability and reliability of eCryptfs.

I must say that living with eCryptfs was excellent. It is reliable and a performance hit was always minimal. At the time I started using it the only published benchmark was from Phoronix. Results were pretty good, most of the time performance impact was less than 2%. Last week they published an updated benchmark and this time results are not looking too good. I can't say what happened, I never noticed it in my own day to day operation, but Phoronix is more or less a credible source. It could be worse though, just days ago LUKS users were locked out of their homes.

Linux Magazine published a big article on eCryptfs just yesterday. It was written by Dustin Kirkland, eCryptfs developer, and it's one of the best on the subject.

I didn't write anything for a month. I was reading a lot. I found some good books but one of them I can't get rid of, I think about Cloud Atlas every day.

This book by David Mitchell is composed of several shorter stories, all connected by subtle links. None of them were bad, some were good but some were pure genius. Starting in the 19th century. Going trough the 20th trough a few stories and then all the way forward to the end of civilization. Story following the journalist Luisa Rey was an interesting thriller. Adventures of Timothy Cavendish had me turning pages in suspense and laughing like crazy on occasion. The story about Robert Frobisher, the composer, was beautiful, but the one about clones in the future superstate of Korea was genius.

I found a copy in the local library after failing to find an e-book in English. You should invest some time in reading it, even if you don't like it, it is undeniably an epic novel.

Much to my surprise vicious became very popular in just over a month. About a dozen people grab the code every day, and they should be able to preview the repo and browse trough it - so I decided to serve it with cgit, a fast web interface for git. Connecting it to lighttpd revealed some quirks, so I'll describe my setup in short.

On my web server all web sites are served from "/var/www" so when building cgit I decided to install it in "/var/www/cgit":

$ make CGIT_SCRIPT_PATH=/var/www/cgit

Cgit binary, default css style sheet and logo will be stored there, configuration file is "/etc/cgitrc" and cache (if enabled later) is stored in "/var/cache/cgit".

After creating a new sub-domain I proceeded to configure lighttpd.

# {{{ git.webhost.tld
$HTTP["host"] == "git.webhost.tld" {
  server.document-root = "/var/www/cgit"
  index-file.names     = ( "cgit.cgi" )
  cgi.assign           = ( "cgit.cgi" => "/var/www/cgit/cgit.cgi" )
  url.rewrite-once     = (
    "^/([^?/]+/[^?]*)?(?:\?(.*))?$"   => "/cgit.cgi?url=$1&$2",
)}
# }}}

Before restarting lighttpd I did some quick changes on the default cgitrc. Since web sites are stored in "/var/www" I decided to keep public git repos in "/var/git" (as you might notice from the example repo below). Here are only the most relevant parts. Pay attention to the virtual-root which, together with the above rewrite line, fixes the cgit cache - otherwise it would constantly serve one and the same page.

# URL used as root for all cgit links
#   - fixes caching with the above rewrite
virtual-root=/

# Specify some default clone prefixes
#   - repos are served only trough http
clone-prefix=http://git.sysphere.org

# Specify the css url
css=/cgit.css

# Use a custom logo
logo=http://sysphere.org/images/cgit.png

# Set a custom footer message instead of default "generated by..."
footer=footer.html

## List of repositories
repo.url=myproject
repo.path=/var/git/myproject.git
repo.desc=my project that does something interesting
repo.owner=user
repo.readme=README.html

Serving it this way, beside the cache problem, had other quirks. The png logo cgit tried to serve as yet another repo. I had to link to it directly. The css file on the other hand was OK, custom footer too.


Since I'm talking about git again I'll add a few notes on top of my previous article about it. For years I've been sharing my dotfiles trough a simple directory index, the most popular of them I would convert to HTML. I was getting tired of the whole process (even though Emacs makes it a bit easier with htmlize and scpaste), and now that I'm keeping my dotfiles in git anyway I decided to make that repo public too.

If you read my previous article it is evident that my dotfiles repo could be full of sensitive information, for example a lot of dotfiles contain passwords these days. Once you get that information in there, and publish it (i.e. by mistake) it is hard to get it out. I thought about it for a few days and tried a few approaches. Maybe it would be best to strip all sensitive information and keep a separate repo. But that would require twice as much work, provided I was willing to stay on top of it.

Long story short, I eventually created a new branch called public in my dotfiles.git repo. I push only the public branch to the server and of course I'm careful that it stays clean of all sensitive data. When something changes in master, and it's worth publishing, I only cherry-pick specific commits.

Most applications I use daily are running on my workstation and all dotfiles are written, edited and kept there. From the workstation they are distributed to other machines. What usually happens is that I do massive syncing and then neglect remote systems, sometimes even for months. I'm always trying to improve my environment so often I would end up with completely different working environments. I needed a better solution and git seemed perfect for the job. I guess other distributed version control systems would be just as good but I was already familiar with git. I won't cover the actual git commands, by this point tutorials are just about everywhere, I will rather explain my work-flow.

There are many approaches, most popular seem to be; keeping your dotfiles in a repo and symlinking them back to $HOME or keeping your whole $HOME in git and ignoring mostly everything. Some people also use a bit of trickery to change the git work tree while keeping the repo in another directory and some employ specific tools like git-home. After reviewing all the popular solutions I decided to go with another approach. I created a repo in the "~/dotfiles.git" directory and copied all important dotfiles there. I then use a script called gitup that for each file (or directory) in the current directory checks if there is a dotfile with the same name in $HOME. If the file exists and it changed the script copies it over. An acquaintance uses this approach to easily share his dotfiles via github. It seemed cleaner than symlinks to me, but it does require more work.

I do all modifications on the actual files and later use gitup to get the repo up to date. After I commit they can be distributed further. When I'm setting up a new machine, or an account, I clone the workstation repo directly. Afterwords I can push from the workstation or pull from remote machines when a change happens. Workstation has only the master branch but each remote machine is different and some need to have specific local changes. When that happens I create a new local branch and do my changes there. Most of the time I "live" in the local branch, when something eventually changes in master I rebase it. Whole setup requires some work and it's not perfect, but it certainly beats everything else I tried.

When I wrote about my personal e-mail solution I mentioned that CRM114 Discriminator can be used as a (solid) SPAM filter. I've been using it for about 3 years now, and it served me well. I must admit I never achieved the unbelievable 99.9% accuracy they claim but it is a good solution nonetheless. It is fast, lightweight, scalable and flexible.

I use procmail which pipes all e-mail trough crm114, it is installed system wide, but I keep all "*.crm" and "*.css" files in the "~/.crm114" directory. If you decide to try it you should know that it's very well documented, and the HOWTO document in particular will get you started in no time. There is no need for me to describe the installation and setup process here. What I will talk about is using crm114 together with Alpine.

With the recent "BlameThorstenAndJenny" release the format of .css files has changed for the first time since I've been using it. I decided to rebuild them from scratch. The old ones were not giving perfect results, as I said, and hopefully I can do a better job this time around. Since I get huge amounts of SPAM daily, the first day was a little scary but I managed to get on top of it quickly. I decided not to use mailtrainer.crm, which can be used to train huge amounts of e-mail at once. Instead I rewrote some of my old scripts and did it "manually". The first few batches of SPAM and HAM that I received I trained with my crmtrain script. I used the Export function in Alpine to export each wrongly classified (or unsure) e-mail to the "~/spam" directory and the script took it from there.

Now that things have slowed down I use two scripts to train the filter directly from Alpine. I use the Pipe function in Alpine to pipe each wrongly classified (or unsure) e-mail to the respective script. The crmspam script trains SPAM, while crmham does so for HAM e-mail. I saw that mutt users have a similar setup where they send each e-mail back to them selves, but this time telling the filter how to flag it correctly. Only piping it, while stripping the old CRM114 headers, seems a bit faster and simpler.

It's been a month since my announcement of the vicious library. I kept my self busy and there are 24 widgets in there now... and I'm all out. I went over those from the Todo list and even over the Wishlist. Last changes were dedicated to some cleanup and I removed padding completely. I have no use for it and if you do I'm sure you'll manage to reuse the old code. If you used it only in a widget or two think about using "string.format" instead. More and more people requested the tarball every day and users started sending me e-mail. After a while I gave up and put vicious in git due to popular demand. I serve a bare repo over http now. First, an updated summary:

* Original code modularized
* Widgets ported from Wicked:
  - CPU, Date, FS, MEM, MPD, NET, Uptime
* Widgets written for Vicious:
  - Battery, Battery (acpitool), CPU Information, CPU Frequency,
    Disk I/O, Entropy, HDD Temperature, Mbox, Mbox Count, Maildir,
    Org-Mode, Pacman, Sysload, Thermal, Volume, Weather, Wireless

However, as removal of padding shows, I am still writing this primarily for my self. There could be more major changes in vicious soon, or maybe none at all since I'm not very pleased with the direction of awesome 3.4. I'm waiting to see how things will work out. Some parts of vicious will need to be completely rewritten in case I do switch. I also had other important decisions to make, while writing all these widgets, especially about using libraries that are not in the standard Lua distribution. For example, mail widgets for counting messages in both mbox and maildir folders could really benefit from the LuaFileSytstem library. But I decided against it for the time being. However the next development cycle, for 3.4, could include them.

In the last article I mentioned the obvious project, another modular widget library. They have a very different design, as mentioned, they handle all errors, format the output, setup buttons and timers, progress-bars and graphs... The project saw some major development recently and unfortunately they kept making it more complex going to the opposite extreme of vicious. Recently one vicious user reported a problem with the cpufreq module. It turns out his hardware does not support voltage scaling, yet I bundled it together with frequency scaling. We had a short discussion and I told him that: "with vicious I am giving each user only the framework to create his own stable configuration".

I read some really good books recently, great stories from Spanish authors. I'll mention the last one first. The Secret Supper by Javier Sierra is a wonderful crypto treat. I really enjoyed the story, and all the "secret" messages and early steganography made it into something special. Since Leonardo Da Vinci plays a big role in the book many people associated the work with Dan Brown and The Da Vinci Code - this book is nothing like it, it functions on a completely different level.

In January of 1497, Fray Augustin Leyre, a Dominican Inquisitor and an expert on the interpretation of secret messages is sent to supervise Leonardo Da Vinci's last touches to The Last Supper painting. He was sent by Alejandro VI who had heard that Da Vinci was painting the twelve apostles without their halo of sanctity, that the chalice was missing, and that Leonardo had painted himself in the painting with his back to Jesus. This could have sent him to the inquisition. Why then did he do this?

Was Leonardo Da Vinci a heretic?

I felt that some of the puzzles were a bit easily solved, but I still enjoyed each one. Spending some time with Leonardo, and getting inside the greatest mind that ever lived was also a great experience. I found a good resource while reading the book, using the Arounder you can see the panoramic view of both the church Santa Maria delle Grazie and The Last Supper painting.