26.06.2010 22:08

Notes on systems monitoring

Often it's hard to beat a few lines of shell script to perform a basic monitoring task on a personal system. A system-load, or a file-system monitoring script running from cron is extremely easy to write and setup, while being invaluable in emergencies.

I wrote plenty of those during the years (load, fs, pacct, daemons... monitors), which I use in combination with network and intrusion monitoring. And here's how it works; e-mail alerts from all personal systems are sent out to a third-party system where they are immediately relayed to my local mailbox, and then archived, but not marked as read. That way a backup remains, but that archive mailbox is also available through IMAP. While having e-mail alerts is good, a live alert is better. Live in my case means IM, and if I'm on the road: SMS - which is easily done through a service like email2sms offered by the mobile service provider. Although the IM part is more to the point of this article.

Years ago while learning Python I wrote two bots, an IRC one and a Jabber one. Dealing with a protocol like IRC is great for learning, it involves many different libraries and problems. Text processing, sockets, databases, accessing web resources... When I lost interest (don't we all) I decided maintaining and running the Jabber version is a better choice, and besides, I love that protocol. Sometime later hooking the Jabber bot into the existing alert system was easy thanks to the imaplib library. Bot connects to the IMAP server and relays, to Jabber, all alerts which have the Unseen flag. Once read an alert is flagged as Seen, and that's all there is to it. Live alerts 24/7, over my favorite protocol, in my favorite messaging client.

If you are interested in writing your own bots you can checkout the xmpppy library. Handling the connection, presence and subscriptions can be done in as little as 20 lines of code. To complete the cycle I should also mention some of my favorites for local monitoring. For process monitoring the htop project provides a great interactive replacement for top in procps. Finally, no article would be complete without mentioning nmon - an amazing AIX and Linux performance monitor, developed (and unofficially supported) by an IBM employee.


Written by anrxc | Permalink | Filed under jabber, work, code