end of this blog

After having this blog for some time, two things are obvious to me. I do not write frequently enough and therefore there is no traffic on this blog. I am now considering giving up on this, since I am switching my domain from a web hosting to google apps, but if at least one person actually reads this and finds this useful, I will rehost this blog on a shared blog hosting and continue my infrequent technical writings. If such a person is out there, do comment.

Share This

August 18th, 2008 | Database | 2 Comments »

crontab to english translator

A couple of years ago I have written this script, that takes crontab entries from standard input, parses them and prints english translation. It is definitely not perfect and will bail at a lot of valid crontab entries, but for all it is worth here it is.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135

#!/usr/bin/python
 
import re
import os
import sys
import string
 
class CronJob:
        """A class describing a scheduled job."""
        def __init__(self, str):
                """
                Generate a new object from a crontab line. We should differentiate between the following types of crontabs:
                1. something = something (raise exception)
                2. <schedule> <command> (classic cron shedule)
                3. [!&]word(arg)[,word(arg)...] <schedule> <command> (fcron style schedule)
                4. #somestuff (comment, raise exception)
                5. <empty or space> (empty line, raise exception)
                """
 
                if re.compile("^\s*$").search(str):
                        raise NotACronJobError("EMPTY")
                elif re.compile("^\s*#").search(str):
                        m = re.compile("^\s*#(.*)").search(str)
                        raise NotACronJobError("COMMENT", m.group(1))
                elif re.compile("^\s*\S+\s*=.+").search(str):
                        m = re.compile("^\s*(\S+?)\s*=\s*(.+)").search(str)
                        raise NotACronJobError("VARIABLE", m.group(1), m.group(2))
                elif re.compile("^(\*|\d+)").search(str) or re.compile("^[!&]\w+").search(str):
                        if re.compile("^!.+?\)\s*$").search(str): raise NotACronJobError("GARBAGE", str)
                        self._parseLine(str)
                        return
                else:
                        raise(NotACronJobError("GARBAGE", str))
 
        def _parseLine(self, str):
                if re.compile("^[!&]\w+").search(str):
                        self.type = "fcron"
                        m = re.compile("^\S+\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(.+)").search(str)
                else:
                        self.type = "vixie"
                        m = re.compile("^(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(.+)").search(str)
                self.min = self._parseDateTime(m.group(1), "min")
                self.hr = self._parseDateTime(m.group(2), "hr")
                self.dom = self._parseDateTime(m.group(3), "dom")
                self.mon = self._parseDateTime(m.group(4), "mon")
                self.dow = self._parseDateTime(m.group(5), "dow")
                self.cmd = self._parseCmd(m.group(6))
 
        def _parseDateTime(self, dt, type):
                min = range(0,59)
                hr = range(0,23)
                dom = range(1,31)
                mon = range(1,12)
                dow = range(0-7)
                if dt == "*":
                        return None
                elif re.compile("^\d+$").search(dt):
                        return range(int(dt),int(dt) + 1)
                elif re.compile(",").search(dt):
                        dts = dt.split(",")
                        parsed = [self._parseDateTime(x, type) for x in dts]
                        res = []
                        for x in parsed:
                                if res == None: res = []
                                res = res.extend(x)
                                return res
                elif re.compile("\/").search(dt):
                        m = re.compile("(.+?)/(.+)").search(dt)
                        r = m.group(1)
                        st = m.group(2)
                        if r == "*":
                                r = eval(type)
                        else:
                                (x,y) = r.split("-")
                                r = range(int(x),int(y))
                        return range(r[0], r[-1], int(st))
                elif re.compile("-").search(dt):
                        m = re.compile("(\d+)-(\d+)").search(dt)
                        return range(int(m.group(1)),int(m.group(2)))
                else:
                        raise NotACronJobError("GARBAGE", dt)
 
        def _parseCmd(self, cmd):
                if re.compile("^\s*root\s*").search(cmd):
                        cmd = re.compile("^\s*root\s*").sub("", cmd)
                return cmd
 
        def __str__(self):
                s = "Run %s" % self.cmd
                if self.mon != None:
                        months = ("Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec")
                        s = s + " in " + ",".join([months[x] for x in self.mon])
                if self.dom != None:
                        tmp = ",".join(["%sth" % x for x in self.dom])
                        tmp = tmp.replace("1th", "1st")
                        tmp = tmp.replace("2th", "2nd")
                        tmp = tmp.replace("3th", "3rd")
                        s = s + " on " + tmp + " day"
                        if self.mon == None:
                                s = s + " of every month"
                if self.dow != None:
                        week = ("sunday", "monday", "tuesday", "wednesday", "thirsday", "friday", "saturday")
                        s = s + " on " + ",".join([week[x] for x in self.dow])
                if self.hr != None:
                        if len(self.hr) == 1 and len(self.min) == 1:
                                s = s + " at %s:%s" % (string.zfill(self.hr[0],2),string.zfill(self.min[0],2))
                        else:
                                s = s + " at " + ",".join([str(x) for x in self.hr])
                        if self.dow == None and self.dom == None:
                                s = s + " every day"
                else:
                        s = s + " at %s minutes" % ",".join([str(x) for x in self.min]) + " of every hour "
                return s
 
 
class NotACronJobError(Exception):
        """An exception raised by CronJob to indicate that the line in question doesn't contain a vaild cron schedule information."""
        def __str__(self):
                if self.args[0] == "EMPTY":
                        return "Empty Line"
                elif self.args[0] == "COMMENT":
                        return "A comment: %s" % self.args[1]
                elif self.args[0] == "VARIABLE":
                        return "An environment variable: %s = %s" % (self.args[1], self.args[2])
                elif self.args[0] == "GARBAGE":
                        return "Uncronish thingamabob: %s" % self.args[1]
                else:
                        return "If you don't know how to play with me, go to the other sandbox!"
 
if __name__ == "__main__":
        for line in sys.stdin:
                try:
                        print CronJob(line)
                except NotACronJobError, err:
                        print err

Share This

July 31st, 2008 | Miscelaneous, UNIX | No Comments »

Restoring MySQL databases CLI trick

It is very easy to dump and restore a database using mysql and mysqldump CLI utilities, just

# backup
mysqldump --single-transaction mydb > dump.sql
#restore
mysql mydb < dump.sql

and you are all. Unfortunately, if your database is several gigabytes and takes a long time to restore you might want to have some sort of output, to indicate where in the process your backup or restore is. For backup you just add -v flag to your mysqldump command and it will throw out some information about which table it is backing up. What about restore? While it is definitely possible to just go and check what table is being restored (mysqldump dumps tables in alphabetical order), I came up with a little clever trick to make the restore progress obvious and similar to mysqldump. Just add perl.

cat dump.sql | perl -ne '/Table structure for table \`(.*?)\`/ && do {chomp($t=`date`); print STDERR $t . " loading $1\n";}; print' | mysql mydb

Share This

May 16th, 2008 | Database, UNIX | No Comments »

Why I don’t like Debian based distributions.

I have been happily using Fedora for a while now, but I keep a close eye on Ubuntu development, since it is my humble opinion, that nothing, at the moment, compares to Ubuntu in ease of use, hardware compatibility and general togetherness. I recommend Ubuntu to people who want to try out Linux, I ran Ubuntu myself for a while, I run beta versions of Ubuntu releases and file bugs (well, when I have time). Now I also have an Eee PC laptop running Ubuntu. I like Ubuntu. But I run Fedora as my main OS. The reason for this is Ubuntu being Debian derivative and as such dragging with it all the horrible Debian legacy. I honestly wish Ubuntu chose a different What is horrible about Debian? Well, this post intends to list a few things that annoy the hell out of me and that IMHO should have been fixed ages ago. Yes, I am aware, that I blaspheme.

  1. Package installation procedure - when a list of packages is being installed or upgraded, Debian package manager or DPKG does this procedure in stages. That is it will first unpack all the packages, then run all the pre-install scripts, then install the files, then run the post-install scripts etc. (I am not trying to be correct about the exact steps here). And while this behavior might seem logical therein lies a problem. If, for example, one of the packages’ post-install scripts fails, dpkg reports a problem and quits and all the rest of the packages remain unconfigured. True, dpkg will continue where it left off when the suer resolves whatever problem is causing the script to fail or removes the offending package, but this is not the point. Lets consider a case where actual updated package is broken and script fails because of a syntax error. Once dpkg fails the system ends up in rather strange state. All the services that were to be updated were stopped, but weren’t started again (since that happens in the post-install). New libraries were unpacked, but ldconfig weren’t run. New kernel might have been installed, but new initrd wasn’t generated and boot manager wasn’t updated. Basically we have a broken system that needs careful fixing by a specialist who knows what he is doing. And even if you do know what you are doing, your choices are limited. You need to either fix the script yourself, repackage and reinstall, but that makes your system somewhat inconsistent or you need to completely remove the package, rerun dpkg to finish the install/upgrade of other packages in the queue  and try to reinstall the old version back, but that might not be possible since all the other packages might prevent the old version from being installed, so you need todescend the dependency hell and start selectively uninstalling and downgrading packages to get a working version of whatever software. Yes, some of it is also true about RPMs, but at least when one of the RPM installs fails all the rest of the packages are either NOT installed or installed COMPLETELY nothing except possibly the broken package is done half way.
  2. Package state markings - as I mentioned in previous paragraph, when dpkg fails to do some of its tasks it can be rerun and will proceed from the point it stopped (or fail in the same place). This is done by having very granular records of package state. APT seems to like to mark packages just a little bit too much and annoys its user. Lets say, I have started an install of a package that needs a 100MB of dependencies and suddenly I need to go somewhere. So, I hit CTRL-C, close the laptop and run out. Later I find that my laptop doesn’t have a reader of some sort installed, for example FB reader and I need it right away to read some document. I hit apt-get install fbreader, but suddenly the whole 100MB of stuff starts downloading again. Why? Because APT marked all those packages for installation and will install them unless they are unmarked. Honestly I don’t know how to easily unmark packages marked for install/upgrade short of doing dpkg –get-selections, manually editing the resulting list and piping it into dpkg –set-selections. There maybe a way to do this using GUI interface such as synaptic, but at a glance I couldn’t find it. Other example of this “feature” is when you are trying to remove packages. Sometimes you see a package and think “I don’t need this, why is it installed”, so you dpkg -P it. And suddenly dpkg tells you, that the package is actually a dependence of something or other. But although dpkg proudly reported that it did not remove the package in question because of dependency problems, it DID however mark the package as “to be removed”, so if ever the dependencies change this package might just disappear without any intervention.
  3. SysV scripts - Debian like most other Linux distros uses SysV startup. One feature though seems to be specifically done to annoy the hell out of the user. Every time a service that has a startup script is upgraded, it automatically setup to start at boot. Even if it was manually turned off before. In Fedora I can say chkconfig httpd off and Apache will not start until I say otherwise. On a less sophisticated system I can say something like rm -f /etc/rc?.d/S*httpd to achieve the same result. On Debian I can update-rc.d -f remove apache, but once an upgrade to the apache package is installed it will reinstate itself on its default runlevels and happily start on boot. As far as I know, there is NO way to prevent this. Ridiculous.
  4. Package management command set - this is not as much a problem as a way to a lot of confusion and it is not restricted to the package management system. There is just too much legacy in the Debian native commands. The package system provides a very good illustration. In Fedora I generally use two package management commands, yum and rpm. Yum mostly works with remote repositories and handles package installs, upgrades etc. RPM works with locally installed packages and manages installing from local file, querying the package DB, removing packages, package signing keys etc. In Debian it is not as simple. To install from remote repositories I use either apt-get. To search remote repositories I use apt-cache. To install from local file or remove package I use dpkg. To query package database I use dpkg-query. To manage keys I use apt-key. Each of these has its own specific subcommands and flags.
  5. DEFOMA - the Debian Font manager. Basically this is a convoluted something that is supposed to make all the font management automagical. Unfortunately all it seems to do is confuse anyone who tries to figure out what happens to fonts on the system.

Share This

March 14th, 2008 | Database | 1 Comment »

Yet another post about firefox extensions

Previously I have written about various useful extensions for Firefox. Recently I have tested quite a few extensions that didn’t make my “install these first” list and although I do not think any of these are of the “you are not browsing right if you do not have this” grade, but I find some of them rather nice additions to my web experience.

  • CustomizeGoogle - is one of the subtle, yet extremely powerful extensions. Once you install it, suddenly your experience with Google search, GMail, Google Calendar and other Google products just becomes nicer. You get Google Suggest keywords while you type, GMail auto-redirects to an encrypted version, you get links to other search engines in your search results, Google Images starts to actually point to images etc.
  • SpeedDial - if you ever used Opera, you already know what this is about. Basically, it adds a special location (you can configure it to be your home page) that shows thumbnails of several (nine by default) sites of your choice with handy shortcuts to go straight to these sites. I used to keep a lot of tabs open at all times in my Firefox sessions, in order to have all the reference documentation I need at hand at all times. Now I just assign relevant pages to my Speed Dial and voila, CTRL-1 gives me a tab with Apache 2.2 manuals, CTRL-2 - tab with MySQL reference manual etc. Or I can just open a new tab and click on whatever I need right there. Again, I can see people saying that this is just an unneeded addition to bookmarks and bookmark toolbar. Bookmark toolbar takes screen space. Bookmarks are nice, but since you cannot assign shortcut to a particular bookmark (as far as I know), Speed Dial actually does speed up getting to your favorite sites even if just a little bit. As an alternative, one can always use bookmark keywords (one of the more obscure Firefox features). For example, you can bookmark Slashdot.org and assign keyword slash to it. Then you use CTRL-T to open new tab, CTRL-L to switch focus to the location bar, type slash and hit enter. This is much faster, then browsing bookmarks menu with a mouse (especially for the keyboard oriented people like me), but not as fast or visually friendly as using Speed Dial extension. After all with Speed Dial you do not need to remember keywords. Note: Obviously some of these arguments are useless for people who use mouse more then keyboard. But I would guess that with one of the mouse gesture extensions you should be able to map Speed Dials to gestures.
  • Secure Login - this one is even more subtle. If you are using Password Manager to remember your login information, you might sometimes be annoyed that it fills out your login info weather you actually want it or not. The Secure Login will change this behavior to a more appropriate. Every time there is a login form on the page, Secure Login will search the Password Manager for a fitting login/password combo and if it finds one it will highlight the form fields with yellow, light up an icon in the status bar and may, if configured, even play a sound. It will prevent the P.M. from filling the info into the form. Pressing a shortcut key or clicking a toolbar button will fill the form and submit in one motion (or just fill the form if you are so inclined). It can warn you if the form is attempting to submit something to a domain different from the page the form is located on and will show a popup to indicate where the form will be sent.
  • Resizable Form Fields - does exactly what its name suggests. It allows you to resize text fields, text areas, combo-boxes and lists. Well… most of the time at least. I have seen a few sites where it doesn’t work (probably due to absolute positioning or some other CSS tricks). But where it works it is a nice feature to have.
  • TrashMail.net - will add a menu item “Paste a disposable email address” next to Paste. When used it will use trashmail.net site to generate a temporary email address. This is very useful when trying to read an article from some suspicious site that requires registration.
  • BugMeNot - will use the bugmenot.com login database to login into those annoying sites that require you to register in order to read. New York Times is one of the popular examples. Yes, this is a morally questionable practice, but those compulsive registration dudes are just soooo annoying and I am not a lawyer to be able to properly read their “Privacy Policy” documents :)
  • URL Fixer - this is probably the subtlest one. It will quietly fix basic typos in URLs. Ever typed www.google.con or wwww.gmail.com? No more.
  • ScrapBook - this is one of the more non-obvious and extremely powerful add-ons. ScrapBook will allow you to properly gather and organize the data you mine on the web and will give you some tools to properly work with the materials. On a more particular note, ScrapBook will allow you to save a page or a fragment of a page completely to your hard drive. It will allow you to organize these fragments and pages into folders (same as you would organize bookmarks). It will allow you to mark up (same as with a highlighter pen) parts of the pages you saved and add notes and annotations. Since ScrapBook will actually save the data locally you will not worry about the data going off line or changing at the original location. This is a beautiful tool to do research on the web.

These are the extensions for a common user of Firefox that I have recently added to my add-on arsenal. Stay tuned for my post about some other extensions which are more useful to developers, hackers and power users.

Share This

November 27th, 2007 | Miscelaneous | 2 Comments »

VirtualBox - the VMware alternative

Yesterday I have discovered VirtualBox. In short, VirtualBox is yet another virtualization package. It provides more or less the same function as VMware, Xen, Qemu and VirtualPC. At the moment it is happily running a FreeBSD world build as a guest on my Fedora 8 workstation. I cannot say that my testing of this product is complete, as far as first impressions go, this is fairly favorable. Out of the features Lets split these impressions into three usual categories.

The Good:

  • Support virtualization extensions of the modern CPUs
  • Seems less I/O intensive then VMware
  • Works on FreeBSD

The Bad:

  • The GUI is somewhat clunky
  • No script to automatically configure the kernel module and network

The Ugly:

  • In order to activate the kernel module, I had to guess the location of the module source and run make && make install from CLI.
  • In order to activate bridged networking I had to manually configure ethernet bridging
  • Once the VM crashed without any reason
  • Sometimes FreeBSD guest seems to have some problems with the virtual CPU.

Overall the experience was not all bad. There are some things which I think can be smoother, but it works. Good luck to the developers.

Share This

October 31st, 2007 | Miscelaneous | No Comments »

MySQL features I would kill for.

It seems that nowadays there is a trend in writing “top 10 features I want software X to have”. I have seen at least two such posts about
MySQL, here and here. So, since I have been working with MySQL for a while, here is my list:

  1. File per table backup mode for mysqldump that would work with –single-transaction flag
  2. Clustering without the NDB in memory storage
  3. Ability to turn logs (query, binary, slow queries) on and off without restarting
  4. Ability to setup log filters (such as log queries using particular table into a separate file or log queries scanning more then 10K rows)
  5. Ability to use bound variables in prepared statements properly (such as use variables in LIMIT or pass table names in the variables)
  6. Proper implementation of views (proper, as in not involving running a select every time a view is queried)

    7. Share This

July 31st, 2007 | Database | 1 Comment »

Success of Ubuntu

I think that the existence of this blog post is a clear indication that Linux is succeeding on the Desktop :)

Share This

July 31st, 2007 | Miscelaneous, Rant | No Comments »

Fedora 7 and ATI binary drivers. An Ugly Hack.

There is a known problem with the recently released Fedora 7 and ATI video cards.

  • Most recent driver (version 8.37.6) causes X server to segfault
  • Older drivers do not support new Xorg versioning system (server reports 1.3 and driver expects >7)
  • Xorg open source ATI drivers do not have support for anything past Radeon 9250 (due to ATI not disclosing specs)
  • Xorg VESA driver doesn’t support either 3D acceleration or multi screen and is generally rather slow

All this caused Michael Larabel (who seems to know most about the state of ATI drivers for Linux) to warn people not to upgrade to Fedora 7 just yet.
So, what do you do, if you already upgraded (like me)? Well, if you have single monitor and don’t play games much, you can probably live with VESA driver.
Otherwise you can temporarily downgrade your X server to the supported version. Here is a short HOWTO:

  1. Login as root
    su -
  2. Add freshrpms repository
    rpm -ivh http://ftp.freshrpms.net/pub/freshrpms/fedora/linux/7/freshrpms-release/freshrpms-release-1.1-1.fc.noarch.rpm
  3. Install ATI proprietary drivers
    yum install ati-x11-drv
  4. Start ATI even daemon

    service atieventsd restart
  5. Download and install old version of Xorg server
    wget http://ftp.cica.es/fedora/linux/core/test/6.91/Prime/x86_64/os/Fedora/xorg-x11-server-Xorg-1.2.0-6.fc7.x86_64.rpm
    rpm -U –force xorg-x11-server-Xorg-1.2.0-6.fc7.x86_64.rpm
  6. Uninstall newer Xorg server
    rpm -e xorg-x11-server-Xorg-1.3.0.0-5.fc7
  7. Prevent YUM from upgrading Xorg again
    sed '/metadata/aexclude=xorg-x11-server-Xorg*' /etc/yum.conf
  8. Configure Xorg to use ATI drivers using aticonfig
    1. CTRL-ALT-F1 to switch to console and login as root
    2. telinit 3
    3. aticonfig --initial for single monitor or aticonfig --initial=dual-head for dual monitors
    4. telinit 5

This is it. At this point you should have proper, 3D accelerated setup.
Most of the directions I have taken and adapted from this thread at fedoraforum.org

Update: There is a new release of the ATI drivers that works with Xorg 7.3 (somewhat). It is packaged by both freshrpms and livna and therefore there is no need to downgrade the X server anymore

Share This

June 6th, 2007 | UNIX | 8 Comments »

Quest for web log analysis software

I am currently searching for a web log analysis package for our site. I have to say that the more I look at the available options the more disgusted I get. Basically what I am looking for is wel log analysis software with following features:

  • Reading data from web server logs (not using custom javascript to record hits)
  • Storing log data in a SQL database, so I can use SQL to generate custom reports
  • Capable of generating custom reports with custom graphs and charts
  • Capable of reading custom log formats (such as Apache LogFormat strings)
  • Able to “drill down/zoom in” into the reports for more information
  • Running on Linux, BSD or Solaris.

It seems that to get all of these is close to impossible. Read the rest of this entry »

Share This

May 11th, 2007 | Miscelaneous, Rant | 2 Comments »

« Previous Entries
Close
E-mail It