» blog.e-shell.org

heading for Birmingham

In a few hours I'll be taking the flight to London (Heathrow). It is a 2-hour flight from A Coruña.

Once we reach Heathrow, we will pick up a rental car (we already have a reservation) and will drive another 2 hours to get to Birmingham.

This will be my first time in England, so I expect the 2-hours drive through the midlands to be a very nice experience (if we manage to drive the other way around, remember that, in england, people drive on the left side of the road!).

We will reach Birmingham at 19:30-20:00 (GMT) probably, time to check in at the hotel and prepare ourselves for tomorrow's day one at the europython conference.

@birmingham

Here I'm, at the Ibis City Centre Hotel in Birmingham. The trip from A Coruña to Birmingham was very funny and the best part was for sure the driving experience from London to B'Ham (yes, that's the name they use for Birmingham in traffic signs, seems the name is too large).

We picked up the car at the europcar rental service in Heathrow, a brand-new Peugeot 207:

It took us some time to decide, as they let us choose the one we would like the most (between Ford, Skoda, Seat, Volkswagen, Nissan and some other brands). Finally, the 207 seemed the bigger one, so we took it.

I drove the entire trip (Santi will drive back to London on Friday) and it was quite a... strage experience. Having to drive from the right side of the car, and on the left side of the road... We got lost 4 times before getting the correct path to Birmingham, using the M40 highway, and each time it was because we missed something on cross roads and rotondas (how is the name for that...?). Lucky for us, we brought a GPS with us, so getting back to the road was easy.

Of course, we did stop to get some food:

Nothing really exotic, only some sandwiches at a service area by the highway. But the sandwiches were fine and we could eat them on the countryside.

Some hours later than expected, we arrived at B'Ham and we had to explore a little bit of the city (which is really really nice, you should visit it, seriously) until we reach the Ibis Hotel. Time for the check-in, see the room and get back to the street, just to do a little walk and drink (obviously) a point of beer. The Hotel is located just in the middle of Chinatown, so we are sorrounded by all kinds of asiatic restaurants, shops, etc and the Arcadian is pretty near too, so there is a lot of places to spent some time walking around.

Just to finish this post, this is a picture of what we see from the room's window:

About to begin the europython 2009

Right now I'm sitting on a chair in the Adrian Boult Hall inside the Birmingham Conservatoire. Just some minutes ago we went through registration, we picked up our t-shirts and stuff and got some orange juice.

The crew from the conference is sitting just in front of me, and the welcome talk is about to begin.

(more info to come)

first europython talks day

Finally back to the hotel, it is time to take a look back and analyze the first talks day at europython 2009.

After the opening I've referenced in my previous post, I headed for the first talk of the morning for me, Acceptance Testing with RobotFramework. Pekka Klärck, the lead developer of RobotFramework covered both the needed introduction to the project (a tool to automate code testing) and a more detailed explanation of how it works and how you could benefit from it. Very interesting talk, being the best part of it when he showed us how to combine the RobotFramework with Selenium to perform automatic web testing. Very interesting and practical talk!

Just after this first one ended, we headed to Lecture Room 1, where Luke Leighton was prepared for the The Zen of Pyjamas talk. I've never heard of Pyjamas until the europython, but it is quite an interesting piece of software. So, what it is? This note from the project website is pretty clear:

"pyjamas is a stand-alone python to javascript compiler, an AJAX framework / library and a Widget set API. "

That means that, with pyjamas, you can write python code and then generate javascript code to handle most usual ajax tasks. If you add to that, the fact that there is a Pyjamas Desktop project that allow you to pick up a Pyjamas project (web) and generate a full-bloated desktop application... it is quite a worth to take a deeper look into the project.

The only problem with this talk was related to the fact that it was a 60-min talk, in a full-crowded room (too many people in there). So from time to time wasn't very comfortable. But a nice talk anyway Luke is a real showman!.

From the upstairs room, we had a privileged view of the main hall, where all the stands are installed (google, oracle, plone, oreilly, etc), as well as the white boards to write open space and lightning talk proposals.

After the talk about pyjamas, we had lunch in the cafeteria of the conservatory. I don't know the name of those dishes, but I've learned why everybody outside UK (and a lot of them from inside) say English food is not so good...

After had lunch we went back to Adrian Boult Hall, to attend the Corey Doctorow's KeyNote. He covered an interesting topic that has been there since the origin of the Internet, the copyright/license issues about content in the Internet (or moving through). The room was full of people (I think everybody was there) and it was a joy to listen to him.

At the end of the keynote, he give for free 3 copies of 2 of his books, just throwing them into the air, pointing at the attendes. Just a picture of his keynote (click on it to get a bigger view). I've added a picture of one of the tries the crew from the europython did, just trying to connect with Guido (Guido Van Rossum)

Then I attended a talk about python and the xapian search engine, from Richard Boulton, who was somehow nervous and spent more time talking about the things his flax solution does than to demostrante how to use xapian with python... (IMHO, and do not misunderstood me, I think flax is a nice approach).

Back to Adrian Boult Hall once more, to saw Zeth running a presentation about pixelise, a django-app that allows you to manage XML files efficiently using a DBD XML database.

Finally, I got back to room number 1, picked up a nice place, and relaxed myself a bit, while waiting for the next three talks.

At 19:00 the day ended officially at europython... it was time for pub-parties!.

Sadly for me, there was no party today. I had a lot of work to do to prepare tomorrow's publication of one of our current projects so, after a short walk from the conservatory to the hotel (really near, a 10-minute walk through the city center), and a stop to have some dinner, we arrived somehow tired at the hotel.

I've been working on that project since then, but now seems the perfect time to go to bed. (But let me show you some pics of the trip went back to the hotel first)

Living a day between a laptop and a mobile phone

no pics, no talks, no meetings, no time

Those words represent perfectly my third day in B'Ham. Today was a difficult day, probably one of the most difficult days in my life, so I couldn't attend the talks, the keynotes or the lighting talks.

The only thing I could attend to was the europython dinner, which was really an enjoyable experience, the perfect place to taste different food and meet nice people (more than 400 pythonistas in the same dinning room!).

Let's see what happens tomorrow.

Last conference day at europython 2009

After the storm...

Yes, today I was able to attend some talks. Yesterday was a really bad day. We had a plan to release a new project to the public but, due to some problems, we weren't able to do it, and that was a really a mess. I kept myself pushing hard towards the release, that was the reason why I spent the whole day sitting anywhere near a power plug, working on my laptop, seeing people going by, from here to there, joining talks, open spaces and so on. The rest of the time I've been on the phone, talking with my people (the guys working on the project with me) and with my customer (for whom the project was developed in the first place). A really really frustrating day.

So, back to today, the day begun with Bruce Eckel's Metaprogramming with decorators and metaclasses.

It was really an interesting talk, covering the goods of decorators (with some examples) and the obscure form of Metaprogramming, represented by metaclasses (each time I hear something about them, I also hear 1 million advices about why you shouldn't use them...).

Then I went to Python for system administrators, by John Pinner. I found this one very helpful, as it was more like a tutorial, with a lot of source code examples. John showed us how to create a backup tool using mostly Python's base library. From a find-like tool (that even allow regexp search to filter the results) to the use of the tarfile module to create compressed copies of the data. During the talk, a nice discussion about the best way to handle arguments passed to python scripts took place, with some nice points of view about getopt and argparse, among others. Sadly I don't have nice pics from this one.

Later on, I went to Pythonic filesystem APIs, by Tommy Virtanen, or how we should have a more pythonic APIs to filesystems. I agree with the idea we should have such a thing in Python.

After that talk, we had some time for lunch (yeeeehaaa!), so another chance to get some <ironic>quality-and-delicious english food... </ironic>

Back from the lunch break there was time for the talk from Steve Holden, The PSF and Us. Steve explained what the Python Software Foundation is and how we could get involved with it, promoting Python and helping the Python community.

Another interesting topic covered in this talk, is the idea that the python.org website should be updated, as it is some years old now. Steve explained his point of view for the new website and hi asked us for ideas for it.

Just after the PSF talk the lighting talks took place. Too bad I didn't have the time to prepare mine. Last year, when I was at the djangocon 2008, I see the lighting talks concept for the first time. I've to say I really like the idea of a lot of people giving really quick talks (no more than 5 minutes). In five minutes you have time enough to show a brief introduction to an idea, a project, some code or whatever. It is the perfect place to share information with a lot of people, that will probably contact you later on regarding your talk (for example, the PyCharm guy had a lot of people waiting for him after his LT).

Perhaps next year I'll have some time to prepare a proper LT (Zero14, Beyle... I've so many ideas to discuss...).

Just to finish this post, a picture of one of the squares in the centre of Birmingham. This is the place where we had lunch most of the days, behind the sun (yes, the sun was shinning).

Now it is time to go to bed. Tomorrow we will drive near 2 hours to get back to London, and then back to Spain.

Back home, the end of a crazy week

I'm home. Today was a loooong day. We woke up early, just to have a breakfast before leaving Birmingham. We had a coffee (cafe latte) + a muffin (chocolate muffin for me) at a nice cafe, reading the newspapers (the Birmingham Post for me).

After the breakfast, we picked up the rental car and we drove back to London, doing a short stop at Oxford (we had enough time anyway, that's the reason to wake up so early).

Really a nice place, with a lot of people on the streets (a lot of tourists too :D). Would be nice to have some more time to spend there.

We arrived at Heathrow just in time to leave the rental car and get to the correct terminal, get our tickets and fly back home.

I feel myself tired, a lot, not only because of the long trip today, but because of the whole week. The Europython was really a nice experience, I hope I'll find some time to get back next year (again, in Birmingham).

Just one last picture before going to bed, one of the Oxford Christ City Church buildings:

Saving your work from the python console

One of those things I always wanted to write about, and never found enough time to do it ;D.

One of the things I really like about Python is the python interpreter, usually called python shell or python console too.

It is a very useful feature of the programming language, because (as with any other shell like bash, tcsh or ksh) you can execute any source code in it, checking for results. I use it a lot for debugging/testing purposes (as you can even import modules, methods, etc).

Ok, now imagine you are testing a piece of code, or perhaps just writting some lines to get some information from, for example, data behind some django models. The actual code could be something like:

>>> import socialize.models as sclzmodels
>>> import socialize.utils as sclzutils
>>> sclzmodels.Person.objects.all().count()
>>> typeA = []
>>> typeB = []
>>> typeC = []
>>> for i in sclzmodels.Person.objects.all():
...     real_i = sclzutils.get_object_real_class_from_hierarchy(i, sclzmodels.Person)
...     if real_i.__class__.__name__ == 'TypeA':
...         typeA.append(real_i)
...     elif real_i.__class__.__name__ == 'TypeB':
...         typeB.append(real_i)
...     else:
...         typeC.append(real_i)

Do not pay too much attention on the code, as it is only an example. Basically it will filter a list (really a queryset) of objects and put them into three separate lists depending on some checks.

This is a piece of code really useful for me, that I could use quite often to get some valuable information. So, it could be very nice to put it in a script, just to execute it later without having to re-write those lines every time.

Now comes the handy trick.

As I've executed such code within the python shell, it is already in a buffer that I can easily export into a file, without having to copy/paste anything. The only thing I need to do is, within the same python shell:

>>> import readline
>>> readline.write_history_file('/path/to/where/I/want/to/save/history.txt')

After that you will have a file /path/to/where/I/want/to/save/history.txt that will contain the same code you executed in the python shell (without the >>> and ... promtps):

import socialize.models as sclzmodels
import socialize.utils as sclzutils
sclzmodels.Person.objects.all().count()
typeA = []
typeB = []
typeC = []
for i in sclzmodels.Person.objects.all():
    real_i = sclzutils.get_object_real_class_from_hierarchy(i, sclzmodels.Person)
    if real_i.__class__.__name__ == 'TypeA':
        typeA.append(real_i)
    elif real_i.__class__.__name__ == 'TypeB':
        typeB.append(real_i)
    else:
        typeC.append(real_i)

Nice, isn't it?

Autocompletion for your python shell|console|interpreter

One of the nice features of the django shell (that is, the python shell you get when you call ./manage.py shell inside a django project) is that it has tab autocompletion, that is, you can import a module and use tab-completion to see all the modules within it.

A lot of people are using iPython these days because of the lack of this feature on the default Python shell, but this is only by default, and if you are using iPython only because of the tab completion, forget about installing extra stuff, you only have to do this little trick:

import rlcompleter, readline
readline.parse_and_bind('tab:complete')

and you can use tab completion, for example:

>>> import smtplib
>>> smtplib.
smtplib.CRLF                      smtplib.SSLFakeFile               smtplib.__setattr__(
smtplib.LMTP                      smtplib.__all__                   smtplib.__sizeof__(
smtplib.LMTP_PORT                 smtplib.__class__(                smtplib.__str__(
smtplib.OLDSTYLE_AUTH             smtplib.__delattr__(              smtplib.__subclasshook__(
smtplib.SMTP                      smtplib.__dict__                  smtplib._have_ssl
smtplib.SMTPAuthenticationError(  smtplib.__doc__                   smtplib.base64
smtplib.SMTPConnectError(         smtplib.__file__                  smtplib.email
smtplib.SMTPDataError(            smtplib.__format__(               smtplib.encode_base64(
smtplib.SMTPException(            smtplib.__getattribute__(         smtplib.hmac
smtplib.SMTPHeloError(            smtplib.__hash__(                 smtplib.quoteaddr(
smtplib.SMTPRecipientsRefused(    smtplib.__init__(                 smtplib.quotedata(
smtplib.SMTPResponseException(    smtplib.__name__                  smtplib.re
smtplib.SMTPSenderRefused(        smtplib.__new__(                  smtplib.socket
smtplib.SMTPServerDisconnected(   smtplib.__package__               smtplib.ssl
smtplib.SMTP_PORT                 smtplib.__reduce__(               smtplib.stderr
smtplib.SMTP_SSL                  smtplib.__reduce_ex__(
smtplib.SMTP_SSL_PORT             smtplib.__repr__(

In this example, after importing smtplib I just wrote smtplib. in the shell and hit tab twice, to get all the available options in that module.

Nice, isn't it?

Ok, now it would be even nicer if we do not have to set tab completion each time we run the python shell, but this is an easy one.

Each time you run the python shell|console|interpreter, it will search for a variable in your environment called PYTHONSTARTUP, where you can set a file that will be executed after starting the shell (the perfect place to put some init-lines).

So, all you have to do is, first, add this two lines to a file, for example, called .pythonrc in your home:

$ cat ~/.pythonrc
import rlcompleter, readline
readline.parse_and_bind('tab:complete')
$

Second, add the PYTHONSTARTUP variable to your shell profile file. If you are using csh, tcsh or similar shells, this will do it:

echo "setenv PYTHONSTARTUP ~/.pythonrc" >> ~/.cshrc

If your shell is sh, bash, ksh or similars, this will do it:

echo "export PYTHONSTARTUP=~/.pythonrc" >> ~/.profile

(check your OS info, perhaps instead of .profile it could be .kshrc, .bashrc or something like that).

After that, you will be able to use tab completion each time in your default python shell, with no additional dependencies!

NOTE: Of course, you should take a look at iPython anyway, as it has a lot more features than tab-completion, it is a powerful tool you may find very very useful.

File locking in python

So, let's keep up with the practical techie posts.

If you do software development, almost any kind of it, at one point you will find this scenario where you are running some code in a process in a cron job (or any kind of periodic scheduler). Now, the crontab entry sets that this process has to be run, for example, each 5 minutes. One day 5 minutes is not enough for that process (that usually takes less than a minute) to finish... and there it goes the next call to run that code.

"What could go wrong?" (TM)

Well, depending on the code, maybe nothing happens, maybe an ugly mess will turn a nice day into a nightmare or maybe you will get a call in the middle of the night urging you to fix it ASAP.

Probably there will be a gazillion ways to fix it properly, and the fix will depend a lot on the code, what it does and how it was written to begin with.

In my case the solution seemed to be file locking, that is, create a locked file that could be checked before running the process. If the file is locked, that means another process is already running, so the current process should not start executing that code.

I was working on some python code (what else?), and python itself comes with some handy libraries and utilities to handle file locking: fcntl and specially fcntl.lockf. lockf has the benefit of being quite good at handling deadlocks (that is, something goes wrong with the process running the locking code and the lock file is left in a locked state, even if the code that locked it is not running anymore).

There are more options available, like zc.lockfile for example, but I usually prefer modules/code included in the standard library, unless the external package has too many benefits over it. Feel free to take a look at those options through Google or pypi.

Using lockf is really easy:

import fcntl
import sys

lock_filename = '/tmp/sample-locking.lock'
lock_file = open(lock_filename, 'w')

try:
    fcntl.lockf(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
except IOError:
    print('Cannot lock: ' + lock_filename)
    sys.exit(1)

print('Locked! Running code...')

quit = False
while quit is not True:
    quit = input('Press q to quit ')
    quit = str(quit) == 'q'

print('Bye!')
sys.exit(0)

This sample code tries to lock a file (/tmp/sample-locking.lock) and, if it can set the lock, it goes on executing some more code. In this case, a very simple loop waiting for a command to finish/quit (this is enough for the scope of this post, keeping the process running).

If you want to learn more about lockf, please refer to the official documentation here: https://docs.python.org/3/library/fcntl.html#fcntl.lockf | https://docs.python.org/2/library/fcntl.html#fcntl.lockf

Now, if we run this code in a python interpreter:

$ python lockexample.py
Locked! Running code...
Press q to quit q

The process will be running until we press the key q. Now, try to run it in a new python interpreter:

$ python lockexample.py
Cannot lock: /tmp/sample-locking.lock
$

As the first process was still running, the second process cannot acquire/set the lock, and so it stops running, showing an error message to the user.

Note: This code should run fine with both python 2.x and 3.x*.

This is a very simple example, but I guess you got the point here.

lock_file = open(lock_filename, 'w')

with open(lock_filename, 'w') as lock_file:
    # Rest of the code goes indented here

def lock(filename):
    lock_file = open(filename, 'w')
    try:
        fcntl.lockf(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except IOError:
        return False
    return True

import fcntl
import sys

def lock(filename):
    lock_file = open(filename, 'w')
    try:
        fcntl.lockf(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except IOError:
        return False
    return True

lock_filename = '/tmp/sample-locking.lock'
locked = lock(lock_filename)

if not locked:
    print('Cannot lock: ' + lock_filename)
    sys.exit(1)

print('Locked! Running code...')

quit = False
while quit is not True:
    quit = input('Press q to quit ')
    quit = str(quit) == 'q'

print('Bye!')
sys.exit(0)

import fcntl
import threading

def task():
    lock_filename = '/tmp/sample-locking.lock'
    lock_file = open(lock_filename, 'w')
    try:
        fcntl.lockf(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except IOError:
        print('Cannot lock: ' + lock_filename)
        return False
    print('Locked! Running code...')
    quit = False
    while quit is not True:
        quit = raw_input('Press q to quit ')
        quit = str(quit) == 'q'
    print 'Bye!'
    return True

if __name__ == '__main__':
    print('creating threads')
    first_thread = threading.Thread(target=task, args=())
    second_thread = threading.Thread(target=task, args=())

    print('starting threads')
    first_thread.start()
    second_thread.start()

    print('joining threads')
    first_thread.join()
    second_thread.join()

    print('closing')

$ python lockexample-same-code.py
creating threads
starting threads
joining threads
 Locked! Running code...
Press q to quit Locked! Running code...

q
Bye!
Press q to quit