» blog.e-shell.org

PyConES 2015

El pasado fin de semana tuvo lugar una nueva edición de la reunión anual de la comunidad española de python. Esta vez, la tercera, nos reunimos en Valencia durante tres días (viernes-domingo) durante los cuales disfrutamos de talleres, charlas, lightning talks y muchisimos buenos momentos.

Personalmente he asistido a las tres ediciones (aunque reconozco que no he escrito nada sobre las anteriores, en Madrid y Zaragoza respectivamente) y tengo que reconocer que, para mi, esta ha sido la mejor PyConES hasta la fecha.

El viaje a valencia fue divertido, Oscar y yo nos fuimos para allá en avión desde Santiago de Compostela (vuelo directo con Ryanair). Sin incidencias ni a la idea ni a la vuelta, y eso que cierto miedo irracional de Oscar me tenía un poco precupado.

En valencia nos quedamos en casa de Omar, un amigo de los viejos tiempos del IRC Hispano al que hacía por lo menos 10 años que no veíamos en persona. Tanto él como el resto de la tropa linuxera de valencia fueron unos perfectos anfitriones y, a pesar de no venir a la pycon, se unieron al grupo a la hora de cenar y tomar algo.

Pero volvamos a la PyConES 2015!

La organización se merece un 10. Siempre atentos a cualquier detalle, cualquier cosa que pudiesemos necesitar.

El espacio en el que se dieron las charlas (la UPV - Universidad Politécnica de Valencia) estaba bien, espacioso (quizás el aula del track avanzado un poco pequeña) y bien equipado (baños, café, bebidas... WIFI sin cortes!!).

El catering muy bueno, tanto las comidas de sabado y domingo (en una cafetería universitaria, con menús muy completos y opciones vegerianas/veganas) como los cafés (y bollos!!) de los descansos.

El formato se mantuvo con respecto a las anteriores ediciones. Tres tracks principales (básico, avanzado y científico) repartidos en tres aulas separadas. El contenido de las charlas seleccionadas fue variado, cubriendo desde temas bastante core (tracemalloc por poner un ejemplo) a desarrollo web (algo de django por aqui, algo más de django por allá). También hubo bastante de carácter científico, lo que ya es algo habitual en todas las PyConES.

Si tuviese que decir un término que se repitió por encíma de los demas, diría que fue el de programación asíncrona, especialmente operaciones de entrada/salida asíncronas con asyncio.

Durante esta edición de la PyConES también llevamos a cabo una asamblea extraordinaria de la asociación python españa, en la que debatimos varios temás de interés para el futuro de la asociación. Para mi esta reunión sirvió, además, para poner cara a gente con la que he cruzado correos y conversaciones online durante los últimos meses (\o/).

No voy a daros números de charlas, asistentes, etc (podeis ver esos datos en twitter, si buscais por el hashtag #PyConES15), pero lo que si puedo deciros es que, una vez más, ha sido un placer poder formar parte de un evento como este.

Personalmente me he vuelto a casa con la sensación de que ha sido la PyConES en la que he conocido a más gente, he participado un poco más y he hablado de más temas. He conocido a un montón de gente de pequeños grupos de python de otros lugares del pais (Granada, Valencia, Mallorca, Barcelona... y sobre todo Vigo!) y, por supuesto, me he vuelto a encontrar con viejos amigos a los que hacía casi un año que no veía en persona.

Todavía recuerdo la primera edición, en Madrid. Después de un montón de tiempo en la comunidad de python, todos mis contactos en la comunidad estaban fuera de españa (incluso había asistido a eventos como la djangocon US o la Europython), asi que conocer a gente nueva y en un entorno más cercano geográficamente sonaba realmente atractivo.

Tres años (y tres PyConES) más tarde, no tendría espacio suficiente aquí para mencionar a todas y cada una de las increíbles personas que he encontrado dentro de esta comunidad.

Gracias a todos por hacer esto posible, vamos a seguir trabajando entre todos para seguir haciendo una PyConES un poquito mejor cada año.

Store python PersistentMapping objects into an objects TreeSet

I've been working with ZODB for quite some time now. First when I was doing Zope based web development (oh, the ol' good^hard days), then in some courses I gave about persistence and data storage in python, nowadays working with pyramid and still using ZODB for some apps.

Some days ago, working on one of those pyramid based projects, I found something really weird, something that was not working as expected, and it took me some time to figure out what was it. I'd like to share it, just in case it happens to any of you.

When working with ZODB, there are a couple of packages you will probably use: persistent and BTrees. Both are related to adding persistence to python objects, so you can store them easily in the objects database. Long story short, they make your life a bit easier.

Let's show some sample source code:

from persistent.mapping import PersistentMapping
from functools import total_ordering

@total_ordering
class Box(PersistentMapping):
    __parent__ = __name__ = None
    def __init__(self, name):
        self.name = name
        super(Box, self).__init__()
    def __repr__(self):
        return "<%s.%s %s>" % (self.__class__.__module__,
                               self.__class__.__name__,
                               self.name)
    def __str__(self):
        return self.name

    def __hash__(self):
        return hash(str(self))

    def __eq__(self, other):
        return self.__hash__() == other.__hash__()

    def __lt__(self, other):
        return self.__hash__() > other.__hash__()

    def __gt__(self, other):
        return self.__hash__() < other.__hash__()

Just a model called Box. I'm going to skip all the ZODB setup here, as it is not needed to show you the problem I've found. The class has the usual python methods __init__, __repr__, __str__ and a single name attribute.

It also has a __hash__ method, which is used by the builtin hash() when comparison operations are performed by things like a dictionary (for example to check for duplicates when adding items to a dict).

The __hash__ method is also used in the __eq__, __lt__ and __gt__ methods, just as example in this sample code. These methods should contain some more extensive logic that could let us know if two Box instances are equal or if one is less/greater than the other one.

Finally, the @total_ordering decorator is applied to the Box class. This decorator will extend our comparison methods (__eq__, __lt__ and __gt__) adding the ones that are missing (__le__, __ge__, etc). Basically less code typing.

Let's play a bit with this model in a python interpreter/shell:

>>> from models import Box
>>> sweets = Box(name='sweets')
>>> apples = Box(name='apples')
>>> onions = Box(name='onions')
>>> sweets == apples
False
>>> apples == onions
False
>>> onions == onions
True
>>> sweets > apples
True
>>> apples > sweets
False
>>> apples < sweets
True
>>> boxes = []
>>> boxes.append(sweets)
>>> boxes.append(apples)
>>> boxes.append(onions)
>>> boxes
>>> [<models.Box sweets>, <models.Box apples>, <models.Box onions>]
>>> boxes.append(sweets)
>>> boxes
[<models.Box sweets>, <models.Box apples>, <models.Box onions>, <models.Box sweets>]
>>>

We can create boxes, compare them (well, the great than/less than comparisons are a bit naive as I mentioned, but for the purposes of this, it works) and even add them into a list of boxes. We can the same Box instance multiple times to the boxes list. This works in the same way for both python 2.7.x and 3.x.

Now, when storing those lists of objects into a ZODB, in some situations using a OOTreeSet from the BTrees package is more efficient than storing a plain list. Let's play a bit more with a python 3 shell:

>>> from BTrees.OOBTree import OOBTree, OOTreeSet
>>> boxes_ts = OOTreeSet()
>>> boxes_ts.add(sweets)
1
>>> boxes_ts.add(apples)
1
>>> boxes_ts.add(onions)
1
>>> boxes_ts
<BTrees.OOBTree.OOTreeSet object at 0x10c1c30d0>
>>> list(boxes_ts)
[<models.Box sweets>, <models.Box apples>, <models.Box onions>]
>>> boxes_ts.add(sweets)
0
>>> list(boxes_ts)
[<models.Box sweets>, <models.Box apples>, <models.Box onions>]
>>>

As expected, we can add the different instances of Box to the treeset, but we cannot add duplicates there, so the last attempt to add the sweets box does not work.

Now, same thing in python 2:

>>> from BTrees.OOBTree import OOBTree, OOTreeSet
>>> boxes_ts = OOTreeSet()
>>> boxes_ts.add(sweets)
1
>>> boxes_ts.add(apples)
0
>>> list(boxes_ts)
[<models.Box sweets>]
>>> boxes_ts.add(onions)
0
>>> list(boxes_ts)
[<models.Box sweets>]
>>>

"WTF!" - was my first thought when I notice I couldn't add more Box instances into the treeset. They are different objects, different instances of the same class, but our class has all the needed methods so anything trying to compare both objects can find out that they are actually not the same object, right?

Well, actually not, at least for python 2.

It took me a while to figure out was happening, but first the help from ztane from the #pyramid channel in freenode, then after looking a bit at the source code from the BTrees package, I was able to find it.

The code that performs the comparison operation between two objects, before adding them to the TreeSet was using a cmp() function imported from BTrees._compat. Taking a look in the file BTrees/_compat.py I found:

if sys.version_info[0] < 3: #pragma NO COVER Python2

    PY2 = True
    PY3 = False

    from StringIO import StringIO
    BytesIO = StringIO

    int_types = int, long
    xrange = xrange
    cmp = cmp

    ...

else: #pragma NO COVER Python3

    PY2 = False
    PY3 = True

    from io import StringIO
    from io import BytesIO

    int_types = int,
    xrange = range

    def cmp(x, y):
        return (x > y) - (y > x)

    ...

So, if we run this code in python 3, it relies on a cmp() function defined in that module withing BTrees, which performs some gt/lt operations (covered by our __lt__ and __gt__ methods), but if we run it in python 2 it relies on the builtin cmp from the standard library.

The reason for doing it like that is explained here:

http://python3porting.com/problems.html#unorderable-types-cmp-and-cmp

Now, let's go back to the python 2 shell:

>>> cmp(sweets, apples)
0
>>> cmp(sweets, sweets)
0
>>>

Effectively, for the builtin cmp in python 2, those objects are the same.

Easy to fix, as cmp will rely on the __cmp__ method, if our class provides it. So we only have to extend our class code a bit:

from persistent.mapping import PersistentMapping
from functools import total_ordering

@total_ordering
class Box(PersistentMapping):
    __parent__ = __name__ = None
    def __init__(self, name):
        self.name = name
        super(Box, self).__init__()
    def __repr__(self):
        return "<%s.%s %s>" % (self.__class__.__module__,
                               self.__class__.__name__,
                               self.name)
    def __str__(self):
        return self.name

    def __hash__(self):
        return hash(str(self))

    def __eq__(self, other):
        return self.__hash__() == other.__hash__()

    def __lt__(self, other):
        return self.__hash__() > other.__hash__()

    def __gt__(self, other):
        return self.__hash__() < other.__hash__()

    def __cmp__(self, other):
        if self.__lt__(other):
            return -1
        if self.__eq__(other):
            return 0
        if self.__gt__(other):
            return 1

Now, back to the python 2 shell:

>>> from models import Box
>>> sweets = Box('sweets')
>>> apples = Box('apples')
>>> cmp(sweets, apples)
-1
>>> cmp(apples, sweets)
1
>>> cmp(apples, apples)
0
>>>

Much better, now let's try adding to a TreeSet:

>>> from BTrees.OOBTree import OOBTree, OOTreeSet
>>> boxes_ts = OOTreeSet()
>>> boxes_ts.add(sweets)
1
>>> boxes_ts.add(sweets)
0
>>> boxes_ts.add(apples)
1
>>> boxes_ts.add(apples)
0
>>> boxes_ts.add(sweets)
0
>>>

It works!

Now, some open questions...

According to the official python 2 docs for __cmp__:

Called by comparison operations if rich comparison (see above) is not defined.

So, shouldn't the builtin cmp() use the rich comparison methods in case __cmp__ is not available?.

If @total_ordering is supposed to fill in our code with the rest of the needed comparison methods (considering we already provided __eq__ and one of __lt__ or __gt__), shouldn't it add a __cmp__ method too in python 2?

Sentry email notifications not arriving?

I've been using Sentry for quite some time now in some projects, mostly all of them running python based applications and systems, and recently I did a major upgrade of a Sentry server to the latest version (8.3.1). It was an update with lots of changes, both in the Sentry internals and the dependencies needed by it (like the addition of Redis) but everything went more or less smoothly.

Once I had the server ready, I did upgrade the Raven client accordingly and I modified something in our staging server so an exception (IOError actually) was raised and the proper notification was sent to the Sentry server.

It worked just fine, the error message was sent to the server and it appeared on the Sentry web interface. Just perfect... or not?

Not really. After the error arrived in the Sentry server, a notification should have been sent to a mailing list where developers/support would see it and act accordingly. But that email never arrived and there was no error on the Sentry server logs.

When I took a look at the SMTP server logs, I found this

12:53:12 sm-mta[1534]: NOQUEUE: connect from my.sentry.server [xxx.xxx.xxx.xxx]
12:53:12 sm-mta[1534]: AUTH: available mech=SCRAM-SHA-1 DIGEST-MD5 OTP CRAM-MD5 ANONYMOUS, allowed mech=LOGIN PLAIN
12:53:12 sm-mta[1534]: u4JArC4c001534: Milter (bogom): init success to negotiate
12:53:12 sm-mta[1534]: u4JArC4c001534: Milter (opendkim): init success to negotiate
12:53:12 sm-mta[1534]: u4JArC4c001534: Milter (opendmarc): init success to negotiate
12:53:12 sm-mta[1534]: u4JArC4c001534: Milter: connect to filters
12:53:12 sm-mta[1534]: u4JArC4c001534: milter=bogom, action=connect, continue
12:53:12 sm-mta[1534]: u4JArC4c001534: milter=opendkim, action=connect, continue
12:53:13 sm-mta[1534]: u4JArC4c001534: milter=opendmarc, action=connect, continue
12:53:17 sm-mta[1534]: u4JArC4c001534: my.sentry.server [xxx.xxx.xxx.xxx] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA

Which basically means there was an incoming connection which was suddently closed by the client.

Weird.

So, what was happening? Good question indeed. First thing to try was the email testing tool that comes bundled with Sentry. In the admin panel in the web interface, there is a Mail section, at the bottom of the page you can find such tool.

I clicked there, nothing on the sentry logs, same lines in the SMTP server logs. Not really useful.

And so I started to look at Sentry's source code, to see how those emails are sent. I ended up looking at this file:

https://github.com/getsentry/sentry/blob/8.1.3/src/sentry/utils/email.py

It seems that, to open the SMTP connection, it relies on the get_connection method from django, available here:

https://github.com/django/django/blob/1.6.11/django/core/mail/__init__.py

which, in this case, will use the code from the smtp backend from here:

https://github.com/django/django/blob/1.6.11/django/core/mail/backends/smtp.py

Please note I'm linking to the files for versions 8.1.3 of sentry, which has django 1.6.11 as a dependency at the moment of writing this article.

A bit further down the chain, django uses smtplib.SMTP from the base library of python 2. Now, looking at the documentation, it mentions a socket timeout:

" ... The optional timeout parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). If the timeout expires, socket.timeout is raised. ... "

Interesting. now, let's try to run Sentry's email sending code in a python shell/console. I'm running sentry inside a virtualenv, so after activating the environment, I start a shell and give it a try:

> sentry shell
Python 2.7.11 (default, Mar  1 2016, 11:50:03)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>

>>> from sentry.utils.email import send_mail
>>> send_mail('subject', 'body message', 'from@my.domain', ['to@my-other.domain.'])
Traceback (most recent call last):
  File "", line 1, in
  File "/home/sentry-env/lib/python2.7/site-packages/sentry/utils/email.py", line 393, in send_mail
    connection=get_connection(fail_silently=fail_silently),
  File "/home/sentry-env/lib/python2.7/site-packages/django/core/mail/__init__.py", line 50, in send_mail
    connection=connection).send()
  File "/home/sentry-env/lib/python2.7/site-packages/django/core/mail/message.py", line 276, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/home/sentry-env/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 87, in send_messages
    new_conn_created = self.open()
  File "/home/sentry-env/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 48, in open
    local_hostname=DNS_NAME.get_fqdn())
  File "/usr/local/lib/python2.7/smtplib.py", line 256, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/local/lib/python2.7/smtplib.py", line 317, in connect
    (code, msg) = self.getreply()
  File "/usr/local/lib/python2.7/smtplib.py", line 365, in getreply
    + str(e))
SMTPServerDisconnected: Connection unexpectedly closed: timed out
>>>

At the same time, I was able to see the usual lines in the SMTP server logs.

So, indeed there was a time out. Now, let's check the SMTP configuration. That server runs sendmail, which has an interesting feature called greet_pause, which basically makes the SMTP to hold on for some time before sending the greeting message to the client, after the initial connection. This is a nice spam protection measure, as most spam bots do not wait for that greeting message before starting to send commands, and sendmail will simply drop the connection if the client do not wait for that greeting message before start talking.

But this was not the case here, the SMTP logs should show a total different thing for that, something like

sm-mta[92223]: u5525oOE092223: rejecting commands from [113.132.2.5]
[113.132.2.5] due to pre-greeting traffic after 1 seconds

In this case it seemed the timeout was happening on the client side, as if the connection were dropped by the client after wait for some time.

Interesting, let's go back to a sentry shell, check the default socket timeout there:

>>> import socket
>>> socket.getdefaulttimeout()
5.0
>>>

5 seconds!, now let's look at the sendmail configuration file...

FEATURE(`greet_pause', 5000)

Exactly 5 seconds too. So, basically, python's smtplib was opening a connection with a time out of 5 seconds and sendmail was waiting for 5 seconds after the initial connection was established to send that greeting message. Obviously, python's smtplib was correctly and politely waiting for that greeting message before sending any commands to the SMTP server... but by the time the greeting message was about to be sent by sendmail, python closed the connection with a timeout.

WOW

Now, how to fix this?, easy, raise that timeout a bit on the client side. All we need is to use setdefaulttimeout to adjust it. As sentry is a django-based app, and still has a django-style configuration file, which is a python script that is loaded every time you run the app, all we need is to add the following two lines to sentry.conf.py

# Set the default socket timeout to a value that prevents connections
# to our SMTP server from timing out, due to sendmail's greeting pause
# feature.
import socket
socket.setdefaulttimeout(10)

(Yeah, better put a good comment there, for the next guy looking into the config file and wondering why the f*** we set that timeout there)

In this example I've raised it to 10 seconds, adjust that value to whatever fits for you.

A restart of the sentry server and et voilà, email notifications were back!