Log of last night
July 2020
Sun Mon Tue Wed Thu Fri Sat
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31  
This site is an effort to share some of the base knowledge I have gathered through all this years working with Linux, FreeBSD, OpenBSD, Python or Zope, among others. So, take a look around and I hope you will find the contents useful.
Recent Entries
Recent Comments
Recent Trackbacks
OpenBSD (9 items)
BSD (0 items)
FreeBSD (19 items)
Linux (3 items)
Security (3 items)
Python (22 items)
Zope (13 items)
Daily (144 items)
e-shell (9 items)
Hacks (14 items)
PostgreSQL (3 items)
OSX (8 items)
Nintendo DS (0 items)
enlightenment (0 items)
Apache (3 items)
Nintendo Wii (1 items)
Django (24 items)
Music (12 items)
Plone (7 items)
Varnish (0 items)
Lugo (2 items)
Sendmail (0 items)
europython (7 items)
Cherokee (1 items)
self (1 items)
Nature (1 items)
Hiking (0 items)
uwsgi (0 items)
nginx (0 items)
cycling (10 items)
Networking (1 items)
DNS (0 items)

Syndicate this site (XML)

RSS/RDF 0.91

06 octubre

Log of last night

The reasons for me to get in bed at 05:00

(Previously, on the most stupid sysadmin in the world)

  • Some days ago a friend asked me to set up a subversion repo + trac in one of my servers, so he could host some projects there

  • I checked I had everything necessary (subversion 1.5, trac 0.10.4, etc)

  • I tried to create a repo:

    svnadmin create /path/to/friends/svn/repo

    and I got an ugly error about some "shared library not found".

  • I tried to reinstall/upgrade subversion, which leads me to subversion 1.5 being removed and subversion 1.6 installed

  • Of course I had to upgrade the python bindings for svn and trac itself

  • the new trac doesn't like python 2.4 sqlite version, so I had to upgrade sqlite and the python bindings for it, which means, in the end, upgrading python too.

All of those points led me to yesterday's chapter in this story The sysadmin nightmare!

UPDATE: Ah!, and I almost forgot a very nice part of the story... When I tried #python in freenode, just to get the same answer from some people:

  • Your problem is mod_python itself
  • You should not really use mod_python
  • mod_python is nothing compared to mod_wsgi

:( C'mon guys, that reminds me so much of that time where every answer for every question in that channel was "just use twisted for that" (it wasn't a matter if you were trying to develop your own web server or a simple script to say hello, world). Some times people should just keep their fingers off the keyboard if they do not know the answer to a question, intead of bothering with nonsense stuff, really.

1- After upgrading python to 2.6 (keeping 2.4 installed because I need that for Zope) I realized that I needed to upgrade mod_python too, to use the new interpreter.

2- I upgraded mod_python, not really an upgrade, just recompile/reinstall version 3.3.1 against python 2.6

3- I restarted apache... and all my python-based webapps and websites were gone. All I got instead was this error in the error-log files:

python_handler: Can't get/create interpreter.


4- I used google to search for answers, nothing that could make sense

5- I decided then to reinstall/upgrade apache, from 2.2.8 to 2.2.13, and as I was going to upgrade it, I replaced the prefork mpm with the worker mpm, adding this to /etc/make.conf:


6- Obviously, after upgrading apache, I had to upgrade some extra modules again. At this point I decided not to install php (in the end, I do not need it anymore)

7- Once everything was recompiled and upgraded, I tried to restart apache again. This time apache won't start at all, no errors in the log files... wtf..

8- I found in some mailing list that, since the upgrade to apache 2.2.13, you need a new kernel module in FreeBSD, so I load it:

sudo kldload accf_data.ko

Something similar to what I already did with the accf_http.ko module in previous versions of the web server. This wasn't anyway the reason of the fail.

9- After almost 1 hour trying to start apache in debug mode, in foreground mode, raising LogLevel and I can't remember how many more things, I decided to get the server back to the prefork mpm, just in case anything was wrong with my FreeBSD 6-STABLE server.

10- After recompiled apache again I tried once again to restart the server, same result, it won't boot at all and no errors in the log files. I started to become crazy.

11- I decided to reinstall apache again, with the worker mpm (as using the threaded model didn't made it fail) but this time removing, by hand, any file that could be left over by pkg_delete/make deinstall. It didn't work either.

12- Last try, I removed /usr/local/etc/apache22 and reinstalled it again using the port. Then I tried to restart apache, and it worked!

13- Ok, I put all the important parts of my old configuration files into the new ones (load mod_python, the virtualhosts, etc) and restarted apache. It worked again.

14- I checked the python webapps/websites, same error as before:

python_handler: Can't get/create interpreter

ARGGGG, ok, stay cool, you need to focus... I checked the main error-log from the apache server, just to find...

[Tue Oct 06 04:41:53 2009] [error] make_obcallback: could not call init.\n
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/site-packages/mod_python/apache.py", line 937, in init
from mod_python import importer
  File "/usr/local/lib/python2.6/site-packages/mod_python/importer.py", line 24, in
from mod_python import publisher
  File "/usr/local/lib/python2.6/site-packages/mod_python/publisher.py", line 50, in
from cache import ModuleCache, NOT_INITIALIZED
  File "/usr/local/lib/python2.6/site-packages/mod_python/cache.py", line 27, in
import urllib2
  File "/usr/local/lib/python2.6/urllib2.py", line 91, in
import hashlib
  File "/usr/local/lib/python2.6/hashlib.py", line 136, in
md5 = __get_builtin_constructor('md5')
  File "/usr/local/lib/python2.6/hashlib.py", line 63, in __get_builtin_constructor
import _md5
ImportError: No module named _md5
[Tue Oct 06 04:41:53 2009] [error] get_interpreter: no interpreter callback found.

AHA!, found it.

15- Google, again, just to find some mails in some mailing lists pointing to some problems about the version of openssl used to compile python, mod_python and apache. Not my problem, kept myself searching...

16- Found another thread in another mailing that pointed me to some problems when importing hashlib (the replacement in +python2.5 for md5, sha, etc), this seems a value lead for me.

17- Search through mod_python's source code to see what's the point where the exception is raised and the legacy _md5 module was imported. Once I found that piece of code, I tried to recreate the problem in a python shell, just to find that even if I had python 2.6 and mod_python correctly installed, it crashes in /usr/local/lib/python2.6/hashlib.py (line 107):

new = __hash_new

Shit. Seemed like I would not be able to fix this...

18- Then I tried to fight it from another angle. The problem is that there is no _md5 module... but there is a copy of the legacy md5 one... long life ln!

lrwxr-xr-x   1 root  wheel       6 Oct  6 04:43 _md5.py -> md5.py

19- Discovered that not only _md5 was the problem, after solved that, it began complaining about _sha... same fix:

lrwxr-xr-x   1 root  wheel       6 Oct  6 04:43 _sha.py -> sha.py

20- UhOh, real problem now, began to complain about _sha256.py, but there was no legacy sha256.py in the python 2.6 lib dir. f***!

21- I began to search for all kinds of ports and packages, trying to find something... I went through security/py-openssl and some others until I found security/py-hashlib:

Python secure hash and message digest module MD5, SHA1, SHA224, SHA256,
SHA384 and SHA512 (backported from Python 2.5 for use on 2.3 and 2.4)

WWW: http://code.krypto.org/python/hashlib/

Nothing to loose, I installed it, and et voilá, the needed legacy modules were in the python 2.4 installation, now back to the power of ln:

lrwxr-xr-x   1 root  wheel      37 Oct  6 04:44 _sha256.so -> ../python2.4/site-packages/_sha256.so
lrwxr-xr-x   1 root  wheel      37 Oct  6 04:44 _sha512.so -> ../python2.4/site-packages/_sha512.so

22- Restarted apache, once again

23- Crossed my fingers... tried one of the python webapps... IT WORKED! ;D

24- Ok, nice. Now it is working I went back to the svn+trac issue. I reinstalled trac 0.11.X and set the environment.

25- 04:50 Everything seems to be running ok now, finally I can go to sleep...

Posted by wu at 19:53 | Comments (0) | Trackbacks (0)
<< Riding my bike this morning | Main | Request for your oppinions: E63 or E71? >>
Re: Log of last night

I know, I know... it could happen to me... but ROFLMAO!!!!

Sometimes the "don't touch the damn thing because it just works" it's one of the best advices ever :D

Posted by: Juanjo at octubre 06,2009 20:17
Re: Log of last night

Ains, yes, that's True. But I have an even better advice for "me from the future". Keep the fscking servers up-to-date oftenly.

I don't think that would happen if this server wasn't running so old versions of everything...

Of course, my solution is only a dirty gordisima ñapa, so I should start searching for a real solution to this problem.

P.S.: It is even more curious, because I have some other servers running apache 2.2.x + mod_python3 + python 2.6.x, without any problem, IMHO the problem has something to do with the fact I have 2 different python versions in the same server.

Posted by: Wu at octubre 06,2009 21:21
Please send trackback to:http://blog.e-shell.org/178/tbping
There are no trackbacks.
Post a comment