Log of last night
(Previously, on the most stupid sysadmin in the world)
Some days ago a friend asked me to set up a subversion repo + trac in one of my servers, so he could host some projects there
I checked I had everything necessary (subversion 1.5, trac 0.10.4, etc)
I tried to create a repo:
svnadmin create /path/to/friends/svn/repo
and I got an ugly error about some "shared library not found".
I tried to reinstall/upgrade subversion, which leads me to subversion 1.5 being removed and subversion 1.6 installed
Of course I had to upgrade the python bindings for svn and trac itself
the new trac doesn't like python 2.4 sqlite version, so I had to upgrade sqlite and the python bindings for it, which means, in the end, upgrading python too.
All of those points led me to yesterday's chapter in this story The sysadmin nightmare!
UPDATE: Ah!, and I almost forgot a very nice part of the story... When I tried #python in freenode, just to get the same answer from some people:
- Your problem is mod_python itself
- You should not really use mod_python
- mod_python is nothing compared to mod_wsgi
:( C'mon guys, that reminds me so much of that time where every answer for every question in that channel was "just use twisted for that" (it wasn't a matter if you were trying to develop your own web server or a simple script to say hello, world). Some times people should just keep their fingers off the keyboard if they do not know the answer to a question, intead of bothering with nonsense stuff, really.
1- After upgrading python to 2.6 (keeping 2.4 installed because I need that for Zope) I realized that I needed to upgrade mod_python too, to use the new interpreter.
2- I upgraded mod_python, not really an upgrade, just recompile/reinstall version 3.3.1 against python 2.6
3- I restarted apache... and all my python-based webapps and websites were gone. All I got instead was this error in the error-log files:
python_handler: Can't get/create interpreter.
4- I used google to search for answers, nothing that could make sense
5- I decided then to reinstall/upgrade apache, from 2.2.8 to 2.2.13, and as I was going to upgrade it, I replaced the prefork mpm with the worker mpm, adding this to /etc/make.conf:
WITH_MPM=worker WITH_THREADS=yes WITH_DAV_MODULES=yes WITH_LDAP_MODULES=yes WITH_PROXY_MODULES=yes WITH_CACHE_MODULES=yes WITH_SSL_MODULES=yes WITH_AUTH_MODULES=yes WITH_MISC_MODULES=yes WITH_CUSTOM_EXPERIMENTAL=ext_filter
6- Obviously, after upgrading apache, I had to upgrade some extra modules again. At this point I decided not to install php (in the end, I do not need it anymore)
7- Once everything was recompiled and upgraded, I tried to restart apache again. This time apache won't start at all, no errors in the log files... wtf..
8- I found in some mailing list that, since the upgrade to apache 2.2.13, you need a new kernel module in FreeBSD, so I load it:
sudo kldload accf_data.ko
Something similar to what I already did with the accf_http.ko module in previous versions of the web server. This wasn't anyway the reason of the fail.
9- After almost 1 hour trying to start apache in debug mode, in foreground mode, raising LogLevel and I can't remember how many more things, I decided to get the server back to the prefork mpm, just in case anything was wrong with my FreeBSD 6-STABLE server.
10- After recompiled apache again I tried once again to restart the server, same result, it won't boot at all and no errors in the log files. I started to become crazy.
11- I decided to reinstall apache again, with the worker mpm (as using the threaded model didn't made it fail) but this time removing, by hand, any file that could be left over by pkg_delete/make deinstall. It didn't work either.
12- Last try, I removed /usr/local/etc/apache22 and reinstalled it again using the port. Then I tried to restart apache, and it worked!
13- Ok, I put all the important parts of my old configuration files into the new ones (load mod_python, the virtualhosts, etc) and restarted apache. It worked again.
14- I checked the python webapps/websites, same error as before:
python_handler: Can't get/create interpreter
ARGGGG, ok, stay cool, you need to focus... I checked the main error-log from the apache server, just to find...[Tue Oct 06 04:41:53 2009] [error] make_obcallback: could not call init.\n Traceback (most recent call last): File "/usr/local/lib/python2.6/site-packages/mod_python/apache.py", line 937, in init from mod_python import importer File "/usr/local/lib/python2.6/site-packages/mod_python/importer.py", line 24, in from mod_python import publisher File "/usr/local/lib/python2.6/site-packages/mod_python/publisher.py", line 50, in from cache import ModuleCache, NOT_INITIALIZED File "/usr/local/lib/python2.6/site-packages/mod_python/cache.py", line 27, in import urllib2 File "/usr/local/lib/python2.6/urllib2.py", line 91, in import hashlib File "/usr/local/lib/python2.6/hashlib.py", line 136, in md5 = __get_builtin_constructor('md5') File "/usr/local/lib/python2.6/hashlib.py", line 63, in __get_builtin_constructor import _md5 ImportError: No module named _md5 [Tue Oct 06 04:41:53 2009] [error] get_interpreter: no interpreter callback found.
AHA!, found it.
15- Google, again, just to find some mails in some mailing lists pointing to some problems about the version of openssl used to compile python, mod_python and apache. Not my problem, kept myself searching...
16- Found another thread in another mailing that pointed me to some problems when importing hashlib (the replacement in +python2.5 for md5, sha, etc), this seems a value lead for me.
17- Search through mod_python's source code to see what's the point where the exception is raised and the legacy _md5 module was imported. Once I found that piece of code, I tried to recreate the problem in a python shell, just to find that even if I had python 2.6 and mod_python correctly installed, it crashes in /usr/local/lib/python2.6/hashlib.py (line 107):
new = __hash_new
Shit. Seemed like I would not be able to fix this...
18- Then I tried to fight it from another angle. The problem is that there is no _md5 module... but there is a copy of the legacy md5 one... long life ln!
lrwxr-xr-x 1 root wheel 6 Oct 6 04:43 _md5.py -> md5.py
19- Discovered that not only _md5 was the problem, after solved that, it began complaining about _sha... same fix:
lrwxr-xr-x 1 root wheel 6 Oct 6 04:43 _sha.py -> sha.py
20- UhOh, real problem now, began to complain about _sha256.py, but there was no legacy sha256.py in the python 2.6 lib dir. f***!
21- I began to search for all kinds of ports and packages, trying to find something... I went through security/py-openssl and some others until I found security/py-hashlib:
Python secure hash and message digest module MD5, SHA1, SHA224, SHA256, SHA384 and SHA512 (backported from Python 2.5 for use on 2.3 and 2.4) WWW: http://code.krypto.org/python/hashlib/
Nothing to loose, I installed it, and et voilá, the needed legacy modules were in the python 2.4 installation, now back to the power of ln:lrwxr-xr-x 1 root wheel 37 Oct 6 04:44 _sha256.so -> ../python2.4/site-packages/_sha256.so lrwxr-xr-x 1 root wheel 37 Oct 6 04:44 _sha512.so -> ../python2.4/site-packages/_sha512.so
22- Restarted apache, once again
23- Crossed my fingers... tried one of the python webapps... IT WORKED! ;D
24- Ok, nice. Now it is working I went back to the svn+trac issue. I reinstalled trac 0.11.X and set the environment.
25- 04:50 Everything seems to be running ok now, finally I can go to sleep...