Upgrade Plone buildout from 3.1.7 to 3.3.3
I'm not going to explain all the details (as that would be a perfect topic for a doc perhaps one day I should write) but you can imagine the basic idea. Apache will accept requests for www.somedomain.com and will handle them with a VirtualHost that will ask Varnish for the info. Varnish will send back the info directly if posible (thnx to the proxy features) or will ask one of the four Zope instances for the info (using round-robin load balancing). The Zope instances will query the ZODB database served by the ZEO instance.
Well, as I was saying, I set this whole thing up back in april, 2009, and I didn't have the time to keep the versions of all the dependencies in there up-to-date. At the time I decided myself I had to upgrade the buildout, it was running Plone 3.1.7 using a buildout.cfg file (a slightly modified version without the [versions] stuff) to set the basic options of the buildout and a deploy.cfg file for the production environment stuff (creating the ZEO and 4 Zope instances, the varnish recipe, etc). One of the benefits of using buildout is that you can keep track of the needed dependencies in an isolated way, just installing everything you need within the buildout directory, you can even set the versions you want to use of certain parts of the buildout and you can leave the decission on which versions to install directly on the buildout machinery. This is very nice, except when it fails.
Let me guide you through all the pain-in-the-ass upgrade process, a marvelous walk from the omfg-why-is-it-broken to the final lets-reinstall-everything step where I found a solution to my problem ;D
Before going any further, I have to say that I wanted to upgrade the buildout but that was not the only reason to rebuild the buildout env, I needed to add a new site to the server, which depends on two new products/eggs developed by our company. This means I had to add the sources to the src/ directory within the buildout and add the needed stuff to the buildout.cfg file (in that example, the pxgo. stuff).
Ok, after adding that to my buildout, I just tried to run ./bin/buildout without internet access and telling it not to install or upgrade to new versions:
./bin/buildout -No -c deploy.cfg
I'm still wondering why, but it crashes, giving me an error when building simplejson. That was weird, because it is supposed not to rebuild any dependencies... but I thought it could be nice if I ran buildout without the -No options, upgrading the buildout.
./bin/buildout -c deploy.cfg
That was even worse, as it tried to install the latest versions of everything, that is, it tried to install Plone 4-alpha, which has a dependency on Zope 2.12.x, which is not compatible with... you can read more about this here:
As a lot of people were telling me in #plone in freenode that it was my fault since the beginning, because I wasn't pinning the versions of the dependencies in my buildout.cfg file (more on this here: http://www.netsight.co.uk/blog/2009/11/17/configuring-buildout-to-prefer-stable-releases).
Ok, so, at any point it became a common practice to set fixed versions in the buildout.cfg file (I missed that one, I should be more careful and do not let so much time to pass by since the last time I read about buildouts).
Then I tried to add some [versions] stuff into the buildout.cfg (the stuff you see in the linked file) and I tried to rebuild the buildout again:
./bin/buildout -c deploy.cfg
Still got some errors due to problems with versions and dependencies. I kept myself doing tests and more tests and changes and more changes to the config file until someone in #plone (thnx esteele) pointed me to the file .installed.cfg. That file save information about how your buildout was configured, including versions of the stuff in there.
Removing the .installed.cfg file helped, but then (after re-running ./bin/buildout again) I got stucked in another error:
More dependencies versioning problems...
At that point I was very dissapointed. First because the buildout isn't supposed to crash so easily, second because it was kinda hard to find useful information (as it is sometimes with specific errors).
Nevertheless, I tried to approach the problem from a different angle, and I started a new buildout from scratch. I updated the ZopeSkel package for Python 2.4 and then I use the paster tool to create the new buildout:
paster create -t plone3_buildout mangosta_buildout_new
It asked me some questions, like the version of plone I would like to use (by default 3.3.2). Then I use the bootstrap to create the buildout script and all the needed stuff:
cd mangosta_buildout_new && python2.4 bootstrap.py
And finally I build the buildout:
Everything went perfectly well ;D
So, once I had the new buildout I edited its default buildout.cfg file so it seemed something like this:
and I ran :/bin/buildout again... and it crashes again! :( with this error:
The error was related to Products.PloneFormGen and the only way I solved it was editing the buildout.cfg file and replacing the Plone version from 3.3.2 to 3.3.3. After that everything worked as expected (probably because Plone 3.3.3 depends on plone.app.locales 3.3.7 and not 3.3.5).
Fine, I was almost there. I added my deploy.cfg file and re-run the buildout:
./bin/buildout -v -c deploy.cfg
OK! It worked, and I was able to start the zeoserver and the four Zope instances!
Finally, there was only one last step, that is, moving all data from the old buildout to this one. This was really easy, After stopping all the instances from both buildouts I just copied the Data.fs and Data.fs.index files from one instance to another:
cp mangosta_buildout/var/filestorage/Data.fs mangosta_buildout/var/filestorage/Data.fs.index mangosta_buildout_new/var/filestorage/
After copying the files, I restarted the new buildout zeo and zope instances, went to the ZMI and used the portal_migrations tool to upgrade the plone sites within the ZODB to match the src version. Once I finished doing that, I checked every website, everything was working smoothly.
I've (re-)learned some things during this travel:
1- Since Plone 3.2 there is a file that contains a list of versions of all the dependencies for a given Plone version. For example, the file for Plone 3.3 is:
This is very useful and you should use it in your buildout.cfg files as you will ensure yourself that you are installing the correct versions for all the dependencies.
2- Because of  the buildout sintaxis had changed, so if you are upgrading from 3.1.x to 3.3.x I recommend you to create a new buildout, migrate your data and use the old one as a backup.
3- When you are working with Open Source Software, it is a bad idea to be out-of-the-game for too long. Once you are back, it will take some time to catch up and you could find some disgusting surprises.
4- Open Source Software Communities are great! If you are in trouble, you have the project website, the official docs, the blog entries from people working on the project, the mailing lists and, of course, IRC. In this case, the Plone community is always there to help.