Packaging Python Modules

2009-08-07T23:48:13Z

Tools like EasyInstall and PIP are excellent for frequently changing environments. I use virtualenv extensively in my development environments where I can easy_install or pip -E install away. These tools, however, are not appropriate for production environments. In production, repeatability and minimal dependency on development tools are essential. When rolling out hundreds of servers, installing MySQL-python bindings with easy_install would require setting up a build environment on each host. The operating system's package manager is sufficient for managing the installation of these modules.

What if you need different versions of the same package on a particular host? Assuming there are two applications running on a host that require different versions of the same package: move one of the applications to a separate host or VM. If one application needs two versions of a particular package: fix the application. In an emergency, running an application out of a temporary virtualenv can do the job at the expense of operational headaches.

I run Redhat based systems so RPMs are the way to go for me. python setup.py bdist_rpm is handy but there are a number of things I don't like about it:

Packages are not python-versioned. pyOpenSSL-0.8-1.i386.rpm does not indicate which python it is built against. Also, what if you want that package for both python2.4 and python2.5?
Similarly, I prefer that python library packages be clearly identified in the list of installed packages. A 'python-' prefix is effective. rrdtool-1.2.27-5.i386.rpm is the application package, python25-rrdtool.i386.rpm are the python2.5 bindings.

Since those things are not easy to override in the stock bdist_rpm, I use a lightly modified distutils/command/bdist_rpm.py arbitrarily named bdist_rpm_ver.py. Building production packages is usually as easy as:

easy_install --build-directory ~/rpm/BUILD/ --editable <pkg>
cd ~/rpm/BUILD/<pkg>
pythonXX setup.py bdist_rpm_ver --fix-python --binary-only

This certainly is not perfect yet - sometimes I have to break down and do a --spec-only and tweak things by hand. Fortunately, this happens once and only takes a few minutes to fix and then I have a stable package ready for deployment. Also, I handle dependencies separately from the rpms currently but they can be added in step 3 with --requires, --conflicts, etc.

I'm interested in hearing more about:

I’ve been meaning to write a post on why I think using system packaging for libraries is counter-productive, but that’ll wait for another time.

Ian Bicking - A Few Corrections To “On Packaging”.

In my experience, system packaging is the way to go for simple, repeatable installations to a single host or 1000s of hosts.

g.raphaelli's weblog

Packaging Python Modules