how to install python MySQLdb on mac os x 10.5 (leopard)

I was happy to find that someone’s already posted instructions on how to install mysql-python on mac os x 10.5.

how to install lxml python module on mac os 10.5 (leopard)

lxml is an xml library for python that doesn’t suck. It needs a recent libxml2 and libxslt. Mac OS X does not come with recent versions, and 10.5 breaks completely if you force it to try and use a recent version.

I used to use MacPorts for everything (which comes with py25-lxml) but ran into some issues with 10.5. So back to manual installs, it is.

• assuming your python is the mac os x default…
• make sure no traces of other pythons in your $PATH
• download and install libxml2:

     ./configure --prefix=/usr/local/libxml2-2.7.0
     make
     sudo make install
     cd /Library/Python/2.5/site-packages
     sudo ln -s /usr/local/libxml2-2.7.0/lib/python2.5/site-packages/* .
  

• download and install libxslt:

     ./configure --prefix=/usr/local/libxslt-1.1.24 --with-libxml-prefix=/usr/local/libxml2-2.7.0
     make
     sudo make install
     cd /Library/Python/2.5/site-packages
     sudo ln -s /usr/local/libxslt-1.1.24/lib/python2.5/site-packages/* .
  

• download and install lxml:

     sudo python setup.py install \
       --with-xml2-config=/usr/local/libxml2-2.7.0/bin/xml2-config \
       --with-xslt-config=/usr/local/libxslt-1.1.24/bin/xslt-config
  

Simulating erratic failures in python

Imagine you’re using some library for interfacing with MySQL from python. And MySQL is dying from under your application. And you’re not dealing with it gracefully. So you get greeted with:

OperationalError: (2006, 'MySQL server has gone away')

If you re-start your app, the problem will go away for a bit. So how can you test (unit test, even) that you’re degrading gracefully?

Here’s one example. Rather more clean and generic and simple than what you typically do in something statically typed like java.

import random
from MySQLdb import OperationalError

class Store:
    def get_val(self, a):
        return a.lower()
    def execute(self, *args, **kwargs):
        return "OK"

class ErroringStore:
    """Wrapper around something that has an execute(…) to make it error."""
    def __init__(self, delegate):
        self.delegate = delegate
    def __getattr__(self, name):
        if name == "execute":
            if int(random.random()*2) == 1:
                def execute_error(*args, **kwargs):
                    raise OperationalError(2006, ‘MySQL server has gone away’)
                return execute_error
            else:
                return getattr(self.delegate, name)
        elif hasattr(self.delegate, name):
            return getattr(self.delegate, name)
        raise AttributeError, name
    def __setattr__(self, name, value):
        if name == "delegate":
            self.__dict__[name] = value
        else:
            setattr(self.delegate, name, value)

store = Store()
erroring = ErroringStore(store)

# these are ok
erroring.get_val("BLAAT")

# this doesn’t exist
try:
    erroring.nomethod()
    raise "should result in AttributeError"
except AttributeError:
    pass

# this sometimes throws an exception
errors = 0
for i in range(1, 100):
    try:
        erroring.execute("SELECT TRUE;")
    except:
        errors += 1
print "Failed %d times out of 100" % errors

App Engine is an interesting start

So I looked at google’s App Engine today. Some thoughts:

1. We need to know what it costs.
2. Yay for WSGI.
3. Yay for Apache 2 licensing.
4. Let me deep-link to the code.
5. Amazon provides more power. For example, queues.
6. As others have said the lock-in risk needs some attention. I’m sure someone will solve it soon.
7. I still don’t believe we’ll stop distributing entity relationships. They are useful. Facebook at least provides subselects. Amazon doesn’t.

All in all, if I were to launch some new web app today, I would probably still do that on Amazon S3/EC2.

Leo is…

I have always been jealous of Sam for having an idea of what Ruby is. No more.

Leo is…:

• A general data management environment. Leo shows user-created relationships among any kind of data: computer programs, web sites, etc.
• An outlining editor for programmers. Leo embeds the noweb and CWEB markup languages in an outline context.
• A flexible browser for projects, programs, classes or any other data.
• A project manager. Leo provides multiple views of a project within a single outline. Leo naturally represents tasks that remain up-to-date.
• Portable. Leo runs on Windows, Linux and MacOS X.
• 100% pure Python. Leo uses Tk/tcl to draw the screen.
• Fully scriptable using Python. Leo’s outline files are XML format.
• Open Software, distributed under the Python License.

Leo is certainly a lot like me. Deals with lots of data and their interconnections, very flexible, heavily focussed on python, heavily focussed on people, open source, pretty smart, and a lot of fun to play with.

Our choices for python web applications

So, at work, we’re doing some “next generation” versions of a bunch of our backoffice tooling. That involves producing a bunch of cute little web applications, that often control not so cute and not so little processes (like transcoding and publishing and whatnot). The course-grained architecture pattern is pretty simple and familiar: database with information about files, jobs, tasks and metadata, some common libraries for interacting with the database, some web application middleware using those libraries, and a web server frontend serving up the middleware.

Pretty much normal bread-and-butter stuff. It’s not quite like document-based CMS work (you don’t really want to store many-gigabyte video in a JCR repo), but a lot of the technology choices are still similar.

Customizing a snake

Based on the various tech we have deployed today, and the skills of the people working on this kind of thing, we’re trying to standardize around two main server-side technologies: java and python. This post explains the choices we made for the python universe. At the moment those choices are actually not so easy, since there’s so much happening and so many projects are moving so fast. We scouted the web quite a bit to figure out what to do.

Lower layers

• OS: ubuntu 7.10 (still some nodes on 6.10)
• Database: MySQL 5.0.45 (comes with ubuntu, a bit reconfigured of course) with some little bits of replication
• Python: Python 2.4.4 (2.5 is not on ubuntu 6.10 and not on all our developer workstations, but we’re testing with it and will upgrade eventually)
• WSGI server: right now we have a slightly customized cherrypy wsgi server (so that it accepts signals, restarts itself, runs from /etc/init.d, logs in all the right places, etc) behind an apache httpd 2.2 ProxyPass, which also handles SSL/AAA. We want to try and move to mod_wsgi but first we need its mac install to suck a bit less, and so far, cherry is not quite falling over on us. If mod_wsgi doesn’t work out it’ll probably be back to twisted, probably also behind apache for SSL reasons.

Application glue

• Database access layer: storm, our own slightly modified version. We really like storm, and every now we find we are pushing it a bit beyond its limits, which leads to some bits of patching (by people smarter than me!). Fortunately it seems the guys working on it are quite responsive on IRC. I expect there’ll be a few (more) patches from us that flow back upstream. I really hope someone implements support for forking out reads and writes to different nodes (like you get for free with MySQL Connector/J), either in MySQLdb or inside storm.
• Python web glue: We’re trying to do everything completely WSGI-based, though most everything at the moment is actually inside CherryPy 3.1b1 handlers. The WSGI pattern works just fine and scales nicely enough in our tests.
• Templating: Genshi 0.4.4 (we had to pick one, there’s a few good choices here)
• XML bits and pieces: lxml 1.3.6. It’s the best XML support in python so far, but it still isn’t quite as good as what you get in java. All the various bits and pieces just aren’t quite as mature, and the underlying libxml2 doesn’t quite do XML schema support as well, and I also miss something like XMLBeans for python.

Out of the box?

We took a look at a bunch of the web frameworks out there. We didn’t seriously consider zope, but we took a long stare at pylons, turbogears and django before deciding not to bother with them. We’re not using much of paste either. Basically we missed one or more of

• good support for storm out of the box
• doing everything the WSGI way
• good and correct documentation
• easy to scale / make efficient
• stable core with excellent compatibility and bugfixing

And perhaps a few other things, and on the balance we guessed it would be easier to roll our own and integrate components, rather than strip something else down, and maintain lots of vendor branches.

Key point: standardization good

Two years ago I would’ve picked twisted without blinking and invented another fancy wheel on top of it, but I’m happy I don’t have to do that anymore. Twisted has quite a learning curve, not just for app developers, but also for the people that need to deploy and scale the beast.

Two good things happened to the python webapp world: competition and standardization. Now things are progressing rapidly.

Progress is good, but it can result in various kinds of chaos that don’t help the application developer that likes to plan ahead a bit. The new scripting language based mega frameworks seem to attract a certain kind of developer and they probably work for a certain set of use cases, but standardizing on patterns and interfaces is much more useful for (opinioned!) people like us (with subtly deviating use cases). So framework authors: please do keep working on bridging the gap between all of them by cutting ‘em down into tiny little WSGI middleware bits and pieces, and turn frameworks into libraries where you can.