stripping illegal characters out of xml in python

XML 1.0 does not allow all characters in unicode:

Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

It often trips up developers (like, today, me) that end up having, say, valid unicode, with valid characters like VT (\x1B), or ESC (\x1B), and suddenly they are producing invalid XML. A decent way to deal with this is to strip out the invalid characters. For example this stack overflow post shows how to do this with perl:

$str =~ s/[^\x09\x0A\x0D\x20-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]//go;

Unfortunately the equivalent does not quite work with Python, since \x{10000}-\x{10FFFF} needs to be expressed as \U00010000-\U0010FFFF which not all versions of python seem to accept as part of a regular expression character class.

So people end up doing messy-looking things in python. But I figured out that if I invert the character class, the biggest character I need to write is \uFFFF, which the python regex engine does acccept. Yay:

import re

# xml 1.0 valid characters:
#    Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
# so to invert that, not in Char ::
#       x0 - x8 | xB | xC | xE - x1F 
#       (most control characters, though TAB, CR, LF allowed)
#       | #xD800 - #xDFFF
#       (unicode surrogate characters)
#       | #xFFFE | #xFFFF |
#       (unicode end-of-plane non-characters)
#       >= 110000
#       that would be beyond unicode!!!
_illegal_xml_chars_RE = re.compile(u'[\x00-\x08\x0b\x0c\x0e-\x1F\uD800-\uDFFF\uFFFE\uFFFF]')

def escape_xml_illegal_chars(val, replacement='?'):
    """Filter out characters that are illegal in XML.
    Looks for any character in val that is not allowed in XML
    and replaces it with replacement ('?' by default).
    >>> escape_illegal_chars("foo \x0c bar")
    'foo ? bar'
    >>> escape_illegal_chars("foo \x0c\x0c bar")
    'foo ?? bar'
    >>> escape_illegal_chars("foo \x1b bar")
    'foo ? bar'
    >>> escape_illegal_chars(u"foo \uFFFF bar")
    u'foo ? bar'
    >>> escape_illegal_chars(u"foo \uFFFE bar")
    u'foo ? bar'
    >>> escape_illegal_chars(u"foo bar")
    u'foo bar'
    >>> escape_illegal_chars(u"foo bar", "")
    u'foo bar'
    >>> escape_illegal_chars(u"foo \uFFFE bar", "BLAH")
    u'foo BLAH bar'
    >>> escape_illegal_chars(u"foo \uFFFE bar", " ")
    u'foo   bar'
    >>> escape_illegal_chars(u"foo \uFFFE bar", "\x0c")
    u'foo \x0c bar'
    >>> escape_illegal_chars(u"foo \uFFFE bar", replacement=" ")
    u'foo   bar'
    return _illegal_xml_chars_RE.sub(replacement, val)

Setting up a new mac

I got a new macbook pro. Rather than migrate my settings automatically I decided I’d go for a little spring cleaning by buillding it up from scratch.

Initial setup stuff

  • start software update
  • go through system preferences, add appropriate paranoia
  • open keychain utility, add appropriate paranoia
  • run software update
  • insert Mac OS X CD, install XCode
  • remove all default bookmarks from safari
  • open iTunes, sign in, turn off annoying things like ping
  • download, extract, install, run once, enter serial (where needed)
    • Chrome
    • Firefox
      • set master password
    • Thunderbird
      • set master password
    • Skype
    • OmniGraffle
    • OmniPlan
    • TextMate
    • VMware Fusion
    • Colloquy
    • IntelliJ IDEA; disable most enterpriseish plugins; then open plugin manager and install
      • python plugin
      • ruby plugin
      • I have years of crap in my intellij profile which I’m not adding to this machine
    • VLC
    • Things
  • download stunnel, extract, sudo mkdir /usr/local && sudo chown $USER /usr/local, ./configure --disable-libwrap, make, sudo make install
  • launch terminal, customize terminal settings (font size, window size, window title, backgroundc color) and save as default

Transfer from old mac

  • copy over secure.dmg, mount, set to automount on startup
  • import certs into keychain, firefox, thunderbird
  • set up thunderbird with e-mail accounts
  • copy over ~/.ssh and ~/.subversion
  • set up stunnel for colloquy, run colloquy, add localhost as server, set autojoin channels, change to tabbed interface
  • copy over keychains
  • copy over Office, run office setup assistant
  • copy over documents, data
  • copy over virtual machines
  • open each virtual machine, selecting “I moved it”
  • copy over itunes library
  • plug in, pair and sync ipad and iphone


  • set up time machine

Development packages

I like to know what is in my /usr/local, so I don’t use MacPorts or fink or anything like it.

  • download and extract pip, python install
  • pip install virtualenv Django django-piston south django-audit-log django-haystack httplib2 lxml
    • Ah mac 10.6 has a decent libxml2 and libxslt so lxml just installs. What a breeze.
  • download and install 64-bit mysql .dmg from, also install preference pane
  • Edit ~/.profile, set CLICOLOR=yes, set PATH, add /usr/local/bin/mysql to path, source ~/.profile
    • again, I have years of accumulated stuf in my bash profile that I’m dumping. It’s amazing how fast bash starts when it doesn’t have to execute a gazillion lines of shell script…
  • code>pip install MySQL-python, install_name_tool -change libmysqlclient.16.dylib /usr/local/mysql/lib/libmysqlclient.16.dylib /Library/Python/2.6/site-packages/
  • take care of the many dependencies for yum, mostly standard ./configure && make && make install of
    • pkg-config
    • gettext
    • libiconv
    • gettext again, --with-libiconv-prefix=/usr/local
    • glib, --with-libiconv=gnu
    • popt
    • db-5.1.19.tar.gz, cd build_unix && ../dist/configure --enable-sql && make && make install, cp sql/sqlite3.pc /usr/local/lib/pkgconfig/, cd /usr/local/BerkeleyDB.5.1/include && mkdir db51 && cd db51 && ln -s ../*.h .
    • neon, ./configure --without-gssapi --with-ssl
    • rpm (5.3), export CPATH=/usr/local/BerkeleyDB.5.1/include, sudo mkdir /var/local && sudo chown $USER /var/local, ./configure --with-db=/usr/local/BerkeleyDB.5.1 --with-python --disable-nls --disable-openmp --with-neon=/usr/local && make && make install
    • pip install pysqlite pycurl
    • urlgrabber, sudo mkdir /System/Library/Frameworks/Python.framework/Versions/2.6/share && chown $USER sudo mkdir /System/Library/Frameworks/Python.framework/Versions/2.6/share && python install
    • intltool
    • yum-metadata-parser, export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/lib/pkgconfig; python install
  • install yum. Yum comes without decent install scripts (the idea is you install it from rpm). I did some hacks to get somewhere reasonable:
    cat Makefile | sed -E -e 's/.PHONY(.*)/.PHONY\1 install/' >
    mv Makefile
    cat etc/Makefile | sed -E -e 's/install -D/install/' > etc/
    mv etc/ etc/Makefile
    make DESTDIR=/usr/local PYLIBDIR=/Library/Python/2.6 install
    mv /usr/local/Library/Python/2.6/site-packages/* /Library/Python/2.6/site-packages/
    mv /usr/local/usr/bin/* /usr/local/bin/
    mkdir /usr/local/sbin
    mv /usr/local/usr/sbin/* /usr/local/sbin/
    rsync -av /usr/local/usr/share/ /usr/local/share/
    rm -Rf /usr/local/usr/
    cat /usr/local/bin/yum | sed -E -e 's|/usr/share/yum-cli|/usr/local/share/yum-cli|' > /tmp/
    mv /tmp/ /usr/local/bin
    chmod +x /usr/local/bin/yum

That’s how far I got last night, resulting in 86GB of disk use (most of which is VMs and iTunes library), just enough to be productive on my current $work project. I’m sure there’s weeks of tuning in my future.