I was putting together notes using reStructured Text and kept thinking that it’d be nice to generate both slides and longer written documentation from the same source. I’ve used docutils’ S5 generator in the past, but was looking for something a little more polished looking. Something like the HTML5 Slides.

So I wrote a hieroglyph, a Sphinx builder for generating HTML5 Slides. I presented hieroglyph at the Sunday morning lightning talks at PyCon: you can see the slides, the reStructured Text source, as well as the HTML documentation generated from the same source.

I’m really happy with the output — it looks great in the browser, projects well, and because I’m using the html5slides CSS, looks great on mobile devices, too. I’m even happier that I’m able to work on my content in plain text. You can find the source on github.

class A(object):
    def __init__(self):
        super(self.__class__, self).__init__()

class B(A):
    def __init__(self):
        super(B, self).__init__()

The surprise comes when I try to use my sub-class, B. Instantiating B() blows up the stack with:

RuntimeError: maximum recursion depth exceeded while calling a Python object.

What?

According to the Python 2.7.2 standard library documentation, super “return[s] a proxy object that delegates method calls to a parent or sibling class of type.” So in the case of single inheritance, it delegates access to the super class, it does not return an instance of the super class. In the example above, this means that when you instantiate B, the follow happens:

1. enter B.__init__()
2. call super on B and call __init__ on the proxy object
3. enter A.__init__()
4. call super on self.__class__ and call __init__ on the proxy object

The problem is that when we get to step four, self still refers to our instance of B, so calling super points back to A again. In technical terms: Ka-bloom.

TL;DR: super(self.__class__, self) may look like a neat trick, but it’s the end of the line for sub-classing.

Further reading: Raymond Hettinger’s excellent blog post on super provides a great overview of super and shows off the improved Python 3 syntax, which removes the need to write the class name as part of the super statement. I was really pleased to find the Python standard library documentation links directly to it.

Setting up coverage is on the todo list, but it appears that our particular combination of libraries has at least one strange issue: when cc.license uses Jinja2 to load a template, coverage thinks it’s a Python source file (maybe it uses an import hook or something? haven’t looked) and tries to tokenize it when generating the xml report. Ka-boom. (This has apparently already been reported.)

Another item in the “maybe/someday” file is using Tox to run the tests using multiple versions of Python (example configuration for Tox + Hudson exists). I can see that this is a critical part of the process when releasing libraries for others to consume. We have slightly less surface area — all the servers run the same version of Python — but it’d be great to know exactly what our possible deployment parameters are.

Overall Hudson already feels like it’s adding to our sanity. I just received my copy of Continuous Delivery, so I think this is the start of something wonderful.

UPDATE 30 January 2012: Frédéric Grosshans has provided an updated version that supports nested maildirs; you can find it at github. YMMV.

My script for converting maildir to mbox continues to be one of the most popular pages on yergler.net (according to Google Analytics). Of course, even after I updated it slightly in February, it still had a couple of bugs, likely introduced when I converted the page from MoinMoin to WordPress. This afternoon I finally decided to clear out the pending comments about those bugs, and update it.

While looking at the Python documentation for the mailbox package included in the standard library, I realized it could probably be simplified even further by using the library’s native mbox support. I’m also more comfortable using the standard library’s implementation of mbox rather than my hacked up raw file implementation (who knows, the standard library may do exactly what my script did: I’m not an mbox expert by any stretch of the imagination).

The new script is below. I should note that it’s received very little testing, and I make no guarantees. I also should note that there is nothing creative or original about this. It just uses Python’s excellent standard library. As they say, “batteries included”.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

# -*- coding: utf-8 -*-
 
"""
maildir2mbox.py
Nathan R. Yergler, 6 June 2010
 
This file does not contain sufficient creative expression to invoke
assertion of copyright.  No warranty is expressed or implied; use at
your own risk.
 
---
 
Uses Python's included mailbox library to convert mail archives from
maildir [http://en.wikipedia.org/wiki/Maildir] to 
mbox [http://en.wikipedia.org/wiki/Mbox] format.
 
See http://docs.python.org/library/mailbox.html#mailbox.Mailbox for 
full documentation on this library.
 
---
 
To run, save as maildir2mbox.py and run:
 
$ python maildir2mbox.py [maildir_path] [mbox_filename]
 
[maildir_path] should be the the path to the actual maildir (containing new, cur, tmp);
 
[mbox_filename] will be newly created.
"""
 
import mailbox
import sys
import email
 
# open the existing maildir and the target mbox file
maildir = mailbox.Maildir(sys.argv [-2], email.message_from_file)
mbox = mailbox.mbox(sys.argv[-1])
 
# lock the mbox
mbox.lock()
 
# iterate over messages in the maildir and add to the mbox
for msg in maildir:
    mbox.add(msg)
 
# close and unlock
mbox.close()
maildir.close()

The PDF preview of chapter 3 (basic unit testing) looks good, and the table of contents looks like a good overview of basic tools and techniques. I’m particularly looking forward to reading about integrating Django with other testing tools (chapter 5), and using Django with pdb (chapter 9). I’m looking forward to learning more about testing my Django applications — and hopefully how I can form better habits around testing.

The talk reminded of a blog post I saw late last year and never got around to commenting on, Ruby for Desktop Applications? Yes we can. Now I’m only a year late in commenting on it. This post caught my eye for two reasons. First, the software they discuss was commissioned by the AGI Goldratt Institute. I had heard about Goldratt from my father, whose employer, Trusted Manufacturing, was working on implementing constraints-based manufacturing as a way to reduce costs and distinguish themselves from the rest of the market. More interesting, though, was their discussion of how they built the application, and how it seemed to resonate with some of the work I did in my early days at CC.

Atomic wrote three blog posts (at least that I saw), and the one with the most text (as determined by my highly unscientific “page down” method) was all about how they “rolled” the JRuby application: how they laid out the source tree, how they compile Ruby source into Java JARs, and how they distribute a single JAR file with their application and its dependencies. I thought this was interesting because even though it uses a different language (Python instead of Ruby), GUI framework (wx instead of Swing/Batik), and runtime strategy (bundled interpreter instead of bytecode archive), the thing I spent the most time on when I was developing CC Publisher was deployment.

Like Atomic and Phatch, we had a single code base that we wanted to work across the major platforms (Windows, Linux, and Mac OS X in our case). The presentation about Phatch has some great information about making desktop-specific idioms work in Python, so I’ll let them cover that. Packaging and deployment was the biggest challenge, one we never quite got right.

On Windows, we used py2exe to bundle our Python runtime with the source code and dependencies. This worked most of the time, unless we forget to specify a sub-package in our manifest, in which case it blew up in amazing and spectacular ways (not really). Like Atomic, we used NSIS for the Windows installer portion. On Mac OS X, we used py2app to do something similar, and distributed a disk image. On Linux… well, on Linux, we punted. We experimented with a cx-freeze and flirted with autopackage. But nothing ever worked quite right [enough], so we would up shipping tarballs.

The really appealing thing about Atomic’s approach is that by using a single JAR, you get to leverage a much bigger ecosystem of tools: the Java community has either solved, or has well defined idioms for, launching Java applications from JARs. You get launch4j and izpack, which look like great additions to a the desktop developer’s toolbox.

For better or for worse, we [Creative Commons] decided CC Publisher wasn’t the best place to put our energy and time. This was probably the right decision, but it was a fun project to work on. (We do have rebooting CC Publisher listed as a suggested project for Google Summer of Code, if someone else is interested in helping out.) Given the maturity of Java’s desktop tool chain, and the vast improvements in Jython over the past year or two, I can imagine considering an approach very much like Atomic’s were I working on it today. Even though it seems like the majority of people’s attention is on web applications these days, I like seeing examples of interesting desktop applications being built with dynamic languages.

When I started, the project used the following setup to manage dependencies and the build process:

• Dependencies which have an egg or setuptools-compatible sdist available are specified in install_requires in setup.py

  setup(
      name = "soursop",
  
      # ... details omitted
  
      install_requires = ['setuptools',
                          'zope.interface',
                          'zope.component',
                          'PILwoTK',
                          'flup',
                          ],
  
      )
  

• A buildout configuration that uses djangorecipe to install Django, and zc.recipe.egg to install the application egg and its dependencies
  

  [buildout]
  develop = .
  parts = django scripts
  unzip = true
  eggs = soursop
  
  [django]
  recipe = djangorecipe
  version = 1.1.1
  settings = settings
  eggs = ${buildout:eggs}
  project = soursop
  
  [scripts]
  recipe = zc.recipe.egg
  eggs =
       ${buildout:eggs}
  interpreter = python
  dependent-scripts = true
  extra-paths =
     ${django:location}
  initialization =
     import os
     os.environ['DJANGO_SETTINGS_MODULE'] = '${django:project}.${django:settings}'
  

• Dependencies that didn’t easily install using setuptools (either they didn’t have a sane source-tree layout or weren’t available from PyPI) are either specified as git submodules or imported into the repository.

All this worked pretty well (although I’ve never really loved git submodules).

gp.recipe.pip is a buildout recipe which allows you to install a set of Python packages using pip. gp.recipe.pip builds on zc.recipe.egg, so it inherits all the functionality of that recipe (installing dependencies declared in setup.py, generating scripts, etc). So in that respect, I could simply replace the recipe line in the scripts part and start using pip requirements to install from source control, create editable checkouts, etc.

Previously, I used the ${buildout:eggs} setting to share a set of packages to install between the django part (which I used to generate a Django management script) and the scripts part (which I used to resolve the dependency list and install scripts defined as entry points). I didn’t spend much time looking into replicating this with gp.recipe.pip; it wasn’t immediately clear to me how to get a working set out of it that’s equivalent to an eggs specification (I’m not even sure it makes sense to expect such a thing).

Ignoring the issue of the management script, I simplified my buildout configuration, removing the django part and using gp.recipe.pip:

[buildout]
develop = .
parts = soursop
unzip = true
eggs = soursop
django-settings = settings
django-project = soursop

[soursop]
recipe = gp.recipe.pip
interpreter = python
eggs = ${buildout:eggs}
sources-directory = vendor

initialization =
   import os
   os.environ['DJANGO_SETTINGS_MODULE'] = '${buildout:django-project}.${buildout:django-settings}'

This allowed me to start specifying the resources I previously included as git submodules as pip requirements:

[soursop]
recipe = gp.recipe.pip
interpreter = python
install =
	-r requirements.txt
eggs = ${buildout:eggs}
sources-directory = vendor

The install parameter specifies a series of pip dependencies that buildout will install when it runs. These can include version control URLs, recursive requirements (in this case, a requirements file, requirements.txt), and editable dependencies. In this case I’ve also specified a directory, vendor, in which editable dependencies will be installed.

That actually works pretty well: I can define my list of dependencies in a text file on its own, and I can move away from git submodules and vendor imports to specifying [D]VCS urls that pip will pull.

Unfortunately, I’m still missing my manage script. I wound up creating a small function and entry point to cause the script to be generated. In soursop/scripts.py, I created the following function:

def manage():
    """Entry point for Django manage command; assumes
    DJANGO_SETTINGS_MODULE has been set in the environment.

    This is a convenience for getting a ./bin/manage console script
    when using buildout."""

    from django.core import management
    from django.utils import importlib
    import os

    settings = importlib.import_module(os.environ.get('DJANGO_SETTINGS_MODULE'))

    management.execute_manager(settings)

In setup.py, I added an entry point:

    entry_points = {
        'console_scripts' : [
            'manage = soursop.scripts:manage',
            ]
        },

Re-run buildout, and a manage script appears in the bin directory. Note that I’m still using the environment variable, DJANGO_SETTINGS_MODULE, to specify which settings module we’re using. I could specify the settings module directly in my manage script wrapper. I chose not to do this because I wanted to emulate the behavior of djangorecipe, which lets you change the settings module in buildout.cfg (i.e., from development to production settings). This is also the reason I have custom initialization code specified in my buildout configuration.

Generally I really like the way this works. I’ve been able to eliminate the tracked vendor code in my project, as well as the git submodules. I can easily move my pip requirements into a requirements file and specify it with -r in the install line, separating dependency information from build information.

There are a couple things that I’m ambivalent about. Primarily, I now have two different places where I’ve declared some of my dependencies, setup.py and a requirements file, and each has advantages (which are correspondingly disadvantages for the other). Specifying the requirements in the pip requirements file gives me more flexibility — I can install from subversion, git, or mercurial without even thinking about it. But if someone installs my package from a source distribution using easy_install or pip, the dependencies won’t necessarily be satisfied^{1, 2}. And conversely, specifying the requirements in setup.py allows everyone to introspect them at installation time, but sacrifices the flexibility I’ve gained from pip.

I’m not sure that we’ll end up using Mingus for koucou, but I think we’ll stick with gp.recipe.pip. The disadvantage is a small one (at least in this situation), and it’s not really any worse than the previous situation.

¹ I suppose I could provide a bundle for pip that includes the dependencies, but the documentation doesn’t make that seem very appealing.

² Inability to install my Django application from an sdist isn’t really a big deal: the re-use story just isn’t good enough (in my opinion) to have it make sense. Generally, however, I like to be able to install a package and pull in the dependencies as well.

I’ve read the first four chapters of Grok 1.0 Web Development, and so far there’s a lot to like. It’s the sort of documentation I wish I’d had when I ported the Creative Commons license chooser to Grok¹. I’m looking forward to reading the rest, and will post a proper review when I return from Nairobi. In the mean time, check out Grok, Zope 3 for cavemen.

You can download a preview from Grok 1.0 Web Development, Chapter 5: Forms.

¹ The CC license chooser has evolved a lot over the years; shortly after Grok was launched we adopted many of its features as a way to streamline the code. Grok’s simplified support for custom traversal, in particular, was worth the effort.

Part of this awkwardness is probably the impedance mismatch of the framework and the toolchain: Django applications are Python packages. The Python tools for packaging and installing (distutils, setuptools, distribute, and pip, I think, although I have the least experience with it) work on “module distributions“¹: some chunk of code with a setup.py. This is as much a “social” issue as a technology one: the documentation and tools don’t encourage the “right” kind of behavior, so talk of re-usable applications is often just hand waving or, at best, reinvention².

In both cases we consciously chose Django for what I consider its killer app: the admin interface. But there have been re-use headaches. [NB: What follows is based on our experience, which is setuptools and buildout based] The first one you encounter is that not every developer of a reusable app has made it available on PyPI. If they’re using Subversion you can still use it with setuptools, but when re-using with git, we have some additional work (a submodule or another buildout recipe). I understand pip just works with the most commons [D]VCS, but haven’t used it myself. Additionally, they aren’t all structured as projects, and those that are don’t always declare their dependencies properly³. And finally there’s the “real” issues of templates, URL integration, etc.

I’m not exactly sure what the answer is, but it’s probably 80% human (as opposed to technology). Part of it is practicing good hygiene: writing your apps with relocatable URLs, using proper URL reversal when generating intra-applications URLs, and making sure your templates are somewhat self-contained. But even that only gets you so far. Right now I have to work if I want to make my app easily consumable by others; work, frankly, sucks.

Reuse is one area where I think Zope 3 (and it’s derived frameworks, Grok and repoze.bfg) have an advantage: if you’re re-using an application that provides a particular type of model, for example, all you need to do is register a view for it to get a customized template. The liberal use of interfaces to determine context also helps smooth over some of the URL issues⁴. Just as, or more, importantly, they have a strong culture of writing code as small “projects” and using tools like buildout to assemble the final product.

Code reuse matters, and truth in advertising matters just as much or more. If we want to encourage people to write reusable applications, the tools need to support that, and we need to be explicit about what the benefits we expect to reap from reuse are.

¹ Of course you never actually see these referred to as module distributions; always projects, packages, eggs, or something else.

² Note that I’m not saying that Pylons gets the re-use story much better; the author admits choosing Django at least in part because of the perceived “vibrant community of people writing apps” but found himself more productive with Pylons. Perhaps he entered into that with different expectations? I think it’s worth noting that we chose Django for a project, in part, for the same reason, but with different expectations: not that the vibrant community writing apps would generate reusable code, but that they would education developers we could hire when the time came.

³ This is partially due to the current state of Python packaging: setuptools and distribute expect the dependency information to be included in setup.py; pip specifies it in a requirements file.

⁴ At least when dealing with graph-based traversal; it could be true in other circumstances, I just haven’t thought about it enough.

Seeing a comparison between Java and Clojure (albeit one you can read as more about succinctness than clarity), I was reminded that this lack of magic — boilerplate, ceremony, whatever — is still important.